kamisama0002 opened a new pull request, #14197:
URL: https://github.com/apache/gravitino/pull/14197
### What changes were proposed in this pull request?
Use the full create-table API in the Paimon Spark adapter: `primary-key`
becomes one composite index, and `bucket` / `bucket-key` become distribution
metadata. Convert `CLUSTERED BY` through the same path and reject conflicting
property values. Bucket transforms no longer appear in the partition list.
Return the key and bucket properties when Spark creates or loads the table.
Keep the declared key order and normalize primary-key columns to non-nullable,
as Paimon does.
### Why are the changes needed?
For `TBLPROPERTIES ('primary-key'='id', 'bucket'='-1')`, the connector
currently sends reserved properties and no index. Creation fails with
`Properties or property prefixes are reserved and cannot be set`. `CLUSTERED
BY` also fails because its bucket transform is treated as a Paimon partition.
Fix: #14196
Fixes #10367.
Related to #10721, which added Flink bucket conversion and server-side
support.
### Does this PR introduce _any_ user-facing change?
Yes. `CREATE TABLE` accepts Paimon primary-key and bucket properties,
including dynamic buckets without an explicit bucket key. It also accepts
`CLUSTERED BY ... INTO ... BUCKETS`. `SORTED BY` remains unsupported.
Server-side reserved-property checks stay in place.
### How was this patch tested?
- Windows / JDK 17: all 297 Spark 3.5 module tests passed, including 51 new
Paimon cases. Command: `./gradlew :spark-connector:spark-3.5:test -PskipITs
--configure-on-demand --offline --max-workers=4`. Scoped Spotless checks and
`git diff --check` passed.
- CentOS / JDK 17: the 51 regression cases and both SQL integration tests
passed with an embedded Gravitino server and an isolated local-filesystem
fixture. The SQL tests create and write a dynamic-bucket table whose primary
key excludes the partition column, verify key replay, and read/write a table
created with `CLUSTERED BY` and identity partitions.
- The new regressions reproduced the reserved-property error and rejected
bucket transforms before the fix. Native Paimon tests also check create/load
metadata and primary-key nullability.
Full-repository checks could not finish: the Windows offline run lacked
Netty artifacts, and the Linux run lacked the JDK 24 toolchain for Trino.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
