zhuxiangyi opened a new pull request, #9372:
URL: https://github.com/apache/paimon/pull/9372
### Purpose
`primary-key` and `bucket-key` are plain strings in `TBLPROPERTIES`, so they
reach Paimon exactly as
the user typed them and are then matched against the schema exactly. Every
other identifier in the
same statement follows `spark.sql.caseSensitive` — the partition columns in
particular, because
Spark's analyzer has already resolved them by the time the catalog sees
them. That leaves a single
statement obeying two different rules:
```sql
-- accepted: the partition column resolves case-insensitively
CREATE TABLE t (Id INT, Pt STRING) PARTITIONED BY (pt);
-- rejected
CREATE TABLE t (Id INT, A INT) TBLPROPERTIES ('primary-key'='id',
'bucket'='1');
-- IllegalStateException: Table column [Id, A] should include all primary
key constraint [id]
-- rejected
CREATE TABLE t (Id INT, A INT) TBLPROPERTIES ('bucket-key'='id',
'bucket'='1');
-- RuntimeException: Field names [Id, A] should contains all bucket keys
[id].
```
Since `spark.sql.caseSensitive` is `false` by default, `Id` and `id` are the
same identifier
everywhere else in Spark, so this is surprising.
**Fix.** Resolve both properties against the schema in `SparkCatalog`,
honouring the session's
case-sensitivity setting:
- `toInitialSchema` — for the `primary-key` list and the `bucket-key` option
value.
- `alterTable` — for `bucket-key` only; `primary-key` cannot reach there
because
`validateAlterProperty` already rejects altering it. The table is loaded
only when the property
being set is `bucket-key`, so other `SET TBLPROPERTIES` calls are
unaffected.
Two deliberate choices:
- **With `spark.sql.caseSensitive=true` nothing changes** — the exact-match
requirement stays.
- **A name matching no column is passed through untouched**, so Paimon still
reports it exactly as
the user wrote it instead of a rewritten one.
**Why this stays on the Spark side.** The exact matching in core is
*correct* for Flink: its
identifiers are case-sensitive, so `Id` and `id` really are two different
identifiers there, and its
primary keys come from the resolved `PRIMARY KEY` DDL constraint
(`FlinkCatalog`
`builder.primaryKey(table.primaryKeys())`) rather than an option string.
Changing core would break
that. This mirrors how Iceberg handles it — a case-sensitivity flag plumbed
through the connector
(`SparkUtil.caseSensitive`, `PartitionSpec.Builder#caseSensitive`) rather
than exact matching pushed
down into the format. Paimon already has the same precedent on the Spark
write path, where
`SchemaEvolutionHelper` reads `conf.caseSensitiveAnalysis`.
**Side effect worth noting.** `bucket-key` values are now split on `,` and
trimmed before being
stored, which `primary-key` already did. `'bucket-key' = 'id, sub'`
previously failed, because
`TableSchema#originalBucketKeys` splits on `,` without trimming and then
looked for a column named
`" sub"`. It now works, making the two properties consistent.
Not covered here: other options whose values name columns (for example
`sequence.field`) have the
same shape and could be given the same treatment as a follow-up.
### Tests
New `KeyPropertyCaseResolutionTestBase` with concrete suites for Spark 3.2,
3.3, 3.4, 3.5, 4.0 and
4.1 — 7 cases, asserting the stored key uses the column's real spelling and
that the table is
actually writable and readable afterwards:
- `primary-key` / `bucket-key` resolve like every other identifier
- multi-column and partitioned keys resolve (`'primary-key'='id,sub,pt'`)
- `ALTER TABLE SET TBLPROPERTIES` resolves `bucket-key`
- an exact spelling keeps working
- an unknown column still reports the original name
- `spark.sql.caseSensitive=true` keeps the exact-match requirement
Verified green on Spark 3.2, 3.3, 3.4, 3.5 and 4.0 (7/7 each), and the full
`paimon-spark-ut` module
shows no new failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
