deepakpanda93 opened a new pull request, #19505: URL: https://github.com/apache/hudi/pull/19505
### Describe the issue this Pull Request addresses Closes #15589 / [HUDI-5263](https://issues.apache.org/jira/browse/HUDI-5263). Spark SQL accepts a `CREATE TABLE` that declares partition columns and, at the same time, configures a key generator that never produces a partition path: ```sql create table hudi_cow_pt_tbl ( id bigint, name string, ts bigint, dt string ) using hudi tblproperties ( type = 'cow', primaryKey = 'id', hoodie.table.keygenerator.class = 'org.apache.hudi.keygen.NonpartitionedKeyGenerator' ) partitioned by (dt) ``` The statement succeeds and persists a table config that disagrees with itself. The partition fields are taken from the partition columns in `initHoodieTable`, while the key generator is taken from the table properties, which override the value `extraTableConfig` inferred. Nothing cross checks the two. The resulting table cannot be written to at all. Running against master, every write path is rejected identically: ``` create table: SUCCEEDED (inconsistent config persisted) sql insert: FAILED - Config conflict: PartitionPath: <empty> vs dt dsWrite(partitionpath=dt): FAILED - Config conflict: PartitionPath: <empty> vs dt dsWrite(nonpartitioned): FAILED - Config conflict: PartitionPath: <empty> vs dt ``` So the behaviour has moved on from the original report: the null partition column is no longer reachable, because a later guard blocks the write. What remains is that the broken table is still created, and the eventual failure names neither the key generator nor the partition columns, which are the two things actually in conflict. ### Summary and Changelog Creating a table with a non partitioned key generator and partition columns now fails immediately, with a message that names both, instead of producing a table that cannot be written to. - Adds `validateKeyGeneratorForPartitionColumns` to `HoodieCatalogTable`, invoked from `parseSchemaAndConfigs` on the path that creates a table. - Extracts `resolvePartitionColumns` from `initHoodieTable` so the validation and the value that gets persisted are derived the same way and cannot drift apart. - Covers `NonpartitionedKeyGenerator` and `NonpartitionedAvroKeyGenerator`, resolved through `KeyGeneratorType.getKeyGeneratorClassName` so the key generator class and key generator type properties are both honoured. - The check is deliberately limited to a table being created. An existing table already in this state stays readable and can still be registered in the catalog, so nobody is locked out of data they already have. - Tests added to `TestCreateTable`: the rejection, for both non partitioned key generator classes, and a companion test asserting a genuinely non partitioned table is still created and still writable, so the guard cannot over fire. The new error reads: ``` Cannot create table 'spark_catalog.default.hudi_cow_pt_tbl' partitioned by 'dt' using key generator 'org.apache.hudi.keygen.NonpartitionedKeyGenerator', which does not generate a partition path. Either drop the partition columns, or configure a partitioned key generator through 'hoodie.table.keygenerator.class'. ``` ### Impact A `CREATE TABLE` that previously succeeded now fails. This breaks no working workflow: a table created that way is rejected on every subsequent write, as shown above, so the statement only ever produced an unusable table. The failure simply moves to the statement that causes it. Confined to the Spark SQL table creation path in `HoodieCatalogTable`. No change to the read path, the write path, or the datasource API. Tables that already exist in this state are untouched. ### Risk Level low The validation runs only while creating a table, and only fires on a combination that is provably unusable. Verified that no existing test pairs a non partitioned key generator with partition columns; the whole `org.apache.spark.sql.hudi.ddl` package passes. ### Documentation Update None. No new config, no public API change. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [x] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
