danny0405 opened a new pull request, #20496: URL: https://github.com/apache/hudi/pull/20496
### Describe the issue this Pull Request addresses Part of #20408 and #20401 Phase 2 B1. Keyed SDP flows need exact-result coverage for precombine, ordering, delete markers and reinsertion, including existing tables and invalid configurations. The new missing-partition scenario also exposed a catalog gap: a nonexistent partition field was accepted at table creation and the flow could commit data before Spark failed while resolving that field. ### Summary and Changelog - Validate declared partition columns against a supplied nonempty table schema before initializing/registering a Hudi table. Preserve schema-less existing-table registration; use Spark's case-sensitivity resolver. - Add a catalog regression proving an invalid partition definition creates neither a catalog entry nor hoodie.properties. - Add a real CLI/Connect scenario over new and pre-existing partitioned COW targets, with explicit EVENT_TIME_ORDERING and boolean delete markers. - Check exact contents after duplicate-key precombine, a newer update, a lower-order update, deletion, reinsertion and restart. - Reject missing key/order/partition fields, string delete markers and a persisted-identity change. Assert no completed data commits on invalid new targets and unchanged valid outputs after every rejection. Validation: Spark 4.1.1 / Scala 2.13 / JDK 17 candidate reactor package passed; the full CLI positive/negative matrix passed against that candidate. `TestHoodieCatalogDDL`: 10 passed, 0 failed. Python syntax, git diff --check and repository PR compliance validation passed. No code was copied from an external project. ### Impact Malformed catalog partition definitions fail before metadata creation or writes. No new configuration, public API or storage format. The test uses existing upsert/merge semantics and does not add Auto CDC or imply hard-delete propagation through upstream incremental reads. No CI wiring changes while #20453 is under discussion. ### Risk Level low. The production change checks only explicitly supplied nonempty schemas and uses Spark's resolver. Focused catalog and real CLI negative tests cover the discovered write-before-failure defect. The integration matrix disables metadata indexing and uses isolated local sources/targets. ### Documentation Update Add KEYED_FLOWS.md with runnable commands, explicit record-merger semantics and coverage boundaries. This is validation of existing features and an invalid-input bug fix, with no new supported runtime feature or configuration requiring a website guide. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
