danny0405 opened a new pull request, #19845: URL: https://github.com/apache/hudi/pull/19845
### Describe the issue this Pull Request addresses Tables that omit `_hoodie_record_key` currently cannot retain `_hoodie_commit_seqno` and `_hoodie_partition_path`: those columns are populated only by `ALL`. Add `ALL_EXCEPT_RECORD_KEY` so a table can retain commit time, commit sequence number, partition path, and file name without storing the record-key meta column. ### Summary and Changelog - Add `ALL_EXCEPT_RECORD_KEY` to `MetaFieldsMode`, with explicit `commitSeqnoPopulated` and `partitionPathPopulated` members and getters. - Include both flags in `isWiderThan`, so transitions that add either column are treated as widening. - Populate the selected columns in the Avro and Spark Parquet writers and Spark row-create handle. - Support the new mode in Flink's Parquet writer, Lance writer, and row-create handle, including generated and preserved commit metadata. Write the destination file name when that column is enabled. - Update the configuration description and CLI help. Update Spark concat, Flink COW insert/update/concat, bulk-insert, and preservation assertions to use each column's population flag instead of assuming sequence number and partition path are ALL-only. - Use named column access in the Spark metadata-preservation test. No code was copied from external sources. ### Impact Users can set `hoodie.meta.fields.mode=ALL_EXCEPT_RECORD_KEY` on supported Spark and Flink COPY_ON_WRITE tables. This adds a public enum value and two public population predicates, and changes physical column population only for tables selecting the new mode. Existing modes, defaults, and file schemas are unchanged. The compatibility boundary is opt-in: older builds cannot parse the new enum value. The new mode inherits the existing selective-mode restrictions, including the Java-writer and MERGE_ON_READ restrictions, and maps to `false` for the deprecated `hoodie.populate.meta.fields` boolean. Record-key-dependent features remain subject to existing validation. Storage savings were not benchmarked. ### Risk Level medium The change touches Avro, Spark, and Flink metadata population and table-mode transitions. Validation on the branch rebased onto master `eccfee6dfe7d` passed 218 cases: - Flink 2.2 reactor: 143 cases across `TestMetaFieldsMode` (75), `TestHoodieWriteConfigMetaFieldsMode` (24), `TestHoodieRowDataCreateHandle` (14), `TestHoodieRowDataLanceWriter#testMetaFieldsMode` (6), `TestBulkInsertWriteHelper#testMetaFieldsMode` (12), and the COW metadata/concat methods of `TestFlinkWriteClientFunctional` (12). The mode matrices exercise all six modes, including insert, update, delete, concat, bulk insert, and metadata preservation. - Spark 3.5: the full `TestMetaFieldsModeE2E` class (62 cases) and `TestHoodieRowCreateHandle#testAllExceptRecordKeyPreservesMetadata` (2 cases). This includes mode persistence across table versions, row and non-row writes, concat, clustering, incremental reads, and multi-partition upserts. - Common/Flink dependency configuration: `TestHoodieAvroParquetWriter` (2 cases) and `TestHoodieTableConfigMetaFieldsMode` (9 cases). Compilation, applicable Checkstyle checks, and `git diff --check` passed. The standalone Avro test initially encountered a missing Hadoop runtime dependency under the Spark profile; it passed under the common/Flink configuration. Streamer assertions were updated, but its suite and the full repository suite were not run. ### Documentation Update Updated `HoodieTableConfig.META_FIELDS_MODE`, `MetaFieldsMode` Javadoc, and CLI help to describe the new mode and preserve master's Spark/Flink COPY_ON_WRITE support documentation. A Hudi website documentation follow-up is still needed to list `ALL_EXCEPT_RECORD_KEY`, its populated columns, and the selective-mode restrictions; website changes are not included in this code PR. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
