danny0405 opened a new pull request, #19845: URL: https://github.com/apache/hudi/pull/19845
### Describe the issue this Pull Request addresses Tables that omit `_hoodie_record_key` currently cannot retain `_hoodie_commit_seqno` and `_hoodie_partition_path`: those columns are populated only by `ALL`. Add `ALL_EXCEPT_RECORD_KEY` so a table can retain commit time, commit sequence number, partition path, and file name without storing the record-key meta column. ### Summary and Changelog - Add `ALL_EXCEPT_RECORD_KEY` to `MetaFieldsMode`, with explicit `commitSeqnoPopulated` and `partitionPathPopulated` members and their getters. - Include both population flags in `isWiderThan`, so transitions that add these columns are treated as widening. - Populate the selected columns in the Avro Parquet writer, Spark Parquet writer, and Spark row-create handle. Preserve source commit time and sequence number when metadata preservation is requested, and write the destination file name. - Update the configuration description and CLI help with the new mode. - Extend enum, Avro writer, Spark end-to-end, row-handle, and Streamer coverage to assert each meta column independently. No code was copied from external sources. ### Impact Users can set `hoodie.meta.fields.mode=ALL_EXCEPT_RECORD_KEY` on supported selective-mode tables. This adds a public enum value and two public population predicates, and changes physical column population only for tables selecting the new mode. The Parquet schema, existing modes, and default behavior are unchanged. The new mode inherits the existing selective-mode restrictions and maps to `false` for the deprecated `hoodie.populate.meta.fields` boolean. Writers/readers that do not recognize the new enum value cannot resolve that configuration. Record-key-dependent features remain subject to the existing validation. Storage savings from omitting the record key were not benchmarked. ### Risk Level medium The change touches metadata population in Avro and Spark write paths and the table-mode transition relation. Focused validation passed across 109 test cases: - `TestMetaFieldsMode`: 75 cases, covering all population predicates, resolution overloads, legacy behavior, and all 36 ordered mode pairs. - `TestHoodieTableConfigMetaFieldsMode`: 9 cases. - `TestHoodieWriteConfigMetaFieldsMode`: 18 cases. - `TestHoodieAvroParquetWriter`: 2 cases, including on-disk values and absence of record-key footer statistics. - `TestMetaFieldsModeE2E#allExceptRecordKeyPopulatesOtherColumns`: 3 cases covering row-writer bulk insert, Avro records, and Spark records. - `TestHoodieRowCreateHandle#testAllExceptRecordKeyPreservesMetadata`: 2 cases covering generated and preserved commit metadata. Compilation, applicable Checkstyle checks, and `git diff --check` passed. The Streamer coverage was updated but its suite was not run. The full repository suite was not run. ### Documentation Update Updated `HoodieTableConfig.META_FIELDS_MODE`, `MetaFieldsMode` Javadoc, and CLI help to describe the new mode. A Hudi website documentation follow-up is still needed to list `ALL_EXCEPT_RECORD_KEY`, its populated columns, and the existing selective-mode restrictions; website changes are not included in this code PR. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
