danny0405 opened a new pull request, #19858: URL: https://github.com/apache/hudi/pull/19858
### Describe the issue this Pull Request addresses Flink rejects selective metadata-field modes for COPY_ON_WRITE tables, and its RowData writers only distinguish full metadata population from disabled population. Enable COMMIT_TIME_ONLY, FILE_NAME_ONLY, and COMMIT_TIME_AND_FILE_NAME for Flink COW writes, including upserts and clustering. The merge and concat paths must extract record keys from business fields when record-key metadata is not populated. Clustering must also preserve selected metadata without assuming that the metadata record key and partition path are present. ### Summary and Changelog - Resolve and persist the full metadata-field mode in Flink configuration and table initialization. Continue rejecting selective modes for MOR tables and the Java engine. - Populate selected fields in Parquet, Lance, and bulk RowData writers, with explicit ALL and NONE branches. - Cache record keys extracted through RecordContext on HoodieFlinkRecord, and use the actual reader schema in Flink concat handles. Remove the shared merge-handle key-generator validation that does not apply to Flink's record-context extraction. - Account for multiple partition fields when extracting a single virtual record key, matching the existing complex-key encoding used by the covered writer paths. - Use RowDataKeyGen during clustering when record-key metadata is absent, independently of metadata preservation. - Add metadata-mode, key-cache, COW merge, concat, and clustering-writer coverage, including Spark COW upserts with one record-key field and multiple partition fields. Move the existing Flink client functional test to the datasource module so it uses the real Flink readers. Add checkWrittenDataNoMeta for business-column assertions. No code was copied from external projects. ### Impact Flink COW users can select which metadata fields are populated. Existing defaults remain unchanged. COW merge/concat files retain metadata columns in their schemas for NONE; regular Flink append writes retain their existing metadata-free schema behavior. No table-version or storage-format migration is introduced. The shared RecordContext change also affects virtual-key extraction for Spark and other consumers. This does not resolve every custom key-generator or encoding configuration; those remain outside this PR. The public constructors of HoodieRowDataParquetWriter and HoodieRowDataLanceWriter now accept MetaFieldsMode instead of a boolean. Direct callers must update, which is the source-incompatible change indicated by the title. Mode resolution is cached per writer; no performance improvement is claimed without benchmarking. ### Risk Level medium The change touches metadata population, shared key extraction, and Flink merge/clustering paths. Focused validation passed: - Flink 2.2: metadata-mode COW merges across legacy, file-group-reader, and LSM paths, multipartition upserts, concat across checkpoints, bulk writes with and without metadata preservation, and client functional tests. - Lance RowData writer: all five metadata modes and vector/footer coverage (6 tests). - Spark 3.5: NONE row/non-row writes, selective concat, and single-key/multipartition COW upserts (10 tests). - Temporary full-pipeline clustering verification passed for ALL and all three selective modes, including data and completed replace-commit assertions. That temporary test modification is not included in the PR; permanent coverage is in TestBulkInsertWriteHelper. - Added configuration and key-cache tests, plus git diff --check. The full project test suite and cross-version downgrade tests were not run. ### Documentation Update Updated the hoodie.meta.fields.mode configuration description to include Flink COPY_ON_WRITE support. No configuration key or default is added or changed. The Hudi website documentation needs a follow-up update describing selective Flink COW support and the existing MOR restriction; that website change is not included here. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
