nsivabalan opened a new pull request, #19323: URL: https://github.com/apache/hudi/pull/19323
### Describe the issue this Pull Request addresses Phase 3 of 3 in the selective meta-fields series. Follow-up to #19205 (CoW writer + reader + config) and #19206 (`hudi-cli table set-meta-fields-mode`). This PR extends the mode to Merge-on-Read tables and adds a fail-fast validation for indexes that depend on `_hoodie_record_key`. Without this PR, MoR tables silently ignore `hoodie.meta.fields.mode` on the log-write path (base file honors it, log block does not — merge produces mixed-mode records), and enabling RLI or secondary index on a non-ALL mode silently corrupts index maintenance (index keyed on nulls, updates target phantom rows). ### Summary and Changelog Two additive changes, one commit each: **1. `feat(writer): extend hoodie.meta.fields.mode to MoR log-write and incremental-read paths`** - **Log-write:** `HoodieAppendHandle.populateMetadataFields()` now honors the mode when `populate.meta.fields=false`. Under selective modes the append handle writes only the opted-in meta columns to log blocks, matching the base-file behavior from phase 1. - **Incremental-read:** `MergeOnReadIncrementalRelationV1/V2` (and CoW `IncrementalRelationV1/V2`) now gate on `tableConfig.isCommitTimePopulated()` instead of `populateMetaFields()`. Tables in `COMMIT_TIME_ONLY` / `COMMIT_TIME_AND_FILE_NAME` now serve incremental queries correctly; `FILE_NAME_ONLY` / `NONE` fail fast with an actionable error message that names the mode and points at remediation. - `TableSchemaResolver` from phase 1 already emits meta columns whenever the mode is not NONE, so the log-block reader picks them up without further changes. **2. `feat(writer): fail-fast when RLI/SI are enabled with a non-record-key mode`** - New `checkArgument`s in `HoodieWriteConfig.Builder.validate()` reject the combination of (RLI or SI enabled) with any mode that leaves `_hoodie_record_key` unpopulated (NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME). - Predicates mirror `HoodieMetadataConfig.isRecordLevelIndexEnabled()` / `isSecondaryIndexEnabled()` exactly — including the metadata-enabled gate and (for SI) the column-non-empty gate. Inlined rather than delegating to the accessor because `validate()` runs before the `HoodieWriteConfig` instance's `metadataConfig` field is materialized. - Error messages name the mode explicitly and point at the two remediation options (disable the index, or use ALL mode). ### Impact **Behavior change (MoR tables only, opt-in):** MoR tables that set `hoodie.meta.fields.mode` explicitly now respect that mode in log blocks. Previously the mode was silently dropped on the log-write path — this is a bug fix, not a compatibility break. Tables that never set the mode see no change. **Behavior change (RLI/SI validation, all engines):** Attempting to enable RLI or SI on a table with a non-ALL mode now fails at write-config build time with a clear error. Previously it would silently produce incomplete / phantom index entries. No impact on tables that use RLI/SI with the default (ALL) mode. **Read-side error:** MoR incremental queries on `FILE_NAME_ONLY` / `NONE` tables that previously returned zero rows silently now throw with an actionable message. This is the same guard already present for CoW in phase 1 — extended to MoR here. ### Risk Level **Low.** All changes are additive and guarded by the mode enum introduced in phase 1. The MoR log-write path change only fires when the mode is set (opt-in), and the RLI/SI guard only fires when the user actively enables those indexes on a non-ALL mode — both cases previously produced silent data-correctness issues, so the fail-fast behavior is strictly safer. Verification: new unit + integration tests covering all 5 modes on MoR (bulk-insert, upsert, incremental) plus parameterized RLI + SI rejection across all four non-ALL modes. All existing tests still pass (17 in `TestHoodieWriteConfigMetaFieldsMode`, 32 in `TestMetaFieldsMode`, 10 in `TestHoodieStreamerMetaFieldsMode`). ### Documentation Update None. The `hoodie.meta.fields.mode` config was documented in phase 1 (#19205). This PR extends its scope from CoW to MoR and adds validation guards; no new user-facing config or feature. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
