nsivabalan opened a new pull request, #19323: URL: https://github.com/apache/hudi/pull/19323
## Phase 3 of 3 in the selective meta-fields series Follow-up to #19205 (CoW writer + reader) and #19206 (hudi-cli command). This PR extends the mode to Merge-on-Read tables and adds a fail-fast validation for indexes that depend on `_hoodie_record_key`. **Draft** because CI hasn't run yet and I want #19205 to land first — the top two commits are the only phase-3 delta. Everything else in the diff is shared with the earlier two PRs. ## Scope (new in phase 3) ### `feat(writer): extend hoodie.meta.fields.mode to MoR log-write and incremental-read paths` - **Log-write path**: `HoodieAppendHandle.populateMetadataFields()` now honors the mode when `populate.meta.fields=false`. Under selective modes, the append handle writes only the opted-in meta columns to log blocks — matching the base-file behavior added in phase 1. - **Incremental-read path**: `MergeOnReadIncrementalRelationV1/V2` (and CoW `IncrementalRelationV1/V2`, retroactively) now gate on `tableConfig.isCommitTimePopulated()` instead of `populateMetaFields()`. That lets `COMMIT_TIME_ONLY` and `COMMIT_TIME_AND_FILE_NAME` tables serve incremental queries correctly; `FILE_NAME_ONLY` and `NONE` fail fast with an actionable error message. - **TableSchemaResolver** in phase 1 already emits meta columns whenever the mode is not NONE, so the log-block reader picks them up without further changes. ### `feat(writer): fail-fast when RLI/SI are enabled with a non-record-key mode` - New `checkArgument`s in `HoodieWriteConfig.Builder.validate()` reject the combination of (RLI or SI enabled) with any mode that leaves `_hoodie_record_key` unpopulated (NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME). - Predicates mirror `HoodieMetadataConfig.isRecordLevelIndexEnabled()` / `isSecondaryIndexEnabled()` exactly — including the metadata-enabled and (for SI) column-non-empty gates. Inline rather than delegating to the accessor because `validate()` runs before the `HoodieWriteConfig` instance's `metadataConfig` field is materialized. - Error messages name the mode explicitly and point at the two remediation options (disable the index, or use ALL mode). Rationale: without this guard, RLI would silently key on null record-keys and SI updates would target phantom rows — a mysterious query-time miss instead of a build-time failure. ## Testing Test suites added and re-run locally on this branch: | Module | Test | Result | |---|---|---| | hudi-hadoop-common | `TestHoodieMetaFieldsMode`, `TestHoodieTableConfig` | 102 / 0 fail | | hudi-client-common | `TestHoodieWriteConfigMetaFieldsMode` (incl. 8 new RLI/SI negative + 1 positive) | 17 / 0 fail | | hudi-cli | `TestTableCommand#testSetMetaFieldsMode*` | 9 / 0 fail | | hudi-spark | `TestMetaFieldsMode` (incl. new MoR incremental negative tests), `TestHoodieDatasetBulkInsertHelper`, `TestHoodieSparkSqlWriter[WithTestFormat]` | 32 / 0 fail (1 known count()-optimization flake, passes on retry) | | hudi-utilities | `TestHoodieStreamerMetaFieldsMode` | 10 / 0 fail | ### New tests added in this PR - `TestHoodieWriteConfigMetaFieldsMode` — parameterized RLI + SI rejection across all four non-ALL modes, plus a positive ALL-mode passthrough. - `TestMetaFieldsMode.morIncrementalQueryRejectedUnderFileNameOnly` / `morIncrementalQueryRejectedUnderNone` — verify the incremental read fails fast with a message that names the mode. - MoR bulk-insert / upsert coverage for all 5 modes plus `morIncrementalQueryWorksUnderCommitTimeOnly` end-to-end (all already added in the phase-3 base commit). ## Known follow-ups (out of scope for this PR) - MoR incremental `count()` under selective modes bypasses the range filter — end-to-end read via `collect()` returns correct rows, but `.count()` may under-report. Optimization for another PR. - MoR log-compaction paths under selective modes aren't dedicated-tested (the upsert path does exercise the append handle, which is the write side; compaction reuses the same populators). - Non-Spark writers (Flink, Java client) don't yet honor selective modes; this is guarded by an explicit `checkArgument` that rejects `engineType != SPARK` with a non-ALL/NONE mode. ## Test plan - [ ] CI green on all matrix combinations - [ ] Verify PR #19205 lands first (this PR is stacked on top) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
