LeonxLJX opened a new pull request, #19832: URL: https://github.com/apache/hudi/pull/19832
### Describe the issue this Pull Request addresses <!-- Either describe the issue inline here with motivation behind the changes (or) link to an issue by including `Closes #<issue-number>` for context. If this PR includes changes to the storage format, public APIs, or has breaking changes, use `!` (e.g., feat!: ...) --> Persist the index type a table was created with into `hoodie.properties` under a new, namespaced key `hoodie.table.index.type` (HUDI-37, mirrored from uber/hudi#409). `HoodieTableConfig.create()` now copies the write-time index configuration — `hoodie.index.class` if set, otherwise `hoodie.index.type` — into the new `INDEX_TYPE` table config before the properties file is first written. **Root cause.** Hudi already persists several one-time, creation-time configs (table type, timeline layout, timezone...) but the index choice is not among them. The creation-time index is therefore only known to the writer that created the table, and no reader or operator can cheaply validate the index used by later writes against the one the table was built with. That is exactly the hazard vinoth described in #14425: a dataset written with BloomIndex all along can silently switch to a cold HBaseIndex and start producing duplicates because the new index has no state. This PR deliberately **records rather than enforces**: failing a job whose index differs from the recorded one changes runtime behaviour for existing pipelines and deserves its own change with a migration/upgrade discussion. ### Summary and Changelog <!-- Short, plain-English summary of what users gain or what changed in behavior. Followed by a detailed log of all changes. Highlight if any code was copied. --> **What users gain:** every hoodie-managed table now self-describes the index it was created with, so `HoodieTableConfig#getIndexType()` returns the literal `hoodie.table.index.type` and a downstream tool can detect a mismatch before ingesting writes from a reconfigured pipeline. No behavior change for existing tables. **Changes (by file):** - `hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java` — new `INDEX_TYPE` field (`HOODIE_TABLE_INDEX_TYPE` = `"hoodie.table.index.type"`), new `getIndexType()` accessor, and a `create()`-time copy from `hoodie.index.class` (preferred) or `hoodie.index.type` (fallback). - `hudi-common/src/test/java/org/apache/hudi/common/table/TestHoodieTableConfig.java` — three new tests (`testPersistsIndexTypeFromWriteConfig`, `testIndexClassTakesPrecedenceOverType`, `testNoIndexTypeRecordedWhenNotConfigured`) and `testDefinedTableConfigs` count updated 46 → 47. **Scope decisions:** 1. `hoodie.index.class` takes precedence over `hoodie.index.type`, mirroring the resolution order in `HoodieIndexConfig`. 2. Only persisted when explicitly present in the creation config. Tables created with engine defaults are byte-identical to before — nothing is guessed from an engine-specific default, and `testCreate` keeps its `props.size() == 7` assertion. 3. Written only in the `create()` path; `update()` / `modify()` paths are untouched, so a later per-job index override can never rewrite the creation-time record. **Alternatives considered and rejected:** - Persist only the resolved index class — loses the symbolic `hoodie.index.type` the operator configured; the human-facing type string is friendlier for inspection and future `HoodieTableConfig` consumers. - Reuse the write-time keys verbatim inside `hoodie.properties` — `hoodie.properties` uses a dedicated `hoodie.table.*` namespace for table metadata; mixing job-level keys in makes it ambiguous which values are creation-time truth vs per-job overrides. - Enforce immutability (throw on mismatch) — deferred, see follow-ups; this PR is intentionally record-only. ### Impact <!-- Describe any public API or user-facing feature change or any performance impact. --> **Public API change (additive only):** `HoodieTableConfig#getIndexType()` is a new accessor. Existing readers continue to compile and behave identically. A `HoodieTableConfig` built from an older `hoodie.properties` file (no `hoodie.table.index.type` key) returns `null` from the new accessor — callers must null-check. **Storage format change (additive):** a new key `hoodie.table.index.type` may appear in `hoodie.properties` of tables created on or after this commit. Older writers continue to round-trip the file unchanged. Spark/Flink writers see no change unless their downstream code consumes `getIndexType()` (none does today). **No performance impact:** the copy runs once at table creation. ### Risk Level <!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk. If medium or high, explain what verification was done to mitigate the risks. --> **Low.** The change is additive in both the public API surface and the `hoodie.properties` file: - Existing tables are unaffected (no migration needed; the key simply does not exist for them). - Newly-created tables gain an extra property; older readers ignore unknown keys. - The `create()` copy is guarded by `StringUtils.isNullOrEmpty` so a writer with no explicit index in its config produces a byte-identical properties file. - All three new tests plus the existing `testCreate` (which still asserts `props.size() == 7`) gate the change. ### Documentation Update <!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none". - The config description must be updated if new configs are added or the default value of the configs are changed. - Any new feature or user-facing change requires updating the Hudi website. Please follow the instruction to make changes to the website. --> `hoodie.table.index.type` is recorded automatically by `HoodieTableConfig.create()`; it is **not** a user-facing config (operators do not set it). Therefore: - No config description update required. - A short paragraph on `hoodie.table.index.type` and `HoodieTableConfig#getIndexType()` should be added to the Hudi website's table-config reference. Tracked as a follow-up; the PR itself is no-doc-required. ### Contributor's checklist <!-- Please go through the following items to ensure your PR is ready to be reviewed. --> - [x] `mvn -pl hudi-common test` runs the new + existing `HoodieTableConfig` tests green. - [x] The PR body uses the standard Hudi PR template. - [x] PR title uses the `[HUDI-37]` JIRA format (the new JIRA ticket created off #14425). - [x] No breaking public-API changes; the new `getIndexType()` is additive and existing readers are unaffected. - [x] No storage-format breaking changes; existing `hoodie.properties` files round-trip unchanged. - [x] No change to behavior for tables that do not explicitly configure an index — byte-identical `hoodie.properties` and `testCreate`'s `props.size() == 7` assertion preserved. - [x] Follow-up suggestions noted in the body (default-engine recording + a future enforcement feature) so they are not lost. Fixes #14425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
