deepakpanda93 opened a new pull request, #19650: URL: https://github.com/apache/hudi/pull/19650
### Describe the issue this Pull Request addresses Closes #17068. (JIRA: [HUDI-9550](https://issues.apache.org/jira/browse/HUDI-9550).) The issue asks for the tech spec to cover four table version 9 index changes: the new internal `indexVersion` field, the new secondary index partition scheme, null value handling, and what upgrade/downgrade does to indexes. While working through it I found that the page does not merely omit the new partition scheme — **it currently describes it backwards**. Fixing that is the most important change here, so it goes first. ### The correction `learn/tech-specs.md` says today: > Table version 9 introduces `V2`, which **shards records by the primary (record) key rather than by the secondary key**. > This makes secondary-index updates cheaper on writes with skewed secondary values, **at the cost of secondary-key range > scans having to visit more file groups**. Both halves are inverted. The dispatch is unambiguous: ```java // MetadataPartitionType.java (SECONDARY_INDEX) public SerializableBiFunction<String, Integer, Integer> getFileGroupMappingFunction(HoodieIndexVersion indexVersion) { return HoodieTableMetadataUtil.getSecondaryKeyToFileGroupMappingFunction(indexVersion.greaterThanOrEquals(HoodieIndexVersion.V2)); } ``` ```java // HoodieTableMetadataUtil.java public static SerializableBiFunction<String, Integer, Integer> getSecondaryKeyToFileGroupMappingFunction(boolean needsSecondaryKeyExtraction) { if (needsSecondaryKeyExtraction) { String secondaryKey = SecondaryIndexKeyUtils.getUnescapedSecondaryKeyPrefixFromSecondaryIndexKey(recordKey); return mapRecordKeyToFileGroupIndex(secondaryKey, numFileGroups); } return HoodieTableMetadataUtil::mapRecordKeyToFileGroupIndex; } ``` and its javadoc states it plainly: *"For secondary index partitions (version >= 2) … the unescaped secondary key portion is used for hashing. Otherwise, the full record key is used."* So `V1` hashes the whole `<secondary>$<primary>` key, scattering one secondary value across every file group; `V2` hashes only the `<secondary>$` prefix, so all entries for a secondary value sit in one file group and a lookup by secondary value alone reads exactly one. `V2` is the improvement for secondary-key lookups, and the published text says the opposite. <!-- SCREENSHOT 4 HERE: rendered Secondary Index -> Partitioning + Limitations --> ### Summary and Changelog One file, `website/learn/tech-specs.md` (+96/−10). `learn/` is an unversioned Docusaurus plugin, so there is a single copy and no versioned duplicates; the page therefore has to describe both table version 8 and 9 behaviour, which it now does. **1. Index Definitions.** The definition path and the `hoodie.table.index.defs.path` property were already documented, but the JSON schema was not, and the secondary index section pointed at `#indexing-functions` for "the on-disk shape" — a two-line RFC stub. That dangling cross-reference now resolves to this new section. <!-- SCREENSHOT 1 HERE: rendered Index Definitions section --> **2. Index Versions.** `version` holds a `HoodieIndexVersion`, introduced at table version 9, which lets an index's physical layout change without a table-version upgrade. Worth stating explicitly that this is **not** secondary-index specific: `column_stats`, `partition_stats` and `expr_index` also default to `V2` at table version 9, while `record_index`, `bloom_filters` and `files` stay `V1`. Also that a table version 8 definition may omit `version`, but from version 9 a definition without one is invalid. <!-- SCREENSHOT 2 HERE: rendered Index Versions section with the per-index-type table --> **3. Table Upgrade and Downgrade.** The issue phrases this as "downgrade/upgrade will drop new/old SI index", but only downgrade drops anything: - Upgrading 8 → 9 rebuilds and drops nothing. `EightToNineUpgradeHandler#populateIndexVersionIfMissing` stamps version-less definitions with `V1`, so existing indexes keep their layout. - Downgrading 9 → 8 drops every partition whose version is above `V1` (`UpgradeDowngradeUtils#dropNonV1IndexPartitions`), plus `partition_stats` when a `V2` `column_stats` partition is dropped, since partition stats may have no definition of their own to inspect. <!-- SCREENSHOT 3 HERE: rendered Table Upgrade and Downgrade section --> **4. Secondary index limitations.** One indexed column; the supported column types; and the column-type-change restriction. **Null handling** (issue item 3) needed no change — the page already documents `\0` escaping correctly. ### Relationship to the abandoned PR #13713 #13713 is the open PR linked from the issue. It is **CONFLICTING/DIRTY** and untouched since Feb 2026. The reason it conflicts is worth recording: its target file `website/src/pages/tech-specs-1point0.md` **no longer exists**. Commit `34f4d7bd9433` (#14221) moved tech specs to `website/learn/tech-specs.md`. So it cannot simply be rebased; the content has to be re-applied against the current file, which is what this PR does. I used it as a reference, not as the answer, and re-derived every fact. Two of its statements do not hold on `release-1.2.0`: | #13713 says | Verified behaviour | |---|---| | SI supports "string, double, timestamp and any integral types" | `HoodieIndexUtils#isSecondaryIndexSupportedType` admits `string, int, long, float, double, date, time`, and `timestamp` **only when `isUtcAdjusted`**. Local timestamps are rejected. Its list omits float/date/time and misses the timestamp caveat. | | "We don't allow schema evolution on columns with a secondary index" | `HoodieTable` throws `SchemaCompatibilityException` on a column **type** change, but explicitly `continue`s when only nullability differs. So the restriction is narrower than stated. | Also not carried over: its `.gitignore` change (`.vscode/`, `.metals/`), which is unrelated to the docs, and its `## Indexes` → `## Indices` rename, which introduced typos ("Indicies", "speciffies", "compatibiity"). **On its review feedback**, being precise rather than implying there was more than there was: #13713 has exactly **one** reviewer comment and **zero** inline threads (confirmed via the GraphQL `reviewThreads` API, including resolved and outdated ones): > **@vinothchandar:** *"lets remove the release notes .md file.. and notify the RM.. @yihua on the update."* The author replied "done" and the file is absent from that diff. This PR honours it by adding no release-notes file, and the second half is the reason @yihua is tagged below. ### Verification `npm run build` passes with the warning block **byte-identical** to a baseline built from the same base commit (`eaa8dfd89a38`), 13,265 lines each. Rendered under `npm run serve` and checked over HTTP: all four new headings resolve with correct anchors, 7 TOC entries are generated, the `index.json` block and per-index-type table render, and the inverted `"shards records by the primary"` sentence is confirmed gone. The screenshots above are from that local render. Every claim was checked against `release-1.2.0` source rather than carried over: `HoodieIndexVersion`, `HoodieIndexDefinition`, `SecondaryIndexKeyUtils`, `HoodieTableMetadataUtil`, `MetadataPartitionType`, `EightToNineUpgradeHandler`, `NineToEightDowngradeHandler`, `UpgradeDowngradeUtils`, `HoodieIndexUtils` and `HoodieTable`. One limitation worth stating: this is verified by reading source, not by running an upgrade/downgrade against a real table. A committer confirming the upgrade/downgrade wording in particular would be valuable, since that is the part users will act on. ### Impact Documentation only. No code, config, or behaviour change. It does correct a statement that is currently misleading about which layout is better for secondary-key lookups. ### Risk Level none ### Documentation Update This PR is the documentation update — the tech spec, https://hudi.apache.org/learn/tech-specs. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable cc @yihua (per the RM notification asked for on #13713), @Davis-Zhang-Onehouse (author of #13713), @vinothchandar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
