hudi-agent commented on code in PR #19650:
URL: https://github.com/apache/hudi/pull/19650#discussion_r3796394307
##########
website/learn/tech-specs.md:
##########
@@ -532,12 +596,32 @@ For example, a secondary index on the `city` column, for
a record with `city = C
chennai$id1 -> {"isDeleted": false}
```
-Each secondary-index partition is tagged with a `HoodieIndexVersion` (stored
on the corresponding `HoodieIndexDefinition`).
-Table version 8 constrained every secondary-index partition to `V1` (the
encoding described above). Table version 9
-introduces `V2`, which shards records by the primary (record) key rather than
by the secondary key. This makes secondary-index
-updates cheaper on writes with skewed secondary values, at the cost of
secondary-key range scans having to visit more file
-groups. Readers pick their scan strategy from the per-partition
`HoodieIndexVersion`. New tables created on version 9
-default to `V2` for secondary indexes; existing `V1` partitions from upgraded
tables continue to be read with the V1 encoding.
+**Partitioning** decides which file group of the index partition an entry is
written to. Hudi hashes a portion of the
+record key and takes that value modulo the number of file groups. Which
portion is hashed is governed by the
+[`HoodieIndexVersion`](#index-versions) recorded on the index's
`HoodieIndexDefinition`:
+
+* **`V1`**, the default for table version 8, hashes the whole
`<escaped-secondary-key>$<escaped-primary-key>` key.
+ Entries sharing a secondary value are distributed across all file groups,
so resolving a secondary value to its
+ records reads every file group unless the primary key is already known.
+* **`V2`**, the default for table version 9, hashes only the leading
`<escaped-secondary-key>$` portion. All entries
+ sharing a secondary value therefore reside in one file group, and a lookup
by secondary value alone reads that
+ single file group.
+
+The strategy is selected per partition from its recorded version, so `V1`
partitions on an upgraded table continue to be
+read as `V1` while indexes created afterwards on the same table use `V2`.
+
+#### Limitations
+
+* A secondary index may be defined on **exactly one column**. Attempting
more fails with
+ `Only one column can be indexed for functional or secondary index.`
+* The indexed column must be one of `string`, `int`, `long`, `float`,
`double`, `date`, `time`, or a
Review Comment:
🤖 This supported-types list appears to be missing the short/byte integer
types. `TestSecondaryIndexDataTypes` explicitly creates and validates secondary
indexes on `smallint` and `tinyint` columns (alongside the types listed here)
and asserts they succeed, while `decimal`, `boolean`, and `binary` are in its
unsupported set. It might help to add `smallint` and `tinyint` (or phrase it as
"integral types") so users aren't led to believe those columns can't be indexed.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
