linliu-code opened a new pull request, #707:
URL: https://github.com/apache/hudi-rs/pull/707
## Description
First increment of ENG-47771. With a metadata partition now readable across
several file slices (#705), a key lookup still opened **every** slice. A
partition that shards — `record_index` has ten file groups in the fixture,
`secondary_index_rider_idx` ten — hashes each key to exactly one, so nine of
those ten reads return nothing.
**The routing is cherry-picked from `wip/mdt-file-group-routing`, not
rewritten**, as the ticket asks. It ports Java's `String.hashCode` over UTF-16
code units with wrapping `i32` arithmetic, then `abs(abs(h) % n)`, with vectors
taken from Java's published values — including `polygenelubricants`, whose hash
is exactly `i32::MIN` and is the only input the doubled `abs` exists for. Two
traps it avoids are silent rather than loud: iterating Rust `char`s hashes a
non-BMP character as its scalar and routes it to a different file group, and
computing in `i64` diverges once the hash exceeds `i32`.
`slices_for_keys` turns that hash into a selection:
- **Slices are sorted by file id first.** `file_group_index` returns a
position *among the shards*, while the listing order is the storage's. Hudi
embeds the shard number in the file id (`record-index-0003-0`), so a
lexicographic sort recovers it. Indexing an unsorted list sends a key to the
wrong shard, which returns **no rows rather than an error** — the failure this
sort exists to prevent.
- **Shards are deduplicated**, so a repeated key does not open its shard
twice.
- **An empty key set still opens every slice.** That is not an omission: a
prefix lookup cannot know which shard holds a match, because sharding is by the
*full* key. Java says the same at `getRecordsByKeyPrefixes:239`.
Concurrency is unchanged — the selected slices go through the same bounded
fan-out from #705, on the same `hoodie.read.file.slice.read.concurrency`.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
**Fan-out width is asserted as slices selected, not inferred from rows
returned**, which is what the ticket specifically asks for — and the reason
matters: a wrong shard and an empty shard both return no rows, so counting rows
cannot tell them apart.
Four tests over ten synthetic shards:
| test | what it pins |
| --- | --- |
| `a_key_lookup_opens_only_the_shards_its_keys_route_to` | one key selects
one slice, **and it is the shard the hash names** — not merely some shard |
| `distinct_shards_are_opened_once_each` | two keys on different shards
select two; the same key twice selects one |
| `an_empty_key_set_opens_every_slice` | a scan opens all ten |
| `selection_does_not_depend_on_listing_order` | reversing the input does
not change the selected shard |
The second test asks the routing function which keys land on different
shards rather than hard-coding a hash, so it is not checking the implementation
against itself.
**Mutation-checked with a positive control**, each caught by a different
test:
| mutation | result |
| --- | --- |
| drop the file-id sort | FAILED (1 test) |
| drop the shard deduplication | FAILED (1 test) |
| always scan every slice | FAILED (3 tests) |
| unmutated | 8 passed |
```
cargo test -p hudi-core --lib 1413
passed, 0 failed
cargo test -p hudi-core --lib --no-default-features 1393
passed, 0 failed
cargo test --workspace --all-targets --all-features 0
failing suites
cargo clippy -p hudi-core --all-targets -- -D warnings clean
cargo clippy -p hudi-core --lib --no-default-features -- -D warnings clean
cargo fmt --all -- --check clean
```
**Not covered, stated rather than implied.** The shards in these tests are
synthetic — `FileSlice`s with shard-ordered file ids. Nothing yet reads the
fixture's ten real `record_index` shards and confirms the keys each holds are
the ones routing sends there, because the pruner is still pinned to the `files`
partition. That check is the one that would catch an off-by-one in the modulo,
and it belongs with unpinning the partition.
**Remaining in ENG-47771**, none of it in this PR: unpinning the pruner, a
decoded record type per partition, and the exact-match valid-instant filter —
the last being the one that prevents a record from a pending instant driving
pruning, which is a silently wrong query result rather than a failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
