linliu-code opened a new pull request, #694:
URL: https://github.com/apache/hudi-rs/pull/694
## Description
**Stacked on #691, review that one first.** Until it merges the diff here
shows its commits too; the delta is `fd712c9..HEAD`, six files.
Hudi's metadata table is sorted by record key, and Hudi's readers seek into
it: a reader that supports a key predicate is asked for an exact key set or a
key prefix, and a full scan is the fallback for readers that do not. Full scan
is allowed for the `files` partition only; column statistics, bloom filters and
the indexes are point lookups by design, so this is not an optimisation for the
metadata table, it is how the metadata table is meant to be read. Version two
had row-filter pushdown for parquet, which skips rows while still reading every
block, and nothing that seeks.
The seek turns out to need no new I/O machinery. The ranged HFile reader
added in #691 already fetches an arbitrary subset of data blocks in one call,
keyed by index entry, so the seek is choosing which entries to pass it. Two
functions select them from the block index, which is a sorted map from a
block's first key to its entry, and the existing batched read fetches exactly
those ranges.
Correctness rests on the selection over-including rather than missing. A
block's index key is a lower bound on its first real key, since the writer may
shorten it, so taking the last entry at or below a wanted key can return a
block that turns out not to hold it. That costs one block read. The synchronous
path already names the same case when it lands before a block's first real key.
The caller filters the records the over-included blocks bring back.
One case needs the block before the selected one as well. Row keys are not
required to be unique, and when a key's copies straddle a block boundary the
writer's midpoint falls back to the right hand cell, so the separator equals
the key itself; selecting only the entry at or below it would take the later
block and drop the copies in the earlier one. So an exact hit on a separator
takes the preceding block too, which is one extra block read against silently
losing rows. No fixture here straddles, so that rule has a test of its own
rather than relying on one noticing.
The predicate rides on the base file read options beside the parquet row
filter, and the same rule applies: the HFile reader honours it, other formats
ignore it and return every row.
A byte and call counter is added to the ranged fetcher, because the claim
being made is about what is read rather than about what is returned, and there
was no way to observe it. Bytes is the number to judge a narrowed read on, and
it is exact on a backend that reads each range as asked, which the local
filesystem store does. It is not exact everywhere, and the reason matters more
than the counter: the default ranged read coalesces ranges closer together than
a megabyte into a single request and slices the result, so on the cloud
backends the counter reports what was asked for and the transfer can be larger.
**Selecting blocks therefore only reduces transfer on those backends when the
selected blocks sit more than that threshold apart.** None of the fixtures here
are large enough for that to bite, which is exactly why it is written down
rather than measured. Calls counts calls into the object store API and is named
for what it can observe, since coalescing and any fan-out beneath
it are invisible from the fetcher: a full scan of a thousand adjacent blocks
reports one call and a scattered three block seek also reports one.
Building a probe key needed one fix in passing. A key's ordering compares
its content, which sits after a two byte length prefix, so constructing one
from raw key bytes reads the first two characters as a length and compares the
rest as content. A constructor that prepends the prefix is added. The same
construction exists in the synchronous seek, where it is masked by that path
walking forward or backward from wherever it lands; that is left alone here and
tracked separately.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Measured, not asserted by row count. On the deep index fixture a three key
seek reads 418 bytes where the full scan reads 211,524, on the shortened index
key fixture 2,873 against 110,363, and on the least favourable 49,089 against
295,734. Every wanted key is still returned in all three.
The property the whole design rests on gets the exhaustive case, over every
fixture rather than one: roughly seventy nine thousand keys across nine files
spanning one, two and three index levels, each key selected for individually,
each having to land in a block that holds it. A three key sample cannot rule
out an off by one bound, and the multi level index walk is a second population
path for the block index and the one most likely to break the invariant, so
covering only a single level fixture would have covered the easier half.
Ten tests in all, and every rule has a mutation that fails the test naming
it: selection returning every block, the range walked the wrong way, the probe
key built without its length prefix, the prefix selection forgetting the block
holding its lower bound, the boundary rule removed, selection widened by a
block per key, the reader seeking without filtering, the predicate dropped
between the options and the decode, and the count only projection answering
from the trailer again.
One mutation is **not** caught, and is filed rather than left implied:
making the reader ignore the predicate entirely still passes. No fixture here
can catch it. Every multi-block HFile in the repository is a raw HBase file
with no Avro schema, which the base file reader refuses outright, and every
HFile carrying an Avro schema is a metadata file with exactly one data block.
With one block, seeking and filtering are indistinguishable in rows and in
bytes however the test is written. So the seek is proven at the decoder and the
wiring is proven at the reader, and their join needs a fixture that does not
exist yet.
Run locally on both legs: default features and `--no-default-features`,
clippy clean on both with warnings denied, fmt clean, and `cargo check
--workspace --all-targets --all-features` clean including the Python and C++
bindings. No CI run has happened: fork pull requests sit at `action_required`
until a committer approves them, so all of the above is local only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
