yihua opened a new pull request, #907:
URL: https://github.com/apache/hudi-rs/pull/907
## Description
closes #904
Stacked on #869, whose sharded index lookup this read depends on; review
that first.
`Table::get_file_slices_inner` now asks the record-level index which file
groups hold the keys an `=` or `IN` filter on the record key field (or
`_hoodie_record_key`) names, and plans only those file slices; the row-level
mask still applies the filter to what is read. The lookup runs only for a
snapshot read at the latest commit, on a table with a single record key field
and a populated index; a time-travel read, a composite key or an index that is
off keep the full scan. Keys are encoded the way the writer stores them; a key
generator that prefixes the field name is handled on table version 9, where
that encoding is fixed. The index must describe this reader's snapshot: when
the metadata table's latest commit is newer, or an index base file was
compacted after the snapshot, the read falls back to the full scan.
Reading one key out of a 10M-row table went from a scan of every file slice
to one file slice.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`table_read_record_index_tests` writes three file groups whose key ranges
overlap, so footer statistics cannot separate them, and checks: one planned
slice and the right row for a key, two for an `IN` over two groups, the
intersection of two key filters, no slices for an unknown key, both groups read
for a read as of an earlier commit, every group read when the index is off or
`hoodie.metadata.enable=false` is set on the reader, a reader opened before a
delete still reading the deleted key from its own snapshot, and the merged row
for a merge-on-read table whose update sits in a log file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
