yihua opened a new pull request, #1012:
URL: https://github.com/apache/hudi-rs/pull/1012
## Description
closes #1009
A `Table` handle now reads the metadata table as of the data snapshot it
opened on:
- The valid-instant set for a read is built from the listing the handle's
timeline was loaded from (pending instants of every action, completed rollbacks
and restores), not from fresh listings.
- The metadata table is opened pinned to the snapshot: every metadata
compaction that may have folded in an instant the snapshot cannot see is
dropped from its timeline, so reads take the file slice from before it.
Planning, the public metadata read APIs and table stats all read this pinned
table.
- When the metadata table has archived the instants that earlier slice
needs, or its cleaner removed the slice, the handle refuses the metadata table
and plans from a storage listing. When a pinned `files` partition read fails,
as after a clean that runs once the table is pinned, that plan lists storage
too. Write paths keep the latest view.
One deliberate deviation from Java: the set is capped at the newest instant
time the handle listed, so a metadata-only delta commit newer than the snapshot
is excluded. Java loads the metadata timeline together with the data timeline
and needs no cap; here the metadata table is opened at the first read, and an
instant above the cap was created after the snapshot.
Impact: a plan no longer lists the data timeline again for pending and
rollback instants, which is 2 fewer LISTs per plan. A handle whose metadata
table has nothing past its snapshot does no extra I/O; the metadata partitions
are listed for the cleaned-slice check only when pinning drops a compaction.
Known limits: a partially cleaned earlier slice is caught only when reading
it fails, so a clean interrupted after removing the slice's logs but not its
base reads without error and can miss the files those logs recorded. Storage
listing cannot see a base file a later data clean deleted, so a plan that falls
back can come back short, as on a table without a metadata table.
Over half of the added lines are tests. This textually conflicts with #985
and #1011 in the timeline loader, and needs a one-character change after #984.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
The `snapshot_fence` tests cover a stale handle after a commit and clean,
after a metadata compaction past its snapshot, with a metadata table archived
or cleaned past its snapshot, and with its pinned slice cleaned after the pin
(which fails with `No schema available for merge output` without the planning
fallback). They also check that a compaction that ran before an unseen commit
completed is kept, that a fresh handle reads the latest metadata, and that a
data instant pending at open stays excluded once it completes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]