AadhiKat opened a new pull request, #861:
URL: https://github.com/apache/incubator-xtable/pull/861
Closes #860.
getTableChangeForCommit reconstructs the table snapshot
(deltaLog.getSnapshotAt) for every commit,
which re-reads the Delta checkpoint each time. It's fine when the checkpoint
is small, but on a table
with a large checkpoint it becomes the bottleneck for incremental catch-up
(see #860).
This caches the snapshot, along with the table and file format derived from
it, and reloads it only
when a commit carries a Metadata or Protocol action. Apart from those, the
snapshot is only used for
schema, partitioning, file format, and the table base path, which don't
change between commits.
getActionsForVersion returns the unfiltered action list, so a
schema/protocol change is visible to
the check and forces a reload for that version onward.
The one thing that varies per commit and comes from the snapshot is
InternalTable.latestCommitTime
(snapshot.timestamp()). With reuse, intermediate commits report the
timestamp of the last reload
rather than their own. If keeping that exact matters, it can come from the
commit's CommitInfo
instead — happy to change it if you'd prefer.
Testing:
- Existing ITDeltaConversionSource passes unchanged (15 tests).
testAddColumns is the relevant one:
it adds a column mid-backlog, so if the cache weren't invalidated on the
metadata change the later
commits would carry the old schema and it would fail.
- Added getTableChangeForCommitReconstructsSnapshotOnceForAppendOnlyBacklog:
runs a multi-commit
append backlog through a source built with a spied DeltaLog, checks the
table changes still
validate, and asserts getSnapshotAt is called once rather than once per
commit.
On the ~45M-file table from the issue, incremental catch-up went from ~2-3
min/commit to ~3 s/commit
(around 210 commits in ~10 min), with driver memory flat near 5 GB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
