AadhiKat opened a new issue, #860: URL: https://github.com/apache/incubator-xtable/issues/860
When a Delta source is synced incrementally, DeltaConversionSource.getTableChangeForCommit reconstructs the table snapshot with deltaLog.getSnapshotAt(version) for each commit in the backlog. Reconstructing a snapshot reads the table's checkpoint, so catching up N commits re-reads the checkpoint N times. That's the simplest correct thing to do and it's cheap when the checkpoint is small, but on a table with a large checkpoint it ends up dominating the sync. I ran into this converting a table with ~45M files, where the checkpoint is 454 Parquet files (~6.5 GB). Each commit in the backlog took 2-3 minutes, almost entirely in the checkpoint read, and the backlog barely kept ahead of the table's ingestion. More CPU (4 -> 16 cores) made no difference, which fits — the cost is object-store I/O, not compute. The snapshot here is only used for schema, partitioning, file format, and the table base path. None of those change between commits unless the commit carries a metadata or protocol change, so the snapshot can be loaded once and reused across the backlog, reloading only when such a change shows up. DeltaIncrementalChangesState keeps the full action list per version, so those changes are visible. One caveat: the snapshot is also where InternalTable.latestCommitTime comes from (snapshot.timestamp()), so with reuse intermediate commits would report the timestamp of the last reload rather than their own. If that needs to stay exact, it can be read from the commit's CommitInfo action without reconstructing the snapshot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
