Akash3121 commented on PR #938: URL: https://github.com/apache/incubator-xtable/pull/938#issuecomment-6052963535
> @Akash3121. I tried this change on a cluster in one of our internal environments and saw a few performance regressions, so we need to make the deletion vector check lightweight before this lands. Right now every incremental sync on a deletion-vector-enabled table scans all active files of the latest snapshot, through allFiles().toLocalIterator() in the legacy reader and a full getScanFiles read in the Kernel reader, even when there are no new commits. > > Can we remove the snapshot scan in getCommitsBacklog and document in the 0.5.0 release notes that tables synced before the upgrade with deletion vectors need a full rebuild? I’ve rebased the PR onto the latest main and addressed the resulting compatibility issue. All requested deletion-vector performance, logging, documentation, and test changes are now included. Please take another look when you get a chance. Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
