vinishjail97 commented on PR #861: URL: https://github.com/apache/incubator-xtable/pull/861#issuecomment-5197807893
@AadhiKat Before I approve, I tried reproducing this on S3 first and couldn't a ~10 TB delta table showed no per-commit penalty, which I take to mean checkpoint part count drives it rather than bytes. Could you share a bit about the table behind the 454-part / ~6.5 GB checkpoint (schema width, how long the history runs between checkpoints, Delta and Spark versions)? Mainly I'd like to know whether the 2-3 min per commit went into getLogSegmentForVersion's _delta_log listing or the checkpoint Parquet read after it -- reading 2.4.0, the only expensive call in here looks like snapshot.metadata()? The direction of the optimization looks right either way; but want to see validate it thoroughly once before approving because it's a change in the xtable core path used by users in production. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
