vinishjail97 commented on PR #843: URL: https://github.com/apache/incubator-xtable/pull/843#issuecomment-5037460318
In addition to the `ITXTableSparkRuntimeBundle` matrix, I ran the bundle end-to-end against real cloud storage. Setup: - Built the shaded bundle from this branch and ran `XTableSparkSync` via `spark-submit` on Spark 3.5.2 (local mode). Engines were supplied on a flat classpath from `target/engine-classpath.txt` (Hudi 0.14, Iceberg 1.9.2, Delta), plus `hadoop-aws` for S3A access. - Source data was a 100 GB TPC-DS dataset on S3: 24 Hudi (COW) tables, synced in a single submit using the `--datasetconfig` multi-table mode. - Source format HUDI, targets ICEBERG and DELTA. On Spark 3.5 the Delta side ran through the Delta Kernel path (auto-enabled). Result: - 24 of 24 tables synced, exit 0, no failures. - Verified the Iceberg (`metadata/`) and Delta (`_delta_log/`) metadata were written on S3 alongside the existing Hudi data files; no data was rewritten. Module docs (README plus a Spark quickstart, including a `--datasetconfig` TPC-DS example) are in #847. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
