Hi all, I'd like to propose a new artifact, xtable-spark-runtime, to make XTable dramatically easier to use inside existing Spark pipelines.
Problem ------- XTable's modules are published to Maven, so today a user can pull in xtable-core and assemble their own runtime, but everyone has to solve the same shading/classpath problem themselves, or fall back to the xtable-utilities bundle, which ships every dependency unshaded (~1GB) and is impractical to add to a Spark job. Running it also means building the fat jar and invoking a separate sync process with several config files. There's no maintained, thin, drop-in artifact for the common case: "I already write this table with Spark, keep it in sync in other formats." Proposal -------- A thin, relocated runtime jar that a user adds to an existing Spark job: spark-submit --packages org.apache.xtable:xtable-spark-runtime_2.12:<version>...and activates purely through config, no code change: spark.sql.queryExecutionListeners = org.apache.xtable.spark.XTableSyncListener spark.xtable.tables = /warehouse/db/orders spark.xtable.orders.sourceFormat = HUDI spark.xtable.orders.targets = ICEBERG,DELTA After each successful write to the source table, XTable incrementally translates its metadata to the target formats on the driver, so the table becomes readable by other engines with no extra job or orchestration. This follows the bundle model Hudi and Iceberg already use: engine deps marked provided, a curated dependency allowlist, and relocations so the jar coexists with the cluster's own guava/jackson/avro. The result should be a bundle in the tens of MB, not ~1GB. Why it helps the community -------------------------- - Lowest-friction on-ramp: add one dependency + a few configs to a pipeline you already run, instead of hand-rolling a shaded runtime. - One maintained, tested runtime instead of everyone reinventing shading. - No standalone job to schedule or babysit. A couple of things worth settling here: 1. Config-declared tables (explicit) vs. auto-detecting the written table from the query plan (more magic, less predictable). 2. Async vs Blocking sync for batch jobs? (a batch driver can exit before an async sync finishes). 3. Whether to also ship an Iceberg-style CALL xtable.sync(...) procedure in the same jar for on-demand/backfill use. I'll open a GitHub issue with the detailed design. Feedback welcome on the overall direction and the three questions above. Thanks, Vinish
