Really look forward to this.
On 2026/07/01 01:37:25 Vinish Reddy Pannala wrote: > Hi all, > > I'd like to propose a new artifact, xtable-spark-runtime, to make XTable > dramatically easier to use inside existing Spark pipelines. > > Problem > ------- > XTable's modules are published to Maven, so today a user can pull in > xtable-core and assemble their own runtime, but everyone has to solve the > same shading/classpath problem themselves, or fall back to the > xtable-utilities bundle, which ships every dependency unshaded (~1GB) and > is impractical to add to a Spark job. Running it also means building the > fat jar and invoking a separate sync process with several config files. > There's no maintained, thin, drop-in artifact for the common case: "I > already write this table with Spark, keep it in sync in other formats." > > Proposal > -------- > A thin, relocated runtime jar that a user adds to an existing Spark job: > spark-submit --packages > org.apache.xtable:xtable-spark-runtime_2.12:<version>...and activates > purely through config, no code change: > > spark.sql.queryExecutionListeners = > org.apache.xtable.spark.XTableSyncListener > spark.xtable.tables = /warehouse/db/orders > spark.xtable.orders.sourceFormat = HUDI > spark.xtable.orders.targets = ICEBERG,DELTA > > After each successful write to the source table, XTable incrementally > translates its metadata to the target formats on the driver, so the table > becomes readable by other engines with no extra job or orchestration. > > This follows the bundle model Hudi and Iceberg already use: engine deps > marked provided, a curated dependency allowlist, and relocations so the jar > coexists with the cluster's own guava/jackson/avro. The result should be a > bundle in the tens of MB, not ~1GB. > > Why it helps the community > -------------------------- > - Lowest-friction on-ramp: add one dependency + a few configs to a pipeline > you already run, instead of hand-rolling a shaded runtime. > - One maintained, tested runtime instead of everyone reinventing shading. > - No standalone job to schedule or babysit. > > A couple of things worth settling here: > 1. Config-declared tables (explicit) vs. auto-detecting the written table > from the query plan (more magic, less predictable). > 2. Async vs Blocking sync for batch jobs? (a batch driver can exit before > an async sync finishes). > 3. Whether to also ship an Iceberg-style CALL xtable.sync(...) procedure in > the same jar for on-demand/backfill use. > > I'll open a GitHub issue with the detailed design. Feedback welcome on the > overall direction and the three questions above. > > Thanks, > Vinish >
