+1 for this, Looking forward to more discussions on this. I have added a comment over the issue.
On Wed, Jul 1, 2026 at 6:28 PM Tim Brown <[email protected]> wrote: > +1 to this package. I think even without the listeners this will provide a > more clear way to use code in your current spark pipelines. > > -Tim > > On Wed, Jul 1, 2026 at 1:30 AM Vinoth Chandar <[email protected]> wrote: > > > Really look forward to this. > > > > On 2026/07/01 01:37:25 Vinish Reddy Pannala wrote: > > > Hi all, > > > > > > I'd like to propose a new artifact, xtable-spark-runtime, to make > XTable > > > dramatically easier to use inside existing Spark pipelines. > > > > > > Problem > > > ------- > > > XTable's modules are published to Maven, so today a user can pull in > > > xtable-core and assemble their own runtime, but everyone has to solve > the > > > same shading/classpath problem themselves, or fall back to the > > > xtable-utilities bundle, which ships every dependency unshaded (~1GB) > and > > > is impractical to add to a Spark job. Running it also means building > the > > > fat jar and invoking a separate sync process with several config files. > > > There's no maintained, thin, drop-in artifact for the common case: "I > > > already write this table with Spark, keep it in sync in other formats." > > > > > > Proposal > > > -------- > > > A thin, relocated runtime jar that a user adds to an existing Spark > job: > > > spark-submit --packages > > > org.apache.xtable:xtable-spark-runtime_2.12:<version>...and activates > > > purely through config, no code change: > > > > > > spark.sql.queryExecutionListeners = > > > org.apache.xtable.spark.XTableSyncListener > > > spark.xtable.tables = /warehouse/db/orders > > > spark.xtable.orders.sourceFormat = HUDI > > > spark.xtable.orders.targets = ICEBERG,DELTA > > > > > > After each successful write to the source table, XTable incrementally > > > translates its metadata to the target formats on the driver, so the > table > > > becomes readable by other engines with no extra job or orchestration. > > > > > > This follows the bundle model Hudi and Iceberg already use: engine deps > > > marked provided, a curated dependency allowlist, and relocations so the > > jar > > > coexists with the cluster's own guava/jackson/avro. The result should > be > > a > > > bundle in the tens of MB, not ~1GB. > > > > > > Why it helps the community > > > -------------------------- > > > - Lowest-friction on-ramp: add one dependency + a few configs to a > > pipeline > > > you already run, instead of hand-rolling a shaded runtime. > > > - One maintained, tested runtime instead of everyone reinventing > shading. > > > - No standalone job to schedule or babysit. > > > > > > A couple of things worth settling here: > > > 1. Config-declared tables (explicit) vs. auto-detecting the written > table > > > from the query plan (more magic, less predictable). > > > 2. Async vs Blocking sync for batch jobs? (a batch driver can exit > before > > > an async sync finishes). > > > 3. Whether to also ship an Iceberg-style CALL xtable.sync(...) > procedure > > in > > > the same jar for on-demand/backfill use. > > > > > > I'll open a GitHub issue with the detailed design. Feedback welcome on > > the > > > overall direction and the three questions above. > > > > > > Thanks, > > > Vinish > > > > > >
