Note: "Config-declared tables (explicit) vs. auto-detecting the written table".
Can already be achieved thanks to https://github.com/apache/incubator-xtable/pull/830 (by me). On Sun, 5 Jul 2026, 18:25 Vaibhav Kumar, <[email protected]> wrote: > +1 for this, Looking forward to more discussions on this. I have added a > comment over the issue. > > On Wed, Jul 1, 2026 at 6:28 PM Tim Brown <[email protected]> wrote: > > > +1 to this package. I think even without the listeners this will provide > a > > more clear way to use code in your current spark pipelines. > > > > -Tim > > > > On Wed, Jul 1, 2026 at 1:30 AM Vinoth Chandar <[email protected]> wrote: > > > > > Really look forward to this. > > > > > > On 2026/07/01 01:37:25 Vinish Reddy Pannala wrote: > > > > Hi all, > > > > > > > > I'd like to propose a new artifact, xtable-spark-runtime, to make > > XTable > > > > dramatically easier to use inside existing Spark pipelines. > > > > > > > > Problem > > > > ------- > > > > XTable's modules are published to Maven, so today a user can pull in > > > > xtable-core and assemble their own runtime, but everyone has to solve > > the > > > > same shading/classpath problem themselves, or fall back to the > > > > xtable-utilities bundle, which ships every dependency unshaded (~1GB) > > and > > > > is impractical to add to a Spark job. Running it also means building > > the > > > > fat jar and invoking a separate sync process with several config > files. > > > > There's no maintained, thin, drop-in artifact for the common case: "I > > > > already write this table with Spark, keep it in sync in other > formats." > > > > > > > > Proposal > > > > -------- > > > > A thin, relocated runtime jar that a user adds to an existing Spark > > job: > > > > spark-submit --packages > > > > org.apache.xtable:xtable-spark-runtime_2.12:<version>...and activates > > > > purely through config, no code change: > > > > > > > > spark.sql.queryExecutionListeners = > > > > org.apache.xtable.spark.XTableSyncListener > > > > spark.xtable.tables = /warehouse/db/orders > > > > spark.xtable.orders.sourceFormat = HUDI > > > > spark.xtable.orders.targets = ICEBERG,DELTA > > > > > > > > After each successful write to the source table, XTable incrementally > > > > translates its metadata to the target formats on the driver, so the > > table > > > > becomes readable by other engines with no extra job or orchestration. > > > > > > > > This follows the bundle model Hudi and Iceberg already use: engine > deps > > > > marked provided, a curated dependency allowlist, and relocations so > the > > > jar > > > > coexists with the cluster's own guava/jackson/avro. The result should > > be > > > a > > > > bundle in the tens of MB, not ~1GB. > > > > > > > > Why it helps the community > > > > -------------------------- > > > > - Lowest-friction on-ramp: add one dependency + a few configs to a > > > pipeline > > > > you already run, instead of hand-rolling a shaded runtime. > > > > - One maintained, tested runtime instead of everyone reinventing > > shading. > > > > - No standalone job to schedule or babysit. > > > > > > > > A couple of things worth settling here: > > > > 1. Config-declared tables (explicit) vs. auto-detecting the written > > table > > > > from the query plan (more magic, less predictable). > > > > 2. Async vs Blocking sync for batch jobs? (a batch driver can exit > > before > > > > an async sync finishes). > > > > 3. Whether to also ship an Iceberg-style CALL xtable.sync(...) > > procedure > > > in > > > > the same jar for on-demand/backfill use. > > > > > > > > I'll open a GitHub issue with the detailed design. Feedback welcome > on > > > the > > > > overall direction and the three questions above. > > > > > > > > Thanks, > > > > Vinish > > > > > > > > > >
