vinishjail97 commented on issue #836: URL: https://github.com/apache/incubator-xtable/issues/836#issuecomment-4897413436
Thanks @vaibhavk1992 — flowchart's right, one tweak: `sync()` doesn't run on Spark's listener callback thread (blocking the `ListenerBus` risks dropped events). The listener just resolves the table and hands off to a debounced, single-flight-per-table executor; `ConversionController.sync()` runs there, same driver JVM. **Spark versions:** Right now the version matters mainly via the `delta-core` path (Delta-on-Spark, in-process). But we already have a full Delta **Kernel** implementation (`org.apache.xtable.kernel.*`, Spark-free) — @vaibhavk1992, since you authored the Kernel path: **what are the remaining blockers to making Delta Kernel the default for Delta conversion?** If we can close those, this runtime stops depending on a specific Spark line for Delta, and a single `xtable-spark-runtime_2.12` bundle could serve both 3.5 and 4 (the listener APIs are stable across versions) — which would make the whole "per-Spark-version artifact" question mostly moot. Until Kernel is the default, we'd target Spark 3.5 first with Spark 4 as a follow-up. **`CALL xtable.sync(...)`:** ad-hoc/imperative path — onboarding an existing table, re-sync after a schema change, notebook/`spark-sql`, CI. Complements the listener; optional/stretch. **Full sync:** yes — engine already has `SyncMode.FULL` vs `INCREMENTAL` (and INCREMENTAL auto-falls back to full when unsafe). "Rebuild from scratch" is just `syncMode=FULL`. **Catalog tables (UC/HMS by name):** engine already supports name-based target catalogs (`syncTableAcrossCatalogs`, `CatalogSync`, Glue/HMS modules). Listener MVP is path-based; resolving name→path via the session catalog is a clean follow-up, with UC (resolution + permissions) post-MVP. **Concurrency:** - Per table we keep at most one sync running at a time. If more writes to table X land while X's sync is still in flight, we don't start a parallel sync and we don't queue one-per-write — we just set a "rerun needed" marker. When the current sync finishes, if that marker is set we run exactly one more sync, and because sync is incremental that single catch-up run picks up *all* the commits that accumulated in the meantime. So many writes collapse into at most one follow-up run, nothing is dropped — and even a fully missed trigger self-heals on the next commit. - Bounded pool keyed per table: different tables sync in parallel, each individual table stays serialized. - Batch shutdown: hook drains pending syncs with a bounded timeout before JVM exit. Considering a `spark.xtable.sync.blocking=true` toggle for jobs that must sync before exit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
