vaibhavk1992 commented on issue #836: URL: https://github.com/apache/incubator-xtable/issues/836#issuecomment-4886726530
Hi @vinishjail97
Thanks for initiating this discussion.
I have made a flowchart please confirm if my understanding is correct. I’ve
also listed a few questions below.
---
```
┌─────────────────────────────────────────────────────┐
│ Spark Driver JVM │
│ │
│ Your job code │
│ df.write.format("hudi").save("/orders") │
│ │ │
│ ▼ write completes │
│ Spark fires onSuccess() │
│ │ │
│ ▼ ← still inside same JVM │
│ XTableSyncListener │
│ ConversionController.sync() │
│ │ │
│ ▼ │
│ Iceberg/Delta metadata written │
│ │
└─────────────────────────────────────────────────────┘
```
- Also which Spark versions will be supported — will there be separate
artifacts for Spark 3.3, 3.4, 3.5 like Hudi does
(spark3.3-bundle, spark3.4-bundle)?
- If CALL xtable.sync(table, targets) is shipped in the same jar, what is
the intended use case — ad-hoc backfill,
testing, or something else?
- Would the procedure support full sync (not just incremental) for cases
where you need to rebuild from scratch?
- How does this handle catalog-based tables (Unity Catalog, Hive
Metastore) where the table is referenced by name rather
than path?
**On the single-flight / concurrency model**
- If a sync is already running for table X and another write completes for
table X, is the second sync queued or dropped?
If dropped, how is it guaranteed to run later?
- What is the thread model for the driver executor — is it a single
dedicated thread per table or a shared pool?
- For batch jobs, what is the shutdown guarantee — if the job finishes
while a sync is still running, does Spark wait for
it or kill it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
