salimTutuncu opened a new issue, #928: URL: https://github.com/apache/incubator-xtable/issues/928
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 ### Describe the bug Running a sync on a Delta source that has **no new commits** since the last sync always fails. This breaks the documented pattern of calling the sync after every write / on a schedule: any quiet interval makes the scheduled job exit nonzero, even though there is nothing to do. The Spark Runtime documentation states the sync "is safe to call after every write" (https://xtable.apache.org/docs/how-to-spark-runtime), and `XTableSyncService`'s javadoc says the call is "idempotent and self-healing"; neither holds for this case. Hudi and Parquet sources handle the same no-op correctly, so the expected behavior is already defined within the project. No corruption occurs: the watermark is untouched and the next sync after a real commit succeeds. ### Root cause `getCommitsBacklog` computes the incremental range as `start = versionAtLastSyncInstant + 1`, `end = latestVersion`, with no `start <= end` guard, and passes it to Kernel's `getChanges`: https://github.com/apache/incubator-xtable/blob/e8f22867a6cb297f58b248150220fce71188d16a/xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java#L174-L218 https://github.com/apache/incubator-xtable/blob/e8f22867a6cb297f58b248150220fce71188d16a/xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelIncrementalChangesState.java#L58-L89 When the table is unchanged, `start = end + 1` and Kernel rejects the inverted interval: ```text io.delta.kernel.exceptions.KernelException: Invalid version range: requested table changes for version range [5, 4] ``` ### To Reproduce Environment: JDK 17, `pyspark==3.5.5`; jars: xtable-spark-runtime 0.4.0, iceberg-spark-runtime-3.5 1.9.2, delta-spark 3.3.0 + delta-storage 3.3.0 (to write), delta-kernel-api/defaults 4.0.0 + delta-storage (to sync). The generator script is included at the bottom of this issue. ```bash # 1. a small partitioned Delta table spark-submit --jars delta-spark_2.12-3.3.0.jar,delta-storage-3.3.0.jar \ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \ gen_delta.py /tmp/orders_delta # 2. first sync: succeeds SYNC_JARS=delta-kernel-api-4.0.0.jar,delta-kernel-defaults-4.0.0.jar,delta-storage-3.3.0.jar,iceberg-spark-runtime-3.5_2.12-1.9.2.jar spark-submit --master 'local[2]' --class org.apache.xtable.spark.XTableSparkSync \ --jars "$SYNC_JARS" xtable-spark-runtime_2.12-0.4.0-incubating.jar \ --basepath /tmp/orders_delta --sourceformat DELTA --targets ICEBERG # "Sync is successful for the following formats ICEBERG" # 3. immediately re-run the SAME command with NO new Delta commits spark-submit --master 'local[2]' --class org.apache.xtable.spark.XTableSparkSync \ --jars "$SYNC_JARS" xtable-spark-runtime_2.12-0.4.0-incubating.jar \ --basepath /tmp/orders_delta --sourceformat DELTA --targets ICEBERG echo "exit=$?" ``` Observed on the second run: ```text io.delta.kernel.exceptions.KernelException: Invalid version range: requested table changes for version range [1, 0] exit=1 ``` ### Expected behavior A sync with no new source commits is a successful no-op (as it already is for Hudi and Parquet sources): detect `start > end`, log that there is nothing to sync, return success. ### Affected versions `0.4.0-incubating` artifact; the same unguarded range computation is present on current `main` (`DeltaKernelConversionSource.getCommitsBacklog` / `resetState`, re-checked 2026-09-04). #### Reproduction scripts <details><summary>gen_delta.py</summary> ```python import sys from pyspark.sql import SparkSession, functions as F path = sys.argv[1] spark = SparkSession.builder.getOrCreate() (spark.range(0, 1000).withColumn("order_id", F.col("id")) .withColumn("order_date", F.lit("2026-08-01")).drop("id") .write.format("delta").mode("overwrite").partitionBy("order_date").save(path)) print("wrote 1000 rows ->", path) ``` </details> ### Are you willing to submit PR? - [ ] I am willing to submit a PR! - [ ] I am willing to submit a PR but need help getting started! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
