Akash3121 opened a new pull request, #938:
URL: https://github.com/apache/incubator-xtable/pull/938
This PR is to work on #657
## What is the purpose of the pull request
This pull request prevents XTable from silently producing inconsistent
target metadata when a Delta source contains deletion vectors that conversion
targets cannot represent.
By default, synchronization now fails with a clear error when an active
Delta deletion vector is detected. Users can temporarily opt into the previous
lossy behavior by setting
`xtable.delta.source.allow_unsupported_deletion_vectors=true`, or
`allowUnsupportedDeletionVectors: "true"` in the bundled utility configuration.
When enabled, XTable logs a warning and continues while ignoring the deletion
vectors.
The protection applies to both the legacy Delta reader and the Delta Kernel
reader, for full and incremental synchronization. Incremental synchronization
also inspects the active snapshot, so deletion vectors are detected even when
they predate the synchronization watermark or there are no new commits.
## Brief change log
- Added fail-by-default detection for active Delta deletion vectors.
- Added an explicit warning-and-continue configuration override.
- Applied the behavior to legacy Delta and Delta Kernel full and
incremental sync paths.
- Aligned Delta Kernel handling of same-path remove/add actions with the
legacy Delta reader.
- Ensured one-shot CLI runs propagate unsupported-deletion-vector failures
instead of reporting success.
- Kept continuous mode running while logging failed synchronization
attempts.
- Added strict, fail-closed YAML parsing so only the literal value `true`
enables the unsafe override.
- Added source-only configuration support for bundled sync, catalog sync,
and REST service requests.
- Preserved compatibility with the existing `DeltaConversionSourceConfig`
constructor.
- Documented the override, data-consistency risk, and required full-rebuild
recovery procedure.
## Verify this pull request
This change added tests and can be verified as follows:
- Added unit tests for the default rejection policy, warning behavior,
configuration defaults, parsing, and constructor compatibility.
- Added utility tests for strict YAML parsing and source-only catalog
configuration.
- Added service tests for forwarding the Delta source configuration
property.
- Extended `ITDeltaDeleteVectorConvert` to create a real Delta table with
deletion vectors and verify both legacy Delta and Delta Kernel readers across
full and incremental synchronization.
- Verified that strict mode rejects an active deletion vector even when it
predates the synchronization watermark and the incremental backlog is empty.
- Verified that permissive mode retains the expected active files and
produces an empty file diff for a same-path deletion-vector update.
- Ran the Delta/Spark integration test in a Linux Docker container with
Java 11: 2 tests passed with 0 failures and 0 errors.
- Ran the focused core tests: 17 tests passed with 0 failures and 0 errors.
- Ran the focused utility tests: 4 tests passed with 0 failures and 0
errors.
- Ran the service tests: 3 tests passed with 0 failures and 0 errors.
- Ran Spotless checks for the affected Java modules.
*to validate the change/PR used AI assistance*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
