slachiewicz opened a new issue, #907:
URL: https://github.com/apache/incubator-xtable/issues/907
### Describe the bug
When a Delta commit references a data file by absolute path, XTable
concatenates that path onto
the table base instead of using it. The resulting Iceberg manifest entry
points at a location
that does not exist. The entry carries `record_count=0` and is pruned during
scan planning, so
no reader ever tries to open it — the sync reports success and the converted
table silently
returns fewer rows than the source, with no error anywhere.
The Delta protocol permits absolute paths. From `PROTOCOL.md`, Add File
action:
> A relative path to a data file from the root of the table **or an absolute
path to a file**
> that should be added to the table.
The same wording appears for the Remove File action.
### Root cause
`DeltaActionsConverter.getFullPathToFile` is the whole of the resolution
logic:
```java
static String getFullPathToFile(String tableBasePath, String dataFilePath) {
if (dataFilePath.startsWith(tableBasePath)) {
return dataFilePath;
}
return tableBasePath + Path.SEPARATOR + dataFilePath;
}
```
`startsWith` is doing prefix matching where path containment is meant, so
any absolute path not
lexically under the table base is treated as relative. Calling the method
directly:
| tableBasePath | dataFilePath | result |
|---|---|---|
| `s3a://bucket/tab` | `part-0.parquet` | ok |
| `s3a://bucket/tab` | `s3a://bucket/tab/part-0.parquet` | ok |
| `s3a://bucket/tab` | `s3://bucket/tab/part-0.parquet` |
`s3a://bucket/tab/s3://bucket/tab/part-0.parquet` |
| `s3a://bucket/tab` | `s3a://other-bucket/x/part-0.parquet` |
`s3a://bucket/tab/s3a://other-bucket/x/part-0.parquet` |
| `abfss://[email protected]/tab` | `gs://bucket/t/part-0.parquet`
| concatenated |
| `s3a://bucket/tab` | `file:///local/t/part-0.parquet` | concatenated |
Note the third and fourth rows are the same defect, not two: a scheme alias
(`s3://` against an
`s3a://` base) and a same-scheme different-location path both fail the
prefix test. The second
shape is what a shallow clone produces.
### To Reproduce
1. Write a Delta table. Write a parquet file somewhere outside it.
2. Add that file to the table by absolute URI. With `deltalake` 1.6.3:
```python
from deltalake import DeltaTable
from deltalake.transaction import AddAction
dt = DeltaTable(table_path)
dt.create_write_transaction(
[AddAction(path="file:///abs/path/outside.parquet", size=...,
partition_values={},
modification_time=..., data_change=True, stats=...)],
"append", schema, partition_by=[])
```
The absolute URI is persisted verbatim — no validation, no rewriting.
3. Run `RunSync` DELTA → ICEBERG against the table.
Observed, with a 2-file / 5-row source:
```
Sync is successful for the following formats ICEBERG exit 0
manifest entries:
status=ADDED records=0
file:/.../table/file:///.../elsewhere/part-00000-....parquet <- base
+ absolute URI
status=ADDED records=3
file:/.../table/part-00000-....parquet
snapshot summary: total-data-files=2 total-records=3
```
PyIceberg's `plan_files` returns only the second entry — the mangled one is
pruned before any
I/O is attempted — and the scan returns **3 rows**. The two rows behind the
absolute path are
absent, with no error at any point.
### Expected behavior
An absolute path in the Delta log should be used as-is. If XTable cannot
support a data file
outside the table root, it should fail rather than emit a manifest entry
pointing at a
concatenated path.
### On how reachable this is
Being straight about the evidence, since it affects how much this matters:
- The protocol permits absolute paths, quoted above.
- delta-spark passes such an entry through to XTable — that is how the
conversion sees it.
- delta-rs persists one through its public API without complaint, shown
above. Its own writers
emit relative paths, so it is not a spontaneous producer, but it is a seam.
What I cannot show is a product that emits absolute paths by default.
Databricks shallow clone
is the commonly cited producer and I have no way to test it. So this may be
a latent defect
rather than one users are hitting today — but the input is legal, the
failure is silent, and the
fix is small.
### Related, but not the same
#323 covers the reverse direction — the Delta *target* converting absolute
paths to relative in
`DeltaDataFileUpdatesExtractor`, and asks for that conversion to be
customisable. This report is
about the Delta *source* resolving a logged path in `DeltaActionsConverter`.
Same subject area,
different code path, and neither fix would address the other. #171 and #230
are also path-related
but concern the table base path and the Delta target respectively.
### Environment
- XTable `main`, version 0.5.0-SNAPSHOT, bundled jar, Corretto 11
- `deltalake` 1.6.3 / PyArrow 25.0.1 wrote the fixture; PyIceberg read the
result
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
*This issue was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
