slachiewicz opened a new issue, #905:
URL: https://github.com/apache/incubator-xtable/issues/905
### Describe the bug
Running the bundled jar exactly as the README documents fails for any
dataset with
`sourceFormat: DELTA`:
```
java -jar
xtable-utilities/target/xtable-utilities_2.12-0.5.0-SNAPSHOT-bundled.jar \
--datasetConfig my_config.yaml
```
```
ERROR org.apache.xtable.utilities.RunSync:188 - Error running sync for <path>
org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND]
Failed to find the data source: delta.
Caused by: java.lang.ClassNotFoundException: delta.DefaultSource
```
Delta as a *target* format is unaffected — `ICEBERG` -> `DELTA` succeeds
with the same jar.
Only reading a Delta source is broken.
### Root cause
The shaded jar's
`META-INF/services/org.apache.spark.sql.sources.DataSourceRegister` contains
only Spark's own 12 entries. Delta's entry,
`org.apache.spark.sql.delta.sources.DeltaDataSource`, is absent, so Spark's
short-name lookup
for `delta` finds nothing and falls back to the legacy `delta.DefaultSource`
guess, which does
not exist.
The classes themselves are present and correct — 1,616 `io/delta/*` entries
and
`org/apache/spark/sql/delta/sources/DeltaDataSource.class` are all in the
jar. Only the
ServiceLoader registration is missing, so this is a packaging problem rather
than a missing
dependency.
### To Reproduce
1. Build: `mvn -pl xtable-utilities -am package -DskipTests`
2. Point a dataset config at any Delta table with `sourceFormat: DELTA`.
3. Run the bundled jar as above.
Confirm the registration is missing without running anything:
```
unzip -p xtable-utilities/target/*-bundled.jar \
META-INF/services/org.apache.spark.sql.sources.DataSourceRegister | grep
-c delta
# 0
unzip -l xtable-utilities/target/*-bundled.jar | grep -c
"sql/delta/sources/DeltaDataSource"
# 2
```
### Expected behavior
The merged `DataSourceRegister` should carry Delta's provider alongside
Spark's, and
`sourceFormat: DELTA` should work from the bundled jar without extra
classpath entries.
### Workaround
Put a directory ahead of the jar on the classpath holding a one-line service
file:
```
mkdir -p fix/META-INF/services
echo org.apache.spark.sql.delta.sources.DeltaDataSource \
> fix/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
java -cp fix:xtable-utilities/target/*-bundled.jar
org.apache.xtable.utilities.RunSync \
--datasetConfig my_config.yaml
```
With this in place the same sync succeeds.
### Environment
- `main` at ef88e02, version 0.5.0-SNAPSHOT
- Temurin/Corretto 11, Linux x86_64
- Reproduced against both real S3 (`s3a://`) and ADLS Gen2 (`abfss://`); the
storage backend is
irrelevant, the failure is at data source resolution before any I/O.
### Two related observations
**`RunSync` exits 0 when a dataset fails to sync.** Every failure above
returned exit status 0;
the error appears only in the log. This is already in scope for #889, noting
it here only
because it is why the failure is easy to miss in a scripted run.
**The `FileSystem` service registration does not resolve either.** A bare
`org.apache.hadoop.fs.FsShell` off the bundled jar reports `No FileSystem
for scheme "file"`
unless `fs.file.impl` is set explicitly.
`xtable-utilities/src/main/resources/xtable-hadoop-defaults.xml`
already sets `fs.file.impl`, `fs.s3a.impl` and the GCS equivalents by hand,
which suggests this
was worked around at some point rather than fixed. It may share a root cause
with the Delta
registration.
Related: #736 is the same shape — a class present in the dependency tree but
absent from the
shaded jar at runtime.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
*This issue was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
