slachiewicz opened a new pull request, #906:
URL: https://github.com/apache/incubator-xtable/pull/906
Closes #905.
## What is the purpose of the pull request
The shade plugin in `xtable-utilities` had no `ServicesResourceTransformer`,
so every
`META-INF/services/*` file in the bundled jar was last-one-wins: whichever
artifact shade
processed last silently overwrote all the others' registrations.
## Brief change log
- Added `ServicesResourceTransformer` to the shade transformer list. One
line, no source
changes.
## Why it matters
Two user-visible bugs share this single cause.
**The bundled jar could not read Delta sources at all.** Delta's
`org.apache.spark.sql.delta.sources.DeltaDataSource` lost its
`DataSourceRegister` entry to
Spark's copy, so `sourceFormat: DELTA` failed with
`ClassNotFoundException: delta.DefaultSource` — despite the class and 1,616
`io/delta/*`
entries being present in the jar. Delta as a *target* was unaffected, which
is why this went
unnoticed.
**Only one Hadoop `FileSystem` registration survived.** That is very likely
why
`xtable-utilities/src/main/resources/xtable-hadoop-defaults.xml` has to name
`fs.file.impl`,
`fs.s3a.impl`, `fs.gs.impl` and the GCS `AbstractFileSystem` classes by hand
— a bare
`FsShell` off the jar reports `No FileSystem for scheme "file"` without
them. Those defaults
are left in place here; this PR does not attempt to remove them.
## Verify this pull request
Registrations recovered in the bundled jar:
| | before | after |
|---|---:|---:|
| `DataSourceRegister` delta entries | 0 | 1 |
| `org.apache.hadoop.fs.FileSystem` entries | 5 | 14 |
| total `META-INF/services/` files | 135 | 141 |
End to end against a real S3 bucket, same fresh Delta table, same command,
no workaround on the
classpath:
```
# before: exit 0, but
org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND]
Failed to find the data source: delta
# 0 metadata files written
# after:
INFO ConversionController:215 - Sync is successful for the following formats
ICEBERG
# 6 metadata files written
```
Verified: `java -jar xtable-utilities-*-bundled.jar --datasetConfig <delta
source> --hadoopConfig <s3a>` → `Sync is successful`, against live S3 on
Corretto 11.
Worth a reviewer's eye: merging service files is strictly more registration
than before, so an
engine or filesystem that was previously masked is now visible to
`ServiceLoader`. I did not
find a case where that is unwanted, but I have only exercised the Delta,
Iceberg, S3A and ABFS
paths.
Note for whoever merges: #897 also edits `xtable-utilities/pom.xml`, in the
`artifactSet` and
`filters` sections rather than `transformers`, so the two should not
conflict textually.
*This change was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
