dev-donghwan opened a new pull request, #9814:
URL: https://github.com/apache/paimon/pull/9814
### Purpose
`paimon-s3` registers only the `s3` scheme, so a table whose location starts
with `s3a://` cannot be opened by the plugin:
```
discovered schemes = [s3, viewfs, file, hdfs]
org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io
implementation
for scheme 's3a' in the classpath. Hadoop FileSystem also cannot access this
path
's3a://<bucket>/t/x.txt'.
```
`s3a://` is not an exotic spelling, it is what a Hadoop 3 deployment
produces. The S3 and S3N clients were removed from Hadoop 3, and the Hadoop
copy bundled inside this very plugin shows it: `core-default.xml` declares
`fs.s3a.impl` only, and the one remaining `s3n` class fails with `The s3n://
client to Amazon S3 is no longer available: please migrate to the s3a://
client`. A Hive Metastore therefore stores `s3a://` locations, and every engine
that resolves a table through HMS hands that path to Paimon.
Engines that ship their own `FileIO` can paper over this with a fallback
loader, but the ones that do not, Spark and StarRocks for example, have nothing
to fall back to (see #7835 for the same failure on OBS). The documented
alternative, `location-in-properties`, moves the location out of the Hive
`StorageDescriptor` into table properties, which keeps the path away from
Hive's own filesystem but also hides it from every reader that looks at the
standard field, and it has its own history of problems (#2729, #3309).
The fix is small because both schemes already share one implementation:
- `S3FileIO.createFileSystem` builds `new S3AFileSystem()` and initializes
it with the URI as given, so nothing in the plugin depends on the scheme
spelling.
- `S3AFileSystem.getScheme()` returns a field that is set from that URI, not
a constant.
- `S3FileIO.CONFIG_PREFIXES` already accepts `s3.`, `s3a.` and `fs.s3a.`,
and maps all of them onto `fs.s3a.*`.
So this PR adds an `S3ALoader` that reports `s3a` and reuses `S3Loader`
unchanged. `FileIO.discoverLoaders` keys loaders by scheme and only rejects
duplicates of the same scheme, so registering a second loader needs no
interface change.
For what it is worth, PyPaimon already treats the schemes as equivalent
(`pyarrow_file_io.py`: `elif scheme in {"s3", "s3a", "s3n"}`), so this also
brings the Java side in line.
### Tests
`S3ASchemeTest`, backed by the existing MinIO container, five cases:
- `s3` and `s3a` are both present in the loader table and map to their own
loaders
- write, read, list and delete over `s3a://`
- an object written through `s3://` is readable through `s3a://`
- the Hadoop filesystem actually created is `S3AFileSystem` for both
schemes, asserted by unwrapping the plugin `FileIO`
```
[s3] hadoop FileSystem = org.apache.hadoop.fs.s3a.S3AFileSystem,
getScheme() = s3, getUri() = s3://bucket-jhrebf
[s3a] hadoop FileSystem = org.apache.hadoop.fs.s3a.S3AFileSystem,
getScheme() = s3a, getUri() = s3a://bucket-jhrebf
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
```
The test module needed `hadoop-hdfs-client` and `paimon-hadoop-shaded-3.4`
in test scope, because `CatalogContext` loads `HdfsConfiguration`.
### API and Backward Compatibility
No interface change and no behaviour change for `s3://` paths. Paths that
used to fail with `UnsupportedSchemeException` now resolve to the same `FileIO`
the `s3` scheme has always used.
### Documentation
The supported filesystems table now lists `s3://, s3a://` for S3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
