tom-s-powell opened a new pull request, #17594:
URL: https://github.com/apache/iceberg/pull/17594
`SupportsPrefixOperations.listPrefix` is a generic API whose callers feed
each `FileInfo.location()` straight back into a `FileIO` (or use it to compare
against Iceberg metadata locations). The two other cloud implementations honour
this contract by returning fully-qualified URIs:
* `S3FileIO.listPrefix` → `s3://bucket/key`
* `GCSFileIO.listPrefix` → `gs://bucket/blob`
`ADLSFileIO.listPrefix`, however, was returning the raw `PathItem.getName()`
from the Azure SDK, which is only the container-relative path (e.g. dir/file).
That means the location it emits:
* cannot be re-parsed by `ADLSLocation` (its regex requires an
abfs[s]/wasb[s] scheme, so anything passed back through
`newInputFile`/`deleteFile` throws a `ValidationException`),
* is not round-trippable through `deletePrefix` the way S3/GCS are,
* silently misbehaves in generic prefix walkers like
`FileSystemWalker.listDirRecursivelyWithFileIO` and downstream actions like
`DeleteOrphanFilesSparkAction` / `RemoveOrphanFilesProcedure`, which rely on
`FileInfo.location()` being a usable location string.
### What changed
* `ADLSLocation` now captures and exposes the URI scheme (abfs, abfss, wasb,
wasbs) so it can be reconstituted on the way out.
* `ADLSFileIO.listPrefix` reconstructs a fully-qualified URI
(`<scheme>://<container>@<host>/<path>`, or `<scheme>://<host>/<path>` when no
container is present) from the input prefix's scheme/host/container and the
`PathItem` name, bringing it in line with `S3FileIO` and `GCSFileIO`.
* Tests updated to assert the fully-qualified form and to cover
`ADLSLocation#scheme()`.
### Compatibility
This is a behaviour change for any caller that was relying on the previous
container-relative return value. Given the existing behaviour was inconsistent
with the interface contract and with every other `DelegateFileIO`
implementation - and would break in-tree callers such as orphan-file cleanup on
ADLS - this is a bug fix rather than a semantic break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
