aarushigupta132 opened a new pull request, #17795: URL: https://github.com/apache/iceberg/pull/17795
## Summary This PR extends `SupportsPrefixOperations` with delimited prefix listing. The new `listPrefix(prefix, delimiter)` method returns files directly below a prefix and common sub-prefixes grouped by the delimiter. Results are returned lazily as pages so implementations can retain the pagination behavior of the underlying storage service. The `supportsPrefixListingWithDelimiter(prefix, delimiter)` method lets a `FileIO` report whether it supports a specific delimiter. The new operation has default implementations, so existing `SupportsPrefixOperations` implementations remain compatible. Delimited prefix listing is implemented for S3, GCS, Azure Data Lake Storage, and Hadoop `FileIO`. Wrapper implementations, including `ResolvingFileIO` and `EncryptingFileIO`, delegate the capability to their underlying `FileIO`. ## Motivation When `prefix_listing=true`, `remove_orphan_files` currently lists all files from the Spark driver on one thread. For tables with many files, this creates a driver bottleneck and requires the driver to construct the listed-file dataset before Spark can use it in the metadata anti-join. The Hadoop listing path already limits driver-side recursive listing. It discovers directories to a configured depth and then distributes the remaining directory listings across Spark executors. This PR applies the same approach to prefix-based listing. ## Spark behavior The new `prefix_listing_max_seed_depth` argument controls how many levels of common sub-prefixes are discovered on the driver before listing is distributed to Spark executors. When `prefix_listing_max_seed_depth=0`, which is the default, the table location is used as a single seed prefix. Listing therefore uses one Spark partition, but runs on an executor instead of the driver. When the value is greater than zero, the driver uses delimited prefix listing to discover seed prefixes up to that depth. Spark then distributes those prefixes, subject to the configured listing parallelism, so executors can list them concurrently. Files found during seed discovery are retained and combined with the executor-side results. The resulting listed-file dataset remains available to Spark because it is subsequently used in the anti-join against files referenced by Iceberg metadata. ## Testing Unit tests are added for all affected `FileIO` implementations and the Spark 4.1 `RemoveOrphanFilesProcedure`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
