Fokko commented on PR #3974: URL: https://github.com/apache/iceberg-python/pull/3974#issuecomment-5720085732
Thanks for bringing this up, this is a very sensitive topic 😅 Iceberg is _very_ opinionated about targeting object stores, rather than filesystems. In object stores, list operations (list-prefix) are both slow and expensive because they are paged responses. Therefore, we don't want to make them first-class citizens in the FileIO. The list operation is also [missing from the reference implementation](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/io/FileIO.java). We could follow the same path there, where we have something like [SupportPrefixOperation](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/io/SupportsPrefixOperations.java). I can see PyIceberg being used to clean up small, low-volume orphan files. Ideally, I think you want to use something storage-specific, like [storage-inventory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
