Trying to catch up and follow where this discussion is going, but I generally agree with JB's or Russels's comment that we should use the table's FileIO to perform these operations.
Falling back to the spark/hadoop config is a poor workaround for being able to perform a directory based traversal. On the other hand, I'm not in favor of adding a delimiter to the prefix listing as that hides a larger problem with how we perform the listing operations and it won't scale. WIth layouts like object store layout, you will end up performing huge numbers of list operations (which is already done with with the file system approach). I've discussed this in the past, and the right way to address this is to break up the listing namespace and use the prefix based listing to identify all objects under the table location. Recursive listing doesn't scale with object store layout. -Dan On Mon, Nov 9, 2026 at 5:17 AM Sebastian Daberdaku via dev < [email protected]> wrote: > Thank you, JB, for your input. > > So with this suggestion, if I understand correctly, prefix_listing would > become true by default (and maybe be dropped completely as an argument from > the remove_orphan_files procedure), and Hadoop possibly be removed > completely from the action, right? Or do you see applications where Hadoop > will still be required by this action, as in using a custom io-impl that > does not support prefix_listing? In this case, should I rewrite my PR to > derive the HadoopConfig from the io-impl if present, as Dan suggested? Or > is it not worth it and I should just close the PR and wait for #17790 to be > merged? > > Thanks again, > > Sebastian > > On Mon, Nov 9, 2026 at 11:49 AM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi Sebastian and all, >> >> First, I'm happy to be back on Iceberg (I was busy with other stuff and >> didn't have much time for Iceberg, I'm trying to optimize my time to >> balance between projects and the foundation :)). >> >> Back on this thread, the cross account scenario is clear, thanks >> Sebastian. I agree it's a real problem for multi-tenant deployments. >> >> That said, I agree with Russell and Dan: we should not fix it by giving >> the listing a Hadoop configuration when the table FileIO is S3FileIO. Your >> workaround needs the same role declared twice, once for the AWS SDK and >> once for s3a. That shows the real issue imho: the Hadoop walk is a second >> IO path that doesn't use the table FileIO. Adding catalogName() or >> hadoopConf() to the action would make that second path official instead of >> removing it. >> >> For me, the right fix is the option 2: make the FileIO listing path >> scale, so the listing goes through table.io() and uses the same >> credentials as the reads and deletes. I saw that PR #17790 adds >> delimiter-based listing to FileIO (S3, GCS, ADLS and Hadoop), which is the >> missing piece. With it, the action can walk the first levels on the driver >> and hand the sub-prefixes to the executors with the table FileIO, the same >> way the Hadoop walk does today. I think we should focus on #17790 (I will >> do the review :)), then, with that, we can build the parallel FileIO walk >> on top of it. Once done, we can also discuss using FileIO listing by >> default when table.io() supports it. >> >> I'm not in favor of option 1. Making ResolvingFileIO the Glue default >> would change the default for every Glue user and pull Hadoop onto the >> classpath, just to fix one Spark maintenance action. >> >> Regards >> JB >> > > *This communication, including any attachments, is intended solely for the > recipient(s) addressed. * > *If you have received this email in error, please notify the sender > immediately and delete it from your system. * > *Any unauthorized use, disclosure, or distribution, of this communication > is strictly prohibited.* >
