Hi Daniel and Russell, Thanks for taking another look. I checked whether reading the configuration off the FileIO would work, and it runs into two problems:
1. S3FileIO (as well as GCSFileIO and ADLSFileIO) does not implement HadoopConfigurable, so there is no Hadoop configuration to read from table.io(). The setup in #17860 uses io-impl = S3FileIO, so for the reported case the check would fall through to the session configuration and the listing would still fail with AccessDenied. 2. Where the FileIO is HadoopConfigurable (HadoopFileIO, ResolvingFileIO), the configuration it holds is the one SparkCatalog.buildIcebergCatalog passes in, which is already SparkUtil.hadoopConfCatalogOverrides(spark, name). In those cases the two approaches produce the same configuration. The spark.sql.catalog.<name>.hadoop.* overrides are the only place a user can tell the Hadoop listing how to authenticate when the catalog's FileIO is not Hadoop-based, which is why the PR routes through them. Best, Sebastian On Fri, Oct 30, 2026 at 12:11 AM Russell Spitzer <[email protected]> wrote: > Sorry "bit disconnected" > > On Thu, Oct 29, 2026 at 6:10 PM Russell Spitzer <[email protected]> > wrote: > >> I would check out Dan's suggestion. We should try any solutions we can >> that avoid expanding the API, especially when the connection here between >> the behavior and the API is a bad disconnected. >> >> On Thu, Oct 29, 2026 at 4:53 PM Daniel Weeks <[email protected]> wrote: >> >>> Sebastian, >>> >>> It's been a little while since I looked at this, but wouldn't it make >>> more sense to interrogate the FileIO to see if it's HadoopConfigurable and >>> grab the exact configuration used by the IO rather than go through this >>> catalog path? >>> >>> At that point, you'd have the exact configuration and overrides for that >>> specific table. >>> >>> -Dan >>> >>> On Wed, Oct 28, 2026 at 1:15 AM Sebastian Daberdaku via dev < >>> [email protected]> wrote: >>> >>>> Hi all, >>>> >>>> I'd like to get some eyes on a PR that fixes a credentials mismatch in >>>> remove_orphan_files, and agree on the approach before I port it across >>>> Spark versions: >>>> >>>> https://github.com/apache/iceberg/pull/17862 (closes #17860) >>>> >>>> Problem >>>> >>>> Unless prefix_listing is enabled, DeleteOrphanFilesSparkAction lists >>>> the table location with a Hadoop FileSystem built from the Spark session >>>> configuration alone, while every other file operation in the action goes >>>> through table.io(). When a catalog reaches its storage with >>>> credentials other than the cluster's (e.g. a catalog that assumes a role in >>>> another AWS account), reads and deletes succeed but the listing fails with >>>> AccessDenied or silently sees no files. >>>> >>>> Change >>>> >>>> - DeleteOrphanFilesSparkAction gets a new catalogName(String) method. >>>> When set, the listing uses the session Hadoop configuration plus that >>>> catalog's spark.sql.catalog.<name>.hadoop.* overrides >>>> (SparkUtil.hadoopConfCatalogOverrides). When not set, behavior is >>>> unchanged. >>>> - RemoveOrphanFilesProcedure passes the catalog it was invoked on, so >>>> CALL <catalog>.system.remove_orphan_files(...) picks up that catalog's >>>> overrides with no new procedure argument. >>>> - spark-procedures.md documents which configuration the listing uses. >>>> >>>> The catalog is passed explicitly rather than inferred from >>>> Table.name(): as @yangshangqing95 pointed out in review, a table name's >>>> first segment isn't guaranteed to identify the Spark catalog it came from, >>>> and resolving it via CatalogManager can initialize unrelated catalogs as a >>>> side effect. >>>> >>>> Tests cover the procedure path across the test catalogs and the action >>>> API with both SparkCatalog and SparkSessionCatalog, using a marker >>>> FileSystem registered only in the catalog's hadoop.* overrides. >>>> >>>> Questions >>>> >>>> 1. Is a catalogName(String) method on the action an acceptable public >>>> API addition, or would people prefer a different shape? >>>> 2. The PR was opened against Spark 4.1 before the 4.2 module landed. If >>>> the direction is agreed, I'll move the change to 4.2 and backport it to >>>> 4.1, 4.0 and 3.5. >>>> >>>> Thanks, >>>> Sebastian Daberdaku >>>> -- >>>> >>>> >>>> >>>> >>>> Sebastian Daberdaku >>>> >>>> Head of Platform & Data Engineering >>>> >>>> Via San Francesco d >>>> <https://www.google.com/maps/search/Via+San+Francesco+d+Assisi+15,+20122+Milan,+Italy?entry=gmail&source=g> >>>> 'Assisi 15, 20122 Milan, Italy >>>> <https://www.google.com/maps/search/Via+San+Francesco+d+Assisi+15,+20122+Milan,+Italy?entry=gmail&source=g> >>>> >>>> 40 New Bond St >>>> <https://www.google.com/maps/search/40+New+Bond+St?entry=gmail&source=g>., >>>> W1S 2DE, London, United Kingdom >>>> >>>> Vaso Pasha St., Rainbow Center, Tirana, Albania Musa Tolaj St., Lam A, >>>> 2/104, Prishtina, Kosova >>>> <https://www.google.com/maps/search/104,+Prishtina,+Kosova?entry=gmail&source=g> >>>> >>>> www.cardoai.com >>>> >>>> >>>> *This communication, including any attachments, is intended solely for >>>> the recipient(s) addressed. * >>>> *If you have received this email in error, please notify the sender >>>> immediately and delete it from your system. * >>>> *Any unauthorized use, disclosure, or distribution, of this >>>> communication is strictly prohibited.* >>>> >>> -- Sebastian Daberdaku Head of Platform & Data Engineering Via San Francesco d'Assisi 15, 20122 Milan, Italy 40 New Bond St., W1S 2DE, London, United Kingdom Vaso Pasha St., Rainbow Center, Tirana, Albania Musa Tolaj St., Lam A, 2/104, Prishtina, Kosova www.cardoai.com -- *This communication, including any attachments, is intended solely for the recipient(s) addressed. * *If you have received this email in error, please notify the sender immediately and delete it from your system. * *Any unauthorized use, disclosure, or distribution, of this communication is strictly prohibited.*
