I would check out Dan's suggestion. We should try any solutions we can that avoid expanding the API, especially when the connection here between the behavior and the API is a bad disconnected.
On Thu, Oct 29, 2026 at 4:53 PM Daniel Weeks <[email protected]> wrote: > Sebastian, > > It's been a little while since I looked at this, but wouldn't it make more > sense to interrogate the FileIO to see if it's HadoopConfigurable and grab > the exact configuration used by the IO rather than go through this catalog > path? > > At that point, you'd have the exact configuration and overrides for that > specific table. > > -Dan > > On Wed, Oct 28, 2026 at 1:15 AM Sebastian Daberdaku via dev < > [email protected]> wrote: > >> Hi all, >> >> I'd like to get some eyes on a PR that fixes a credentials mismatch in >> remove_orphan_files, and agree on the approach before I port it across >> Spark versions: >> >> https://github.com/apache/iceberg/pull/17862 (closes #17860) >> >> Problem >> >> Unless prefix_listing is enabled, DeleteOrphanFilesSparkAction lists the >> table location with a Hadoop FileSystem built from the Spark session >> configuration alone, while every other file operation in the action goes >> through table.io(). When a catalog reaches its storage with credentials >> other than the cluster's (e.g. a catalog that assumes a role in another AWS >> account), reads and deletes succeed but the listing fails with AccessDenied >> or silently sees no files. >> >> Change >> >> - DeleteOrphanFilesSparkAction gets a new catalogName(String) method. >> When set, the listing uses the session Hadoop configuration plus that >> catalog's spark.sql.catalog.<name>.hadoop.* overrides >> (SparkUtil.hadoopConfCatalogOverrides). When not set, behavior is unchanged. >> - RemoveOrphanFilesProcedure passes the catalog it was invoked on, so >> CALL <catalog>.system.remove_orphan_files(...) picks up that catalog's >> overrides with no new procedure argument. >> - spark-procedures.md documents which configuration the listing uses. >> >> The catalog is passed explicitly rather than inferred from Table.name(): >> as @yangshangqing95 pointed out in review, a table name's first segment >> isn't guaranteed to identify the Spark catalog it came from, and resolving >> it via CatalogManager can initialize unrelated catalogs as a side effect. >> >> Tests cover the procedure path across the test catalogs and the action >> API with both SparkCatalog and SparkSessionCatalog, using a marker >> FileSystem registered only in the catalog's hadoop.* overrides. >> >> Questions >> >> 1. Is a catalogName(String) method on the action an acceptable public API >> addition, or would people prefer a different shape? >> 2. The PR was opened against Spark 4.1 before the 4.2 module landed. If >> the direction is agreed, I'll move the change to 4.2 and backport it to >> 4.1, 4.0 and 3.5. >> >> Thanks, >> Sebastian Daberdaku >> -- >> >> >> >> >> Sebastian Daberdaku >> >> Head of Platform & Data Engineering >> >> Via San Francesco d >> <https://www.google.com/maps/search/Via+San+Francesco+d+Assisi+15,+20122+Milan,+Italy?entry=gmail&source=g> >> 'Assisi 15, 20122 Milan, Italy >> <https://www.google.com/maps/search/Via+San+Francesco+d+Assisi+15,+20122+Milan,+Italy?entry=gmail&source=g> >> >> 40 New Bond St >> <https://www.google.com/maps/search/40+New+Bond+St?entry=gmail&source=g>., >> W1S 2DE, London, United Kingdom >> >> Vaso Pasha St., Rainbow Center, Tirana, Albania Musa Tolaj St., Lam A, 2/104, >> Prishtina, Kosova >> <https://www.google.com/maps/search/104,+Prishtina,+Kosova?entry=gmail&source=g> >> >> www.cardoai.com >> >> >> *This communication, including any attachments, is intended solely for >> the recipient(s) addressed. * >> *If you have received this email in error, please notify the sender >> immediately and delete it from your system. * >> *Any unauthorized use, disclosure, or distribution, of this communication >> is strictly prohibited.* >> >
