Sorry "bit disconnected" On Thu, Oct 29, 2026 at 6:10 PM Russell Spitzer <[email protected]> wrote:
> I would check out Dan's suggestion. We should try any solutions we can > that avoid expanding the API, especially when the connection here between > the behavior and the API is a bad disconnected. > > On Thu, Oct 29, 2026 at 4:53 PM Daniel Weeks <[email protected]> wrote: > >> Sebastian, >> >> It's been a little while since I looked at this, but wouldn't it make >> more sense to interrogate the FileIO to see if it's HadoopConfigurable and >> grab the exact configuration used by the IO rather than go through this >> catalog path? >> >> At that point, you'd have the exact configuration and overrides for that >> specific table. >> >> -Dan >> >> On Wed, Oct 28, 2026 at 1:15 AM Sebastian Daberdaku via dev < >> [email protected]> wrote: >> >>> Hi all, >>> >>> I'd like to get some eyes on a PR that fixes a credentials mismatch in >>> remove_orphan_files, and agree on the approach before I port it across >>> Spark versions: >>> >>> https://github.com/apache/iceberg/pull/17862 (closes #17860) >>> >>> Problem >>> >>> Unless prefix_listing is enabled, DeleteOrphanFilesSparkAction lists the >>> table location with a Hadoop FileSystem built from the Spark session >>> configuration alone, while every other file operation in the action goes >>> through table.io(). When a catalog reaches its storage with credentials >>> other than the cluster's (e.g. a catalog that assumes a role in another AWS >>> account), reads and deletes succeed but the listing fails with AccessDenied >>> or silently sees no files. >>> >>> Change >>> >>> - DeleteOrphanFilesSparkAction gets a new catalogName(String) method. >>> When set, the listing uses the session Hadoop configuration plus that >>> catalog's spark.sql.catalog.<name>.hadoop.* overrides >>> (SparkUtil.hadoopConfCatalogOverrides). When not set, behavior is unchanged. >>> - RemoveOrphanFilesProcedure passes the catalog it was invoked on, so >>> CALL <catalog>.system.remove_orphan_files(...) picks up that catalog's >>> overrides with no new procedure argument. >>> - spark-procedures.md documents which configuration the listing uses. >>> >>> The catalog is passed explicitly rather than inferred from Table.name(): >>> as @yangshangqing95 pointed out in review, a table name's first segment >>> isn't guaranteed to identify the Spark catalog it came from, and resolving >>> it via CatalogManager can initialize unrelated catalogs as a side effect. >>> >>> Tests cover the procedure path across the test catalogs and the action >>> API with both SparkCatalog and SparkSessionCatalog, using a marker >>> FileSystem registered only in the catalog's hadoop.* overrides. >>> >>> Questions >>> >>> 1. Is a catalogName(String) method on the action an acceptable public >>> API addition, or would people prefer a different shape? >>> 2. The PR was opened against Spark 4.1 before the 4.2 module landed. If >>> the direction is agreed, I'll move the change to 4.2 and backport it to >>> 4.1, 4.0 and 3.5. >>> >>> Thanks, >>> Sebastian Daberdaku >>> -- >>> >>> >>> >>> >>> Sebastian Daberdaku >>> >>> Head of Platform & Data Engineering >>> >>> Via San Francesco d >>> <https://www.google.com/maps/search/Via+San+Francesco+d+Assisi+15,+20122+Milan,+Italy?entry=gmail&source=g> >>> 'Assisi 15, 20122 Milan, Italy >>> <https://www.google.com/maps/search/Via+San+Francesco+d+Assisi+15,+20122+Milan,+Italy?entry=gmail&source=g> >>> >>> 40 New Bond St >>> <https://www.google.com/maps/search/40+New+Bond+St?entry=gmail&source=g>., >>> W1S 2DE, London, United Kingdom >>> >>> Vaso Pasha St., Rainbow Center, Tirana, Albania Musa Tolaj St., Lam A, >>> 2/104, >>> Prishtina, Kosova >>> <https://www.google.com/maps/search/104,+Prishtina,+Kosova?entry=gmail&source=g> >>> >>> www.cardoai.com >>> >>> >>> *This communication, including any attachments, is intended solely for >>> the recipient(s) addressed. * >>> *If you have received this email in error, please notify the sender >>> immediately and delete it from your system. * >>> *Any unauthorized use, disclosure, or distribution, of this >>> communication is strictly prohibited.* >>> >>
