sdaberdaku opened a new issue, #17860:
URL: https://github.com/apache/iceberg/issues/17860
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
We run a single Spark Connect cluster (Spark 4.1.2) serving several tenants.
Each tenant has its own Glue catalog in its own AWS account, reached with
`AssumeRoleAwsClientFactory`:
```
spark.sql.catalog.tenant
org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.tenant.type glue
spark.sql.catalog.tenant.io-impl
org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.tenant.client.factory
org.apache.iceberg.aws.AssumeRoleAwsClientFactory
spark.sql.catalog.tenant.client.assume-role.arn
arn:aws:iam::111122223333:role/lakehouse-access
spark.sql.catalog.tenant.client.assume-role.region us-east-2
spark.sql.catalog.tenant.glue.id 111122223333
```
The cluster itself runs on EKS as an IRSA role in a different account, which
deliberately has no access to the tenant buckets:
```
spark.hadoop.fs.s3.impl
org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.aws.credentials.provider
com.amazonaws.auth.WebIdentityTokenCredentialsProvider
```
Nightly maintenance calls `expire_snapshots` and then `remove_orphan_files`
on each table. The first one works, the second one gets a 403, seconds later,
on the same table:
```sql
-- ok
CALL tenant.system.expire_snapshots(table => 'db.my_table', older_than =>
TIMESTAMP '2026-08-21 00:00:00');
-- 403
CALL tenant.system.remove_orphan_files(table => 'db.my_table', older_than =>
TIMESTAMP '2026-08-25 00:00:00');
```
```
java.io.UncheckedIOException: java.nio.file.AccessDeniedException:
s3://my-lakehouse-bucket/iceberg_tables/my_table: listObjects() on
s3://my-lakehouse-bucket/iceberg_tables/my_table:
software.amazon.awssdk.services.s3.model.AccessDeniedException:
User: arn:aws:sts::444455556666:assumed-role/spark-cluster-irsa/... is not
authorized to perform:
s3:ListBucket on resource: "arn:aws:s3:::my-lakehouse-bucket" (Service:
S3, Status Code: 403)
at
org.apache.iceberg.util.FileSystemWalker.listDirRecursivelyWithHadoop(FileSystemWalker.java:150)
at
org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.listedFileDS(DeleteOrphanFilesSparkAction.java:440)
at
org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.doExecute(DeleteOrphanFilesSparkAction.java:257)
at
org.apache.iceberg.spark.procedures.RemoveOrphanFilesProcedure.call(RemoveOrphanFilesProcedure.java:153)
```
The principal in that error is the cluster's own IRSA role, which appears
nowhere in the catalog config. So the listing is not using the catalog's
credentials, while the rest of the procedure is. The cause is that the action
takes its Hadoop config from the session and drops the catalog it came from:
```java
// DeleteOrphanFilesSparkAction.java:139
this.hadoopConf = new
SerializableConfiguration(spark.sessionState().newHadoopConf());
```
`newHadoopConf()` only sees `spark.hadoop.*`. Everything else in the
procedure goes through `table.io()`, which the catalog built with the
assume-role factory, which is why `expire_snapshots` is fine.
I found #11541, which describes this exact coupling, and PR #12254 which
closed it by adding the `FileIO` listing path. But that path is opt-in
(`prefix_listing`, `usePrefixListing` defaults to `false` at
`DeleteOrphanFilesSparkAction.java:133`), so the Hadoop path still behaves this
way and it is what you get unless you know to ask for the other one.
`prefix_listing => true` does fix the credentials for us, but we can't use
it: it lists serially on the driver and then does `parallelize(matchingFiles,
1)` (#16932, #17387). On Spark Connect there's an extra ceiling, since that
single partition puts the whole listing under `spark.rpc.message.maxSize`,
which is a server startup setting we can't raise from a client session. So
we're stuck with the Hadoop path, and the Hadoop path has the wrong credentials.
What took me longest to work out is that there is no per-catalog knob for
this at all. `SparkUtil.hadoopConfCatalogOverrides(SparkSession, catalogName)`
already exists and applies `spark.sql.catalog.<name>.hadoop.*` on top of the
session config, which is exactly what's needed here, but it only has two
callers, both in `SparkCatalog` (`SparkCatalog.java:142` and `:717`). No action
or procedure uses it. So setting `spark.sql.catalog.tenant.hadoop.fs.s3a.*` is
accepted, looks right, and is then ignored by the walk. I assumed I had a typo
for a while.
Proposal: build the action's Hadoop config with
`SparkUtil.hadoopConfCatalogOverrides(spark, catalogName)` instead of
`spark.sessionState().newHadoopConf()`, with a `catalogName()` setter on the
action alongside the existing ones and `RemoveOrphanFilesProcedure` passing
`tableCatalog().name()` (`BaseProcedure` already holds it). Nothing changes for
anyone who doesn't set `spark.sql.catalog.<name>.hadoop.*`.
I'm happy to open a PR for this, plus the v4.0 and v3.5 backports, if that
sounds like the right direction. If the preference is to leave the Hadoop path
alone because the FileIO one is where things are heading, then at minimum the
docs should say that the listing uses the session's Hadoop config and not the
catalog's, because nothing in the config surface hints at it.
For anyone else hitting the same 403, the workaround is to give S3A the same
role, scoped to the buckets:
```
spark.hadoop.fs.s3a.bucket.my-lakehouse-bucket.aws.credentials.provider
org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
spark.hadoop.fs.s3a.bucket.my-lakehouse-bucket.assumed.role.arn
arn:aws:iam::111122223333:role/lakehouse-access
spark.hadoop.fs.s3a.assumed.role.credentials.provider
com.amazonaws.auth.WebIdentityTokenCredentialsProvider
```
Per-bucket options key on the bucket name from the URI
(`fs.s3a.bucket.<bucket>.`), independent of the scheme, so this covers `s3://`
locations too. The role's trust policy has to allow the cluster principal to
assume it, which it already does if the catalog is working.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
