Sbaia opened a new issue, #3916:
URL: https://github.com/apache/amoro/issues/3916
### What happened?
Hi,
I'm trying to configure a Paimon catalog in Amoro with S3 storage, using
authentication via IAM Role ARN and Kubernetes Service Account (WebIdentity).
My setup works with explicit AK/SK credentials, but DOES NOT work when
switching to role-based authentication.
Here is the configuration I've used:
```{
"name": "Paimon",
"type": "hadoop",
"authConfig": {
"auth.type": "custom"
},
"storageConfig": {
"storage.type": "S3",
"fs.s3a.region": "eu-west-1",
"fs.s3a.aws.credentials.provider":
"com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
},
"properties": {
"warehouse": "s3a://my-bucket-in-aws/"
},
"tableProperties": {},
"tableFormatList": ["PAIMON"]
}```
- With AK/SK credentials, it works fine.
- Using ARN/ServiceAccount, setting fs.s3a.aws.credentials.provider to
com.amazonaws.auth.WebIdentityTokenCredentialsProvider, I always get access
error or "Anonymous" authentication.
- The S3A scheme is required for Hadoop integration, otherwise I get a
missing filesystem implementation error. (Not sure if that's related to the ARN
issue.)
- The IAM role is attached and correctly configured; Flink can write to S3
with the same role and setup.
### Affects Versions
master/0.8.1
### What table formats are you seeing the problem on?
Paimon
### What engines are you seeing the problem on?
AMS
### How to reproduce
try to configure a paimon catalog using AWS Role ARN
### Relevant log output
```shell
2025-11-07 16:06:23,832 ERROR [JettyServerThreadPool-61]
[org.apache.amoro.server.dashboard.DashboardServer] [] - An error occurred
while processing the url:http://amoro.tools.com/api/ams/v1/catalogs
│
│ java.io.UncheckedIOException: java.nio.file.AccessDeniedException:
s3a://my-bucket-in-aws/user.sys: getFileStatus on
s3a://my-bucket-in-aws/user.sys:
software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3 │
│ , Status Code: 403, Request ID: JVF0V003PHTCA7R6, Extended Request ID:
gjOri4E7uwBNMnoamvlGasM5rzYFBzGlBC7kQhRHVZaoXtGPbG3Yybtq1oy9teKAcOEE2783VpRUDYDzNIwnR/iei6w2wzRn):null
│
│ at
org.apache.paimon.privilege.FileBasedPrivilegeManager.getTable(FileBasedPrivilegeManager.java:371)
~[paimon-bundle-1.2.0.jar:1.2.0]
│
│ at
org.apache.paimon.privilege.FileBasedPrivilegeManager.getUserTable(FileBasedPrivilegeManager.java:352)
~[paimon-bundle-1.2.0.jar:1.2.0]
│
│ at
org.apache.paimon.privilege.FileBasedPrivilegeManager.privilegeEnabled(FileBasedPrivilegeManager.java:118)
~[paimon-bundle-1.2.0.jar:1.2.0]
│
│ at
org.apache.paimon.privilege.PrivilegedCatalog.tryToCreate(PrivilegedCatalog.java:72)
~[paimon-bundle-1.2.0.jar:1.2.0]
│
│ at
org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:74)
~[paimon-bundle-1.2.0.jar:1.2.0]
│
│ at
org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:67)
~[paimon-bundle-1.2.0.jar:1.2.0]
│
│ at
org.apache.amoro.formats.paimon.PaimonCatalogFactory.paimonCatalog(PaimonCatalogFactory.java:87)
~[amoro-format-paimon-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]
│
│ at
org.apache.amoro.formats.paimon.PaimonCatalogFactory.create(PaimonCatalogFactory.java:79)
~[amoro-format-paimon-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]
│
│ at
org.apache.amoro.formats.paimon.PaimonCatalogFactory.create(PaimonCatalogFactory.java:42)
~[amoro-format-paimon-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]
│
│ at
org.apache.amoro.CommonUnifiedCatalog.initializeFormatCatalogs(CommonUnifiedCatalog.java:212)
~[amoro-common-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]
│
│ at
org.apache.amoro.CommonUnifiedCatalog.<init>(CommonUnifiedCatalog.java:60)
~[amoro-common-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]
│
│ at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52)
~[javalin-4.6.8.jar:4.6.8]
│
│ at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at org.eclipse.jetty.server.Server.handle(Server.java:516)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
[jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
[jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
[jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
│
│ at java.lang.Thread.run(Thread.java:829) [?:?]
│
│ Caused by: java.nio.file.AccessDeniedException:
s3a://my-bucket-in-aws/user.sys: getFileStatus on
s3a://my-bucket-in-aws/user.sys:
software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status
Code: 403, │
│ Request ID: JVF0V003PHTCA7R6, Extended Request ID:
gjOri4E7uwBNMnoamvlGasM5rzYFBzGlBC7kQhRHVZaoXtGPbG3Yybtq1oy9teKAcOEE2783VpRUDYDzNIwnR/iei6w2wzRn):null
│
│ at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:266)
~[hadoop-aws-3.4.0.jar:?]
│
│ at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:155)
~[hadoop-aws-3.4.0.jar:?]
```
### Anything else
Always
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
