Hi Thomas, did you try to login to EKS node and run some aws command like : aws s3 ls <backend bucket>? It sounds like EKS issue, but not 100% sure. Best
On Sat, 25 Sept 2021 at 22:12, Ingo Bürk <i...@ververica.com> wrote: > Hi Thomas, > > I think you might be looking for this: > https://github.com/apache/flink/pull/16717 > > > Best > Ingo > > On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote: > >> Hi, >> >> I'm using the official docker image: >> apache/flink:1.12.1-scala_2.11-java11 >> >> I'm trying to run a Flink job on an EKS cluster. The job is running under >> a k8s service account that is tied to an IAM role. If I'm not using s3 as >> RocksDB checkpoint backend, everything works just fine. However, when I >> enabled s3 as RocksDB checkpoint backend, I got permission denied. >> >> The IAM role tied to the service account has the appropriate permissions >> to s3. However the underlying role tied to the EKS node doesn't. After >> debugging with AWS support, it looks like the request to s3 was made under >> the EKS node role, not the role tied to the service account. Thus the >> permission denial. >> >> With the same Flink application, I'm also making requests to AWS Secrets >> Manager to get some sensitive information and those requests were made >> explicitly with AWS Java SDK 2.x bundled in the same application Jar file. >> Those requests were made correctly with the IAM role tied to the service >> account. >> >> Based on the info above, I suspect Flink may be using an older version of >> the AWS SDK that doesn't support assuming an IAM role via an IODC web >> identity token file. Please see AWS doc here: >> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html >> >> Could someone help me confirm this bug and maybe have it fixed some time? >> Thanks. >> >> Thomas >> > -- Xiangyu Su Java Developer xian...@smaato.com Smaato Inc. San Francisco - New York - Hamburg - Singapore www.smaato.com Germany: Barcastraße 5 22087 Hamburg Germany M 0049(176)43330282 The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
