Austin - In my case the set up is such that services are deployed on Kubernetes with Docker, running on EKS. There is also an istio service mesh. So all the services communicate and access AWS resources like S3 using the service account. Service account is associated with IAM roles. I have verified that the service account has access to S3, by running a program that connects to S3 to read a file also aws client when packaged into the pod is able to access S3. So that means the roles and policies are good.
When I am running flink, I am following the same configuration for job manager and task manager as provided here: https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/standalone/kubernetes.html The exception we are getting is - org.apache.flink.fs.s3presto.shaded.com.amazonaws.SDKClientException: Unable to load credentials from service end point. This happens in the EC2CredentialFetcher class method fetchCredentials - line number 66, when it tries to read resource, effectively executing CURL 169.254.170.2/AWS_CONTAINER_CREDENTIALS_RELATIVE_URI I am not setting the variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI because its not the right way to do it for us, we are on EKS. Similarly any of the ~/.aws/credentials file approach will also not work for us. Atm, I haven't tried the kuberenetes service account property you mentioned above. I will try and let you know how it goes. Question - do i need to provide any parameters while building the docker image or any configuration in the flink config to tell flink that for all purposes it should be using the service account and not try to get into the EC2CredentialFetcher class. One more thing - we were trying this on the 1.6 version of Flink and not the 1.12 version. Regards, Swagat On Sun, Apr 4, 2021 at 8:56 AM Sameer Wadkar <sam...@axiomine.com> wrote: > Kube2Iam needs to modify IPtables to proxy calls to ec2 metadata to a > daemonset which runs privileged pods which maps a IP Address of the pods > and its associated service account to make STS calls and return temporary > AWS credentials. Your pod “thinks” the ec2 metadata url works locally like > in an ec2 instance. > > I have found that mutating webhooks are easier to deploy (when you have no > control over the Kubernetes environment - say you cannot change iptables or > run privileged pods). These can configure the ~/.aws/credentials file. The > webhook can make the STS call for the service account to role mapping. A > side car container to which the main container has no access can even renew > credentials becoz STS returns temp credentials. > > Sent from my iPhone > > On Apr 3, 2021, at 10:29 PM, Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > > > If you’re just looking to attach a service account to a pod using the > native AWS EKS IAM mapping[1], you should be able to attach the service > account to the pod via the `kubernetes.service-account` configuration > option[2]. > > Let me know if that works for you! > > Best, > Austin > > [1]: > https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html > [2]: > https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html#kubernetes-service-account > > On Sat, Apr 3, 2021 at 10:18 PM Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Can you describe your setup a little bit more? And perhaps how you use >> this setup to grant access to other non-Flink pods? >> >> On Sat, Apr 3, 2021 at 2:29 PM Swagat Mishra <swaga...@gmail.com> wrote: >> >>> Yes I looked at kube2iam, I haven't experimented with it. >>> >>> Given that the service account has access to S3, shouldn't we have a >>> simpler mechanism to connect to underlying resources based on the service >>> account authorization? >>> >>> On Sat, Apr 3, 2021, 10:10 PM Austin Cawley-Edwards < >>> austin.caw...@gmail.com> wrote: >>> >>>> Hi Swagat, >>>> >>>> I’ve used kube2iam[1] for granting AWS access to Flink pods in the past >>>> with good results. It’s all based on mapping pod annotations to AWS IAM >>>> roles. Is this something that might work for you? >>>> >>>> Best, >>>> Austin >>>> >>>> [1]: https://github.com/jtblin/kube2iam >>>> >>>> On Sat, Apr 3, 2021 at 10:40 AM Swagat Mishra <swaga...@gmail.com> >>>> wrote: >>>> >>>>> No we are running on aws. The mechanisms supported by flink to connect >>>>> to resources like S3, need us to make changes that will impact all >>>>> services, something that we don't want to do. So providing the aws secret >>>>> key ID and passcode upfront or iam rules where it connects by executing >>>>> curl/ http calls to connect to S3 , don't work for me. >>>>> >>>>> I want to be able to connect to S3, using aws Api's and if that >>>>> connection can be leveraged by the presto library, that is what I am >>>>> looking for. >>>>> >>>>> Regards, >>>>> Swagat >>>>> >>>>> >>>>> On Sat, Apr 3, 2021, 7:37 PM Israel Ekpo <israele...@gmail.com> wrote: >>>>> >>>>>> Are you running on Azure Kubernetes Service. >>>>>> >>>>>> You should be able to do it because the identity can be mapped to the >>>>>> labels of the pods not necessary Flink. >>>>>> >>>>>> On Sat, Apr 3, 2021 at 6:31 AM Swagat Mishra <swaga...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I think flink doesn't support pod identity, any plans tk achieve it >>>>>>> in any subsequent release. >>>>>>> >>>>>>> Regards, >>>>>>> Swagat >>>>>>> >>>>>>> >>>>>>>
