YanshuoH opened a new issue #9424: URL: https://github.com/apache/pulsar/issues/9424
Hi, **Describe the bug** As deploying Pulsar inside AWS EKS (Kubernetes hosted by AWS), while trying to use the WebIdentityTokenFile to assume the AWS role in order to offload ledgers into S3 Bucket, I've encountered exceptions like: ``` java.util.concurrent.CompletionException: org.jclouds.aws.AWSResponseException: request POST https://pulsar-cluster.s3.amazonaws.com/f3889033-79f9-4273-9f50-82fa0f8992e4-ledger-6908?uploads HTTP/1.1 failed with code 400 ...logs omitted for clearity and safety (see screenshot below)... code='InvalidToken', message='The provided token is malformed or otherwise invalid.' ``` To assume a role inside a pod inside EKS, we rely on a environement variable called `AWS_WEB_IDENTITY_TOKEN_FILE` which is pretty much a standard IAM authorization process to communicate with AWS resources. After looking in to the code, I've realized that if not defining the `s3ManagedLedgerOffloadRole` (or left empty), the `org.apache.bookkeeper.mledger.offload.jcloud.provider.JCloudBlobStoreProvider#AWS_CREDENTIAL_BUILDER` will use `DefaultAWSCredentialsProviderChain` which contains already the `WebIdentityTokenCredentialsProvider`. Strange things is that with the following command inside the container, we've succeeded to get the role identity: ``` root@pulsar-broker-0:/pulsar# aws sts get-caller-identity { "UserId": "AROAZPRHQZGMZXSZDNXWZ:botocore-session-1612264367", "Account": "xxx", "Arn": "xxx" } ``` Yet the offloader still has trouble to upload to S3. **To Reproduce** Steps to reproduce the behavior: 1. Deploy Pulsar Broker within a EKS pod, if that is too costy, you can inject environement variables like this: ``` env: - name: AWS_DEFAULT_REGION value: cn-northwest-1 - name: AWS_REGION value: cn-northwest-1 - name: AWS_ROLE_ARN value: arn:aws-cn:iam::xxx:role/your-iam-role-with-s3-privileges - name: AWS_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token ``` Here I assume you have the knowledge of using web identity token, but if that's not the case, the file `AWS_WEB_IDENTITY_TOKEN_FILE` contains a JWT token as such: ``` eyJhbGciOiJSUzI1NiIsImtpZCI6IjU2MzgwMzI2NDQ1YTcxNjAwOGU0YWY0ZTFmYzc4ODIwN2YzZTE0ZTQifQ.eyJhdWQiOlsic3RzLmFtYXpvbmF3cy5jb20iXSwiZXhwIjoxNjEyMzQ3MTk0LCJpYXQiOjE2MTIyNjA3OTQsImlzcyI6Imh0dHBzOi8vb2lkYy5la3MuY24tbm9ydGh3ZXN0LTEuYW1hem9uYXdzLmNvbS5jbi9pZC9EOEYzMDMyOUIxQjU5ODBEREJCNUFDMTc2MDI4MUVDOCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoicHVsc2FyIiwicG9kIjp7Im5hbWUiOiJwdWxzYXItYnJva2VyLTAiLCJ1aWQiOiIwZWIxMGZhOC00NjUzLTQ0NmYtOWY1OS0wMDllY2E2MzZmNDMifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6InB1bHNhci1icm9rZXIiLCJ1aWQiOiIzNDRmZGYwZi0wZjZkLTRlODgtODZiYi04YTc0ZjNlMzkxOWEifX0sIm5iZiI6MTYxMjI2MDc5NCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnB1bHNhcjpwdWxzYXItYnJva2VyIn0.mJfqLrCNvQwS8Yg8pFlo9TA8pbp4OFodBNiQScB0NcXMDz0wWSr2SkzsSqjnkzRk91QukkT8eUuiEvt3KEM4HmLfjjnKU-qIrAP4HcwOD9LaSWxURzDg5rramsvpgZu1nOB-igTBH3dTGpjEDu3iHWtCKHU9G-OOdX5w-eEcU7S-TpwdEEgrnYejCXnr-M7sJF3zQfT3rbZFhb6bRWt7_67pjpritfACRh5xw9eabXcaDL_Xi9Xla21bSlAIiPVJO-nMGvNLLoXOzUEEB8gfHp-ZJh598kaCNvp72sdQa9UJWNUdN3aGoXPNpsaIwXTU0XXh63uzNUAvq3W0 zqFyuQroot ``` 2. Start Pulsar Broker with offload config as such: ``` managedLedgerOffloadDriver: "aws-s3" s3ManagedLedgerOffloadBucket: "pulsar-cluster" s3ManagedLedgerOffloadRegion: "cn-northwest-1" ``` 3. Run some test messages under a topic while setting retention policies to `-1` (avoid deleting the messages too early) 4. With `pulsar-admin`, do offload on that topic 5. Check the logs and there will be exceptions quoted above. **Expected behavior** With WebIdentityTokenFile, a successful upload to S3 bucket **Screenshots** <img width="1600" alt="Screen Shot 2021-02-02 at 7 21 37 PM" src="https://user-images.githubusercontent.com/6973092/106593522-e0b8d080-658b-11eb-9842-8b67514d4dee.png";> **Desktop (please complete the following information):** - OS: Docker Image of `apachepulsar/pulsar-all:2.6.3` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/6973092/106593522-e0b8d080-658b-11eb-9842-8b67514d4dee.png"](https://user-images.githubusercontent.com/6973092/106593522-e0b8d080-658b-11eb-9842-8b67514d4dee.png"){kind=link}