michaelkoepf commented on PR #1245: URL: https://github.com/apache/fluss/pull/1245#issuecomment-3588404528
In general, this PR is ready for review/testing. We need to figure out 2 things. 1. If AWS Credential Providers with Temporary Credentials (e.g., the IAM Credential Provider) finally work. @polyzos can provide AWS access. We need somebody who has experience in setting up different temporary authentication methods with S3 and EC2. 2. There is some anomaly that I currently cannot make sense of (see below). The idea of the PR is to deactivate token delegation to be able to use other credential providers. Without token delegation, users should be forced to set the credential provider to avoid misconfiguration or use unsafe credential providers that use long-term credentials. A more detailed description is in the [update docs of the PR](https://github.com/apache/fluss/blob/2e4c770240817ee51b5e34747a0330d58b16ffec/website/docs/maintenance/filesystems/s3.md). To force users to set the credential provider when token delegation is deactivated, I set the credential provider config options to blank, see [here](https://github.com/apache/fluss/blob/2e4c770240817ee51b5e34747a0330d58b16ffec/fluss-filesystems/fluss-fs-s3/src/main/java/org/apache/fluss/fs/s3/S3FileSystemPlugin.java#L148). If you check out the PR and follow the instructions under `Example usage with MinIO` (build Fluss from source, build Docker images with the specified names, etc.) you can see that `s3.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` is set in the Coordinator and Tablet Server, and everything works. Data is written to MinIO. Now the anomaly: Shut down the Docker Compose stack. Remove `s3.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` from the Coordinator and Tablet server configuration **but leave the credentials (access key, secret) there**. Spin up the Docker Compose stack again. Follow the instructions in the example. For some reason, data is still written to MinIO. This is a contradiction to me, because I set the credential providers by default to blank. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
