wcmolin opened a new issue, #18267:
URL: https://github.com/apache/dolphinscheduler/issues/18267
### Search before asking
- [x] I had searched in the issues and found no similar issues.
### What happened
We use a self-hosted MinIO service in our internal network as S3-compatible
storage for remote logging.
Example configuration:
remote.logging:
target: S3
base.dir: task-logs
thread.pool.size: 10
aws:
s3:
credentials.provider.type: AWSStaticCredentialsProvider
access.key.id: xxx
access.key.secret: xxx
region: internal-region
bucket.name: dolphinscheduler
endpoint: http://10.10.20.30:9000
After a task finishes, DolphinScheduler tries to upload the task log to
MinIO, but creating the S3 client fails:
java.lang.IllegalArgumentException: Cannot create enum from internal-region
value!
at com.amazonaws.regions.Regions.fromName(Regions.java:101)
at
org.apache.dolphinscheduler.authentication.aws.AmazonS3ClientFactory.createAmazonS3Client(AmazonS3ClientFactory.java:35)
The problem is that AmazonS3ClientFactory still calls
Regions.fromName(region) even when a custom S3 endpoint is configured.
For self-hosted MinIO or other S3-compatible storage, the region can be a
custom value and may not be an official AWS region.
### What you expected to happen
When aws.s3.endpoint is configured, DolphinScheduler should allow custom
region values for S3-compatible storage.
It should only require official AWS regions when no custom endpoint is
configured and the client is connecting to AWS S3 directly.
### How to reproduce
1. Deploy a self-hosted MinIO service in an internal network, for example
http://10.10.20.30:9000.
2. Configure MinIO with a custom region, for example internal-region.
3. Enable DolphinScheduler remote logging.
4. Configure remote.logging.target as S3.
5. Configure aws.yaml with the MinIO endpoint and custom region.
6. Run a task and wait for it to finish.
7. The remote log upload fails when creating the S3 client.
### Anything else
A possible fix is to use the raw configured region string when a custom
endpoint is present, and only call Regions.fromName(region) when no endpoint is
configured.
### Version
dev
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
