geosmart opened a new issue #5599:
URL: https://github.com/apache/dolphinscheduler/issues/5599
I need to change hdfs to minio , but the endpoint config `fs.s3a.endpoint`
not working
## test case
```java
public static void main(String[] args) throws Exception {
test_minio_s3a();
}
private static void test_minio_s3a() throws IOException {
Configuration configuration = new HdfsConfiguration();
configuration.set("fs.defaultFS", "s3a://bucketname");
configuration.set("fs.s3a.endpoint", "http://ip:32030";);
configuration.set("fs.s3a.access.key", "ak");
configuration.set("fs.s3a.secret.key", "sk");
configuration.set("fs.s3a.impl",
"org.apache.hadoop.fs.s3a.S3AFileSystem");
FileSystem fs = FileSystem.get(configuration);
System.out.println(fs.listFiles(new Path("/"), false));
}
```
## error log
```java
8:50:55.856 [main] DEBUG
org.apache.http.impl.conn.PoolingClientConnectionManager - Connection released:
[id: 0][route: {s}->https://bucketname.s3.amazonaws.com:443][total kept alive:
0; route allocated: 0 of 15; total allocated: 0 of 15]
18:50:55.861 [main] DEBUG com.amazonaws.request - Received error response:
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS
Service: null, AWS Request ID: X41SEJ67XC25KHPD, AWS Error Code: null, AWS
Error Message: Bad Request, S3 Extended Request ID:
KKy0QdemQ+TggcpuXXQKgg8R+Yac2BRgBO6Vs2Ed524epxg2RZnxy6ObR4Wpq0xMscXcJlb/Mec=
Exception in thread "main"
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS
Service: Amazon S3, AWS Request ID: X41SEJ67XC25KHPD, AWS Error Code: null, AWS
Error Message: Bad Request, S3 Extended Request ID:
KKy0QdemQ+TggcpuXXQKgg8R+Yac2BRgBO6Vs2Ed524epxg2RZnxy6ObR4Wpq0xMscXcJlb/Mec=
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:154)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at
org.apache.dolphinscheduler.common.dboard.HadoopUtilsTest.test_minio_s3a(HadoopUtilsTest.java:35)
at
org.apache.dolphinscheduler.common.dboard.HadoopUtilsTest.main(HadoopUtilsTest.java:24)
Disconnected from the target VM, address: '127.0.0.1:37581', transport:
'socket'
```
Any help would be appreciated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
