sambhav gupta created HUDI-4154:
-----------------------------------
Summary: Unable to write HUDI Tables to S3 via Flink SQL
Key: HUDI-4154
URL: https://issues.apache.org/jira/browse/HUDI-4154
Project: Apache Hudi
Issue Type: Bug
Components: connectors
Reporter: sambhav gupta
When trying to write Hudi Tables into MinIO(S3) via Flink SQL we are facing
issues .
The configuration is as follows:
1) MinIO S3 working on localhost:9000 - Latest docker image
2) Flink 1.13.6
3) Hudi - hudi-flink-bundle_2.11-0.10.1.jar
4) etc/core/site.xml set with S3 properties for access key, secret key and
endpoint already
When we create a MOR Hudi table as follows and try to insert records in it we
face an issue.
> create table t1s3hudi(id int PRIMARY KEY, name varchar(50)) with ('connector'
> = 'hudi', 'path' = 's3a://test123/t1s3hudi', 'table.type' = 'MERGE_ON_READ',
> 'hoodie.aws.access.key' = 'minioadmin', 'hoodie.aws.secret.key' =
> 'minioadmin');
> insert into t1s3hudi values(1,'one number s3');
The exception that we get in error logs is:
*Caused by: org.apache.hudi.exception.HoodieException: Error while checking
whether table exists under path:s3a://test123/t1s3hudi*
*at org.apache.hudi.util.StreamerUtil.tableExists(StreamerUtil.java:292)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:258)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:164)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:194)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*... 18 more*
*Caused by: java.nio.file.AccessDeniedException:
s3a://test123/t1s3hudi/.hoodie: getFileStatus on
s3a://test123/t1s3hudi/.hoodie:
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon
S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: XAJMZTMQDGHRWZS8;
S3 Extended Request ID:
qaTd5xTZCvnRwThI9fTSeuWVuzXpuw9H6w7roFGBnBVNQmHe1O7mgHbzEZmEIKNp/bx3Iyb9/Kc=;
Proxy: null), S3 Extended Request ID:
qaTd5xTZCvnRwThI9fTSeuWVuzXpuw9H6w7roFGBnBVNQmHe1O7mgHbzEZmEIKNp/bx3Iyb9/Kc=:403
Forbidden*
*at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:218)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:145)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2184)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2970)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hudi.util.StreamerUtil.tableExists(StreamerUtil.java:290)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:258)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:164)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:194)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*... 18 more*
*Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden
(Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID:
XAJMZTMQDGHRWZS8; S3 Extended Request ID:
qaTd5xTZCvnRwThI9fTSeuWVuzXpuw9H6w7roFGBnBVNQmHe1O7mgHbzEZmEIKNp/bx3Iyb9/Kc=;
Proxy: null)*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1338)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1235)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:280)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1232)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2169)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2970)
~[flink-s3-fs-hadoop-1.13.6.jar:1.13.6]*
*at org.apache.hudi.util.StreamerUtil.tableExists(StreamerUtil.java:290)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.hudi.util.StreamerUtil.initTableIfNotExists(StreamerUtil.java:258)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.hudi.sink.StreamWriteOperatorCoordinator.start(StreamWriteOperatorCoordinator.java:164)
~[hudi-flink-bundle_2.11-0.10.1.jar:0.10.1]*
*at
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.start(OperatorCoordinatorHolder.java:194)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.scheduler.DefaultOperatorCoordinatorHandler.startAllOperatorCoordinators(DefaultOperatorCoordinatorHandler.java:85)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.scheduler.SchedulerBase.startScheduling(SchedulerBase.java:592)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.startScheduling(JobMaster.java:955)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.startJobExecution(JobMaster.java:873)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.jobmaster.JobMaster.onStart(JobMaster.java:383)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStart(RpcEndpoint.java:181)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
*at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StoppedState.start(AkkaRpcActor.java:605)
~[flink-dist_2.11-1.13.6.jar:1.13.6]*
We have tried creating a 'csv' table using the same s3 path and 'filesystem'
connector and we were able to write to MinIO(s3) bucket with the same
credentials provided to the core-site.xml.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
