[
https://issues.apache.org/jira/browse/FLINK-37749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102991#comment-18102991
]
Dhruv Patel commented on FLINK-37749:
-------------------------------------
Hello [~FredTing] Can I work on this if this is still open
Thanks
> FileSystem Connector fails writing to S3 bucket
> -----------------------------------------------
>
> Key: FLINK-37749
> URL: https://issues.apache.org/jira/browse/FLINK-37749
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.20.0, 1.19.2, 1.20.1
> Environment: Docker running:
> - Flink 1.20.1
> - minio
> Reporter: Fred Teunissen
> Priority: Blocker
> Attachments: filesystem-connector-s3-issue.tgz
>
>
> Writing to an output file on S3 using the filesystem connector results in a
> crash.
> The output file is defined as follows.
> {code:java}
> CREATE TABLE <tableName>> (
> bool BOOLEAN,
> `someInt` INT
> )
> PARTITIONED BY (someInt)
> WITH (
> 'connector' = 'filesystem',
> 'path' = 's3a://tdata',
> 'format' = 'csv',
> -- 'format' = 'parquet',
> 'partition.default-name' = 'null'
> );{code}
> Note that the 'path' refers to an existing bucket that is hosted using Minio.
> The job starts OK but as soon as the first checkpoint is being triggered the
> job crashes with the following exception:
> {code:java}
> "Caused by: org.apache.flink.util.SerializedThrowable: java.io.IOException:
> Stream closed." (full details in the attached zip file).{code}
> The issue can be reproduced by unzipping the attached zip file that contains
> a docker-compose file to spin up the Flink cluster and Minio container
> and using the instructions in the README.MD to generate test data resulting
> in the job crash.
> Cause is unknown but I suspect more (or other) .jars need to be added to the
> plugins/s3 directory as documented here:
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
