[
https://issues.apache.org/jira/browse/FLINK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413319#comment-16413319
]
yanxiaobin commented on FLINK-8939:
-----------------------------------
thanks! Because S3 follows the *eventually consistent* principle. When using
S3（not enable consistent view）, sometimes the file is not renamed from
the.inprogress suffix to the.pending suffix.
> Provide better support for saving streaming data to s3
> ------------------------------------------------------
>
> Key: FLINK-8939
> URL: https://issues.apache.org/jira/browse/FLINK-8939
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API, Streaming Connectors
> Reporter: chris snow
> Priority: Major
> Attachments: 18652BC0-DD67-42C3-9A33-12F7BC10F9F3.png
>
>
> Flink seems to struggle saving data to s3 due to the lack of a truncate
> method, and in my test this resulted in lots of files with a .valid-length
> suffix
> I’m using a bucketing sink:
> {code:java}
> return new BucketingSink<Tuple2<String, Object>>(path)
> .setWriter(writer)
> .setBucketer(new DateTimeBucketer<Tuple2<String,
> Object>>(formatString));{code}
>
> !18652BC0-DD67-42C3-9A33-12F7BC10F9F3.png!
> See also, the discussion in the comments here:
> https://issues.apache.org/jira/browse/FLINK-8543?filter=-2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
