The BucketingSink doesn't have well defined semantics when used with S3.
loss is possible but I am not sure whether it is the only problem. There are
plans to rewrite the BucketingSink in Flink 1.6 to enable eventually
file systems .
On Thu, May 17, 2018 at 11:57 AM, Amit Jain <aj201...@gmail.com> wrote:
> We are using Flink to process click stream data from Kafka and pushing
> the same in 128MB file in S3.
> What is the message processing guarantees with S3 sink? In my
> understanding, S3A client buffers the data on memory/disk. In failure
> scenario on particular node, TM would not trigger Writer#close hence
> buffered data can lose entirely assuming this buffer contains data of
> last successful checkpointing.