[
https://issues.apache.org/jira/browse/FLINK-19345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245183#comment-17245183
]
zhuxiaoshang commented on FLINK-19345:
--------------------------------------
HI [~lzljs3620320],when the `auto-compaction` is open,the bucket should not be
committed before the compaction is done.Otherwise,may cause compacting
failure.Correct me if I'm wrong.
When i dig into the code,i found that the bucket is committed in
`AbstractStreamingWriter#notifyCheckpointComplete` no matter whether the
`auto-compaction` is open.
> In Table File Sink, introduce streaming sink compaction
> -------------------------------------------------------
>
> Key: FLINK-19345
> URL: https://issues.apache.org/jira/browse/FLINK-19345
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
> Reporter: Jingsong Lee
> Assignee: Jingsong Lee
> Priority: Major
> Fix For: 1.12.0
>
>
> Users often complain that many small files are written out. Small files will
> affect the performance of file reading and the DFS system, and even the
> stability of the DFS system.
> Target:
> * Compact all files generated by this job in a single checkpoint.
> * With compaction, Users can have smaller checkpoint interval, even to
> seconds.
> Document:
> https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
