Hi,
文件数是和并发有关的,一个并发一次至少写一个文件,还和文件滚动大小有关。


> 2020年12月25日 下午2:10,[email protected] 写道:
> 
> 想请问下,写filesystem的时候依赖checkpoint进行commit,那么做完一次checkpoint的时候可提交的文件数是由并行度parallelism数决定的吗?我发现我的文件提交数都是3个3个的当每次chk结束后。
> 
> 
> 
> 
> 发件人: [email protected]
> 发送时间: 2020-12-24 18:47
> 收件人: user-zh
> 主题: Re: Re: Flink-1.11.1流写filesystem分区提交问题
> 一语点醒梦中人,谢谢回复@冯嘉伟
> 
> 因为我是先在sql-client中进行的提交测试,因此忽略了这个问题,谢谢
> 
> best,
> amenhub
> 
> 
> 
> 发件人: 冯嘉伟
> 发送时间: 2020-12-24 18:39
> 收件人: user-zh
> 主题: Re: Flink-1.11.1流写filesystem分区提交问题
> 有开启checkpoint吗?
> Part files can be in one of three states:
> In-progress : The part file that is currently being written to is
> in-progress
> Pending : Closed (due to the specified rolling policy) in-progress files
> that are waiting to be committed
> Finished : On successful checkpoints (STREAMING) or at the end of input
> (BATCH) pending files transition to “Finished”
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/file_sink.html
> <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/file_sink.html>
>   
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/

回复