Hi, 文件数是和并发有关的,一个并发一次至少写一个文件,还和文件滚动大小有关。
> 2020年12月25日 下午2:10,[email protected] 写道: > > 想请问下,写filesystem的时候依赖checkpoint进行commit,那么做完一次checkpoint的时候可提交的文件数是由并行度parallelism数决定的吗?我发现我的文件提交数都是3个3个的当每次chk结束后。 > > > > > 发件人: [email protected] > 发送时间: 2020-12-24 18:47 > 收件人: user-zh > 主题: Re: Re: Flink-1.11.1流写filesystem分区提交问题 > 一语点醒梦中人,谢谢回复@冯嘉伟 > > 因为我是先在sql-client中进行的提交测试,因此忽略了这个问题,谢谢 > > best, > amenhub > > > > 发件人: 冯嘉伟 > 发送时间: 2020-12-24 18:39 > 收件人: user-zh > 主题: Re: Flink-1.11.1流写filesystem分区提交问题 > 有开启checkpoint吗? > Part files can be in one of three states: > In-progress : The part file that is currently being written to is > in-progress > Pending : Closed (due to the specified rolling policy) in-progress files > that are waiting to be committed > Finished : On successful checkpoints (STREAMING) or at the end of input > (BATCH) pending files transition to “Finished” > https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/file_sink.html > <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/file_sink.html> > > -- > Sent from: http://apache-flink.147419.n8.nabble.com/
