想请问下,写filesystem的时候依赖checkpoint进行commit,那么做完一次checkpoint的时候可提交的文件数是由并行度parallelism数决定的吗?我发现我的文件提交数都是3个3个的当每次chk结束后。
发件人: amen...@163.com 发送时间: 2020-12-24 18:47 收件人: user-zh 主题: Re: Re: Flink-1.11.1流写filesystem分区提交问题 一语点醒梦中人,谢谢回复@冯嘉伟 因为我是先在sql-client中进行的提交测试,因此忽略了这个问题,谢谢 best, amenhub 发件人: 冯嘉伟 发送时间: 2020-12-24 18:39 收件人: user-zh 主题: Re: Flink-1.11.1流写filesystem分区提交问题 有开启checkpoint吗? Part files can be in one of three states: In-progress : The part file that is currently being written to is in-progress Pending : Closed (due to the specified rolling policy) in-progress files that are waiting to be committed Finished : On successful checkpoints (STREAMING) or at the end of input (BATCH) pending files transition to “Finished” https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/file_sink.html <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/file_sink.html> -- Sent from: http://apache-flink.147419.n8.nabble.com/