想请问下,写filesystem的时候依赖checkpoint进行commit,那么做完一次checkpoint的时候可提交的文件数是由并行度parallelism数决定的吗?我发现我的文件提交数都是3个3个的当每次chk结束后。



 
发件人: amen...@163.com
发送时间: 2020-12-24 18:47
收件人: user-zh
主题: Re: Re: Flink-1.11.1流写filesystem分区提交问题
一语点醒梦中人,谢谢回复@冯嘉伟
 
因为我是先在sql-client中进行的提交测试,因此忽略了这个问题,谢谢
 
best,
amenhub
 
 
 
发件人: 冯嘉伟
发送时间: 2020-12-24 18:39
收件人: user-zh
主题: Re: Flink-1.11.1流写filesystem分区提交问题
有开启checkpoint吗?
Part files can be in one of three states:
In-progress : The part file that is currently being written to is
in-progress
Pending : Closed (due to the specified rolling policy) in-progress files
that are waiting to be committed
Finished : On successful checkpoints (STREAMING) or at the end of input
(BATCH) pending files transition to “Finished”
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/file_sink.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/file_sink.html>
  
--
Sent from: http://apache-flink.147419.n8.nabble.com/

回复