gaoyunhaii commented on pull request #18428: URL: https://github.com/apache/flink/pull/18428#issuecomment-1027668111
Hi Fabian~ > I am not really familiar with this part of the code base but so far I did not see any failing tests regarding the TableOperatorWrapperGenerator can you elaborate a bit more why the current solution may cause problems. Sorry I'm also not expert in table / sql , but from the current implementation of TableOperatorWrapperGenerator, it tries to merge multiple transformations into one `MultipleInputTransformation`, and the parallelism of this one is set with the maximum of all the merged transformations. Previously the transformation might returns env.defaultParallelism in this step, but now it might return -1, suppose we have multiple parallelisms [200, env.default, env.default] and env.default = 300, previously the result would be 300 while now the result becomes 200. > Afaict the motivation to update the commit summary is exactly for unaligned checkpointing. In case the checkpoint barrier overtakes the records we only emit an incomplete summary and later need to emit a new summary with the updated amount of seen records. > In general, I am also open to restricting it in the first version and figuring out the unaligned checkpoint use cases later. WDYT? I think with unaligned checkpointing we should still be able to keep one summary for each checkpoint: each operator could only emit the summary after checkpoint complete notified & received all the summary from each input subtask? If this is feasible we could force the current implementation of pre-commit topology to follow this pattern, but not provide the same promise (only one summary to the same checkpont) to the post-commit topology, then we would not have compatibility no matter which pattern we want in the future~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
