lvyanquan commented on PR #4418: URL: https://github.com/apache/flink-cdc/pull/4418#issuecomment-4957351548
One thing I would like to clarify is the exact OOM scenario this PR is targeting. My understanding is that the coordinator state mainly keeps growing during the snapshot phase, as more snapshot splits are assigned and finished. Once the job has successfully entered the binlog/incremental phase, the number of finished snapshot splits should be stable. If a checkpoint containing the full snapshot metadata has already completed successfully, then the coordinator state size of later checkpoints should be roughly the same. So I wonder whether the reported OOM happened while the snapshot metadata was still growing, or after the job had already entered the binlog phase. In the latter case, I agree that keeping and repeatedly serializing such a large state is still wasteful and can keep memory pressure high, but it would be helpful to clarify why it still leads to OOM after the state size has become stable, e.g. checkpoint overlap, delayed GC, direct memory/container RSS pressure, etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
