[GitHub] [iceberg] ayush-san opened a new issue #2900: Flink CDC job getting failed due to G1 old gc and large checkpointing time

[GitBox]

ayush-san opened a new issue #2900:
URL: https://github.com/apache/iceberg/issues/2900


   Hi all,
   
   We are using Flink + iceberg to consume CDC data. We have combined all the 
tables of a single DB in one job. Our job is frequently running into GC issues. 
Earlier it was running default on parallel GC and I have changed it to G1GC. 
G1GC did bring some improvements but still, I am facing the same problem. 
   
   Following are the params on my job - -ytm 5120m -yjm 1024m -yD 
env.java.opts="-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35"
   
   This job is running CDC ingestion for 17 tables with a parallelism of 1 and 
throughput is around ~10k messages for the 10minutes checkpointing interval
   
   I am attaching a part of the thread dump and gc log too
   
   During old GC, the job gets stuck and its checkpointing which is normally 
under 1 sec gets increased exponentially to the timeout threshold. Job either 
get failed due to checkpointing timeout or it failed to get the heartbeat of 
the task manager
   
   <img width="1118" alt="Screenshot 2021-07-29 at 16 08 58" 
src="https://user-images.githubusercontent.com/57655135/127742589-c174fc7b-748e-4c85-b898-37c3f3a14b62.png";>
   <img width="1349" alt="Screenshot 2021-07-29 at 16 09 19" 
src="https://user-images.githubusercontent.com/57655135/127742613-d3c5724b-634e-4d89-928c-30e34a9c0674.png";>
   
   As you can see in this screenshot checkpoint duration which is normally 
under 1 sec, spikes to over 10mins too but throughput is never above 1k per min
   <img width="1342" alt="Screenshot 2021-07-31 at 19 46 26" 
src="https://user-images.githubusercontent.com/57655135/127742660-614c2cf3-863b-43b1-9b86-7c157aea5fc9.png";>
   
   
[thread_dump.txt](https://github.com/apache/iceberg/files/6911309/thread_dump.txt)
   [gc-11.log](https://github.com/apache/iceberg/files/6911310/gc-11.log)
   [gc-13.log](https://github.com/apache/iceberg/files/6911311/gc-13.log)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org