[GitHub] [hudi] gunjdesai commented on issue #6610: [QUESTION] Faster approach for backfilling older data without stopping realtime pipelines

[GitBox]

gunjdesai commented on issue #6610:
URL: https://github.com/apache/hudi/issues/6610#issuecomment-1238051026

   On further inspection, I was able to find out that the query is constantly 
picking up more rows and failing when the size of rows cross the memory 
available causing an OOM. 
   
   <img width="679" alt="Screenshot 2022-09-06 at 5 24 57 PM" 
src="https://user-images.githubusercontent.com/7438622/188628948-7ff0132b-4c8d-4e42-92a1-5e3d8ad69747.png";>
   
   At the 10 min interval, the number of output rows are at 1.6M
   <img width="1792" alt="Screenshot 2022-09-06 at 5 25 27 PM" 
src="https://user-images.githubusercontent.com/7438622/188629007-7488ba83-1e98-4126-ba52-38290d2bfac6.png";>
   
   At the 12 min interval, the number of output rows are at 2.1M
   <img width="528" alt="Screenshot 2022-09-06 at 5 28 22 PM" 
src="https://user-images.githubusercontent.com/7438622/188629212-9035e64a-01d2-4cd3-8544-caadf815cd16.png";>
   
   The processed output keeps on increasing even above 23M rows


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org