jtmzheng commented on issue #2408: URL: https://github.com/apache/hudi/issues/2408#issuecomment-758360941
Thanks Udit! I'd tried setting `hoodie.commits.archival.batch` to 5 earlier today after going through the source code - that got my application back and running again. The first bug definitely seems like the root cause, after turning on more verbose logging I found several 300mb commit files being loaded in for archival before the crash (re: the second bug https://github.com/apache/hudi/blob/e3d3677b7e7899705b624925666317f0c074f7c7/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java#L353 clears the list, which isn't the most intuitive). It seems like these large commit files were generated when I set `hoodie.cleaner.commits.retained` to 1. What is the trade-off in lowering `hoodie.keep.max.commits` and `hoodie.keep.min.commits`? I couldn't find much good documentation on the archival process/configs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
