meatheadmike commented on issue #11960: URL: https://github.com/apache/hudi/issues/11960#issuecomment-2361347079
I can certainly attempt partitioning again, but doesn't that just exacerbate the file group problem? My last attempt at partitioning made the batches take waaaaaaaay too long. If the writing to many files groups truly is the problem, then why does the process go OOM instead of simply taking longer? Is seems that this is something that could be parallelized and sped up. But in my trial increasing cores did not help. Is there a way I can configure this to not use as much ram? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
