azagrebin commented on issue #10180: [FLINK-14631] Account for netty direct allocations in direct memory limit (Netty Shuffle) URL: https://github.com/apache/flink/pull/10180#issuecomment-555570474 Thanks for the further feedback @zhijiangW I am adding a fixup to change the expected number of chunks per arena to 6 for accounting in shuffle memory size computation. It seems to be our best guess atm. We can later adjust it if some bigger scale jobs show us during release testing that it needs an increase. Not sure, I fully understand the first option for configuration setup. It seems the number of really required arenas depend on actual job and cluster. I also agree that yet another config `nettyMemorySize` is too confusing. We can keep the current way of configuration. I am adding more into docs and a wrapping exception in case of `OutOfMemoryError` for allocation from arenas explaining how to tackle it by increasing number of arenas. @zhijiangW @xintongsong would be nice if you have another look into the fixup.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
