azagrebin commented on issue #10180: [FLINK-14631] Account for netty direct allocations in direct memory limit (Netty Shuffle) URL: https://github.com/apache/flink/pull/10180#issuecomment-554891572 @zhijiangW Really thanks for the good insight. We were thinking about a reasonable default value for the number of arenas before we get the zero-copy PR in. It seems that allocating even one arena is too much in most of our testing scenarios. But the artificial tests might be not conclusive enough for that as you just mentioned it can reach even 1Gb. According tor your experience, what would be that reasonable default value for the number of arenas? Should we keep the previous one - number of slots? When you say: `let users config this portion of memory size directly`, do you mean number of arenas? We keep the number of arenas configurable directly and account for them in shuffle memory size. Shuffle memory size is also configurable separately. We also check that the configured shuffle memory size is enough for the configured number of arenas. Did you mean something like this? @pnowojski sure, I will add the description to the commit
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
