[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/4481 Will merge this now. ---
[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers
Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/4481 LGTM ð ---
[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 rebased again - this should be good to go. @StefanRRichter can you continue with this? ---
[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 FYI: I just rebased this PR onto current `master` to make this mergable and support further extensions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 by cherry-picking the commits from #4506, plus some fixes for code which was changed in the wrong way previously, the failing yarn tests should now be fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 ok, one test fixed, the other is not so simple but maybe @tillrohrmann can help with it: Inside `ContaineredTaskManagerParameters#create()`, we calculate the amount of off-heap space that we need and for yarn, we use exactly this amount for setting the `-XX:MaxDirectMemorySize` JVM property without letting room for other components and libraries. This worked so far for the network buffers when memory as a whole was set to off-/on-heap and the flink-reserved memory was not completely used. Now, however, if set to on-heap, the `-XX:MaxDirectMemorySize` is too sharp. I'm unsure about the solutions: 1) remove setting `-XX:MaxDirectMemorySize` and let the JVM adjust automatically, or 2) add some "sane" default to our off-heap usage? The same may apply to Mesos if `ResourceProfile(cpuCores, heapMemoryInMB, directMemoryInMB, nativeMemoryInMB)` is used. At the moment, only the other constructors are used leading to solution 1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #4481: [FLINK-7316][network] always use off-heap network buffers
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/4481 actually, I need to fix the test failures in `ContaineredTaskManagerParametersTest` and some failure in the `flink-yarn-tests` first... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
