Hi Udi, Thanks for your confirmation, and I have opened the PR [1] which addresses all of the issues in BEAM-4938 mentioned. Appreciate if you have time to have a look, welcome any comments :)
Best, Jincheng [1] https://github.com/apache/beam/pull/10463 Udi Meiri <[email protected]> 于2019年12月27日周五 上午2:31写道： > Probably best to take one of these existing update bugs: > https://issues.apache.org/jira/browse/BEAM-4938 > There's no discussion on these bugs, so I'd go with 1.26.0 unless someone > else has an objection. > > On Mon, Dec 23, 2019 at 10:04 PM jincheng sun <[email protected]> > wrote: > >> Hi folks, >> >> When submitting a Python word count job to a Flink session/standalone >> cluster repeatedly, the meta space usage of the task manager of the Flink >> cluster will continuously increase (about 40MB each time). The reason is >> that the Beam classes are loaded with the user class loader(child-first by >> default) in Flink and there is a minor problem with the implementation of >> `ProcessManager`(from Beam) and `ThreadPoolCache`(from Netty) which may >> cause the user class loader could not be garbage collected even after the >> job finished which causes the meta space memory leak eventually. You can >> refer to FLINK-15338[1] for more information. >> >> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to >> track it. Regarding to `ThreadPoolCache`, it is a Netty problem and has >> been fixed in NETTY#8955[3]. Netty 4.1.35 Final has already included this >> fix and GRPC 1.22.0 has already dependents on Netty 4.1.35 Final. So we >> need to bump the version of GRPC to 1.22.0+ (currently 1.21.0). >> >> My proposal is to upgrade the GRPC version to the 1.22.0+ (May be the >> latest 1.26.0?) >> >> I've created JIRA [4], but I'm not sure if there will be any other >> problems with the bump the version of GRPC up. So, I'd like to bring up >> this discussion and welcome your feedback ! >> >> [1] https://issues.apache.org/jira/browse/FLINK-15338 >> [2] https://issues.apache.org/jira/browse/BEAM-9006 >> [3] https://github.com/netty/netty/pull/8955 >> [4] https://issues.apache.org/jira/browse/BEAM-9030 >> >> Best, >> Jincheng >> >
