[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=390883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390883
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 21/Feb/20 20:20
Start Date: 21/Feb/20 20:20
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10463: [BEAM-9030]
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r382787180
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -956,7 +956,7 @@ class BeamModulePlugin implements Plugin {
FileTree exposedClasses = project.zipTree(it).matching {
include "**/*.class"
// BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "META-INF/versions/*/module-info.class"
+ exclude "**/module-info.class"
Review comment:
This started ignoring "module-info.class".
Wiithout this change, we could have detected module-info.class sneaking into
the vendor jar.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390883)
Time Spent: 8h 20m (was: 8h 10m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=390884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-390884
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 21/Feb/20 20:20
Start Date: 21/Feb/20 20:20
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10463: [BEAM-9030]
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r382787413
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -956,7 +956,7 @@ class BeamModulePlugin implements Plugin {
FileTree exposedClasses = project.zipTree(it).matching {
Review comment:
unexpectedlyExposedClasses
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 390884)
Time Spent: 8.5h (was: 8h 20m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=372670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372670 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 16/Jan/20 00:36 Start Date: 16/Jan/20 00:36 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10602: [BEAM-9030] Align version of protoc/protoc-gen-grpc-java to vendored version. URL: https://github.com/apache/beam/pull/10602 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 372670) Time Spent: 8h 10m (was: 8h) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 8h 10m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=372620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372620 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 15/Jan/20 23:11 Start Date: 15/Jan/20 23:11 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10602: [BEAM-9030] Align version of protoc/protoc-gen-grpc-java to vendored version. URL: https://github.com/apache/beam/pull/10602#issuecomment-574900277 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 372620) Time Spent: 8h (was: 7h 50m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=372572=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372572 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 15/Jan/20 21:46 Start Date: 15/Jan/20 21:46 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10602: [BEAM-9030] Align version of protoc/protoc-gen-grpc-java to vendored version. URL: https://github.com/apache/beam/pull/10602 Also add comments as to which grpc/protobuf versions should be used during code generation. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=372573=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372573 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 15/Jan/20 21:46 Start Date: 15/Jan/20 21:46 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10602: [BEAM-9030] Align version of protoc/protoc-gen-grpc-java to vendored version. URL: https://github.com/apache/beam/pull/10602#issuecomment-574871070 R: @sunjincheng121 @youngoli This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 372573) Time Spent: 7h 50m (was: 7h 40m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=372562=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372562 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 15/Jan/20 21:31 Start Date: 15/Jan/20 21:31 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10578: [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 URL: https://github.com/apache/beam/pull/10578 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 372562) Time Spent: 7.5h (was: 7h 20m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=372560=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372560
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 15/Jan/20 21:29
Start Date: 15/Jan/20 21:29
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10578: [BEAM-9030]
Migrate Beam to use beam-vendor-grpc-1_26_0
URL: https://github.com/apache/beam/pull/10578#discussion_r367115675
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1437,14 +1437,14 @@ class BeamModulePlugin implements Plugin {
project.protobuf {
protoc {
// The artifact spec for the Protobuf Compiler
- artifact = "com.google.protobuf:protoc:3.6.0" }
+ artifact = "com.google.protobuf:protoc:$protobuf_version" }
Review comment:
applyGrpcNature works on the unvendored gRPC version and as such is correct
in using $protobuf_version and $grpc_version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 372560)
Time Spent: 7h 10m (was: 7h)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=372561=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-372561
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 15/Jan/20 21:29
Start Date: 15/Jan/20 21:29
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10578: [BEAM-9030]
Migrate Beam to use beam-vendor-grpc-1_26_0
URL: https://github.com/apache/beam/pull/10578#discussion_r367115151
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1515,14 +1515,14 @@ class BeamModulePlugin implements Plugin {
project.protobuf {
protoc {
// The artifact spec for the Protobuf Compiler
- artifact = "com.google.protobuf:protoc:3.11.0" }
+ artifact = "com.google.protobuf:protoc:$protobuf_version" }
Review comment:
applyPortabilityNature uses the vendored grpc artifacts and should use
protoc/protoc-gen-grpc-java related to the vendored grpc version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 372561)
Time Spent: 7h 20m (was: 7h 10m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=371828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371828
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 14/Jan/20 19:28
Start Date: 14/Jan/20 19:28
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10578: [BEAM-9030]
Migrate Beam to use beam-vendor-grpc-1_26_0
URL: https://github.com/apache/beam/pull/10578#discussion_r366500697
##
File path: vendor/sdks-java-extensions-protobuf/build.gradle
##
@@ -23,7 +23,7 @@ applyJavaNature(
shadowClosure: {
dependencies {
include(dependency('com.google.guava:guava:26.0-jre'))
-include(dependency('com.google.protobuf:protobuf-java:3.7.1'))
+include(dependency('com.google.protobuf:protobuf-java:3.11.0'))
Review comment:
Can we expose the protobuf and guava version from
GrpcVendoring_1_26_0.groovy so its obvious that we want to use the versions
related to our vendored gRPC.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 371828)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=371826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371826
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 14/Jan/20 19:28
Start Date: 14/Jan/20 19:28
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10578: [BEAM-9030]
Migrate Beam to use beam-vendor-grpc-1_26_0
URL: https://github.com/apache/beam/pull/10578#discussion_r366496246
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -372,7 +372,7 @@ class BeamModulePlugin implements Plugin {
def google_cloud_bigdataoss_version = "1.9.16"
def google_cloud_core_version = "1.61.0"
def google_cloud_spanner_version = "1.6.0"
-def grpc_version = "1.17.1"
+def grpc_version = "1.26.0"
Review comment:
I don't think you want to update the non-vendored gRPC version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 371826)
Time Spent: 7h (was: 6h 50m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=371827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371827
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 14/Jan/20 19:28
Start Date: 14/Jan/20 19:28
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10578: [BEAM-9030]
Migrate Beam to use beam-vendor-grpc-1_26_0
URL: https://github.com/apache/beam/pull/10578#discussion_r366497864
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1444,7 +1444,7 @@ class BeamModulePlugin implements Plugin {
// An artifact spec for a protoc plugin, with "grpc" as
// the identifier, which can be referred to in the "plugins"
// container of the "generateProtoTasks" closure.
- grpc { artifact = "io.grpc:protoc-gen-grpc-java:1.13.1" }
+ grpc { artifact = "io.grpc:protoc-gen-grpc-java:1.26.0" }
Review comment:
This is also for the non-vendored gRPC version. I also believe it should be
$grpc_version and the protoc should be $protobuf_version.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 371827)
Time Spent: 7h (was: 6h 50m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Improvement
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=371601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371601 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 14/Jan/20 13:37 Start Date: 14/Jan/20 13:37 Worklog Time Spent: 10m Work Description: mxm commented on issue #10578: [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 URL: https://github.com/apache/beam/pull/10578#issuecomment-574177598 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 371601) Time Spent: 6h 50m (was: 6h 40m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=371600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371600 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 14/Jan/20 13:37 Start Date: 14/Jan/20 13:37 Worklog Time Spent: 10m Work Description: mxm commented on issue #10578: [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 URL: https://github.com/apache/beam/pull/10578#issuecomment-574177559 Jenkins had some issues with the build queue piling up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 371600) Time Spent: 6h 40m (was: 6.5h) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=371597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371597 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 14/Jan/20 13:23 Start Date: 14/Jan/20 13:23 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10578: [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 URL: https://github.com/apache/beam/pull/10578#issuecomment-574168723 @lukecwik @mxm It looks that only committers could trigger the test now, could you help to trigger the test? Thanks a lot! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 371597) Time Spent: 6.5h (was: 6h 20m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=371596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371596 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 14/Jan/20 13:17 Start Date: 14/Jan/20 13:17 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10578: [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 URL: https://github.com/apache/beam/pull/10578#issuecomment-574168723 @lukecwik It looks that only committers could trigger the test now, could you help to trigger the test? Thanks a lot! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 371596) Time Spent: 6h 20m (was: 6h 10m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=371435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-371435 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 14/Jan/20 08:33 Start Date: 14/Jan/20 08:33 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10578: [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 URL: https://github.com/apache/beam/pull/10578 As the vendored artifact beam-vendor-grpc-1_26_0 has been released, we can update the existing source of Beam to use it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=368744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368744 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 09/Jan/20 01:43 Start Date: 09/Jan/20 01:43 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-572341918 Thank you for all of your review and merging! I have bring up the discussion about gRPC Vendor Release, the detail can be found in [1]. We can open the second half of the upgrade PR once the release is out. [1] https://lists.apache.org/thread.html/7e4de745a224efc24fd5c82737160ef050cbd18a99b45c4e3434fd56%40%3Cdev.beam.apache.org%3E This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368744) Time Spent: 6h (was: 5h 50m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 6h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=368534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368534 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 09/Jan/20 00:50 Start Date: 09/Jan/20 00:50 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368534) Time Spent: 5h 40m (was: 5.5h) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=368535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368535 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 09/Jan/20 00:50 Start Date: 09/Jan/20 00:50 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-572328778 Linkage checker reported no errors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368535) Time Spent: 5h 50m (was: 5h 40m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=368117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368117 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 08/Jan/20 10:28 Start Date: 08/Jan/20 10:28 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-571987531 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368117) Time Spent: 5.5h (was: 5h 20m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=368113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-368113 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 08/Jan/20 10:26 Start Date: 08/Jan/20 10:26 Worklog Time Spent: 10m Work Description: iemejia commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-571986716 comment just to trigger tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 368113) Time Spent: 5h 20m (was: 5h 10m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=367959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367959 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 08/Jan/20 03:39 Start Date: 08/Jan/20 03:39 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-571877625 I have updated the PR accordingly, would be great if you have another look :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 367959) Time Spent: 5h 10m (was: 5h) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=366791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366791
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 06/Jan/20 17:35
Start Date: 06/Jan/20 17:35
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10463: [BEAM-9030]
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r363386810
##
File path: build.gradle
##
@@ -318,7 +318,7 @@ if (project.hasProperty('javaLinkageArtifactIds')) {
dependsOn
project.getTasksByName('publishMavenJavaPublicationToMavenLocal', true /*
recursively */)
classpath = project.configurations.linkageCheckerJava
main = 'com.google.cloud.tools.opensource.classpath.LinkageCheckerMain'
-args '-a',
project.javaLinkageArtifactIds.split(',').collect({"${project.ext.mavenGroupId}:${it}:${project.version}"}).join(',')
+args '-a',
project.javaLinkageArtifactIds.split(',').collect({"${project.ext.mavenGroupId}:${it}:0.1"}).join(',')
Review comment:
You don't want to make this change permanent.
You could check to see if the artifact id has a `:` and if so skip appending
the version. Then people could specify
`-PjavaLinkageArtifactIds=beam-vendor-grpc-1_26_0:0.1-SNAPSHOT` and then we
would get linkage results `org.apache.beam:beam-vendor-grpc-1_26_0:0.1-SNAPSHOT`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 366791)
Time Spent: 4h 40m (was: 4.5h)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=366793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366793 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 06/Jan/20 17:35 Start Date: 06/Jan/20 17:35 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#discussion_r363393068 ## File path: vendor/README.md ## @@ -0,0 +1,37 @@ + + +# Vendored Dependencies Release + +The upgrading of the vendored dependencies should be performed in two steps: +- Firstly, we need to perform a formal release of the vendored dependency. + The [release process](http://s.apache.org/beam-release-vendored-artifacts) of the vendored dependency + is separate from the release of Apache Beam. +- When the release of the vendored dependency is out, we can migrate Apache Beam to use the newly released + vendored dependency. + +# How to validate the vendored dependencies + +The [linkage tool](https://lists.apache.org/thread.html/eb5d95b9a33d7e32dc9bcd0f7d48ba8711d42bd7ed03b9cf0f1103f1%40%3Cdev.beam.apache.org%3E) +is useful for the vendored dependency upgrades. It reports the linkage errors across multiple Apache Beam artifact ids. + +For example, when we upgrade the vendored gRPC to 1.26.0, we could run the linkage tool as following: +``` +./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-vendor-grpc-1_26_0 :checkJavaLinkage Review comment: You need to add `-PvendoredDependenciesOnly` to enable publishing for `beam-vendor-grpc-1_26_0` Also, please update the comment depending on how you want to solve how to specify the version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 366793) Time Spent: 5h (was: 4h 50m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 5h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=366792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366792
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 06/Jan/20 17:35
Start Date: 06/Jan/20 17:35
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10463: [BEAM-9030]
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r363399248
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy
##
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.gradle
+
+import org.gradle.api.Project
+
+/**
+ * Utilities for working with our vendored version of gRPC. The test
dependency junit and the runtime dependencies
+ * slf4j, commons-logging and log4j are not included in the vendored jar. The
linkage checker complains about
+ * these dependencies could be ignored.
+ */
+class GrpcVendoring_1_26_0 {
+ /** Returns the list of compile time dependencies. */
+ static List dependencies() {
+return [
+ 'com.google.guava:guava:26.0-jre',
+ 'com.google.protobuf:protobuf-java:3.11.0',
+ 'com.google.protobuf:protobuf-java-util:3.11.0',
+ 'com.google.code.gson:gson:2.8.6',
+ 'io.grpc:grpc-auth:1.26.0',
+ 'io.grpc:grpc-core:1.26.0',
+ 'io.grpc:grpc-context:1.26.0',
+ 'io.grpc:grpc-netty:1.26.0',
+ 'io.grpc:grpc-protobuf:1.26.0',
+ 'io.grpc:grpc-stub:1.26.0',
+ 'io.netty:netty-transport-native-epoll:4.1.42.Final',
+ // tcnative version from
https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty
+ 'io.netty:netty-tcnative-boringssl-static:2.0.26.Final',
+ 'com.google.auth:google-auth-library-credentials:0.18.0',
+ 'io.grpc:grpc-testing:1.26.0',
+ 'com.google.api.grpc:proto-google-common-protos:1.12.0',
+ 'io.opencensus:opencensus-api:0.24.0',
+ 'io.opencensus:opencensus-contrib-grpc-metrics:0.24.0',
+ 'io.perfmark:perfmark-api:0.19.0',
+ 'com.github.jponge:lzma-java:1.3',
+ 'com.google.protobuf.nano:protobuf-javanano:3.0.0-alpha-5',
+ 'com.jcraft:jzlib:1.1.3',
+ 'com.ning:compress-lzf:1.0.3',
+ 'net.jpountz.lz4:lz4:1.3.0',
+ 'org.bouncycastle:bcpkix-jdk15on:1.54',
+ 'org.bouncycastle:bcprov-jdk15on:1.54',
+ 'org.conscrypt:conscrypt-openjdk-uber:1.3.0',
+ 'org.eclipse.jetty.alpn:alpn-api:1.1.2.v20150522',
+ 'org.eclipse.jetty.npn:npn-api:1.1.1.v20141010',
+ 'org.jboss.marshalling:jboss-marshalling:1.4.11.Final',
+ 'org.jboss.modules:jboss-modules:1.1.0.Beta1'
+]
+ }
+
+ /**
+ * Returns the list of runtime time dependencies that should be exported as
runtime
+ * dependencies within the vendored jar.
+ */
+ static List runtimeDependencies() {
+return [
+ 'com.google.errorprone:error_prone_annotations:2.3.3',
+ 'commons-logging:commons-logging:1.2',
+ 'org.apache.logging.log4j:log4j-api:2.6.2',
+ 'org.slf4j:slf4j-api:1.7.21'
+]
+ }
+
+ /**
+ * Returns the list of test dependencies.
+ */
+ static List testDependencies() {
+return [
+ 'junit:junit:4.12',
Review comment:
We should make this a provided dependency, and not test since test will only
be part of the test runtime for the specific pom while we want users who are
consuming this dependency to provide junit if they intend to test using the
vendored grpc library.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 366792)
Time Spent: 4h 50m (was: 4h 40m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
>
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=366625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366625 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 06/Jan/20 13:16 Start Date: 06/Jan/20 13:16 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-571134123 @lukecwik @kennknowles @suztomo @iemejia Thanks for your comments. Have addressed your comments in the latest PR. Would be great if you have a look at the update :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 366625) Time Spent: 4.5h (was: 4h 20m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=366623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366623
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 06/Jan/20 13:11
Start Date: 06/Jan/20 13:11
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10463:
[BEAM-9030] Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r363286806
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoringOld.groovy
##
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.gradle
+
+import org.gradle.api.Project
+
+/**
+ * Utilities for working with our vendored version of gRPC.
+ */
+class GrpcVendoringOld {
Review comment:
+1 to use `GrpcVendoring_1_21_0` and `GrpcVendoring_1_26_0`. Have also added
`vendor/README.md` about how to upgrade and validate the vendored dependencies.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 366623)
Time Spent: 4h 20m (was: 4h 10m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=365539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-365539
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 02/Jan/20 23:26
Start Date: 02/Jan/20 23:26
Worklog Time Spent: 10m
Work Description: kennknowles commented on pull request #10463:
[BEAM-9030] Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r362667218
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoringOld.groovy
##
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.gradle
+
+import org.gradle.api.Project
+
+/**
+ * Utilities for working with our vendored version of gRPC.
+ */
+class GrpcVendoringOld {
Review comment:
Instead of new/old, use version suffixes `GrpcVendoring_1_21_0` and
`GrpcVendoring_1_26_0`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 365539)
Time Spent: 4h 10m (was: 4h)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=365534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-365534 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 02/Jan/20 23:24 Start Date: 02/Jan/20 23:24 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-570396347 I agree with Ismaël but also these other linkage tips could live in the wiki developer guide: https://cwiki.apache.org/confluence/display/BEAM/Developer+Guides This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 365534) Time Spent: 4h (was: 3h 50m) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 4h > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=365435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-365435
]
ASF GitHub Bot logged work on BEAM-9030:
Author: ASF GitHub Bot
Created on: 02/Jan/20 18:06
Start Date: 02/Jan/20 18:06
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #10463: [BEAM-9030] Bump
grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#issuecomment-570290191
The gradle linkage checker command assumes that your checking
org.apache.beam:$artifactId:2.19.0 as shown
[here](https://github.com/apache/beam/blob/a87af6ab3ad26bbc984b5cf12835afebe400a109/build.gradle#L321).
You can edit the line to be:
```
args '-a',
project.javaLinkageArtifactIds.split(',').collect({"${project.ext.mavenGroupId}:${it}:0.1"}).join(',')
```
Then the command below should match the version that your publishing:
```
./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-vendor-grpc-1_26_0
:checkJavaLinkage
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 365435)
Time Spent: 3h 50m (was: 3h 40m)
> Bump grpc to 1.26.0
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster
> repeatedly, the meta space usage of the task manager of the Flink cluster
> will continuously increase (about 40MB each time). The reason is that the
> Beam classes are loaded with the user class loader in Flink and there are
> problems with the implementation of `ProcessManager`(from Beam) and
> `ThreadPoolCache`(from netty) which may cause the user class loader could not
> be garbage collected even after the job finished which causes the meta space
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the
> version of GRPC to 1.22.0+ (currently 1.21.0).
>
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9030) Bump grpc to 1.26.0
[ https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=364988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364988 ] ASF GitHub Bot logged work on BEAM-9030: Author: ASF GitHub Bot Created on: 31/Dec/19 13:44 Start Date: 31/Dec/19 13:44 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10463: [BEAM-9030] Bump grpc to 1.26.0 URL: https://github.com/apache/beam/pull/10463#issuecomment-569930822 Hi @lukecwik, Regarding to the linkage check, could you share the command you use? I can not reproduce the result with the command `./gradlew -Ppublishing -PjavaLinkageArtifactIds=beam-sdks-java-core,beam-sdks-java-io-jdbc :checkJavaLinkage`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 364988) Time Spent: 3h 40m (was: 3.5h) > Bump grpc to 1.26.0 > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
