[jira] [Closed] (BEAM-9573) Watermark hold for timer output timestamp is not computed correctly
[ https://issues.apache.org/jira/browse/BEAM-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng closed BEAM-9573. - Resolution: Fixed Close it as the PR is already merged :) > Watermark hold for timer output timestamp is not computed correctly > --- > > Key: BEAM-9573 > URL: https://issues.apache.org/jira/browse/BEAM-9573 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.20.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Blocker > Fix For: 2.20.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > With the introduction of timer output timestamp, a new watermark hold had > been added to the Flink Runner. The watermark computation works on the keyed > state backend which computes a key-scoped watermark hold and not the desired > operator-wide watermark hold. > Computation: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1140 > Key-scoped state: > https://github.com/apache/beam/blob/b564239081e9351c56fb0e7d263495b95dd3f8f3/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L1130 > We need to change this to operate on all keys. This has to be done before > fixing BEAM-9566. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048192#comment-17048192 ] sunjincheng commented on BEAM-9298: --- I think there is no need to make it a blocker of 2.20.0. I have removed the target version. > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.21.0 > > Time Spent: 40m > Remaining Estimate: 0h > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9298: -- Fix Version/s: 2.21.0 > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.21.0 > > Time Spent: 40m > Remaining Estimate: 0h > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045105#comment-17045105 ] sunjincheng commented on BEAM-8618: --- Yes，just remove the tag for now. :) > Tear down unused DoFns periodically in Python SDK harness > - > > Key: BEAM-8618 > URL: https://issues.apache.org/jira/browse/BEAM-8618 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Per the discussion in the ML, detail can be found [1], the teardown of DoFns > should be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8618: -- Fix Version/s: (was: 2.20.0) > Tear down unused DoFns periodically in Python SDK harness > - > > Key: BEAM-8618 > URL: https://issues.apache.org/jira/browse/BEAM-8618 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Per the discussion in the ML, detail can be found [1], the teardown of DoFns > should be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8620) Tear down unused DoFns periodically in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8620: -- Fix Version/s: (was: 2.20.0) > Tear down unused DoFns periodically in Java SDK harness > --- > > Key: BEAM-8620 > URL: https://issues.apache.org/jira/browse/BEAM-8620 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > > Per the discussion in the ML the detail can be found here[1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8620) Tear down unused DoFns periodically in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044119#comment-17044119 ] sunjincheng commented on BEAM-8620: --- Thanks for the reminder, I would like to reset the fix version to 2.21.0, and appreciate if you(or some one) can add unrelease version of 2.21.0. :) for now, reset it as None. > Tear down unused DoFns periodically in Java SDK harness > --- > > Key: BEAM-8620 > URL: https://issues.apache.org/jira/browse/BEAM-8620 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > Per the discussion in the ML the detail can be found here[1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9298: -- Fix Version/s: (was: 2.20.0) > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9299: -- Fix Version/s: (was: 2.20.0) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044074#comment-17044074 ] sunjincheng edited comment on BEAM-9295 at 2/25/20 2:41 AM: I would like reset the fix version to 2.21. Thank you . [~amaliujia] Could you please add the un-release version 2.21.0 ? was (Author: sunjincheng121): I would like reset the fix version to 2.21. Thank you . [~amaliujia] > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9295: -- Fix Version/s: (was: 2.20.0) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044074#comment-17044074 ] sunjincheng commented on BEAM-9295: --- I would like reset the fix version to 2.21. Thank you . [~amaliujia] > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9288) Conscrypt shaded dependency
[ https://issues.apache.org/jira/browse/BEAM-9288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042601#comment-17042601 ] sunjincheng commented on BEAM-9288: --- I agree that we should exclude it from the vendored grpc for now. I will submit a PR to address this issue. > Conscrypt shaded dependency > --- > > Key: BEAM-9288 > URL: https://issues.apache.org/jira/browse/BEAM-9288 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Esun Kim >Assignee: sunjincheng >Priority: Critical > Fix For: 2.20.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Conscrypt is not designed to be shaded properly mainly because of so files. I > happened to see BEAM-9030 (*1) creating a new vendored gRPC shading Conscrypt > (*2) in it. I think this could make a problem when new Conscrypt is brought > by new gcsio depending on gRPC-alts (*4) in a dependency chain. (*5) In this > case, it may have a conflict when finding proper so files for Conscrypt. > *1: https://issues.apache.org/jira/browse/BEAM-9030 > *2: > [https://github.com/apache/beam/blob/e24d1e51cbabe27cb3cc381fd95b334db639c45d/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy#L78] > *3: https://issues.apache.org/jira/browse/BEAM-6136 > *4: [https://mvnrepository.com/artifact/io.grpc/grpc-alts/1.27.0] > *5: https://issues.apache.org/jira/browse/BEAM-8889 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039584#comment-17039584 ] sunjincheng commented on BEAM-9299: --- Currently we always test the last version for flink runners，and I think it's by design for now, and I agree with you [~iemejia], would be better to avoid test all versions for every PR. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039129#comment-17039129 ] sunjincheng commented on BEAM-9298: --- I've brought up a community discussion here: [https://lists.apache.org/thread.html/rfb5ac9d889d0e3f4400471de3c25000a15352bde879622c899d97581%40%3Cdev.beam.apache.org%3E] > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 10m > Remaining Estimate: 0h > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9298) Drop support for Flink`1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039079#comment-17039079 ] sunjincheng commented on BEAM-9298: --- Thanks for the confirm [~mxm] Because the code of Flink 1.7 is too old, the current existence of Flink runner 1.7 will affect the upgrade of Flink run.er 1.8x and 1.9x , more detail can be found in BEAM-9299, so we need to remove the support of Flink runner 1.7 as soon as possible. and I'll bring up the DISCUSSION soon，and open the PR ASAP. > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng > ` Priority: Major > Fix For: 2.20.0 > > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9298: -- Comment: was deleted (was: I see, thank you all! I will do it accodingly after add Flink 1.10 build target. :)) > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038898#comment-17038898 ] sunjincheng commented on BEAM-9299: --- As BEAM-5396 and FLINK-11048 have been completed so far, we have a cleaner solution 3, which is to remove `BeamFlinkRemoteStreamEnvironment` class. I have updated the PR, but the content does not include the changes of `cover all of the version for test` which [~angoenka] mentioned before, I think we can handle it in another PR. What do you think? > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038240#comment-17038240 ] sunjincheng commented on BEAM-9299: --- Thanks for your feedback [~iemejia] [~angoenka]! I will update PR according to solution 2. And I agree that it's better to let the test cover all of versions of runner. But I'm curious. Why only test the latest one before, Is this by design？ > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036700#comment-17036700 ] sunjincheng edited comment on BEAM-9299 at 2/14/20 5:42 AM: I found that the issue FLINK-15844 also exists in Flink 1.8.3. I noticed that FLINK-15844 has provided a fix for 1.9.3. However, AFAIK, there will be no new 1.8 releases any more according to FLink's [release policy|#update-policy-for-old-releases]]. There are two solutions in my mind: - Solution1: As the signature of `JobWithJars.buildUserCodeClassLoader `has changed in both Flink 1.8.3 and 1.9.2 and Flink 1.7.2 still uses the old signature, we could drop the Flink 1.7 support firstly and then update the implementation of `FlinkExecutionEnvironments` to use the new signature. - Solution2: We could make a copy of `FlinkExecutionEnvironments` in each version of Flink runner and update the implementation for each copy according to the Flink version. This solution decouples the drop of Flink 1.7 support and the upgrades of 1.8 and 1.9. Besides, Flink community has made big change for the job submission path(FLINK-13954) in 1.10, e.g. `JobWithJars` has been removed in 1.10. It means that the job submission logic will be separate for 1.8/1.9 and 1.10 anyway. Personally I prefer solution2 and what's your thought? :) [~iemejia] [~mxm] [~thw] was (Author: sunjincheng121): I found that the issue FLINK-15844 also exists in Flink 1.8.3. I noticed that FLINK-15844 has provided a fix for 1.9.3. However, AFAIK, there will be no new 1.8 releases any more according to FLink's [release policy|#update-policy-for-old-releases]]. There are two solutions in my mind: - Solution1: As the signature of `JobWithJars.buildUserCodeClassLoader `has changed in both Flink 1.8.3 and 1.9.2 and Flink 1.7.2 still uses the old signature, we could drop the Flink 1.7 support firstly and then update the implementation of `FlinkExecutionEnvironments` to use the new signature. - Solution2: We could make a copy of `FlinkExecutionEnvironments` in each version of Flink runner and update the implementation for each copy according to the Flink version. This solution decouples the drop of Flink 1.7 support and the upgrades of 1.8 and 1.9. Besides, Flink community has made big change for the job submission path(FLINK-13954) in 1.10, e.g. `JobWithJars` has been removed in 1.10. It means that the job submission logic will be separate for 1.8/1.9 and 1.10 anyway. Personally I prefer solution2 and what's your thought? :) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036700#comment-17036700 ] sunjincheng edited comment on BEAM-9299 at 2/14/20 5:35 AM: I found that the issue FLINK-15844 also exists in Flink 1.8.3. I noticed that FLINK-15844 has provided a fix for 1.9.3. However, AFAIK, there will be no new 1.8 releases any more according to FLink's [release policy|#update-policy-for-old-releases]]. There are two solutions in my mind: - Solution1: As the signature of `JobWithJars.buildUserCodeClassLoader `has changed in both Flink 1.8.3 and 1.9.2 and Flink 1.7.2 still uses the old signature, we could drop the Flink 1.7 support firstly and then update the implementation of `FlinkExecutionEnvironments` to use the new signature. - Solution2: We could make a copy of `FlinkExecutionEnvironments` in each version of Flink runner and update the implementation for each copy according to the Flink version. This solution decouples the drop of Flink 1.7 support and the upgrades of 1.8 and 1.9. Besides, Flink community has made big change for the job submission path(FLINK-13954) in 1.10, e.g. `JobWithJars` has been removed in 1.10. It means that the job submission logic will be separate for 1.8/1.9 and 1.10 anyway. Personally I prefer solution2 and what's your thought? :) was (Author: sunjincheng121): I found that the issue FLINK-15844 also exists in Flink 1.8.3. I noticed that FLINK-15844 has provided a fix for 1.9.3. However, AFAIK, there will be no new 1.8 releases any more according to FLink's [release policy|[https://flink.apache.org/downloads.html#update-policy-for-old-releases]]. There are two solutions in my mind: - Solution1: As the signature of `JobWithJars.buildUserCodeClassLoader `has changed in both Flink 1.8.3 and 1.9.2 and Flink 1.7.2 still uses the old signature, we could drop the Flink 1.7 support firstly and then update the implementation of `FlinkExecutionEnvironments` to use the new signature. -Solution2: We could make a copy of `FlinkExecutionEnvironments` in each version of Flink runner and update the implementation for each copy according to the Flink version. This solution decouples the drop of Flink 1.7 support and the upgrades of 1.8 and 1.9. Besides, Flink community has made big change for the job submission path(FLINK-13954) in 1.10, e.g. `JobWithJars` has been removed in 1.10. It means that the job submission logic will be separate for 1.8/1.9 and 1.10 anyway. Personally I prefer solution2 and what's your thought? :) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036700#comment-17036700 ] sunjincheng commented on BEAM-9299: --- I found that the issue FLINK-15844 also exists in Flink 1.8.3. I noticed that FLINK-15844 has provided a fix for 1.9.3. However, AFAIK, there will be no new 1.8 releases any more according to FLink's [release policy|[https://flink.apache.org/downloads.html#update-policy-for-old-releases]]. There are two solutions in my mind: - Solution1: As the signature of `JobWithJars.buildUserCodeClassLoader `has changed in both Flink 1.8.3 and 1.9.2 and Flink 1.7.2 still uses the old signature, we could drop the Flink 1.7 support firstly and then update the implementation of `FlinkExecutionEnvironments` to use the new signature. -Solution2: We could make a copy of `FlinkExecutionEnvironments` in each version of Flink runner and update the implementation for each copy according to the Flink version. This solution decouples the drop of Flink 1.7 support and the upgrades of 1.8 and 1.9. Besides, Flink community has made big change for the job submission path(FLINK-13954) in 1.10, e.g. `JobWithJars` has been removed in 1.10. It means that the job submission logic will be separate for 1.8/1.9 and 1.10 anyway. Personally I prefer solution2 and what's your thought? :) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036154#comment-17036154 ] sunjincheng edited comment on BEAM-9298 at 2/13/20 11:46 AM: - I see, thank you all! I will do it accodingly after add Flink 1.10 build target. :) was (Author: sunjincheng121): Thank you all! I will do it accodingly after add Flink 1.10 build target. :) > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036154#comment-17036154 ] sunjincheng commented on BEAM-9298: --- Thank you all! I will do it accodingly after add Flink 1.10 build target. :) > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035945#comment-17035945 ] sunjincheng edited comment on BEAM-9299 at 2/13/20 6:20 AM: Thanks for add this input [~iemejia] , I think we can upgrade the 1.9.1 to 1.9.3 when Flink 1.9.3 released due to this upgrade is not urgent. :) And we only upgrade 1.8.x to 1.8.3 in this JIRA. What do you think? was (Author: sunjincheng121): Thanks for add this input [~iemejia] , I think we can upgrade the 1.9.1 to 1.9.3 when Flink 1.9.3 released due to this upgrade is not urgent. :) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035945#comment-17035945 ] sunjincheng commented on BEAM-9299: --- Thanks for add this input [~iemejia] , I think we can upgrade the 1.9.1 to 1.9.3 when Flink 1.9.3 released due to this upgrade is not urgent. :) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9299: -- Comment: was deleted (was: Thanks for add this input [~iemejia] I think we can upgrade the 1.9.1 to 1.9.3 when Flink 1.9.3 released. :) and do not upgrade the 1.9.1 in this JIRA. ) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035941#comment-17035941 ] sunjincheng commented on BEAM-9299: --- Thanks for add this input [~iemejia] I think we can upgrade the 1.9.1 to 1.9.3 when Flink 1.9.3 released. :) and do not upgrade the 1.9.1 in this JIRA. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035890#comment-17035890 ] sunjincheng edited comment on BEAM-9298 at 2/13/20 4:53 AM: Thanks for the reminder [~iemejia] and [~thw] ! I prefer maintain at most three(Even maintaining two is reasonable) versions of Flink runner for Apache Beam community, because Apache Flink only support the current and previous minor release with bugfixes, more detail can be found in [1]. So, I think its better to maintain the three versions of 1.8/1.9/1.10 after add Flink v1.10 build target to Flink runner. I'm not sure if it deserves a discussion in the mailing list as it seems that we already maintain 3 versions of Flink runner in Beam now. Removing Flink runner 1.7 seems straightward if we upgrade Flink runner to 1.10. However, I'm not sure what's the policy before and it's great to also hear [~mxm] 's suggestions? :) [1] [https://flink.apache.org/downloads.html#update-policy-for-old-releases] was (Author: sunjincheng121): Thanks for the reminder [~iemejia] and [~thw] ! I prefer maintain at most three versions of Flink runner for Apache Beam community, because Apache Flink only support the current and previous minor release with bugfixes, more detail can be found in [1]. So, I think its better to maintain the three versions of 1.8/1.9/1.10 after add Flink v1.10 build target to Flink runner. I'm not sure if it deserves a discussion in the mailing list as it seems that we already maintain 3 versions of Flink runner in Beam now. Removing Flink runner 1.7 seems straightward if we upgrade Flink runner to 1.10. However, I'm not sure what's the policy before and it's great to also hear [~mxm] 's suggestions? :) [1] [https://flink.apache.org/downloads.html#update-policy-for-old-releases] > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035890#comment-17035890 ] sunjincheng edited comment on BEAM-9298 at 2/13/20 3:48 AM: Thanks for the reminder [~iemejia] and [~thw] ! I prefer maintain at most three versions of Flink runner for Apache Beam community, because Apache Flink only support the current and previous minor release with bugfixes, more detail can be found in [1]. So, I think its better to maintain the three versions of 1.8/1.9/1.10 after add Flink v1.10 build target to Flink runner. I'm not sure if it deserves a discussion in the mailing list as it seems that we already maintain 3 versions of Flink runner in Beam now. Removing Flink runner 1.7 seems straightward if we upgrade Flink runner to 1.10. However, I'm not sure what's the policy before and it's great to also hear [~mxm] 's suggestions? :) [1] [https://flink.apache.org/downloads.html#update-policy-for-old-releases] was (Author: sunjincheng121): Thanks for the reminder [~iemejia] and [~thw] ! I prefer maintain at most three versions of Flink runner for Apache Beam community, because Apache Flink only support the current and previous minor release with bugfixes, more detail can be found in [1]. So, I think its better to maintain the three versions of 1.8/1.9/1.10 after add Flink v1.10 build target to Flink runner. I'm not sure if it deserves a discussion in the mailing list as it seems that we always maintain 3 versions of Flink runner in Beam. Removing Flink runner 1.7 seems straightward if we upgrade Flink runner to 1.10. However, I'm not sure what's the policy before and it's great to also hear [~mxm] 's suggestions? :) [1] [https://flink.apache.org/downloads.html#update-policy-for-old-releases] > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035890#comment-17035890 ] sunjincheng commented on BEAM-9298: --- Thanks for the reminder [~iemejia] and [~thw] ! I prefer maintain at most three versions of Flink runner for Apache Beam community, because Apache Flink only support the current and previous minor release with bugfixes, more detail can be found in [1]. So, I think its better to maintain the three versions of 1.8/1.9/1.10 after add Flink v1.10 build target to Flink runner. I'm not sure if it deserves a discussion in the mailing list as it seems that we always maintain 3 versions of Flink runner in Beam. Removing Flink runner 1.7 seems straightward if we upgrade Flink runner to 1.10. However, I'm not sure what's the policy before and it's great to also hear [~mxm] 's suggestions? :) [1] [https://flink.apache.org/downloads.html#update-policy-for-old-releases] > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
sunjincheng created BEAM-9299: - Summary: Upgrade Flink Runner to 1.8.3 and 1.9.2 Key: BEAM-9299 URL: https://issues.apache.org/jira/browse/BEAM-9299 Project: Beam Issue Type: Task Components: runner-flink Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.20.0 I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. What do you think? [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9298) Drop support for Flink 1.7
sunjincheng created BEAM-9298: - Summary: Drop support for Flink 1.7 Key: BEAM-9298 URL: https://issues.apache.org/jira/browse/BEAM-9298 Project: Beam Issue Type: Task Components: runner-flink Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.20.0 With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we should consider dropping support for Flink 1.7. Then dropping 1.7 will also decrease the build time. What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
sunjincheng created BEAM-9295: - Summary: Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 Key: BEAM-9295 URL: https://issues.apache.org/jira/browse/BEAM-9295 Project: Beam Issue Type: New Feature Components: runner-flink Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.20.0 Apache Flink 1.10 has completed the final release vote, see [1]. So, I would like to add Flink 1.10 build target and make Flink Runner compatible with Flink 1.10. And I appreciate it if you can leave your suggestions or comments! [1] https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8618: -- Fix Version/s: (was: 2.19.0) 2.20.0 > Tear down unused DoFns periodically in Python SDK harness > - > > Key: BEAM-8618 > URL: https://issues.apache.org/jira/browse/BEAM-8618 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > Per the discussion in the ML, detail can be found [1], the teardown of DoFns > should be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019363#comment-17019363 ] sunjincheng commented on BEAM-8618: --- Thanks for the reminder， reset the fixversion to 2.20. > Tear down unused DoFns periodically in Python SDK harness > - > > Key: BEAM-8618 > URL: https://issues.apache.org/jira/browse/BEAM-8618 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > Per the discussion in the ML, detail can be found [1], the teardown of DoFns > should be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8620) Tear down unused DoFns periodically in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8620: -- Fix Version/s: (was: 2.19.0) 2.20.0 > Tear down unused DoFns periodically in Java SDK harness > --- > > Key: BEAM-8620 > URL: https://issues.apache.org/jira/browse/BEAM-8620 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > Per the discussion in the ML the detail can be found here[1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng resolved BEAM-7951. --- Fix Version/s: 2.19.0 Resolution: Fixed > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 14h > Remaining Estimate: 0h > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9153) The heading level of "Deploy source release to dist.apache.org" in release guide is incorrect
[ https://issues.apache.org/jira/browse/BEAM-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9153: -- Description: Deploy source release to dist.apache.org" in [1] should have the same heading level as "Deploy Python artifacts to PyPI. [1][https://beam.apache.org/contribute/release-guide/#deploy-source-release-to-distapacheorg] was:Deploy source release to dist.apache.org" should have the same heading level as "Deploy Python artifacts to PyPI > The heading level of "Deploy source release to dist.apache.org" in release > guide is incorrect > - > > Key: BEAM-9153 > URL: https://issues.apache.org/jira/browse/BEAM-9153 > Project: Beam > Issue Type: Bug > Components: website >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > Deploy source release to dist.apache.org" in [1] should have the same > heading level as "Deploy Python artifacts to PyPI. > > > [1][https://beam.apache.org/contribute/release-guide/#deploy-source-release-to-distapacheorg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9153) The heading level of "Deploy source release to dist.apache.org" in release guide is incorrect
sunjincheng created BEAM-9153: - Summary: The heading level of "Deploy source release to dist.apache.org" in release guide is incorrect Key: BEAM-9153 URL: https://issues.apache.org/jira/browse/BEAM-9153 Project: Beam Issue Type: Bug Components: website Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.20.0 Deploy source release to dist.apache.org" should have the same heading level as "Deploy Python artifacts to PyPI -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017754#comment-17017754 ] sunjincheng edited comment on BEAM-9137 at 1/17/20 7:14 AM: I have a quick look at of this issue and it seems that the beam_PostCommit_Py_ValCont was broken on 20 Dec, 2019. We can see that it succeed on [19 Dec 2019 |http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c100795298.3239.1576775433670.JavaMail.jenkins@jenkins02%3e]and failed on [20 Dec, 2019|http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c165865149.3447.1576823413985.JavaMail.jenkins@jenkins02%3e]. So, maybe we should focus on the commits of that day. was (Author: sunjincheng121): I have a quick look at of this issue and it seems that the beam_PostCommit_Py_ValCont was broken on 20 Dec, 2019. We can see that it succeed on [19 Dec 2019 |http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c100795298.3239.1576775433670.JavaMail.jenkins@jenkins02%3e]and failed on [20 Dec, 2019|http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c165865149.3447.1576823413985.JavaMail.jenkins@jenkins02%3e]. > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017754#comment-17017754 ] sunjincheng commented on BEAM-9137: --- I have a quick look at of this issue and it seems that the beam_PostCommit_Py_ValCont was broken on 20 Dec, 2019. We can see that it succeed on [19 Dec 2019 |http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c100795298.3239.1576775433670.JavaMail.jenkins@jenkins02%3e]and failed on [20 Dec, 2019|http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c165865149.3447.1576823413985.JavaMail.jenkins@jenkins02%3e]. > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng reassigned BEAM-9137: - Assignee: (was: sunjincheng) > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017747#comment-17017747 ] sunjincheng commented on BEAM-9137: --- It seems that the test error is not caused by[1], more detail log can be found in [2]. [~boyuanz] [1] [https://github.com/apache/beam/pull/10338] [2] [https://github.com/apache/beam/pull/10338#issuecomment-575482269] > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Assignee: sunjincheng >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9055) Unify the config names of Fn Data API across languages
[ https://issues.apache.org/jira/browse/BEAM-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9055: -- Fix Version/s: 2.19.0 > Unify the config names of Fn Data API across languages > -- > > Key: BEAM-9055 > URL: https://issues.apache.org/jira/browse/BEAM-9055 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > > Currently for time-based cache threshold config, the config key is > "data_buffer_time_limit_ms" in the Python SDK harness and > "beam_fn_api_data_buffer_time_limit" in the Java SDK harness. As discussed in > [https://github.com/apache/beam/pull/10246#discussion_r362572952], we should > unify the config names of Fn Data API across languages to have a good user > experience. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9055) Unify the config names of Fn Data API across languages
sunjincheng created BEAM-9055: - Summary: Unify the config names of Fn Data API across languages Key: BEAM-9055 URL: https://issues.apache.org/jira/browse/BEAM-9055 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: sunjincheng Assignee: sunjincheng Currently for time-based cache threshold config, the config key is "data_buffer_time_limit_ms" in the Python SDK harness and "beam_fn_api_data_buffer_time_limit" in the Java SDK harness. As discussed in [https://github.com/apache/beam/pull/10246#discussion_r362572952], we should unify the config names of Fn Data API across languages to have a good user experience. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9030) Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0)
[ https://issues.apache.org/jira/browse/BEAM-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9030: -- Summary: Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0) (was: Metaspace memory leak when running python jobs with flink runner) > Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0) > --- > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9030) Metaspace memory leak when running python jobs with flink runner
[ https://issues.apache.org/jira/browse/BEAM-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17002682#comment-17002682 ] sunjincheng commented on BEAM-9030: --- Discussion details can be found here: [https://lists.apache.org/thread.html/ef5b24766d94d3d389bee9c03e59003b9cf417c81cde50ede5856ad7%40%3Cdev.beam.apache.org%3E] > Metaspace memory leak when running python jobs with flink runner > > > Key: BEAM-9030 > URL: https://issues.apache.org/jira/browse/BEAM-9030 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > > When submitting a Python word count job to a Flink session/standalone cluster > repeatedly, the meta space usage of the task manager of the Flink cluster > will continuously increase (about 40MB each time). The reason is that the > Beam classes are loaded with the user class loader in Flink and there are > problems with the implementation of `ProcessManager`(from Beam) and > `ThreadPoolCache`(from netty) which may cause the user class loader could not > be garbage collected even after the job finished which causes the meta space > memory leak eventually. You can refer to FLINK-15338[1] for more information. > Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track > it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed > in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC > 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the > version of GRPC to 1.22.0+ (currently 1.21.0). > > What do you think? > [1] https://issues.apache.org/jira/browse/FLINK-15338 > [2] https://issues.apache.org/jira/browse/BEAM-9006 > [3] [https://github.com/netty/netty/pull/8955] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9030) Metaspace memory leak when running python jobs with flink runner
sunjincheng created BEAM-9030: - Summary: Metaspace memory leak when running python jobs with flink runner Key: BEAM-9030 URL: https://issues.apache.org/jira/browse/BEAM-9030 Project: Beam Issue Type: Bug Components: java-fn-execution, runner-flink Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.19.0 When submitting a Python word count job to a Flink session/standalone cluster repeatedly, the meta space usage of the task manager of the Flink cluster will continuously increase (about 40MB each time). The reason is that the Beam classes are loaded with the user class loader in Flink and there are problems with the implementation of `ProcessManager`(from Beam) and `ThreadPoolCache`(from netty) which may cause the user class loader could not be garbage collected even after the job finished which causes the meta space memory leak eventually. You can refer to FLINK-15338[1] for more information. Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the version of GRPC to 1.22.0+ (currently 1.21.0). What do you think? [1] https://issues.apache.org/jira/browse/FLINK-15338 [2] https://issues.apache.org/jira/browse/BEAM-9006 [3] [https://github.com/netty/netty/pull/8955] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9006) Meta space memory leak caused by the shutdown hook of ProcessManager
[ https://issues.apache.org/jira/browse/BEAM-9006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-9006: -- Fix Version/s: (was: 2.18.0) 2.19.0 > Meta space memory leak caused by the shutdown hook of ProcessManager > - > > Key: BEAM-9006 > URL: https://issues.apache.org/jira/browse/BEAM-9006 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > > Currently the class `ProcessManager` will add a shutdown hook to stop all the > living processes before JVM exits. The shutdown hook will never be removed. > If this class is loaded by the user class loader, it will cause the user > class loader could not be garbage collected which causes meta space memory > leak eventually. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9006) Meta space memory leak caused by the shutdown hook of ProcessManager
sunjincheng created BEAM-9006: - Summary: Meta space memory leak caused by the shutdown hook of ProcessManager Key: BEAM-9006 URL: https://issues.apache.org/jira/browse/BEAM-9006 Project: Beam Issue Type: Bug Components: java-fn-execution Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.18.0 Currently the class `ProcessManager` will add a shutdown hook to stop all the living processes before JVM exits. The shutdown hook will never be removed. If this class is loaded by the user class loader, it will cause the user class loader could not be garbage collected which causes meta space memory leak eventually. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8935) Fail fast if sdk harness startup failed
sunjincheng created BEAM-8935: - Summary: Fail fast if sdk harness startup failed Key: BEAM-8935 URL: https://issues.apache.org/jira/browse/BEAM-8935 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.19.0 Currently the runner waits for the sdk harness to startup blockingly until the sdk harness is available or timeout occurs. The timeout is 1 or 2 minutes. If the sdk harness startup failed for some reason, the runner may be aware of it after 1 or 2 minutes. This is too long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8929) Remove unnecessary exception handling in FnApiControlClientPoolService
sunjincheng created BEAM-8929: - Summary: Remove unnecessary exception handling in FnApiControlClientPoolService Key: BEAM-8929 URL: https://issues.apache.org/jira/browse/BEAM-8929 Project: Beam Issue Type: Improvement Components: java-fn-execution Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.19.0 The exception handling logic in [FnApiControlClientPoolService|https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java#L102] is unnecessary and could be removed.(Clean up usless code) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8618: -- Fix Version/s: (was: 2.18.0) 2.19.0 > Tear down unused DoFns periodically in Python SDK harness > - > > Key: BEAM-8618 > URL: https://issues.apache.org/jira/browse/BEAM-8618 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > > Per the discussion in the ML, detail can be found [1], the teardown of DoFns > should be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8733) The "KeyError: u'-47'" error from line 305 of sdk_worker.py
[
https://issues.apache.org/jira/browse/BEAM-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng resolved BEAM-8733.
---
Resolution: Fixed
> The "KeyError: u'-47'" error from line 305 of sdk_worker.py
> ---
>
> Key: BEAM-8733
> URL: https://issues.apache.org/jira/browse/BEAM-8733
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> The issue reported by [~chamikara], error message as follows:
> apache_beam/runners/worker/sdk_worker.py", line 305, in get
> self.fns[bundle_descriptor_id],
> KeyError: u'-47'
> {code}
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
> Suppressed: java.lang.IllegalStateException: Already closed.
> at
> org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:93)
> at
> org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
> {code}
> More discussion info can be found here:
> https://github.com/apache/beam/pull/10004
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8620) Tear down unused DoFns periodically in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8620: -- Fix Version/s: (was: 2.18.0) 2.19.0 > Tear down unused DoFns periodically in Java SDK harness > --- > > Key: BEAM-8620 > URL: https://issues.apache.org/jira/browse/BEAM-8620 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > > Per the discussion in the ML the detail can be found here[1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8847) Handle the registration request synchronously in the Java SDK harness
sunjincheng created BEAM-8847: - Summary: Handle the registration request synchronously in the Java SDK harness Key: BEAM-8847 URL: https://issues.apache.org/jira/browse/BEAM-8847 Project: Beam Issue Type: Bug Components: sdk-java-harness Affects Versions: 2.18.0 Reporter: sunjincheng Assignee: sunjincheng Currently the registration request is handled asynchronously in the Java SDK harness. As discussed in BEAM-8733, this JIRA. tries to change it to synchronous. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8846) Force synchronization of the stream observer in BeamFnControlClient
sunjincheng created BEAM-8846: - Summary: Force synchronization of the stream observer in BeamFnControlClient Key: BEAM-8846 URL: https://issues.apache.org/jira/browse/BEAM-8846 Project: Beam Issue Type: Bug Components: sdk-java-harness Affects Versions: 2.18.0 Reporter: sunjincheng Assignee: sunjincheng Currently there is no synchronization to access the stream observer in BeamFnControlClient which is not thread safe. We should fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7952) Make the input queue of the input buffer in Python SDK Harness size limited.
[ https://issues.apache.org/jira/browse/BEAM-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng resolved BEAM-7952. --- Fix Version/s: 2.18.0 Resolution: Duplicate https://issues.apache.org/jira/browse/BEAM-8667 > Make the input queue of the input buffer in Python SDK Harness size limited. > > > Key: BEAM-7952 > URL: https://issues.apache.org/jira/browse/BEAM-7952 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > > At Python SDK harness, the input queue size of the input buffer in Python SDK > Harness is not size limited and also not configurable. This may become a > problem if the data production rate is more than the data consumption rate. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8617) Tear down the DoFns upon the control service termination in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng closed BEAM-8617. - Resolution: Duplicate > Tear down the DoFns upon the control service termination in Python SDK harness > -- > > Key: BEAM-8617 > URL: https://issues.apache.org/jira/browse/BEAM-8617 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Per the discussion in the ML can be found [1], the teardown of DoFns should > be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support to teardown the DoFns upon the control > service termination for Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8557) Clean up useless null check.
[
https://issues.apache.org/jira/browse/BEAM-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng closed BEAM-8557.
-
Resolution: Fixed
> Clean up useless null check.
>
>
> Key: BEAM-8557
> URL: https://issues.apache.org/jira/browse/BEAM-8557
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> I think we do not need null check here:
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
> Because before the the `onNext` call, the `Future` already put into the queue
> in `handle` method.
>
> I found the test as follows:
> {code:java}
> @Test
> public void testUnknownResponseIgnored() throws Exception{code}
> I do not know why we need test this case? I think it would be better if we
> throw the Exception for an UnknownResponse, otherwise, this may hidden a
> potential bug.
> Please correct me if there anything I misunderstand @kennknowles
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8557) Clean up useless null check.
[
https://issues.apache.org/jira/browse/BEAM-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated BEAM-8557:
--
Fix Version/s: 2.18.0
> Clean up useless null check.
>
>
> Key: BEAM-8557
> URL: https://issues.apache.org/jira/browse/BEAM-8557
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> I think we do not need null check here:
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
> Because before the the `onNext` call, the `Future` already put into the queue
> in `handle` method.
>
> I found the test as follows:
> {code:java}
> @Test
> public void testUnknownResponseIgnored() throws Exception{code}
> I do not know why we need test this case? I think it would be better if we
> throw the Exception for an UnknownResponse, otherwise, this may hidden a
> potential bug.
> Please correct me if there anything I misunderstand @kennknowles
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7948) Add time-based cache threshold support in the Java data service
[ https://issues.apache.org/jira/browse/BEAM-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-7948: -- Fix Version/s: 2.18.0 > Add time-based cache threshold support in the Java data service > --- > > Key: BEAM-7948 > URL: https://issues.apache.org/jira/browse/BEAM-7948 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently only size-based cache threshold is supported in data service. It > should also support the time-based cache threshold. This is very important, > especially for streaming jobs which are sensitive to the delay. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7948) Add time-based cache threshold support in the Java data service
[ https://issues.apache.org/jira/browse/BEAM-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng resolved BEAM-7948. --- Resolution: Fixed > Add time-based cache threshold support in the Java data service > --- > > Key: BEAM-7948 > URL: https://issues.apache.org/jira/browse/BEAM-7948 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently only size-based cache threshold is supported in data service. It > should also support the time-based cache threshold. This is very important, > especially for streaming jobs which are sensitive to the delay. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8733) The "KeyError: u'-47'" error from line 305 of sdk_worker.py
[
https://issues.apache.org/jira/browse/BEAM-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980674#comment-16980674
]
sunjincheng commented on BEAM-8733:
---
Hi [~lcwik], thank you for explaining, then it makes sense to me now. I will
submit a PR to change both the Python SDK harness and Java SDK harness to
synchronous.
Does it make sense to you?
> The "KeyError: u'-47'" error from line 305 of sdk_worker.py
> ---
>
> Key: BEAM-8733
> URL: https://issues.apache.org/jira/browse/BEAM-8733
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
>
> The issue reported by [~chamikara], error message as follows:
> apache_beam/runners/worker/sdk_worker.py", line 305, in get
> self.fns[bundle_descriptor_id],
> KeyError: u'-47'
> {code}
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
> Suppressed: java.lang.IllegalStateException: Already closed.
> at
> org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:93)
> at
> org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
> {code}
> More discussion info can be found here:
> https://github.com/apache/beam/pull/10004
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-7952) Make the input queue of the input buffer in Python SDK Harness size limited.
[ https://issues.apache.org/jira/browse/BEAM-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng reassigned BEAM-7952: - Assignee: sunjincheng > Make the input queue of the input buffer in Python SDK Harness size limited. > > > Key: BEAM-7952 > URL: https://issues.apache.org/jira/browse/BEAM-7952 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > > At Python SDK harness, the input queue size of the input buffer in Python SDK > Harness is not size limited and also not configurable. This may become a > problem if the data production rate is more than the data consumption rate. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8733) The "KeyError: u'-47'" error from line 305 of sdk_worker.py
[
https://issues.apache.org/jira/browse/BEAM-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979765#comment-16979765
]
sunjincheng commented on BEAM-8733:
---
I want to confirm a few things with you before making changes as I'm still not
quite familiar with the Beam. Per my understanding, the registration in the
Java SDK harness is also
asynchronous(https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/BeamFnControlClient.java#L138).
Have I missed something? (I am not arguing, just want to have the correct
understanding) :)
> The "KeyError: u'-47'" error from line 305 of sdk_worker.py
> ---
>
> Key: BEAM-8733
> URL: https://issues.apache.org/jira/browse/BEAM-8733
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
>
> The issue reported by [~chamikara], error message as follows:
> apache_beam/runners/worker/sdk_worker.py", line 305, in get
> self.fns[bundle_descriptor_id],
> KeyError: u'-47'
> {code}
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
> Suppressed: java.lang.IllegalStateException: Already closed.
> at
> org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:93)
> at
> org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
> {code}
> More discussion info can be found here:
> https://github.com/apache/beam/pull/10004
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-8733) The "KeyError: u'-47'" error from line 305 of sdk_worker.py
[
https://issues.apache.org/jira/browse/BEAM-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng reassigned BEAM-8733:
-
Assignee: sunjincheng
> The "KeyError: u'-47'" error from line 305 of sdk_worker.py
> ---
>
> Key: BEAM-8733
> URL: https://issues.apache.org/jira/browse/BEAM-8733
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
>
> The issue reported by [~chamikara], error message as follows:
> apache_beam/runners/worker/sdk_worker.py", line 305, in get
> self.fns[bundle_descriptor_id],
> KeyError: u'-47'
> {code}
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
> Suppressed: java.lang.IllegalStateException: Already closed.
> at
> org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:93)
> at
> org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
> {code}
> More discussion info can be found here:
> https://github.com/apache/beam/pull/10004
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8733) The "KeyError: u'-47'" error from line 305 of sdk_worker.py
[
https://issues.apache.org/jira/browse/BEAM-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976225#comment-16976225
]
sunjincheng commented on BEAM-8733:
---
Hi, Thanks for the log info, [~chamikara].
>From the exception log(the line number of RegisterAndProcessBundleOperation),
>it seems that the
>[commit|https://github.com/apache/beam/commit/686833381ecc92f0fbe04e576a582a7640ca7bbd]
> is not included in the DataFlow runner. Could you help to check this as this
>commit ensures that registration is executed successfully before executing
>process bundle request?
> The "KeyError: u'-47'" error from line 305 of sdk_worker.py
> ---
>
> Key: BEAM-8733
> URL: https://issues.apache.org/jira/browse/BEAM-8733
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
>
> The issue reported by [~chamikara], error message as follows:
> apache_beam/runners/worker/sdk_worker.py", line 305, in get
> self.fns[bundle_descriptor_id],
> KeyError: u'-47'
> {code}
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
> at
> org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
> Suppressed: java.lang.IllegalStateException: Already closed.
> at
> org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:93)
> at
> org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
> {code}
> More discussion info can be found here:
> https://github.com/apache/beam/pull/10004
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8733) The "KeyError: u'-47'" error from line 305 of sdk_worker.py
[
https://issues.apache.org/jira/browse/BEAM-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated BEAM-8733:
--
Description:
The issue reported by [~chamikara], error message as follows:
apache_beam/runners/worker/sdk_worker.py", line 305, in get
self.fns[bundle_descriptor_id],
KeyError: u'-47'
{code}
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
at
org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
at
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
Suppressed: java.lang.IllegalStateException: Already closed.
at
org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:93)
at
org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
{code}
More discussion info can be found here:
https://github.com/apache/beam/pull/10004
was:
The issue reported by [~chamikara], error message as follows:
apache_beam/runners/worker/sdk_worker.py", line 305, in get
self.fns[bundle_descriptor_id],
KeyError: u'-47'
{code}
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
at
org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
at
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
Suppressed: java.lang.IllegalStateException: Already closed.
at
org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:93)
at
org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
{code}
> The "KeyError: u'-47'" error from line 305 of sdk_worker.py
> ---
>
> Key: BEAM-8733
> URL: https://issues.apache.org/jira/browse/BEAM-8733
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
>
> The issue reported by [~chamikara], error message as follows:
> apache_beam/runners/worker/sdk_worker.py", line 305, in get
> self.fns[bundle_descriptor_id],
> KeyError: u'-47'
> {code}
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
> at
> org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
> at
> org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
> at
>
[jira] [Created] (BEAM-8733) The "KeyError: u'-47'" error from line 305 of sdk_worker.py
sunjincheng created BEAM-8733:
-
Summary: The "KeyError: u'-47'" error from line 305 of
sdk_worker.py
Key: BEAM-8733
URL: https://issues.apache.org/jira/browse/BEAM-8733
Project: Beam
Issue Type: Bug
Components: sdk-py-harness
Reporter: sunjincheng
Fix For: 2.18.0
The issue reported by [~chamikara], error message as follows:
apache_beam/runners/worker/sdk_worker.py", line 305, in get
self.fns[bundle_descriptor_id],
KeyError: u'-47'
{code}
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
at
org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:330)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
at
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:411)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:380)
at
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)
at
org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)
Suppressed: java.lang.IllegalStateException: Already closed.
at
org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:93)
at
org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:220)
at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:91)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8620) Tear down unused DoFns periodically in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8620: -- Issue Type: Improvement (was: Task) > Tear down unused DoFns periodically in Java SDK harness > --- > > Key: BEAM-8620 > URL: https://issues.apache.org/jira/browse/BEAM-8620 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > > Per the discussion in the ML the detail can be found here[1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8619) Tear down the DoFns upon the control service termination in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8619: -- Issue Type: Improvement (was: Task) > Tear down the DoFns upon the control service termination in Java SDK harness > > > Key: BEAM-8619 > URL: https://issues.apache.org/jira/browse/BEAM-8619 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Affects Versions: 2.18.0 >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > > Per the discussion in the ML, the detail can be found [1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for teardown the DoFns upon the > control service termination in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8618: -- Issue Type: Improvement (was: Task) > Tear down unused DoFns periodically in Python SDK harness > - > > Key: BEAM-8618 > URL: https://issues.apache.org/jira/browse/BEAM-8618 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > > Per the discussion in the ML, detail can be found [1], the teardown of DoFns > should be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8617) Tear down the DoFns upon the control service termination in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8617: -- Issue Type: Improvement (was: Task) > Tear down the DoFns upon the control service termination in Python SDK harness > -- > > Key: BEAM-8617 > URL: https://issues.apache.org/jira/browse/BEAM-8617 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > > Per the discussion in the ML can be found [1], the teardown of DoFns should > be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support to teardown the DoFns upon the control > service termination for Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8620) Tear down unused DoFns periodically in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8620: -- Issue Type: Task (was: Improvement) > Tear down unused DoFns periodically in Java SDK harness > --- > > Key: BEAM-8620 > URL: https://issues.apache.org/jira/browse/BEAM-8620 > Project: Beam > Issue Type: Task > Components: sdk-java-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > > Per the discussion in the ML the detail can be found here[1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8617) Tear down the DoFns upon the control service termination in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8617: -- Issue Type: Task (was: Improvement) > Tear down the DoFns upon the control service termination in Python SDK harness > -- > > Key: BEAM-8617 > URL: https://issues.apache.org/jira/browse/BEAM-8617 > Project: Beam > Issue Type: Task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > > Per the discussion in the ML can be found [1], the teardown of DoFns should > be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support to teardown the DoFns upon the control > service termination for Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8618: -- Issue Type: Task (was: Improvement) > Tear down unused DoFns periodically in Python SDK harness > - > > Key: BEAM-8618 > URL: https://issues.apache.org/jira/browse/BEAM-8618 > Project: Beam > Issue Type: Task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > > Per the discussion in the ML, detail can be found [1], the teardown of DoFns > should be supported in the portability framework. It happens at two places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for tear down the unused DoFns > periodically in Python SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8619) Tear down the DoFns upon the control service termination in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8619: -- Summary: Tear down the DoFns upon the control service termination in Java SDK harness (was: Teardown the DoFns upon the control service termination in Java SDK harness) > Tear down the DoFns upon the control service termination in Java SDK harness > > > Key: BEAM-8619 > URL: https://issues.apache.org/jira/browse/BEAM-8619 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Affects Versions: 2.18.0 >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > > Per the discussion in the ML, the detail can be found [1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for teardown the DoFns upon the > control service termination in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8619) Tear down the DoFns upon the control service termination in Java SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8619: -- Issue Type: Task (was: Improvement) > Tear down the DoFns upon the control service termination in Java SDK harness > > > Key: BEAM-8619 > URL: https://issues.apache.org/jira/browse/BEAM-8619 > Project: Beam > Issue Type: Task > Components: sdk-java-harness >Affects Versions: 2.18.0 >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > > Per the discussion in the ML, the detail can be found [1], the teardown of > DoFns should be supported in the portability framework. It happens at two > places: > 1) Upon the control service termination > 2) Tear down the unused DoFns periodically > The aim of this JIRA is to add support for teardown the DoFns upon the > control service termination in Java SDK harness. > [1] > https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8620) Tear down unused DoFns periodically in Java SDK harness
sunjincheng created BEAM-8620: - Summary: Tear down unused DoFns periodically in Java SDK harness Key: BEAM-8620 URL: https://issues.apache.org/jira/browse/BEAM-8620 Project: Beam Issue Type: Improvement Components: sdk-java-harness Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.18.0 Per the discussion in the ML the detail can be found here[1], the teardown of DoFns should be supported in the portability framework. It happens at two places: 1) Upon the control service termination 2) Tear down the unused DoFns periodically The aim of this JIRA is to add support for tear down the unused DoFns periodically in Java SDK harness. [1] https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8619) Teardown the DoFns upon the control service termination in Java SDK harness
sunjincheng created BEAM-8619: - Summary: Teardown the DoFns upon the control service termination in Java SDK harness Key: BEAM-8619 URL: https://issues.apache.org/jira/browse/BEAM-8619 Project: Beam Issue Type: Improvement Components: sdk-java-harness Affects Versions: 2.18.0 Reporter: sunjincheng Assignee: sunjincheng Per the discussion in the ML, the detail can be found [1], the teardown of DoFns should be supported in the portability framework. It happens at two places: 1) Upon the control service termination 2) Tear down the unused DoFns periodically The aim of this JIRA is to add support for teardown the DoFns upon the control service termination in Java SDK harness. [1] https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness
sunjincheng created BEAM-8618: - Summary: Tear down unused DoFns periodically in Python SDK harness Key: BEAM-8618 URL: https://issues.apache.org/jira/browse/BEAM-8618 Project: Beam Issue Type: Improvement Components: sdk-py-harness Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.18.0 Per the discussion in the ML, detail can be found [1], the teardown of DoFns should be supported in the portability framework. It happens at two places: 1) Upon the control service termination 2) Tear down the unused DoFns periodically The aim of this JIRA is to add support for tear down the unused DoFns periodically in Python SDK harness. [1] https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8617) Tear down the DoFns upon the control service termination in Python SDK harness
sunjincheng created BEAM-8617: - Summary: Tear down the DoFns upon the control service termination in Python SDK harness Key: BEAM-8617 URL: https://issues.apache.org/jira/browse/BEAM-8617 Project: Beam Issue Type: Improvement Components: sdk-py-harness Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.18.0 Per the discussion in the ML can be found [1], the teardown of DoFns should be supported in the portability framework. It happens at two places: 1) Upon the control service termination 2) Tear down the unused DoFns periodically The aim of this JIRA is to add support to teardown the DoFns upon the control service termination for Python SDK harness. [1] https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8594) Remove unnecessary error check of the control service accessing in DataFlow Runner
sunjincheng created BEAM-8594: - Summary: Remove unnecessary error check of the control service accessing in DataFlow Runner Key: BEAM-8594 URL: https://issues.apache.org/jira/browse/BEAM-8594 Project: Beam Issue Type: Improvement Components: runner-dataflow Reporter: sunjincheng Assignee: sunjincheng Fix For: 2.18.0 Currently there are a few places in the DataFlow Runner which checks if there is error reported when accessing the SDK harness's control service. Actually, the error reported by the SDK harness has already been handled in the [FnApiControlClient|https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L152]. There is no need to check it anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8565) Update .test-infra/jenkins/README with missing entries and correct wrong entries
[
https://issues.apache.org/jira/browse/BEAM-8565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated BEAM-8565:
--
Description:
Currently the following tests are missing in the .test-infra/jenkins/README:
{code:java}
beam_PreCommit_BeamSQL_ZetaSQL
beam_PostCommit_CrossLanguageValidatesRunner
beam_PostCommit_Java11_Dataflow_Examples
beam_PostCommit_Java11_Dataflow_Portability_Examples
beam_PostCommit_PortableJar_Flink
beam_PostCommit_Python_MongoDBIO_IT
beam_PostCommit_Website_Test
beam_PerformanceTests_KafkaIOIT
beam_PerformanceTests_MongoDBIOIT
beam_LoadTests_Python_ParDo_Flink_Batch{code}
The following tests are duplicate:
{code:java}
beam_PreCommit_Go
beam_PreCommit_JavaPortabilityApi{code}
The trigger command for the following items are wrong:
{code:java}
beam_PreCommit_Java_Examples_Dataflow
beam_PreCommit_Portable_Python
beam_PreCommit_Python2_PVR_Flink
beam_PerformanceTests_TFRecordIOIT
{code}
This JIRA will address these issues.
was:
Currently the following tests are missing in the .test-infra/jenkins/README:
beam_PreCommit_BeamSQL_ZetaSQL
beam_PostCommit_CrossLanguageValidatesRunner
beam_PostCommit_Java11_Dataflow_Examples
beam_PostCommit_Java11_Dataflow_Portability_Examples
beam_PostCommit_PortableJar_Flink
beam_PostCommit_Python_MongoDBIO_IT
beam_PostCommit_Website_Test
beam_PerformanceTests_KafkaIOIT
beam_PerformanceTests_MongoDBIOIT
beam_LoadTests_Python_ParDo_Flink_Batch
The following tests are duplicate:
beam_PreCommit_Go
beam_PreCommit_JavaPortabilityApi
The trigger command for the following items are wrong:
beam_PreCommit_Java_Examples_Dataflow
beam_PreCommit_Portable_Python
beam_PreCommit_Python2_PVR_Flink
beam_PerformanceTests_TFRecordIOIT
This JIRA will address these issues.
> Update .test-infra/jenkins/README with missing entries and correct wrong
> entries
>
>
> Key: BEAM-8565
> URL: https://issues.apache.org/jira/browse/BEAM-8565
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.18.0
>
>
> Currently the following tests are missing in the .test-infra/jenkins/README:
> {code:java}
> beam_PreCommit_BeamSQL_ZetaSQL
> beam_PostCommit_CrossLanguageValidatesRunner
> beam_PostCommit_Java11_Dataflow_Examples
> beam_PostCommit_Java11_Dataflow_Portability_Examples
> beam_PostCommit_PortableJar_Flink
> beam_PostCommit_Python_MongoDBIO_IT
> beam_PostCommit_Website_Test
> beam_PerformanceTests_KafkaIOIT
> beam_PerformanceTests_MongoDBIOIT
> beam_LoadTests_Python_ParDo_Flink_Batch{code}
>
> The following tests are duplicate:
> {code:java}
> beam_PreCommit_Go
> beam_PreCommit_JavaPortabilityApi{code}
>
> The trigger command for the following items are wrong:
> {code:java}
> beam_PreCommit_Java_Examples_Dataflow
> beam_PreCommit_Portable_Python
> beam_PreCommit_Python2_PVR_Flink
> beam_PerformanceTests_TFRecordIOIT
> {code}
> This JIRA will address these issues.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8565) Update .test-infra/jenkins/README with missing entries and correct wrong entries
[ https://issues.apache.org/jira/browse/BEAM-8565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8565: -- Fix Version/s: 2.18.0 > Update .test-infra/jenkins/README with missing entries and correct wrong > entries > > > Key: BEAM-8565 > URL: https://issues.apache.org/jira/browse/BEAM-8565 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.18.0 > > > Currently the following tests are missing in the .test-infra/jenkins/README: > beam_PreCommit_BeamSQL_ZetaSQL > beam_PostCommit_CrossLanguageValidatesRunner > beam_PostCommit_Java11_Dataflow_Examples > beam_PostCommit_Java11_Dataflow_Portability_Examples > beam_PostCommit_PortableJar_Flink > beam_PostCommit_Python_MongoDBIO_IT > beam_PostCommit_Website_Test > beam_PerformanceTests_KafkaIOIT > beam_PerformanceTests_MongoDBIOIT > beam_LoadTests_Python_ParDo_Flink_Batch > The following tests are duplicate: > beam_PreCommit_Go > beam_PreCommit_JavaPortabilityApi > The trigger command for the following items are wrong: > beam_PreCommit_Java_Examples_Dataflow > beam_PreCommit_Portable_Python > beam_PreCommit_Python2_PVR_Flink > beam_PerformanceTests_TFRecordIOIT > This JIRA will address these issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8565) Update .test-infra/jenkins/README with missing entries and correct wrong entries
sunjincheng created BEAM-8565: - Summary: Update .test-infra/jenkins/README with missing entries and correct wrong entries Key: BEAM-8565 URL: https://issues.apache.org/jira/browse/BEAM-8565 Project: Beam Issue Type: Bug Components: testing Reporter: sunjincheng Assignee: sunjincheng Currently the following tests are missing in the .test-infra/jenkins/README: beam_PreCommit_BeamSQL_ZetaSQL beam_PostCommit_CrossLanguageValidatesRunner beam_PostCommit_Java11_Dataflow_Examples beam_PostCommit_Java11_Dataflow_Portability_Examples beam_PostCommit_PortableJar_Flink beam_PostCommit_Python_MongoDBIO_IT beam_PostCommit_Website_Test beam_PerformanceTests_KafkaIOIT beam_PerformanceTests_MongoDBIOIT beam_LoadTests_Python_ParDo_Flink_Batch The following tests are duplicate: beam_PreCommit_Go beam_PreCommit_JavaPortabilityApi The trigger command for the following items are wrong: beam_PreCommit_Java_Examples_Dataflow beam_PreCommit_Portable_Python beam_PreCommit_Python2_PVR_Flink beam_PerformanceTests_TFRecordIOIT This JIRA will address these issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8557) Clean up useless null check.
[
https://issues.apache.org/jira/browse/BEAM-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated BEAM-8557:
--
Description:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
{code:java}
@Test
public void testUnknownResponseIgnored() throws Exception{code}
I do not know why we need test this case? I think it would be better if we
throw the Exception for an UnknownResponse, otherwise, this may hidden a
potential bug.
Please correct me if there anything I misunderstand @kennknowles
was:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
{code:java}
@Test
public void testUnknownResponseIgnored() throws Exception{code}
I do not know why we need test this case? I think it would be better if we
throw the Exception for an UnknownResponse, otherwise, this may hidden a
potential bug.
What do you think? @kennknowles
> Clean up useless null check.
>
>
> Key: BEAM-8557
> URL: https://issues.apache.org/jira/browse/BEAM-8557
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think we do not need null check here:
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
> Because before the the `onNext` call, the `Future` already put into the queue
> in `handle` method.
>
> I found the test as follows:
> {code:java}
> @Test
> public void testUnknownResponseIgnored() throws Exception{code}
> I do not know why we need test this case? I think it would be better if we
> throw the Exception for an UnknownResponse, otherwise, this may hidden a
> potential bug.
> Please correct me if there anything I misunderstand @kennknowles
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8557) Clean up useless null check.
[
https://issues.apache.org/jira/browse/BEAM-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated BEAM-8557:
--
Description:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
{code:java}
@Test
public void testUnknownResponseIgnored() throws Exception{code}
I do not know why we need test this case? I think it would be better if we
throw the Exception for an UnknownResponse, otherwise, this may hidden a
potential bug.
What do you think? @kennknowles
was:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
{code:java}
@Test
public void testUnknownResponseIgnored() throws Exception{code}
I am do not know why we need test this case? I think it would be better if we
throw the Exception for an UnknownResponse, otherwise, this may hidden a
potential bug.
What do you think? @kennknowles
> Clean up useless null check.
>
>
> Key: BEAM-8557
> URL: https://issues.apache.org/jira/browse/BEAM-8557
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think we do not need null check here:
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
> Because before the the `onNext` call, the `Future` already put into the queue
> in `handle` method.
>
> I found the test as follows:
> {code:java}
> @Test
> public void testUnknownResponseIgnored() throws Exception{code}
> I do not know why we need test this case? I think it would be better if we
> throw the Exception for an UnknownResponse, otherwise, this may hidden a
> potential bug.
> What do you think? @kennknowles
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8557) Clean up useless null check.
[
https://issues.apache.org/jira/browse/BEAM-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated BEAM-8557:
--
Description:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
{code:java}
@Test
public void testUnknownResponseIgnored() throws Exception{code}
I am do not know why we need test this case? I think it would be better if we
throw the Exception for an UnknownResponse, otherwise, this may hidden a
potential bug.
What do you think? @kennknowles
was:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
{code:java}
@Test
public void testUnknownResponseIgnored() throws Exception{code}
I am do not know why we need test this case? @kennknowles
What do you think?
> Clean up useless null check.
>
>
> Key: BEAM-8557
> URL: https://issues.apache.org/jira/browse/BEAM-8557
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think we do not need null check here:
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
> Because before the the `onNext` call, the `Future` already put into the queue
> in `handle` method.
>
> I found the test as follows:
> {code:java}
> @Test
> public void testUnknownResponseIgnored() throws Exception{code}
> I am do not know why we need test this case? I think it would be better if we
> throw the Exception for an UnknownResponse, otherwise, this may hidden a
> potential bug.
> What do you think? @kennknowles
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8557) Clean up useless null check.
[
https://issues.apache.org/jira/browse/BEAM-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated BEAM-8557:
--
Description:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
{code:java}
@Test
public void testUnknownResponseIgnored() throws Exception{code}
I am do not know why we need test this case? @kennknowles
What do you think?
was:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
```
@Test
public void testUnknownResponseIgnored() throws Exception {
String id = "actualInstruction";
String unknownId = "unknownInstruction";
CompletionStage responseFuture =
client.handle(BeamFnApi.InstructionRequest.newBuilder().setInstructionId(id).build());
client
.asResponseObserver()
.onNext(BeamFnApi.InstructionResponse.newBuilder().setInstructionId(unknownId).build());
assertThat(MoreFutures.isDone(responseFuture), is(false));
assertThat(MoreFutures.isCancelled(responseFuture), is(false));
}
```
I am do not know why we need test this case? @kennknowles
What do you think?
> Clean up useless null check.
>
>
> Key: BEAM-8557
> URL: https://issues.apache.org/jira/browse/BEAM-8557
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think we do not need null check here:
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
> Because before the the `onNext` call, the `Future` already put into the queue
> in `handle` method.
>
> I found the test as follows:
> {code:java}
> @Test
> public void testUnknownResponseIgnored() throws Exception{code}
> I am do not know why we need test this case? @kennknowles
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8557) Clean up useless null check.
[
https://issues.apache.org/jira/browse/BEAM-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sunjincheng updated BEAM-8557:
--
Description:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
I found the test as follows:
```
@Test
public void testUnknownResponseIgnored() throws Exception {
String id = "actualInstruction";
String unknownId = "unknownInstruction";
CompletionStage responseFuture =
client.handle(BeamFnApi.InstructionRequest.newBuilder().setInstructionId(id).build());
client
.asResponseObserver()
.onNext(BeamFnApi.InstructionResponse.newBuilder().setInstructionId(unknownId).build());
assertThat(MoreFutures.isDone(responseFuture), is(false));
assertThat(MoreFutures.isCancelled(responseFuture), is(false));
}
```
I am do not know why we need test this case? @kennknowles
What do you think?
was:
I think we do not need null check here:
[https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
Because before the the `onNext` call, the `Future` already put into the queue
in `handle` method.
What do you think?
> Clean up useless null check.
>
>
> Key: BEAM-8557
> URL: https://issues.apache.org/jira/browse/BEAM-8557
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think we do not need null check here:
> [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151]
> Because before the the `onNext` call, the `Future` already put into the queue
> in `handle` method.
>
> I found the test as follows:
> ```
> @Test
> public void testUnknownResponseIgnored() throws Exception {
> String id = "actualInstruction";
> String unknownId = "unknownInstruction";
> CompletionStage responseFuture =
>
> client.handle(BeamFnApi.InstructionRequest.newBuilder().setInstructionId(id).build());
> client
> .asResponseObserver()
>
> .onNext(BeamFnApi.InstructionResponse.newBuilder().setInstructionId(unknownId).build());
> assertThat(MoreFutures.isDone(responseFuture), is(false));
> assertThat(MoreFutures.isCancelled(responseFuture), is(false));
> }
> ```
> I am do not know why we need test this case? @kennknowles
> What do you think?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8442) Unify bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8442: -- Summary: Unify bundle register in Python SDK harness (was: Unfiy bundle register in Python SDK harness) > Unify bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8557) Clean up useless null check.
[ https://issues.apache.org/jira/browse/BEAM-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8557: -- Parent: BEAM-7944 Issue Type: Sub-task (was: Improvement) > Clean up useless null check. > > > Key: BEAM-8557 > URL: https://issues.apache.org/jira/browse/BEAM-8557 > Project: Beam > Issue Type: Sub-task > Components: runner-core, sdk-java-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > > I think we do not need null check here: > [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151] > Because before the the `onNext` call, the `Future` already put into the queue > in `handle` method. > What do you think? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8557) Clean up useless null check.
sunjincheng created BEAM-8557: - Summary: Clean up useless null check. Key: BEAM-8557 URL: https://issues.apache.org/jira/browse/BEAM-8557 Project: Beam Issue Type: Improvement Components: runner-core, sdk-java-harness Reporter: sunjincheng Assignee: sunjincheng I think we do not need null check here: [https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java#L151] Because before the the `onNext` call, the `Future` already put into the queue in `handle` method. What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959462#comment-16959462 ] sunjincheng edited comment on BEAM-7951 at 10/25/19 6:19 AM: - This is not release blocker, thanks for reset the Fix Version. [~kenn] was (Author: sunjincheng121): This is not release blocker, thanks for reset the Fix Version. > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959462#comment-16959462 ] sunjincheng commented on BEAM-7951: --- This is not release blocker, thanks for reset the Fix Version. > Allow runner to configure customization WindowedValue coder such as > ValueOnlyWindowedValueCoder > --- > > Key: BEAM-7951 > URL: https://issues.apache.org/jira/browse/BEAM-7951 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > > The coder of WindowedValue cannot be configured and it’s always > FullWindowedValueCoder. We don't need to serialize the timestamp, window and > pane properties in Flink and so it will be better to make the coder > configurable (i.e. allowing to use ValueOnlyWindowedValueCoder) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7950) Remove the Python 3 warning as it has already been supported
[ https://issues.apache.org/jira/browse/BEAM-7950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959460#comment-16959460 ] sunjincheng commented on BEAM-7950: --- Thanks [~altay] and [~tvalentyn] :) > Remove the Python 3 warning as it has already been supported > > > Key: BEAM-7950 > URL: https://issues.apache.org/jira/browse/BEAM-7950 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.16.0 > > > There are warnings that Python 3 is not fully supported in Beam > (beam/sdks/python/setup.py). As mentioned in the ML, we should remove the > Python 3 warning as it has already been supported as an effort of > https://issues.apache.org/jira/browse/BEAM-1251. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8442) Unfiy bundle register in Python SDK harness
[ https://issues.apache.org/jira/browse/BEAM-8442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng updated BEAM-8442: -- Parent: BEAM-7944 Issue Type: Sub-task (was: Improvement) > Unfiy bundle register in Python SDK harness > --- > > Key: BEAM-8442 > URL: https://issues.apache.org/jira/browse/BEAM-8442 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.17.0 > > Time Spent: 10m > Remaining Estimate: 0h > > There are two methods for bundle register in Python SDK harness: > `SdkHarness._request_register` and `SdkWorker.register.` It should be unfied. -- This message was sent by Atlassian Jira (v8.3.4#803005)
