[jira] [Commented] (BEAM-9999) Remove support for EOLed runners (Apex, etc.)
[ https://issues.apache.org/jira/browse/BEAM-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107792#comment-17107792 ] Thomas Weise commented on BEAM-: Apache Apex itself has moved to attic and there are no users of the Beam Apex runners that I know of. > Remove support for EOLed runners (Apex, etc.) > - > > Key: BEAM- > URL: https://issues.apache.org/jira/browse/BEAM- > Project: Beam > Issue Type: Bug > Components: runner-apex, runner-core >Reporter: Ahmet Altay >Priority: Major > > These runners look EOLed, not maintained: > - Apex (last release 2+ years ago) > - Gearpump (last release 1+ year ago) > Removing support for these could reduce the code base size, reduce flaky > test, and make it easier to add new features. > /cc [~kenn][~tysonjh] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[ https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092348#comment-17092348 ] Thomas Weise commented on BEAM-9811: Last build was successful: [https://builds.apache.org/job/beam_Release_NightlySnapshot/795/] Closing ticket. > beam_Release_NightlySnapshot failing due to "Failed to publish publication > 'mavenJava'" > --- > > Key: BEAM-9811 > URL: https://issues.apache.org/jira/browse/BEAM-9811 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Thomas Weise >Priority: Major > Attachments: Screen Shot 2020-04-23 at 13.57.32.png > > Time Spent: 50m > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_Release_NightlySnapshot/] > > For example, > [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/] > [https://scans.gradle.com/s/ryvtuscii4l5u] > The > :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository] > 1st of 2 > > Failed to publish publication 'mavenJava' to repository > 'apache.snapshots.https' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory)[View > exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0] > The > :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository] > Failed to publish publication 'mavenJava' to repository > 'testPublicationLocal' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > Tomo, is this related to a recent dependency upgrade ? > > cc: [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[ https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-9811. Fix Version/s: Not applicable Resolution: Fixed > beam_Release_NightlySnapshot failing due to "Failed to publish publication > 'mavenJava'" > --- > > Key: BEAM-9811 > URL: https://issues.apache.org/jira/browse/BEAM-9811 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Thomas Weise >Priority: Major > Fix For: Not applicable > > Attachments: Screen Shot 2020-04-23 at 13.57.32.png > > Time Spent: 50m > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_Release_NightlySnapshot/] > > For example, > [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/] > [https://scans.gradle.com/s/ryvtuscii4l5u] > The > :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository] > 1st of 2 > > Failed to publish publication 'mavenJava' to repository > 'apache.snapshots.https' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory)[View > exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0] > The > :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository] > Failed to publish publication 'mavenJava' to repository > 'testPublicationLocal' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > Tomo, is this related to a recent dependency upgrade ? > > cc: [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[ https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-9811. -- > beam_Release_NightlySnapshot failing due to "Failed to publish publication > 'mavenJava'" > --- > > Key: BEAM-9811 > URL: https://issues.apache.org/jira/browse/BEAM-9811 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Thomas Weise >Priority: Major > Fix For: Not applicable > > Attachments: Screen Shot 2020-04-23 at 13.57.32.png > > Time Spent: 50m > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_Release_NightlySnapshot/] > > For example, > [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/] > [https://scans.gradle.com/s/ryvtuscii4l5u] > The > :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository] > 1st of 2 > > Failed to publish publication 'mavenJava' to repository > 'apache.snapshots.https' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory)[View > exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0] > The > :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository] > Failed to publish publication 'mavenJava' to repository > 'testPublicationLocal' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > Tomo, is this related to a recent dependency upgrade ? > > cc: [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[ https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091792#comment-17091792 ] Thomas Weise commented on BEAM-9811: Due to the dynamic construction of publish tasks with target repository names, the job now failed when publishing to the remote repo. Changed this to a less brittle setup: [https://github.com/apache/beam/pull/11520] > beam_Release_NightlySnapshot failing due to "Failed to publish publication > 'mavenJava'" > --- > > Key: BEAM-9811 > URL: https://issues.apache.org/jira/browse/BEAM-9811 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Thomas Weise >Priority: Major > Attachments: Screen Shot 2020-04-23 at 13.57.32.png > > Time Spent: 50m > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_Release_NightlySnapshot/] > > For example, > [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/] > [https://scans.gradle.com/s/ryvtuscii4l5u] > The > :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository] > 1st of 2 > > Failed to publish publication 'mavenJava' to repository > 'apache.snapshots.https' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory)[View > exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0] > The > :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository] > Failed to publish publication 'mavenJava' to repository > 'testPublicationLocal' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > Tomo, is this related to a recent dependency upgrade ? > > cc: [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[ https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091605#comment-17091605 ] Thomas Weise edited comment on BEAM-9811 at 4/24/20, 2:07 PM: -- Build still fails with this change: [https://builds.apache.org/job/beam_Release_NightlySnapshot/794/] I will take another look. was (Author: thw): Build still fails with this change, I will take another look. > beam_Release_NightlySnapshot failing due to "Failed to publish publication > 'mavenJava'" > --- > > Key: BEAM-9811 > URL: https://issues.apache.org/jira/browse/BEAM-9811 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Thomas Weise >Priority: Major > Attachments: Screen Shot 2020-04-23 at 13.57.32.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_Release_NightlySnapshot/] > > For example, > [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/] > [https://scans.gradle.com/s/ryvtuscii4l5u] > The > :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository] > 1st of 2 > > Failed to publish publication 'mavenJava' to repository > 'apache.snapshots.https' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory)[View > exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0] > The > :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository] > Failed to publish publication 'mavenJava' to repository > 'testPublicationLocal' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > Tomo, is this related to a recent dependency upgrade ? > > cc: [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[ https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091605#comment-17091605 ] Thomas Weise commented on BEAM-9811: Build still fails with this change, I will take another look. > beam_Release_NightlySnapshot failing due to "Failed to publish publication > 'mavenJava'" > --- > > Key: BEAM-9811 > URL: https://issues.apache.org/jira/browse/BEAM-9811 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Thomas Weise >Priority: Major > Attachments: Screen Shot 2020-04-23 at 13.57.32.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_Release_NightlySnapshot/] > > For example, > [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/] > [https://scans.gradle.com/s/ryvtuscii4l5u] > The > :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository] > 1st of 2 > > Failed to publish publication 'mavenJava' to repository > 'apache.snapshots.https' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory)[View > exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0] > The > :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository] > Failed to publish publication 'mavenJava' to repository > 'testPublicationLocal' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > Tomo, is this related to a recent dependency upgrade ? > > cc: [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[ https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090948#comment-17090948 ] Thomas Weise edited comment on BEAM-9811 at 4/23/20, 9:36 PM: -- ./gradlew :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository The task fails w/ and w/o that commit (7438265f53b85a774b32a7c65a0901dbd0636a50) It even fails with the commit or the last successful Jenkins run: [https://builds.apache.org/job/beam_Release_NightlySnapshot/785/] (d279f4bd7e00584e65cf04f8d9f6ce7d94f44004) Can someone else please try to run it locally? was (Author: thw): ./gradlew :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository The task fails w/ and w/o that commit (7438265f53b85a774b32a7c65a0901dbd0636a50) It even fails with the commit or the last successful Jenkins run: [https://builds.apache.org/job/beam_Release_NightlySnapshot/785/] Can someone else please try to run it locally? > beam_Release_NightlySnapshot failing due to "Failed to publish publication > 'mavenJava'" > --- > > Key: BEAM-9811 > URL: https://issues.apache.org/jira/browse/BEAM-9811 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Thomas Weise >Priority: Major > Attachments: Screen Shot 2020-04-23 at 13.57.32.png > > > [https://builds.apache.org/job/beam_Release_NightlySnapshot/] > > For example, > [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/] > [https://scans.gradle.com/s/ryvtuscii4l5u] > The > :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository] > 1st of 2 > > Failed to publish publication 'mavenJava' to repository > 'apache.snapshots.https' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory)[View > exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0] > The > :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository] > Failed to publish publication 'mavenJava' to repository > 'testPublicationLocal' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > Tomo, is this related to a recent dependency upgrade ? > > cc: [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[ https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090948#comment-17090948 ] Thomas Weise commented on BEAM-9811: ./gradlew :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository The task fails w/ and w/o that commit (7438265f53b85a774b32a7c65a0901dbd0636a50) It even fails with the commit or the last successful Jenkins run: [https://builds.apache.org/job/beam_Release_NightlySnapshot/785/] Can someone else please try to run it locally? > beam_Release_NightlySnapshot failing due to "Failed to publish publication > 'mavenJava'" > --- > > Key: BEAM-9811 > URL: https://issues.apache.org/jira/browse/BEAM-9811 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Thomas Weise >Priority: Major > Attachments: Screen Shot 2020-04-23 at 13.57.32.png > > > [https://builds.apache.org/job/beam_Release_NightlySnapshot/] > > For example, > [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/] > [https://scans.gradle.com/s/ryvtuscii4l5u] > The > :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository] > 1st of 2 > > Failed to publish publication 'mavenJava' to repository > 'apache.snapshots.https' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory)[View > exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0] > The > :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository > task failed.[View task in console > log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository] > Failed to publish publication 'mavenJava' to repository > 'testPublicationLocal' > java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml > (No such file or directory) > > Tomo, is this related to a recent dependency upgrade ? > > cc: [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9811) beam_Release_NightlySnapshot failing due to "Failed to publish publication 'mavenJava'"
[
https://issues.apache.org/jira/browse/BEAM-9811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090934#comment-17090934
]
Thomas Weise commented on BEAM-9811:
First failing build:
[https://builds.apache.org/job/beam_Release_NightlySnapshot/786/]
contains the change from: [https://github.com/apache/beam/pull/11399]
I'm able to reproduce the issue locally:
{code:java}
$ ./gradlew
:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository
Starting a Gradle Daemon (subsequent builds will be faster)
Configuration on demand is an incubating feature.
> Task
> :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository
> FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository'.
> Failed to publish publication 'mavenJava' to repository 'testPublicationLocal'
> java.io.FileNotFoundException:
/Users/tweise/src/beam/sdks/java/bom/build/publications/mavenJava/pom-default.xml
(No such file or directory)
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
{code}
> beam_Release_NightlySnapshot failing due to "Failed to publish publication
> 'mavenJava'"
> ---
>
> Key: BEAM-9811
> URL: https://issues.apache.org/jira/browse/BEAM-9811
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Chamikara Madhusanka Jayalath
>Priority: Major
> Attachments: Screen Shot 2020-04-23 at 13.57.32.png
>
>
> [https://builds.apache.org/job/beam_Release_NightlySnapshot/]
>
> For example,
> [https://builds.apache.org/job/beam_Release_NightlySnapshot/793/]
> [https://scans.gradle.com/s/ryvtuscii4l5u]
> The
> :sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository
> task failed.[View task in console
> log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToApache.snapshots.httpsRepository]
> 1st of 2
>
> Failed to publish publication 'mavenJava' to repository
> 'apache.snapshots.https' > java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml
> (No such file or directory) >
> /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml
> (No such file or directory)[View
> exception|https://scans.gradle.com/s/ryvtuscii4l5u/failure#top=0]
> The
> :sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository
> task failed.[View task in console
> log|https://scans.gradle.com/s/ryvtuscii4l5u/console-log?task=:sdks:java:bom:publishMavenJavaPublicationToTestPublicationLocalRepository]
> Failed to publish publication 'mavenJava' to repository
> 'testPublicationLocal' > java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml
> (No such file or directory) >
> /home/jenkins/jenkins-slave/workspace/beam_Release_NightlySnapshot/src/sdks/java/bom/build/publications/mavenJava/pom-default.xml
> (No such file or directory)
>
> Tomo, is this related to a recent dependency upgrade ?
>
> cc: [~lcwik]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9474) Environment cleanup is not robust enough and may leak resources
[
https://issues.apache.org/jira/browse/BEAM-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise resolved BEAM-9474.
Fix Version/s: 2.21.0
Resolution: Fixed
> Environment cleanup is not robust enough and may leak resources
> ---
>
> Key: BEAM-9474
> URL: https://issues.apache.org/jira/browse/BEAM-9474
> Project: Beam
> Issue Type: Bug
> Components: java-fn-execution
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.21.0
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> The cleanup code in {{DefaultJobBundleFactory}} and its {{RemoteEnvironment}}
> s may leak resources. This is especially a concern when the execution engines
> reuses the same JVM or underlying machines for multiple runs of a pipeline.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9298) Drop support for Flink 1.7
[ https://issues.apache.org/jira/browse/BEAM-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034809#comment-17034809 ] Thomas Weise commented on BEAM-9298: [~iemejia] yes, this should be on the mailing list. IMO good to communicate intent to dev@ and user@ and also refer to [https://beam.apache.org/documentation/runners/flink/#version-compatibility] > Drop support for Flink 1.7 > --- > > Key: BEAM-9298 > URL: https://issues.apache.org/jira/browse/BEAM-9298 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > With Flink 1.10 around the corner, more detail can be found in BEAM-9295, we > should consider dropping support for Flink 1.7. Then dropping 1.7 will also > decrease the build time. > What do you think? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9060) Flink suppresses stdout/stderr during JobGraph generation from JAR
[
https://issues.apache.org/jira/browse/BEAM-9060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise resolved BEAM-9060.
Fix Version/s: 2.19.0
Resolution: Fixed
> Flink suppresses stdout/stderr during JobGraph generation from JAR
> --
>
> Key: BEAM-9060
> URL: https://issues.apache.org/jira/browse/BEAM-9060
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Flink uses the {{OptimizedPlanEnvironment}} which replaces stdout/stderr
> during job graph creation. This was intended only for previewing the plan,
> but other parts of Flink, e.g. the Rest API have started to use this code as
> well.
> We can work around FLINK-15504 by restoring the original stdout/stderr when
> we detect the {{OptimizedPlanEnvironment}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8962) FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction
[
https://issues.apache.org/jira/browse/BEAM-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise resolved BEAM-8962.
Resolution: Fixed
> FlinkMetricContainer causes churn in the JobManager and lets the web frontend
> malfunction
> -
>
> Key: BEAM-8962
> URL: https://issues.apache.org/jira/browse/BEAM-8962
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The {{FlinkMetricContainer}} wraps the Beam metric container for reporting
> metrics, but also stores them as Flink accumulators. With high parallelism
> jobs with over a thousand tasks and many built-in Beam metrics for every Beam
> step, this can accumulate to over 100MB of serialized data which is stored in
> the JobManager's ExecutionGraph. This then fails to even sent over the wire,
> due to the akka.framesize limit (10MB by default), and manifests in {{500
> Internal Server Error}}s in the web frontend.
> We need to introduce an option to disable the reporting via accumulators. It
> is mostly useful for batch workloads where you can retrieve the final
> accumulator values at the end of the job. It adds a lot of memory and network
> overhead.
> Perhaps we could even turn off the accumulators for streaming jobs, or
> entirely and make them opt-in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8816) Load balance bundle processing w/ multiple SDK workers
[ https://issues.apache.org/jira/browse/BEAM-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-8816. Fix Version/s: 2.19.0 Resolution: Fixed > Load balance bundle processing w/ multiple SDK workers > -- > > Key: BEAM-8816 > URL: https://issues.apache.org/jira/browse/BEAM-8816 > Project: Beam > Issue Type: Improvement > Components: runner-core, runner-flink >Affects Versions: 2.17.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability > Fix For: 2.19.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > We found skewed utilization of SDK workers causing excessive latency with > Streaming/Python/Flink. (Remember that with Python, we need to execute > multiple worker processes on a machine instead of relying on threads in a > single worker, which requires the runner to make a decision to which worker > to give a bundle for processing.) > The Flink runner has knobs to influence the number of records per bundle and > the maximum duration for a bundle. But since the runner does not understand > the cost of individual records, it is possible for the duration of bundles to > fluctuate significantly due to skew in processing time of individual records. > And unless the bundle size is 1, multiple expensive records could be > allocated to a single bundle before the cutoff time is reached. We notice > this with a pipeline that executes models, but there are other use cases > where the cost of individual records can vary significantly. > Additionally, the Flink runner establishes the association between the > subtask managing an executable stage and the SDK worker during > initialization, lasting for the duration of the job. In other words, bundles > for the same executable stage will always be sent to the same SDK worker. > When the execution time skew is tied to specific keys (stateful processing), > it further aggravates the issue. > [https://lists.apache.org/thread.html/59c02d8b8ea849c158deb39ad9d83af4d8fcb56570501c7fe8f79bb2@%3Cdev.beam.apache.org%3E] > Long term this problem can be addressed with SDF. Till then, an (optional) > runner controlled balancing mechanism has shown to improve the performance in > internal testing. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8959) Boolean pipeline options which default to true cannot be set to false
[
https://issues.apache.org/jira/browse/BEAM-8959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16994909#comment-16994909
]
Thomas Weise commented on BEAM-8959:
[https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse]
Maybe the option should just be --disable_metrics if the default is true. I
would find that more intuitive.
> Boolean pipeline options which default to true cannot be set to false
> -
>
> Key: BEAM-8959
> URL: https://issues.apache.org/jira/browse/BEAM-8959
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Maximilian Michels
>Priority: Critical
>
> With the included argument parser, any boolean pipeline options which default
> to true cannot be set to false, e.g. {{--enable_metrics=false}}:
> {noformat}
> error: argument --enable_metrics: ignored explicit argument 'false'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8815) Portable pipeline execution without artifact staging
[ https://issues.apache.org/jira/browse/BEAM-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-8815. Resolution: Fixed > Portable pipeline execution without artifact staging > > > Key: BEAM-8815 > URL: https://issues.apache.org/jira/browse/BEAM-8815 > Project: Beam > Issue Type: Task > Components: runner-core, runner-flink >Affects Versions: 2.17.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability > Fix For: 2.17.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > The default artifact staging implementation relies on a distributed > filesystem. A directory and manifest will be created even when artifact > staging isn't used, and the container boot code will fail retrieving > artifacts, even though there are non. In a containerized environment it is > common to package artifacts into containers. It should be possible to run the > pipeline w/o a distributed filesystem. > [https://lists.apache.org/thread.html/1b0d545955a80688ea19f227ad943683747b17beb45368ad0908fd21@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8815) Portable pipeline execution without artifact staging
[ https://issues.apache.org/jira/browse/BEAM-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982811#comment-16982811 ] Thomas Weise commented on BEAM-8815: [~Ardagan] I'm marking this ticket for 2.17 just so that you see it and decide to merge the PR or not. > Portable pipeline execution without artifact staging > > > Key: BEAM-8815 > URL: https://issues.apache.org/jira/browse/BEAM-8815 > Project: Beam > Issue Type: Task > Components: runner-core, runner-flink >Affects Versions: 2.17.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability > Fix For: 2.17.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > The default artifact staging implementation relies on a distributed > filesystem. A directory and manifest will be created even when artifact > staging isn't used, and the container boot code will fail retrieving > artifacts, even though there are non. In a containerized environment it is > common to package artifacts into containers. It should be possible to run the > pipeline w/o a distributed filesystem. > [https://lists.apache.org/thread.html/1b0d545955a80688ea19f227ad943683747b17beb45368ad0908fd21@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8815) Portable pipeline execution without artifact staging
[ https://issues.apache.org/jira/browse/BEAM-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-8815: --- Fix Version/s: 2.17.0 > Portable pipeline execution without artifact staging > > > Key: BEAM-8815 > URL: https://issues.apache.org/jira/browse/BEAM-8815 > Project: Beam > Issue Type: Task > Components: runner-core, runner-flink >Affects Versions: 2.17.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability > Fix For: 2.17.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > The default artifact staging implementation relies on a distributed > filesystem. A directory and manifest will be created even when artifact > staging isn't used, and the container boot code will fail retrieving > artifacts, even though there are non. In a containerized environment it is > common to package artifacts into containers. It should be possible to run the > pipeline w/o a distributed filesystem. > [https://lists.apache.org/thread.html/1b0d545955a80688ea19f227ad943683747b17beb45368ad0908fd21@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8815) Portable pipeline execution without artifact staging
[ https://issues.apache.org/jira/browse/BEAM-8815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-8815: -- Assignee: Thomas Weise > Portable pipeline execution without artifact staging > > > Key: BEAM-8815 > URL: https://issues.apache.org/jira/browse/BEAM-8815 > Project: Beam > Issue Type: Task > Components: runner-core, runner-flink >Affects Versions: 2.17.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability > > The default artifact staging implementation relies on a distributed > filesystem. A directory and manifest will be created even when artifact > staging isn't used, and the container boot code will fail retrieving > artifacts, even though there are non. In a containerized environment it is > common to package artifacts into containers. It should be possible to run the > pipeline w/o a distributed filesystem. > [https://lists.apache.org/thread.html/1b0d545955a80688ea19f227ad943683747b17beb45368ad0908fd21@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8816) Load balance bundle processing w/ multiple SDK workers
[ https://issues.apache.org/jira/browse/BEAM-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-8816: --- Description: We found skewed utilization of SDK workers causing excessive latency with Streaming/Python/Flink. (Remember that with Python, we need to execute multiple worker processes on a machine instead of relying on threads in a single worker, which requires the runner to make a decision to which worker to give a bundle for processing.) The Flink runner has knobs to influence the number of records per bundle and the maximum duration for a bundle. But since the runner does not understand the cost of individual records, it is possible for the duration of bundles to fluctuate significantly due to skew in processing time of individual records. And unless the bundle size is 1, multiple expensive records could be allocated to a single bundle before the cutoff time is reached. We notice this with a pipeline that executes models, but there are other use cases where the cost of individual records can vary significantly. Additionally, the Flink runner establishes the association between the subtask managing an executable stage and the SDK worker during initialization, lasting for the duration of the job. In other words, bundles for the same executable stage will always be sent to the same SDK worker. When the execution time skew is tied to specific keys (stateful processing), it further aggravates the issue. [https://lists.apache.org/thread.html/59c02d8b8ea849c158deb39ad9d83af4d8fcb56570501c7fe8f79bb2@%3Cdev.beam.apache.org%3E] Long term this problem can be addressed with SDF. Till then, an (optional) runner controlled balancing mechanism has shown to improve the performance in internal testing. was: We found skewed utilization of SDK workers causing excessive latency with Streaming/Python/Flink. (Remember that with Python, we need to execute multiple worker processes on a machine instead of relying on threads in a single worker, which requires the runner to make a decision to which worker to give a bundle for processing.) The Flink runner has knobs to influence the number of records per bundle and the maximum duration for a bundle. But since the runner does not understand the cost of individual records, it is possible for the duration of bundles to fluctuates significantly due to skew in processing time of individual records. And unless the bundle size is 1, multiple expensive records could be allocated to a single bundle before the cutoff time is reached. We notice this with a pipeline that executes models, but there are other use cases where the cost of individual records can vary significantly. Additionally, the Flink runner establishes the association between the subtask managing an executable stage and the SDK worker during initialization, lasting for the duration of the job. In other words, bundles for the same executable stage will always be sent to the same SDK worker. When the execution time skew is tied to specific keys (stateful processing), it further aggravates the issue. [https://lists.apache.org/thread.html/59c02d8b8ea849c158deb39ad9d83af4d8fcb56570501c7fe8f79bb2@%3Cdev.beam.apache.org%3E] > Load balance bundle processing w/ multiple SDK workers > -- > > Key: BEAM-8816 > URL: https://issues.apache.org/jira/browse/BEAM-8816 > Project: Beam > Issue Type: Improvement > Components: runner-core, runner-flink >Affects Versions: 2.17.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability > > We found skewed utilization of SDK workers causing excessive latency with > Streaming/Python/Flink. (Remember that with Python, we need to execute > multiple worker processes on a machine instead of relying on threads in a > single worker, which requires the runner to make a decision to which worker > to give a bundle for processing.) > The Flink runner has knobs to influence the number of records per bundle and > the maximum duration for a bundle. But since the runner does not understand > the cost of individual records, it is possible for the duration of bundles to > fluctuate significantly due to skew in processing time of individual records. > And unless the bundle size is 1, multiple expensive records could be > allocated to a single bundle before the cutoff time is reached. We notice > this with a pipeline that executes models, but there are other use cases > where the cost of individual records can vary significantly. > Additionally, the Flink runner establishes the association between the > subtask managing an executable stage and the SDK worker during > initialization, lasting for the duration of the job. In other words, bundles > for the same executable
[jira] [Created] (BEAM-8816) Load balance bundle processing w/ multiple SDK workers
Thomas Weise created BEAM-8816: -- Summary: Load balance bundle processing w/ multiple SDK workers Key: BEAM-8816 URL: https://issues.apache.org/jira/browse/BEAM-8816 Project: Beam Issue Type: Improvement Components: runner-core, runner-flink Affects Versions: 2.17.0 Reporter: Thomas Weise Assignee: Thomas Weise We found skewed utilization of SDK workers causing excessive latency with Streaming/Python/Flink. (Remember that with Python, we need to execute multiple worker processes on a machine instead of relying on threads in a single worker, which requires the runner to make a decision to which worker to give a bundle for processing.) The Flink runner has knobs to influence the number of records per bundle and the maximum duration for a bundle. But since the runner does not understand the cost of individual records, it is possible for the duration of bundles to fluctuates significantly due to skew in processing time of individual records. And unless the bundle size is 1, multiple expensive records could be allocated to a single bundle before the cutoff time is reached. We notice this with a pipeline that executes models, but there are other use cases where the cost of individual records can vary significantly. Additionally, the Flink runner establishes the association between the subtask managing an executable stage and the SDK worker during initialization, lasting for the duration of the job. In other words, bundles for the same executable stage will always be sent to the same SDK worker. When the execution time skew is tied to specific keys (stateful processing), it further aggravates the issue. [https://lists.apache.org/thread.html/59c02d8b8ea849c158deb39ad9d83af4d8fcb56570501c7fe8f79bb2@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8815) Portable pipeline execution without artifact staging
Thomas Weise created BEAM-8815: -- Summary: Portable pipeline execution without artifact staging Key: BEAM-8815 URL: https://issues.apache.org/jira/browse/BEAM-8815 Project: Beam Issue Type: Task Components: runner-core, runner-flink Affects Versions: 2.17.0 Reporter: Thomas Weise The default artifact staging implementation relies on a distributed filesystem. A directory and manifest will be created even when artifact staging isn't used, and the container boot code will fail retrieving artifacts, even though there are non. In a containerized environment it is common to package artifacts into containers. It should be possible to run the pipeline w/o a distributed filesystem. [https://lists.apache.org/thread.html/1b0d545955a80688ea19f227ad943683747b17beb45368ad0908fd21@%3Cdev.beam.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8471) Flink native job submission for portable pipelines
[
https://issues.apache.org/jira/browse/BEAM-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise updated BEAM-8471:
---
Fix Version/s: (was: 2.18.0)
2.17.0
> Flink native job submission for portable pipelines
> --
>
> Key: BEAM-8471
> URL: https://issues.apache.org/jira/browse/BEAM-8471
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability-flink
> Fix For: 2.17.0
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> There are currently two methods to run a portable pipeline written in a
> non-JVM language to Flink:
> 1) Run the SDK client entry point which will submit the job server, which in
> turn will submit to a Flink cluster using the Flink remote environment
> 2) Run the SDK client entry point to generate a Flink jar file that can be
> used to start the Flink job using any of the Flink client tooling available.
> Either approach requires the SDK client and the job server dependency to be
> present on the client. This doesn't work well in environments such as
> FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
> [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
> This improvement is to provide a new Flink entry point (main method) that
> invokes the SDK client entry point to generate the pipeline and submits the
> resulting Flink job like any other Flink native driver program would, via the
> optimizer plan environment ("[auto]").
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (BEAM-8670) Manage environment parallelism in DefaultJobBundleFactory
[ https://issues.apache.org/jira/browse/BEAM-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-8670. -- Fix Version/s: 2.18.0 Resolution: Fixed > Manage environment parallelism in DefaultJobBundleFactory > - > > Key: BEAM-8670 > URL: https://issues.apache.org/jira/browse/BEAM-8670 > Project: Beam > Issue Type: Task > Components: runner-core >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability > Fix For: 2.18.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Multiple (remote) environments are currently managed through logic in > ExecutableStageContext. This pins ExecutableStage to a given environment for > the lifetime of a job and makes it impossible to implement an alternative > scheduling strategy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8670) Manage environment parallelism in DefaultJobBundleFactory
Thomas Weise created BEAM-8670: -- Summary: Manage environment parallelism in DefaultJobBundleFactory Key: BEAM-8670 URL: https://issues.apache.org/jira/browse/BEAM-8670 Project: Beam Issue Type: Task Components: runner-core Reporter: Thomas Weise Assignee: Thomas Weise Multiple (remote) environments are currently managed through logic in ExecutableStageContext. This pins ExecutableStage to a given environment for the lifetime of a job and makes it impossible to implement an alternative scheduling strategy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8656) flink_master_url usage in flink_runner.py
[ https://issues.apache.org/jira/browse/BEAM-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-8656: --- Issue Type: Bug (was: Improvement) > flink_master_url usage in flink_runner.py > - > > Key: BEAM-8656 > URL: https://issues.apache.org/jira/browse/BEAM-8656 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > flink_master_url was replaced with flink_master in flink_runner.py without > preserving backward compatibility, but it remains documented on the website. > We will either have to update the website (making it clear that the > instructions are for 2.17+) or else make sure that the flink_master_url gets > aliased to flink_master before the 2.17 release is finalized. If anyone's > already using flink_runner.py, this will also break their existing pipeline. > [https://github.com/apache/beam/pull/9946] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8656) flink_master_url usage in flink_runner.py
[ https://issues.apache.org/jira/browse/BEAM-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973934#comment-16973934 ] Thomas Weise commented on BEAM-8656: +1 for updating the website, there are several more things that need to be updated for 2.17. It is probably not very important to have backward compatibility in this case, since flink_runner.py was just added and only used for experimental purposes. > flink_master_url usage in flink_runner.py > - > > Key: BEAM-8656 > URL: https://issues.apache.org/jira/browse/BEAM-8656 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > flink_master_url was replaced with flink_master in flink_runner.py without > preserving backward compatibility, but it remains documented on the website. > We will either have to update the website (making it clear that the > instructions are for 2.17+) or else make sure that the flink_master_url gets > aliased to flink_master before the 2.17 release is finalized. If anyone's > already using flink_runner.py, this will also break their existing pipeline. > [https://github.com/apache/beam/pull/9946] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8591) Exception is thrown while running Beam Pipeline on Kubernetes Flink Cluster.
[
https://issues.apache.org/jira/browse/BEAM-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972922#comment-16972922
]
Thomas Weise commented on BEAM-8591:
Please see the following email thread and linked doc for options on how to run
Beam with Flink on k8s:
[https://lists.apache.org/thread.html/4e377933da8f5abb817413fcbd1de172b81a468c8a4d782255f46a1a@%3Cdev.beam.apache.org%3E]
The LOOPBACK environment won't work in a distributed environment, it is
designed for local execution.
I can see where the confusion may come from and the Flink runner page is going
to receive some updates soonish: BEAM-8243 CC: [~ibzib]
The DOCKER environment is also most likely not going to work, the doc explains
why (unless you are going to setup docker within k8s).
That leaves you with either the PROCESS environment or EXTERNAL (using the
Python SDK worker pool option). Take a look at the doc and ask for
clarification there or here:
[https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/]
> Exception is thrown while running Beam Pipeline on Kubernetes Flink Cluster.
>
>
> Key: BEAM-8591
> URL: https://issues.apache.org/jira/browse/BEAM-8591
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Mingliang Gong
>Priority: Major
>
> h2. Setup Clusters
> * Setup Local Flink Cluster:
> [https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html]
> * Setup Kubernetes Flink Cluster using Minikube:
> [https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/kubernetes.html]
> h2. Verify Clusters
> Execute command “./bin/flink run examples/streaming/WordCount.jar”. Both
> Local and K8S Flink Cluster work fine.
> h2. Using Apache Beam Flink Runner
> Instruction: [https://beam.apache.org/documentation/runners/flink/]
> Sample Pipeline Code:
> {code:java}
> import apache_beam as beam
> from apache_beam.options.pipeline_options import PipelineOptions
> options = PipelineOptions([
> "--runner=PortableRunner",
> "--job_endpoint=localhost:8099",
> "--environment_type=LOOPBACK"
> ])
> with beam.Pipeline(options=options) as pipeline:
> data = ["Sample data",
> "Sample data - 0",
> "Sample data - 1"]
> raw_data = (pipeline
> | 'CreateHardCodeData' >> beam.Create(data)
> | 'Map' >> beam.Map(lambda line : line + '.')
> | 'Print' >> beam.Map(print)){code}
> Verify different environment_type in Python SDK Harness Configuration
> *environment_type=LOOPBACK*
> # Run pipeline on local cluster: Works Fine
> # Run pipeline on K8S cluster, Exceptions are thrown:
> java.lang.Exception: The user defined 'open()' method caused an exception:
> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException:
> UNAVAILABLE: io exception Caused by:
> org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> Connection refused: localhost/127.0.0.1:51017
> *environment_type=DOCKER*
> # Run pipeline on local cluster: Work fine
> # Run pipeline on K8S cluster, Exception are thrown:
> Caused by: java.io.IOException: Cannot run program "docker": error=2, No
> such file or directory.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5187) Create a ProcessJobBundleFactory for non-dockerized SDK harness
[
https://issues.apache.org/jira/browse/BEAM-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972715#comment-16972715
]
Thomas Weise commented on BEAM-5187:
The process environment factory that is part of Beam expects the user to supply
a script. If the script calls the container boot binary, then artifact
retrieval should work (I never tried it because we are not using that class).
The custom factory in the Lyft fork starts the Python worker directly and
doesn't support artifact retrieval. If we had a retrieval implementation in
Java then we could add it.
> Create a ProcessJobBundleFactory for non-dockerized SDK harness
> ---
>
> Key: BEAM-5187
> URL: https://issues.apache.org/jira/browse/BEAM-5187
> Project: Beam
> Issue Type: New Feature
> Components: runner-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 2.7.0
>
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> As discussed on the mailing list [1], we want to giver users an option to
> execute portable pipelines without Docker. Analog to the
> {{DockerJobBundleFactory}}, a {{ProcessJobBundleFactory}} could be added to
> directly fork SDK harness processes.
> Artifacts will be provided by an artifact directory or could be setup similar
> to the existing bootstrapping code ("boot.go") which we use for containers.
> The process-based execution can optionally be configured via the pipeline
> options.
> [1]
> [https://lists.apache.org/thread.html/d8b81e9f74f77d74c8b883cda80fa48efdcaf6ac2ad313c4fe68795a@%3Cdev.beam.apache.org%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-7870) Externally configured KafkaIO / PubsubIO consumer causes coder problems
[ https://issues.apache.org/jira/browse/BEAM-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966364#comment-16966364 ] Thomas Weise edited comment on BEAM-7870 at 11/4/19 2:25 AM: - Thanks for the update. I would also prefer the specific record type (PubsubMessage/KafkaRecord) over generic Row and this approach is a nice compromise. was (Author: thw): Thanks for the update. I would also prefer the specific record type (PubsubMessage/KafkaRecord) over generic Row. > Externally configured KafkaIO / PubsubIO consumer causes coder problems > --- > > Key: BEAM-7870 > URL: https://issues.apache.org/jira/browse/BEAM-7870 > Project: Beam > Issue Type: Bug > Components: runner-flink, sdk-java-core >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > > There are limitations for the consumer to work correctly. The biggest issue > is the structure of KafkaIO itself, which uses a combination of the source > interface and DoFns to generate the desired output. The problem is that the > source interface is natively translated by the Flink Runner to support > unbounded sources in portability, while the DoFn runs in a Java environment. > To transfer data between the two a coder needs to be involved. It happens to > be that the initial read does not immediately drop the KafakRecord structure > which does not work together well with our current assumption of only > supporting "standard coders" present in all SDKs. Only the subsequent DoFn > converts the KafkaRecord structure into a raw KV[byte, byte], but the DoFn > won't have the coder available in its environment. > There are several possible solutions: > 1. Make the DoFn which drops the KafkaRecordCoder a native Java transform in > the Flink Runner > 2. Modify KafkaIO to immediately drop the KafkaRecord structure > 3. Add the KafkaRecordCoder to all SDKs > 4. Add a generic coder, e.g. AvroCoder to all SDKs > For a workaround which uses (3), please see this patch which is not a proper > fix but adds KafkaRecordCoder to the SDK such that it can be used > encode/decode records: > [https://github.com/mxm/beam/commit/b31cf99c75b3972018180d8ccc7e73d311f4cfed] > > See also > [https://github.com/apache/beam/pull/8251|https://github.com/apache/beam/pull/8251:] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7870) Externally configured KafkaIO / PubsubIO consumer causes coder problems
[ https://issues.apache.org/jira/browse/BEAM-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966364#comment-16966364 ] Thomas Weise commented on BEAM-7870: Thanks for the update. I would also prefer the specific record type (PubsubMessage/KafkaRecord) over generic Row. > Externally configured KafkaIO / PubsubIO consumer causes coder problems > --- > > Key: BEAM-7870 > URL: https://issues.apache.org/jira/browse/BEAM-7870 > Project: Beam > Issue Type: Bug > Components: runner-flink, sdk-java-core >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > > There are limitations for the consumer to work correctly. The biggest issue > is the structure of KafkaIO itself, which uses a combination of the source > interface and DoFns to generate the desired output. The problem is that the > source interface is natively translated by the Flink Runner to support > unbounded sources in portability, while the DoFn runs in a Java environment. > To transfer data between the two a coder needs to be involved. It happens to > be that the initial read does not immediately drop the KafakRecord structure > which does not work together well with our current assumption of only > supporting "standard coders" present in all SDKs. Only the subsequent DoFn > converts the KafkaRecord structure into a raw KV[byte, byte], but the DoFn > won't have the coder available in its environment. > There are several possible solutions: > 1. Make the DoFn which drops the KafkaRecordCoder a native Java transform in > the Flink Runner > 2. Modify KafkaIO to immediately drop the KafkaRecord structure > 3. Add the KafkaRecordCoder to all SDKs > 4. Add a generic coder, e.g. AvroCoder to all SDKs > For a workaround which uses (3), please see this patch which is not a proper > fix but adds KafkaRecordCoder to the SDK such that it can be used > encode/decode records: > [https://github.com/mxm/beam/commit/b31cf99c75b3972018180d8ccc7e73d311f4cfed] > > See also > [https://github.com/apache/beam/pull/8251|https://github.com/apache/beam/pull/8251:] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8243) Document behavior of FlinkRunner
[ https://issues.apache.org/jira/browse/BEAM-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965163#comment-16965163 ] Thomas Weise commented on BEAM-8243: I meant older Beam releases, we should probably only advertise the latest for portability. But the "traditional" process of job submission is also too scary for first time user :) > Document behavior of FlinkRunner > > > Key: BEAM-8243 > URL: https://issues.apache.org/jira/browse/BEAM-8243 > Project: Beam > Issue Type: Improvement > Components: runner-flink, website >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > > The Flink runner guide should include a couple details > 1) FlinkRunner pulls the job server jar from Maven by default (need to make > this explicit in case of firewall concerns) > 2) how to override in case the above is a problem -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8243) Document behavior of FlinkRunner
[ https://issues.apache.org/jira/browse/BEAM-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965112#comment-16965112 ] Thomas Weise commented on BEAM-8243: There are IMO enough other reasons to not keep instructions around for old versions (for portability) :) > Document behavior of FlinkRunner > > > Key: BEAM-8243 > URL: https://issues.apache.org/jira/browse/BEAM-8243 > Project: Beam > Issue Type: Improvement > Components: runner-flink, website >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > > The Flink runner guide should include a couple details > 1) FlinkRunner pulls the job server jar from Maven by default (need to make > this explicit in case of firewall concerns) > 2) how to override in case the above is a problem -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8243) Document behavior of FlinkRunner
[ https://issues.apache.org/jira/browse/BEAM-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964541#comment-16964541 ] Thomas Weise commented on BEAM-8243: Looking forward to see the gradle commands disappear from [https://beam.apache.org/documentation/runners/flink/] and it would be good to add back running the wordcount example with FlinkRunner. That might look really user friendly now! > Document behavior of FlinkRunner > > > Key: BEAM-8243 > URL: https://issues.apache.org/jira/browse/BEAM-8243 > Project: Beam > Issue Type: Improvement > Components: runner-flink, website >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > > The Flink runner guide should include a couple details > 1) FlinkRunner pulls the job server jar from Maven by default (need to make > this explicit in case of firewall concerns) > 2) how to override in case the above is a problem -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8243) Document behavior of FlinkRunner
[ https://issues.apache.org/jira/browse/BEAM-8243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964534#comment-16964534 ] Thomas Weise commented on BEAM-8243: [~ibzib] [~robertwb] is this going to cover documentation for BEAM-8372? > Document behavior of FlinkRunner > > > Key: BEAM-8243 > URL: https://issues.apache.org/jira/browse/BEAM-8243 > Project: Beam > Issue Type: Improvement > Components: runner-flink, website >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > > The Flink runner guide should include a couple details > 1) FlinkRunner pulls the job server jar from Maven by default (need to make > this explicit in case of firewall concerns) > 2) how to override in case the above is a problem -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8372) Allow submission of Flink UberJar directly to flink cluster.
[ https://issues.apache.org/jira/browse/BEAM-8372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-8372: --- Fix Version/s: 2.17.0 > Allow submission of Flink UberJar directly to flink cluster. > > > Key: BEAM-8372 > URL: https://issues.apache.org/jira/browse/BEAM-8372 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Labels: portability, portability-flink > Fix For: 2.17.0 > > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8372) Allow submission of Flink UberJar directly to flink cluster.
[ https://issues.apache.org/jira/browse/BEAM-8372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-8372: --- Labels: portability portability-flink (was: ) > Allow submission of Flink UberJar directly to flink cluster. > > > Key: BEAM-8372 > URL: https://issues.apache.org/jira/browse/BEAM-8372 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Labels: portability, portability-flink > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8518) Pipeline options translation fails silently with incompatible jackson-core library
Thomas Weise created BEAM-8518: -- Summary: Pipeline options translation fails silently with incompatible jackson-core library Key: BEAM-8518 URL: https://issues.apache.org/jira/browse/BEAM-8518 Project: Beam Issue Type: Bug Components: runner-core Reporter: Thomas Weise Assignee: Thomas Weise Discovered when executing the job server with other dependencies in the class path. In this case it was jackson-annotations-2.2.3.jar, jackson-core-2.2.3.jar, jackson-databind-2.2.3.jar that came from Hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8471) Flink native job submission for portable pipelines
[
https://issues.apache.org/jira/browse/BEAM-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961628#comment-16961628
]
Thomas Weise commented on BEAM-8471:
If anyone is interested to try this using curl with a local Flink cluster:
Upload the job server jar (can be done ahead of time, when building a
container, for example)
{code:java}
curl -X POST -H "Expect:" -F
"jarfile=@/Users/tweise/src/beam/runners/flink/1.8/job-server/build/libs/beam-runners-flink-1.8-job-server-2.16.0-SNAPSHOT.jar"
http://localhost:8081/jars/upload {code}
The response will be the jar file reference used later.
Job submission payload:
{code:java}
{ "entryClass":"org.apache.beam.runners.flink.FlinkPortableClientEntryPoint",
"programArgs":"--driver-cmd \"python -m apache_beam.dummy_pipeline\"",
"parallelism": 1
} {code}
Run the job:
{code:java}
curl --verbose -X POST -d "@launch.json"
'http://localhost:8081/jars/c1c9747a-42f5-4a97-8871-0dafb809b869_beam-runners-flink-1.8-job-server-2.16.0-SNAPSHOT.jar/run'
{code}
> Flink native job submission for portable pipelines
> --
>
> Key: BEAM-8471
> URL: https://issues.apache.org/jira/browse/BEAM-8471
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability-flink
> Fix For: 2.18.0
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> There are currently two methods to run a portable pipeline written in a
> non-JVM language to Flink:
> 1) Run the SDK client entry point which will submit the job server, which in
> turn will submit to a Flink cluster using the Flink remote environment
> 2) Run the SDK client entry point to generate a Flink jar file that can be
> used to start the Flink job using any of the Flink client tooling available.
> Either approach requires the SDK client and the job server dependency to be
> present on the client. This doesn't work well in environments such as
> FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
> [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
> This improvement is to provide a new Flink entry point (main method) that
> invokes the SDK client entry point to generate the pipeline and submits the
> resulting Flink job like any other Flink native driver program would, via the
> optimizer plan environment ("[auto]").
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (BEAM-8471) Flink native job submission for portable pipelines
[
https://issues.apache.org/jira/browse/BEAM-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise closed BEAM-8471.
--
Fix Version/s: 2.18.0
Resolution: Implemented
> Flink native job submission for portable pipelines
> --
>
> Key: BEAM-8471
> URL: https://issues.apache.org/jira/browse/BEAM-8471
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability-flink
> Fix For: 2.18.0
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> There are currently two methods to run a portable pipeline written in a
> non-JVM language to Flink:
> 1) Run the SDK client entry point which will submit the job server, which in
> turn will submit to a Flink cluster using the Flink remote environment
> 2) Run the SDK client entry point to generate a Flink jar file that can be
> used to start the Flink job using any of the Flink client tooling available.
> Either approach requires the SDK client and the job server dependency to be
> present on the client. This doesn't work well in environments such as
> FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
> [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
> This improvement is to provide a new Flink entry point (main method) that
> invokes the SDK client entry point to generate the pipeline and submits the
> resulting Flink job like any other Flink native driver program would, via the
> optimizer plan environment ("[auto]").
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (BEAM-8471) Flink native job submission for portable pipelines
[
https://issues.apache.org/jira/browse/BEAM-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise updated BEAM-8471:
---
Comment: was deleted
(was: CC: [~robertwb], [~ibzib], [~mxm]
I'm going to upstream this: [https://github.com/lyft/beam/pull/27/files]
)
> Flink native job submission for portable pipelines
> --
>
> Key: BEAM-8471
> URL: https://issues.apache.org/jira/browse/BEAM-8471
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability-flink
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> There are currently two methods to run a portable pipeline written in a
> non-JVM language to Flink:
> 1) Run the SDK client entry point which will submit the job server, which in
> turn will submit to a Flink cluster using the Flink remote environment
> 2) Run the SDK client entry point to generate a Flink jar file that can be
> used to start the Flink job using any of the Flink client tooling available.
> Either approach requires the SDK client and the job server dependency to be
> present on the client. This doesn't work well in environments such as
> FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
> [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
> This improvement is to provide a new Flink entry point (main method) that
> invokes the SDK client entry point to generate the pipeline and submits the
> resulting Flink job like any other Flink native driver program would, via the
> optimizer plan environment ("[auto]").
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8471) Flink native job submission for portable pipelines
[
https://issues.apache.org/jira/browse/BEAM-8471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise updated BEAM-8471:
---
Description:
There are currently two methods to run a portable pipeline written in a non-JVM
language to Flink:
1) Run the SDK client entry point which will submit the job server, which in
turn will submit to a Flink cluster using the Flink remote environment
2) Run the SDK client entry point to generate a Flink jar file that can be used
to start the Flink job using any of the Flink client tooling available.
Either approach requires the SDK client and the job server dependency to be
present on the client. This doesn't work well in environments such as
FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
[https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
This improvement is to provide a new Flink entry point (main method) that
invokes the SDK client entry point to generate the pipeline and submits the
resulting Flink job like any other Flink native driver program would, via the
optimizer plan environment ("[auto]").
was:
There are currently two methods to run a portable pipeline written in a non-JVM
language to Flink:
1) Run the SDK client entry point which will submit the job server, which in
turn will submit to a Flink cluster using the Flink remote environment
2) Run the SDK client entry point to generate a Flink jar file that can be used
to start the Flink job using any of the Flink client tooling available.
Either approach requires the SDK client and the job server dependency to be
present on the client. This doesn't work well in environments such as
FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
[https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
This improvement is to provide an new Flink entry point (main method) that
invokes the SDK client entry point to generate the pipeline and submits the
resulting Flink job like any other Flink native driver program would, via the
optimizer plan environment ("[auto]").
> Flink native job submission for portable pipelines
> --
>
> Key: BEAM-8471
> URL: https://issues.apache.org/jira/browse/BEAM-8471
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability-flink
>
> There are currently two methods to run a portable pipeline written in a
> non-JVM language to Flink:
> 1) Run the SDK client entry point which will submit the job server, which in
> turn will submit to a Flink cluster using the Flink remote environment
> 2) Run the SDK client entry point to generate a Flink jar file that can be
> used to start the Flink job using any of the Flink client tooling available.
> Either approach requires the SDK client and the job server dependency to be
> present on the client. This doesn't work well in environments such as
> FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
> [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
> This improvement is to provide a new Flink entry point (main method) that
> invokes the SDK client entry point to generate the pipeline and submits the
> resulting Flink job like any other Flink native driver program would, via the
> optimizer plan environment ("[auto]").
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-8471) Flink native job submission for portable pipelines
Thomas Weise created BEAM-8471:
--
Summary: Flink native job submission for portable pipelines
Key: BEAM-8471
URL: https://issues.apache.org/jira/browse/BEAM-8471
Project: Beam
Issue Type: Improvement
Components: runner-flink
Reporter: Thomas Weise
Assignee: Thomas Weise
There are currently two methods to run a portable pipeline written in a non-JVM
language to Flink:
1) Run the SDK client entry point which will submit the job server, which in
turn will submit to a Flink cluster using the Flink remote environment
2) Run the SDK client entry point to generate a Flink jar file that can be used
to start the Flink job using any of the Flink client tooling available.
Either approach requires the SDK client and the job server dependency to be
present on the client. This doesn't work well in environments such as
FlinkK8sOperator that rely on the Flink REST API jar run endpoint (see
[https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.x7hki4bhh18l]).
This improvement is to provide an new Flink entry point (main method) that
invokes the SDK client entry point to generate the pipeline and submits the
resulting Flink job like any other Flink native driver program would, via the
optimizer plan environment ("[auto]").
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (BEAM-8417) Expose ExternalWorkerHandler hostname
[ https://issues.apache.org/jira/browse/BEAM-8417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-8417. -- Fix Version/s: 2.17.0 Resolution: Fixed > Expose ExternalWorkerHandler hostname > - > > Key: BEAM-8417 > URL: https://issues.apache.org/jira/browse/BEAM-8417 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Wanqi Lyu >Assignee: Wanqi Lyu >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently `fn_api_runner.ExternalWorkerHandler` endpoints have `localhost` as > their hostname by default, which prevents it from being connected from > external workers started on other machines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8387) Remove sdk-worker-parallelism option from JobServerDriver
[ https://issues.apache.org/jira/browse/BEAM-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-8387. Fix Version/s: 2.17.0 Resolution: Fixed > Remove sdk-worker-parallelism option from JobServerDriver > - > > Key: BEAM-8387 > URL: https://issues.apache.org/jira/browse/BEAM-8387 > Project: Beam > Issue Type: Task > Components: runner-core >Affects Versions: 2.16.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > Fix For: 2.17.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The option was added when it wasn't possible to specify it as pipeline > option, which is no longer the case. The pipeline option has a value of 0, > which means that the runner should pick a suitable value. But this is then > overridden in FlinkJobInvoker with 1 (because that's the default in the > JobServerDriver config. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8387) Remove sdk-worker-parallelism option from JobServerDriver
[ https://issues.apache.org/jira/browse/BEAM-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-8387: --- Status: Open (was: Triage Needed) > Remove sdk-worker-parallelism option from JobServerDriver > - > > Key: BEAM-8387 > URL: https://issues.apache.org/jira/browse/BEAM-8387 > Project: Beam > Issue Type: Task > Components: runner-core >Affects Versions: 2.16.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > > The option was added when it wasn't possible to specify it as pipeline > option, which is no longer the case. The pipeline option has a value of 0, > which means that the runner should pick a suitable value. But this is then > overridden in FlinkJobInvoker with 1 (because that's the default in the > JobServerDriver config. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8183) Optionally bundle multiple pipelines into a single Flink jar
[ https://issues.apache.org/jira/browse/BEAM-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-8183. Fix Version/s: 2.17.0 Resolution: Implemented > Optionally bundle multiple pipelines into a single Flink jar > > > Key: BEAM-8183 > URL: https://issues.apache.org/jira/browse/BEAM-8183 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > Fix For: 2.17.0 > > Time Spent: 3h > Remaining Estimate: 0h > > [https://github.com/apache/beam/pull/9331#issuecomment-526734851] > "With Flink you can bundle multiple entry points into the same jar file and > specify which one to use with optional flags. It may be desirable to allow > inclusion of multiple pipelines for this tool also, although that would > require a different workflow. Absent this option, it becomes quite convoluted > for users that need the flexibility to choose which pipeline to launch at > submission time." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8387) Remove sdk-worker-parallelism option from JobServerDriver
Thomas Weise created BEAM-8387: -- Summary: Remove sdk-worker-parallelism option from JobServerDriver Key: BEAM-8387 URL: https://issues.apache.org/jira/browse/BEAM-8387 Project: Beam Issue Type: Task Components: runner-core Affects Versions: 2.16.0 Reporter: Thomas Weise Assignee: Thomas Weise The option was added when it wasn't possible to specify it as pipeline option, which is no longer the case. The pipeline option has a value of 0, which means that the runner should pick a suitable value. But this is then overridden in FlinkJobInvoker with 1 (because that's the default in the JobServerDriver config. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8115) Overwrite portable Flink application jar pipeline options at runtime
[ https://issues.apache.org/jira/browse/BEAM-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949726#comment-16949726 ] Thomas Weise commented on BEAM-8115: The first case will allow to change all runner and SDK worker options, so that will be valuable. Regarding transform properties, it is correct that there is no solution that will work with status quo. For our custom transform translations, we can interpolate properties to do the replacement. For Beam transforms, I wonder if there could be a different value provider that binds on deserialization in the worker, i.e. property value isn't part of the transform, just the pointer to it. > Overwrite portable Flink application jar pipeline options at runtime > > > Key: BEAM-8115 > URL: https://issues.apache.org/jira/browse/BEAM-8115 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > In the first iteration of portable Flink application jars, all pipeline > options are set at job creation time and cannot be later modified at runtime. > There should be a way to pass arguments to the jar to write/overwrite > pipeline options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-7980) External environment with containerized worker pool
[ https://issues.apache.org/jira/browse/BEAM-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-7980. -- Resolution: Implemented > External environment with containerized worker pool > --- > > Key: BEAM-7980 > URL: https://issues.apache.org/jira/browse/BEAM-7980 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Fix For: 2.16.0 > > Time Spent: 9h > Remaining Estimate: 0h > > Augment Beam Python docker image and boot.go so that it can be used to launch > BeamFnExternalWorkerPoolServicer. > [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.lnhm75dhvhi0] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (BEAM-7980) External environment with containerized worker pool
[ https://issues.apache.org/jira/browse/BEAM-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reopened BEAM-7980: > External environment with containerized worker pool > --- > > Key: BEAM-7980 > URL: https://issues.apache.org/jira/browse/BEAM-7980 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Fix For: 2.16.0 > > Time Spent: 9h > Remaining Estimate: 0h > > Augment Beam Python docker image and boot.go so that it can be used to launch > BeamFnExternalWorkerPoolServicer. > [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.lnhm75dhvhi0] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise resolved BEAM-6829.
Fix Version/s: 2.17.0
Resolution: Fixed
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Fix For: 2.17.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-8351) Support passing in arbitrary KV pairs to sdk worker via external environment config
[ https://issues.apache.org/jira/browse/BEAM-8351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise reassigned BEAM-8351: -- Assignee: Wanqi Lyu > Support passing in arbitrary KV pairs to sdk worker via external environment > config > --- > > Key: BEAM-8351 > URL: https://issues.apache.org/jira/browse/BEAM-8351 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Wanqi Lyu >Assignee: Wanqi Lyu >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Originally, the environment config for environment type of EXTERNAL only > support passing in an url for the external worker pool; We want to support > passing in arbitrary KV pairs to sdk worker via external environment config, > so that the when starting the sdk harness we could get the values from > `StartWorkerRequest.params`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8183) Optionally bundle multiple pipelines into a single Flink jar
[ https://issues.apache.org/jira/browse/BEAM-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943638#comment-16943638 ] Thomas Weise commented on BEAM-8183: BEAM-8115 is indeed orthogonal and applicable for the cases where parameterization can be solved w/o different execution path in the driver program. For the remaining cases bundling multiple protos could be the solution. > Optionally bundle multiple pipelines into a single Flink jar > > > Key: BEAM-8183 > URL: https://issues.apache.org/jira/browse/BEAM-8183 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > > [https://github.com/apache/beam/pull/9331#issuecomment-526734851] > "With Flink you can bundle multiple entry points into the same jar file and > specify which one to use with optional flags. It may be desirable to allow > inclusion of multiple pipelines for this tool also, although that would > require a different workflow. Absent this option, it becomes quite convoluted > for users that need the flexibility to choose which pipeline to launch at > submission time." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8183) Optionally bundle multiple pipelines into a single Flink jar
[
https://issues.apache.org/jira/browse/BEAM-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943170#comment-16943170
]
Thomas Weise commented on BEAM-8183:
{quote}The major issue to me seems to be that we need to execute pipeline
construction code which is environment dependent. To generate new pipelines for
an environment, we need to execute the pipeline submission code in that
environment. And this is where I see a problem. Python pipelines have to
execute user code in python using python sdk to construct the pipeline.
{quote}
You are correct that the Python entry point / driver program would need to be
(re)executed for a fully generic solution. But that's not necessary for the
majority of use cases. Those are artifact + configuration. If there is a way to
parameterize configuration values in the proto, we can address that majority of
use cases with a single job jar artifact.
My fallback for the exception path would be to generate multiple protos into a
single jar, which is why I'm interested in this capability. So that jar would
contain "mypipeline_staging" and "mypipeline_production" and the deployment
would select the pipeline via its configuration (parameter to the Flink entry
point). Similar would work for Spark.
But beyond that we also have (in our infrastructure) the use case of multiple
entry points that the user can pick at submit time.
> Optionally bundle multiple pipelines into a single Flink jar
>
>
> Key: BEAM-8183
> URL: https://issues.apache.org/jira/browse/BEAM-8183
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-flink
>
> [https://github.com/apache/beam/pull/9331#issuecomment-526734851]
> "With Flink you can bundle multiple entry points into the same jar file and
> specify which one to use with optional flags. It may be desirable to allow
> inclusion of multiple pipelines for this tool also, although that would
> require a different workflow. Absent this option, it becomes quite convoluted
> for users that need the flexibility to choose which pipeline to launch at
> submission time."
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8183) Optionally bundle multiple pipelines into a single Flink jar
[ https://issues.apache.org/jira/browse/BEAM-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943015#comment-16943015 ] Thomas Weise commented on BEAM-8183: For context: [https://lists.apache.org/thread.html/2122928a0a5f678d475ec15af538eb7303f73557870af174b1fdef7e@%3Cdev.beam.apache.org%3E] Running the same pipeline in different environments with different parameters is a common need. Virtually everyone has dev/staging/prod or whatever their environments are and they want to use the same build artifact. That normally requires some amount of parameterization. The other use case is bundling multiple pipelines into the same container and select which to run at launch time. I was surprised about the question given prior discussion. Even more so considering that Beam already has the concept of user options. The approach of generating the jar file currently is equivalent to hard coding all pipeline options and asking the user to recompile. Yes, we could generate a new jar file for every option or environment but please not it bloats the container images (job server is > 100MB). We can also create separate Docker images, now we are in the GB range. > Optionally bundle multiple pipelines into a single Flink jar > > > Key: BEAM-8183 > URL: https://issues.apache.org/jira/browse/BEAM-8183 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > > [https://github.com/apache/beam/pull/9331#issuecomment-526734851] > "With Flink you can bundle multiple entry points into the same jar file and > specify which one to use with optional flags. It may be desirable to allow > inclusion of multiple pipelines for this tool also, although that would > require a different workflow. Absent this option, it becomes quite convoluted > for users that need the flexibility to choose which pipeline to launch at > submission time." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8183) Optionally bundle multiple pipelines into a single Flink jar
[ https://issues.apache.org/jira/browse/BEAM-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939795#comment-16939795 ] Thomas Weise commented on BEAM-8183: Hi Kyle, I have setup my Flink image build to use the jar file runner, but currently it can only bundle one pipeline into the job jar. Would need support for multiple Python entry points / different configurations. Just wanted to check how soon you are planning to work on this? > Optionally bundle multiple pipelines into a single Flink jar > > > Key: BEAM-8183 > URL: https://issues.apache.org/jira/browse/BEAM-8183 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-flink > > [https://github.com/apache/beam/pull/9331#issuecomment-526734851] > "With Flink you can bundle multiple entry points into the same jar file and > specify which one to use with optional flags. It may be desirable to allow > inclusion of multiple pipelines for this tool also, although that would > require a different workflow. Absent this option, it becomes quite convoluted > for users that need the flexibility to choose which pipeline to launch at > submission time." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6733) Use Flink 1.6/1.7 prepareSnapshotPreBarrier to replace BufferedOutputManager
[
https://issues.apache.org/jira/browse/BEAM-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936920#comment-16936920
]
Thomas Weise commented on BEAM-6733:
This has an effect on checkpointing that is probably worth discussing.
Checkpoint barriers would be blocked by finishing the bundle, impacting
alignment similar to what we see in Flink under backpressure. Heavy
backpressure in Flink causes checkpoint timeouts and related operational issues.
Regarding latency this should be neutral as we already wait for the bundle to
finish before we let the watermark pass?
> Use Flink 1.6/1.7 prepareSnapshotPreBarrier to replace BufferedOutputManager
>
>
> Key: BEAM-6733
> URL: https://issues.apache.org/jira/browse/BEAM-6733
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Flink 1.6/1.7 provides a hook to execute an action before the snapshot
> barrier is emitted by the operator. At the moment (<=1.5) the Flink Runner
> has to buffer any elements which are emitted during a snapshot because the
> barrier has already been emitted. This leads to a lot of code complexity.
> We can remove the buffering in favor of finishing the current bundle in
> {{DoFnOperator}}'s {{prepareSnapshotPreBarrier}}. The 1.5/1.6/1.7 build setup
> poses a challenge to do that in a way that does not lead to much code
> duplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8157) Key encoding for state requests is not consistent across SDKs
[ https://issues.apache.org/jira/browse/BEAM-8157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936835#comment-16936835 ] Thomas Weise commented on BEAM-8157: I would probably not backport since 2.16 doesn't have this issue (it will only become relevant with Flink 1.9). > Key encoding for state requests is not consistent across SDKs > - > > Key: BEAM-8157 > URL: https://issues.apache.org/jira/browse/BEAM-8157 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.13.0 >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.17.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > The Flink runner requires the internal key to be encoded without a length > prefix (OUTER context). The user state request handler exposes a serialized > version of the key to the Runner. This key is encoded with the NESTED context > which may add a length prefix. We need to convert it to OUTER context to > match the Flink runner's key encoding. > So far this has not caused the Flink Runner to behave incorrectly. However, > with the upcoming support for Flink 1.9, the state backend will not accept > requests for keys not part of any key group/partition of the operator. This > is very likely to happen with the encoding not being consistent. > **NOTE** This is only applicable to the Java SDK, as the Python SDK uses > OUTER encoding for the key in state requests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7722) Simplify running of Beam Python on Flink
[ https://issues.apache.org/jira/browse/BEAM-7722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-7722. Fix Version/s: 2.16.0 Resolution: Implemented > Simplify running of Beam Python on Flink > > > Key: BEAM-7722 > URL: https://issues.apache.org/jira/browse/BEAM-7722 > Project: Beam > Issue Type: Test > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Fix For: 2.16.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > Currently this requires building and running several processes. We should be > able to automate most of this away. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-7962) Drop support for Flink 1.5 and 1.6
[
https://issues.apache.org/jira/browse/BEAM-7962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise updated BEAM-7962:
---
Issue Type: Task (was: Bug)
> Drop support for Flink 1.5 and 1.6
> --
>
> Key: BEAM-7962
> URL: https://issues.apache.org/jira/browse/BEAM-7962
> Project: Beam
> Issue Type: Task
> Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>
> With Flink 1.9 around the corner, we should consider dropping support for
> Flink 1.5 and 1.6.
> This will get rid of Flink 1.5/1.6 specific workarounds, e.g. make use of
> Flink's {{preSnapshotBarrier}} in {{AbstractStreamOperator}} which removes
> the needs to buffer elements during a snapshot.
> Dropping 1.5/1.6 will also decrease the build time.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (BEAM-7730) Add Flink 1.9 build target and Make FlinkRunner compatible with Flink 1.9
[ https://issues.apache.org/jira/browse/BEAM-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926685#comment-16926685 ] Thomas Weise commented on BEAM-7730: Removing 2.16 fix version, this is not a release blocker. > Add Flink 1.9 build target and Make FlinkRunner compatible with Flink 1.9 > - > > Key: BEAM-7730 > URL: https://issues.apache.org/jira/browse/BEAM-7730 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: David Moravek >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Apache Flink 1.9 will coming and it's better to add Flink 1.9 build target > and make Flink Runner compatible with Flink 1.9. > I will add the brief changes after the Flink 1.9.0 released. > And I appreciate it if you can leave your suggestions or comments! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-7730) Add Flink 1.9 build target and Make FlinkRunner compatible with Flink 1.9
[ https://issues.apache.org/jira/browse/BEAM-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-7730: --- Fix Version/s: (was: 2.16.0) > Add Flink 1.9 build target and Make FlinkRunner compatible with Flink 1.9 > - > > Key: BEAM-7730 > URL: https://issues.apache.org/jira/browse/BEAM-7730 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: David Moravek >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Apache Flink 1.9 will coming and it's better to add Flink 1.9 build target > and make Flink Runner compatible with Flink 1.9. > I will add the brief changes after the Flink 1.9.0 released. > And I appreciate it if you can leave your suggestions or comments! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7962) Drop support for Flink 1.5 and 1.6
[
https://issues.apache.org/jira/browse/BEAM-7962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926684#comment-16926684
]
Thomas Weise commented on BEAM-7962:
[~markflyhigh] this issue does not block the 2.16 release. I removed the fix
version.
> Drop support for Flink 1.5 and 1.6
> --
>
> Key: BEAM-7962
> URL: https://issues.apache.org/jira/browse/BEAM-7962
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>
> With Flink 1.9 around the corner, we should consider dropping support for
> Flink 1.5 and 1.6.
> This will get rid of Flink 1.5/1.6 specific workarounds, e.g. make use of
> Flink's {{preSnapshotBarrier}} in {{AbstractStreamOperator}} which removes
> the needs to buffer elements during a snapshot.
> Dropping 1.5/1.6 will also decrease the build time.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (BEAM-7962) Drop support for Flink 1.5 and 1.6
[
https://issues.apache.org/jira/browse/BEAM-7962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise updated BEAM-7962:
---
Fix Version/s: (was: 2.16.0)
> Drop support for Flink 1.5 and 1.6
> --
>
> Key: BEAM-7962
> URL: https://issues.apache.org/jira/browse/BEAM-7962
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>
> With Flink 1.9 around the corner, we should consider dropping support for
> Flink 1.5 and 1.6.
> This will get rid of Flink 1.5/1.6 specific workarounds, e.g. make use of
> Flink's {{preSnapshotBarrier}} in {{AbstractStreamOperator}} which removes
> the needs to buffer elements during a snapshot.
> Dropping 1.5/1.6 will also decrease the build time.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (BEAM-7980) External environment with containerized worker pool
[ https://issues.apache.org/jira/browse/BEAM-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-7980. Fix Version/s: 2.16.0 Resolution: Incomplete > External environment with containerized worker pool > --- > > Key: BEAM-7980 > URL: https://issues.apache.org/jira/browse/BEAM-7980 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Fix For: 2.16.0 > > Time Spent: 8h 50m > Remaining Estimate: 0h > > Augment Beam Python docker image and boot.go so that it can be used to launch > BeamFnExternalWorkerPoolServicer. > [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.lnhm75dhvhi0] > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8137) Worker pool option for Java SDK container
Thomas Weise created BEAM-8137: -- Summary: Worker pool option for Java SDK container Key: BEAM-8137 URL: https://issues.apache.org/jira/browse/BEAM-8137 Project: Beam Issue Type: Improvement Components: sdk-java-harness Reporter: Thomas Weise The worker pool option was added to the Python SDK container in BEAM-7980. Support in the Java SDK container is simpler since it can rely on threading and it should be added for feature parity. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8115) Overwrite portable Flink application jar pipeline options at runtime
[ https://issues.apache.org/jira/browse/BEAM-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919657#comment-16919657 ] Thomas Weise commented on BEAM-8115: I believe this is would have to include support for user pipeline options that are processed within the main entry point, i.e. that become properties of transforms. Examples include JDBC connection URL or Kafka broker address. > Overwrite portable Flink application jar pipeline options at runtime > > > Key: BEAM-8115 > URL: https://issues.apache.org/jira/browse/BEAM-8115 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > In the first iteration of portable Flink application jars, all pipeline > options are set at job creation time and cannot be later modified at runtime. > There should be a way to pass arguments to the jar to write/overwrite > pipeline options. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Closed] (BEAM-8038) Python Precommit fail: 'BeamFnExternalWorkerPoolServicer' has no attribute '_worker_processes'
[ https://issues.apache.org/jira/browse/BEAM-8038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-8038. -- Fix Version/s: Not applicable Resolution: Fixed > Python Precommit fail: 'BeamFnExternalWorkerPoolServicer' has no attribute > '_worker_processes' > -- > > Key: BEAM-8038 > URL: https://issues.apache.org/jira/browse/BEAM-8038 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness, test-failures >Reporter: Ahmet Altay >Assignee: Thomas Weise >Priority: Critical > Fix For: Not applicable > > Time Spent: 1h > Remaining Estimate: 0h > > Logs: https://builds.apache.org/job/beam_PreCommit_Python_Commit/8246/console > 10:14:09 > -- > 10:14:09 XML: > /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/nosetests.xml > 10:14:09 > -- > 10:14:09 Ran 2594 tests in 629.438s > 10:14:09 > 10:14:09 OK (SKIP=520) > 10:14:09 Error in atexit._run_exitfuncs: > 10:14:09 Traceback (most recent call last): > 10:14:09 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs > 10:14:09 func(*targs, **kargs) > 10:14:09 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/worker_pool_main.py", > line 72, in kill_worker_processes > 10:14:09 for worker_process in cls._worker_processes.values(): > 10:14:09 AttributeError: type object 'BeamFnExternalWorkerPoolServicer' has > no attribute '_worker_processes' > 10:14:09 Error in sys.exitfunc: > 10:14:09 Traceback (most recent call last): > 10:14:09 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs > 10:14:09 func(*targs, **kargs) > 10:14:09 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/worker_pool_main.py", > line 72, in kill_worker_processes > 10:14:09 for worker_process in cls._worker_processes.values(): > 10:14:09 AttributeError: type object 'BeamFnExternalWorkerPoolServicer' has > no attribute '_worker_processes' > 10:14:10 py27-cython run-test-post: commands[0] | > /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/scripts/run_tox_cleanup.sh > 10:14:10 ___ summary > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8038) Python Precommit fail: 'BeamFnExternalWorkerPoolServicer' has no attribute '_worker_processes'
[ https://issues.apache.org/jira/browse/BEAM-8038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913571#comment-16913571 ] Thomas Weise commented on BEAM-8038: Ouch, there is a self. missing. Will send a PR now. > Python Precommit fail: 'BeamFnExternalWorkerPoolServicer' has no attribute > '_worker_processes' > -- > > Key: BEAM-8038 > URL: https://issues.apache.org/jira/browse/BEAM-8038 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness, test-failures >Reporter: Ahmet Altay >Assignee: Thomas Weise >Priority: Critical > > Logs: https://builds.apache.org/job/beam_PreCommit_Python_Commit/8246/console > 10:14:09 > -- > 10:14:09 XML: > /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/nosetests.xml > 10:14:09 > -- > 10:14:09 Ran 2594 tests in 629.438s > 10:14:09 > 10:14:09 OK (SKIP=520) > 10:14:09 Error in atexit._run_exitfuncs: > 10:14:09 Traceback (most recent call last): > 10:14:09 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs > 10:14:09 func(*targs, **kargs) > 10:14:09 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/worker_pool_main.py", > line 72, in kill_worker_processes > 10:14:09 for worker_process in cls._worker_processes.values(): > 10:14:09 AttributeError: type object 'BeamFnExternalWorkerPoolServicer' has > no attribute '_worker_processes' > 10:14:09 Error in sys.exitfunc: > 10:14:09 Traceback (most recent call last): > 10:14:09 File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs > 10:14:09 func(*targs, **kargs) > 10:14:09 File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/worker_pool_main.py", > line 72, in kill_worker_processes > 10:14:09 for worker_process in cls._worker_processes.values(): > 10:14:09 AttributeError: type object 'BeamFnExternalWorkerPoolServicer' has > no attribute '_worker_processes' > 10:14:10 py27-cython run-test-post: commands[0] | > /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/scripts/run_tox_cleanup.sh > 10:14:10 ___ summary > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-7980) External environment with containerized worker pool
[ https://issues.apache.org/jira/browse/BEAM-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-7980: --- Description: Augment Beam Python docker image and boot.go so that it can be used to launch BeamFnExternalWorkerPoolServicer. [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.lnhm75dhvhi0] > External environment with containerized worker pool > --- > > Key: BEAM-7980 > URL: https://issues.apache.org/jira/browse/BEAM-7980 > Project: Beam > Issue Type: Improvement > Components: sdk-py-harness >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > > Augment Beam Python docker image and boot.go so that it can be used to launch > BeamFnExternalWorkerPoolServicer. > [https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.lnhm75dhvhi0] > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (BEAM-7980) External environment with containerized worker pool
Thomas Weise created BEAM-7980: -- Summary: External environment with containerized worker pool Key: BEAM-7980 URL: https://issues.apache.org/jira/browse/BEAM-7980 Project: Beam Issue Type: Improvement Components: sdk-py-harness Reporter: Thomas Weise Assignee: Thomas Weise -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (BEAM-7141) Expose kv and window parameters for on_timer
[ https://issues.apache.org/jira/browse/BEAM-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-7141. Resolution: Fixed Fix Version/s: 2.14.0 > Expose kv and window parameters for on_timer > > > Key: BEAM-7141 > URL: https://issues.apache.org/jira/browse/BEAM-7141 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.12.0 >Reporter: Thomas Weise >Assignee: Rakesh Kumar >Priority: Major > Fix For: 2.14.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > We would like to have access to key and window inside the timer callback. > Without, it is also difficult to debug. We run into this while working on > BEAM-7112 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (BEAM-7868) Hidden Flink Runner parameters are dropped in python pipelines
[ https://issues.apache.org/jira/browse/BEAM-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897703#comment-16897703 ] Thomas Weise commented on BEAM-7868: What means "hidden" and what do you expect to happen? > Hidden Flink Runner parameters are dropped in python pipelines > -- > > Key: BEAM-7868 > URL: https://issues.apache.org/jira/browse/BEAM-7868 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > > Hidden pipeline options for Portable flink runner are not interpreted by > Python SDK. > An example of this is > ManualDockerEnvironmentOptions.getRetainDockerContainers which is not > interpreted in python sdk. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Closed] (BEAM-7597) Typo: Correct "it's" to "its"
[ https://issues.apache.org/jira/browse/BEAM-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-7597. -- Resolution: Fixed Fix Version/s: Not applicable Thanks for the contribution! > Typo: Correct "it's" to "its" > - > > Key: BEAM-7597 > URL: https://issues.apache.org/jira/browse/BEAM-7597 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Riona MacNamara >Assignee: Alex Goos >Priority: Trivial > Labels: starter > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > On [https://beam.apache.org/documentation/], under *Choosing a runner*: > > Actual: "for configuring it’s execution" > Expected: "for configuring its execution" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-1318) PipelineOptions should warn if there are unused options
[ https://issues.apache.org/jira/browse/BEAM-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865001#comment-16865001 ] Thomas Weise commented on BEAM-1318: A warning is now logged when options are discarded: [https://github.com/apache/beam/blob/37b76b67b5d0cbd92e6a3fadee67f9fcf93cbc5d/sdks/python/apache_beam/options/pipeline_options.py#L261] I assume this resolves the issue, please reopen if not. > PipelineOptions should warn if there are unused options > --- > > Key: BEAM-1318 > URL: https://issues.apache.org/jira/browse/BEAM-1318 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Ahmet Altay >Priority: Minor > Labels: newbie, starter > Fix For: 2.11.0 > > > Since PipelineOptions uses argparse, it is possible that some options are > actually consumed by the program. In that case a better usage pattern would > be to pass only unconsumed options to PipelineOptions but we cannot enforce > this. > This cannot be an error because of the above reason, but we can show a > warning. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-1318) PipelineOptions should warn if there are unused options
[ https://issues.apache.org/jira/browse/BEAM-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-1318. -- Resolution: Done Fix Version/s: 2.11.0 > PipelineOptions should warn if there are unused options > --- > > Key: BEAM-1318 > URL: https://issues.apache.org/jira/browse/BEAM-1318 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Ahmet Altay >Priority: Minor > Labels: newbie, starter > Fix For: 2.11.0 > > > Since PipelineOptions uses argparse, it is possible that some options are > actually consumed by the program. In that case a better usage pattern would > be to pass only unconsumed options to PipelineOptions but we cannot enforce > this. > This cannot be an error because of the above reason, but we can show a > warning. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7348) Option to expire SDK worker environments
[ https://issues.apache.org/jira/browse/BEAM-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-7348. Resolution: Implemented Fix Version/s: 2.14.0 > Option to expire SDK worker environments > > > Key: BEAM-7348 > URL: https://issues.apache.org/jira/browse/BEAM-7348 > Project: Beam > Issue Type: Improvement > Components: runner-core >Affects Versions: 2.12.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability, portability-flink > Fix For: 2.14.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > We discovered that Python SDK workers are susceptible to memory leaks that > are quite hard to identify and/or fix. This becomes an issue in streaming > pipelines, where the workers run "forever". It would be good if the user has > an option to recycle the workers when there is no other practical way to > address (slow) resource leaks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-2591) Python shim for submitting to FlinkRunner
[ https://issues.apache.org/jira/browse/BEAM-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846080#comment-16846080 ] Thomas Weise commented on BEAM-2591: It is possible to spin up a job server and run Flink in embedded mode, we do it during testing. The runner choice is embedded into the job server though (which seems to make sense). To switch between runners, the user would launch the respective job server, manually or automated. > Python shim for submitting to FlinkRunner > - > > Key: BEAM-2591 > URL: https://issues.apache.org/jira/browse/BEAM-2591 > Project: Beam > Issue Type: Sub-task > Components: runner-flink, sdk-py-core >Reporter: Kenneth Knowles >Priority: Major > Labels: portability > Fix For: Not applicable > > > Whatever the result of https://s.apache.org/beam-job-api, Python users will > need to be able to pass --runner=FlinkRunner and have it work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-2591) Python shim for submitting to FlinkRunner
[ https://issues.apache.org/jira/browse/BEAM-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise closed BEAM-2591. -- Resolution: Resolved Fix Version/s: Not applicable Runners are implicitly identified through the job service endpoint, which the user supplies during pipeline submission. > Python shim for submitting to FlinkRunner > - > > Key: BEAM-2591 > URL: https://issues.apache.org/jira/browse/BEAM-2591 > Project: Beam > Issue Type: Sub-task > Components: runner-flink, sdk-py-core >Reporter: Kenneth Knowles >Priority: Major > Labels: portability > Fix For: Not applicable > > > Whatever the result of https://s.apache.org/beam-job-api, Python users will > need to be able to pass --runner=FlinkRunner and have it work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-7348) Option to expire SDK worker environments
[ https://issues.apache.org/jira/browse/BEAM-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7348 started by Thomas Weise. -- > Option to expire SDK worker environments > > > Key: BEAM-7348 > URL: https://issues.apache.org/jira/browse/BEAM-7348 > Project: Beam > Issue Type: Improvement > Components: runner-core >Affects Versions: 2.12.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability, portability-flink > > We discovered that Python SDK workers are susceptible to memory leaks that > are quite hard to identify and/or fix. This becomes an issue in streaming > pipelines, where the workers run "forever". It would be good if the user has > an option to recycle the workers when there is no other practical way to > address (slow) resource leaks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7348) Option to expire SDK worker environments
[ https://issues.apache.org/jira/browse/BEAM-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-7348: --- Status: Open (was: Triage Needed) > Option to expire SDK worker environments > > > Key: BEAM-7348 > URL: https://issues.apache.org/jira/browse/BEAM-7348 > Project: Beam > Issue Type: Improvement > Components: runner-core >Affects Versions: 2.12.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability, portability-flink > > We discovered that Python SDK workers are susceptible to memory leaks that > are quite hard to identify and/or fix. This becomes an issue in streaming > pipelines, where the workers run "forever". It would be good if the user has > an option to recycle the workers when there is no other practical way to > address (slow) resource leaks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7348) Option to expire SDK worker environments
Thomas Weise created BEAM-7348: -- Summary: Option to expire SDK worker environments Key: BEAM-7348 URL: https://issues.apache.org/jira/browse/BEAM-7348 Project: Beam Issue Type: Improvement Components: runner-core Affects Versions: 2.12.0 Reporter: Thomas Weise Assignee: Thomas Weise We discovered that Python SDK workers are susceptible to memory leaks that are quite hard to identify and/or fix. This becomes an issue in streaming pipelines, where the workers run "forever". It would be good if the user has an option to recycle the workers when there is no other practical way to address (slow) resource leaks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7126) Double encoding of state keys in portable Flink runner
[ https://issues.apache.org/jira/browse/BEAM-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839595#comment-16839595 ] Thomas Weise commented on BEAM-7126: [~mxm] should the fix version be 2.14 or are you going to add this to the 2.13 branch? > Double encoding of state keys in portable Flink runner > -- > > Key: BEAM-7126 > URL: https://issues.apache.org/jira/browse/BEAM-7126 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.11.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability-flink > Fix For: 2.13.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > State keys currently need to be encoded as NESTED. My attempt to use the > ByteString directly in BEAM-7112 caused checkpointing to fail. We should look > into eliminating the redundant key encoding and adjusting > StateRequestHandlers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7171) New bundles may start within snapshotState
[ https://issues.apache.org/jira/browse/BEAM-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-7171: --- Labels: portability-flink (was: ) > New bundles may start within snapshotState > -- > > Key: BEAM-7171 > URL: https://issues.apache.org/jira/browse/BEAM-7171 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.11.0 >Reporter: Thomas Weise >Priority: Major > Labels: portability-flink > > The Flink runner finishes bundles as part of snapshotState. In the portable > runner, it is possible that a new bundle will be started as part of finishing > the bundle when the bundleFinishedCallback is invoked. This happens when the > watermark advances and timers get fired. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7171) New bundles may start within snapshotState
[ https://issues.apache.org/jira/browse/BEAM-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829270#comment-16829270 ] Thomas Weise commented on BEAM-7171: A possible solution is to replace the "finish bundle callback" with a "post snapshot callback" and advance the watermark only then. No additional state needs to be part of the snapshot, since the next watermark is certain to follow. > New bundles may start within snapshotState > -- > > Key: BEAM-7171 > URL: https://issues.apache.org/jira/browse/BEAM-7171 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.11.0 >Reporter: Thomas Weise >Priority: Major > > The Flink runner finishes bundles as part of snapshotState. In the portable > runner, it is possible that a new bundle will be started as part of finishing > the bundle when the bundleFinishedCallback is invoked. This happens when the > watermark advances and timers get fired. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7171) New bundles may start within snapshotState
Thomas Weise created BEAM-7171: -- Summary: New bundles may start within snapshotState Key: BEAM-7171 URL: https://issues.apache.org/jira/browse/BEAM-7171 Project: Beam Issue Type: Bug Components: runner-flink Affects Versions: 2.11.0 Reporter: Thomas Weise The Flink runner finishes bundles as part of snapshotState. In the portable runner, it is possible that a new bundle will be started as part of finishing the bundle when the bundleFinishedCallback is invoked. This happens when the watermark advances and timers get fired. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7141) Expose kv and window parameters for on_timer
[ https://issues.apache.org/jira/browse/BEAM-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825522#comment-16825522 ] Thomas Weise commented on BEAM-7141: WindowedValue passed from Flink side: [https://github.com/apache/beam/blob/b23ef6432935970d6a568a632903ac0d70eb2ec0/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java#L647] > Expose kv and window parameters for on_timer > > > Key: BEAM-7141 > URL: https://issues.apache.org/jira/browse/BEAM-7141 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Affects Versions: 2.12.0 >Reporter: Thomas Weise >Priority: Major > > We would like to have access to key and window inside the timer callback. > Without, it is also difficult to debug. We run into this while working on > BEAM-7112 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7141) Expose kv and window parameters for on_timer
Thomas Weise created BEAM-7141: -- Summary: Expose kv and window parameters for on_timer Key: BEAM-7141 URL: https://issues.apache.org/jira/browse/BEAM-7141 Project: Beam Issue Type: Improvement Components: sdk-py-core Affects Versions: 2.12.0 Reporter: Thomas Weise We would like to have access to key and window inside the timer callback. Without, it is also difficult to debug. We run into this while working on BEAM-7112 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7015) Have only a single definition of standard_coders.yaml
[ https://issues.apache.org/jira/browse/BEAM-7015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise resolved BEAM-7015. Resolution: Fixed Fix Version/s: 2.13.0 > Have only a single definition of standard_coders.yaml > - > > Key: BEAM-7015 > URL: https://issues.apache.org/jira/browse/BEAM-7015 > Project: Beam > Issue Type: Improvement > Components: beam-model, sdk-py-core >Reporter: Luke Cwik >Assignee: Thomas Weise >Priority: Major > Labels: portability, triaged > Fix For: 2.13.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > There are two copies of standard_coders.yaml defined: > * > https://github.com/apache/beam/blob/master/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml > * > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/data/standard_coders.yaml > The Python SDK specific instance should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7058) Python SDK metric process_bundle_msecs reported as zero
[ https://issues.apache.org/jira/browse/BEAM-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825149#comment-16825149 ] Thomas Weise commented on BEAM-7058: [~ajam...@google.com] any update? > Python SDK metric process_bundle_msecs reported as zero > --- > > Key: BEAM-7058 > URL: https://issues.apache.org/jira/browse/BEAM-7058 > Project: Beam > Issue Type: Bug > Components: runner-flink, sdk-py-harness >Reporter: Thomas Weise >Assignee: Alex Amato >Priority: Major > Labels: portability-flink, triaged > > With the portable Flink runner, the metric is reported as 0, while the count > metric works as expected. > [https://lists.apache.org/thread.html/25eec8104bda6e4c71cc6c5e9864c335833c3ae2afe225d372479f30@%3Cdev.beam.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7127) Timer parameter not supported in timer callback
[
https://issues.apache.org/jira/browse/BEAM-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823642#comment-16823642
]
Thomas Weise commented on BEAM-7127:
[~altay] the issue can be reproduced with the Flink runner:
[https://gist.github.com/tweise/7405004c94b913c70c69393029b73e25]
> Timer parameter not supported in timer callback
> ---
>
> Key: BEAM-7127
> URL: https://issues.apache.org/jira/browse/BEAM-7127
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Priority: Major
> Labels: portability
>
> Referencing the timer in its on_timer callback produces a recursive pickle
> error.
> {code:java}
> @userstate.on_timer(timer_spec)
> def process_timer(self, timer_1=beam.DoFn.TimerParam(timer_spec)):
> {code}
> Unit test:
> [https://github.com/apache/beam/blob/cbe4dfbdbe5d0da5152568853ee5e17334dd1b54/sdks/python/apache_beam/transforms/userstate_test.py#L67]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7127) Timer parameter not supported in timer callback
[
https://issues.apache.org/jira/browse/BEAM-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822730#comment-16822730
]
Thomas Weise commented on BEAM-7127:
{code:java}
File
"/Users/tweise/src/beam/sdks/python/apache_beam/examples/flink/flink_state.py",
line 78, in run
| 'statefulCount' >> beam.ParDo(StateTimerFn())
File "apache_beam/transforms/core.py", line 979, in __init__
super(ParDo, self).__init__(fn, *args, **kwargs)
File "apache_beam/transforms/ptransform.py", line 689, in __init__
self.fn = pickler.loads(pickler.dumps(self.fn))
File "apache_beam/internal/pickler.py", line 230, in dumps
s = dill.dumps(o)
File
"/Users/tweise/python-ve/beam/lib/python2.7/site-packages/dill/_dill.py", line
294, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
File
"/Users/tweise/python-ve/beam/lib/python2.7/site-packages/dill/_dill.py", line
287, in dump
pik.dump(obj)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 224, in dump
self.save(obj)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 331, in save
self.save_reduce(obj=obj, *rv)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 396, in save_reduce
save(cls)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 286, in save
f(self, obj) # Call unbound method with explicit self
File "apache_beam/internal/pickler.py", line 107, in wrapper
return fun(pickler, obj)
File
"/Users/tweise/python-ve/beam/lib/python2.7/site-packages/dill/_dill.py", line
1315, in save_type
obj.__bases__, _dict), obj=obj)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 401, in save_reduce
save(args)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 286, in save
f(self, obj) # Call unbound method with explicit self
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 562, in save_tuple
save(element)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 286, in save
f(self, obj) # Call unbound method with explicit self
File "apache_beam/internal/pickler.py", line 198, in new_save_module_dict
return old_save_module_dict(pickler, obj)
File
"/Users/tweise/python-ve/beam/lib/python2.7/site-packages/dill/_dill.py", line
902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 649, in save_dict
self._batch_setitems(obj.iteritems())
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 681, in _batch_setitems
save(v)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 286, in save
f(self, obj) # Call unbound method with explicit self
File
"/Users/tweise/python-ve/beam/lib/python2.7/site-packages/dill/_dill.py", line
1394, in save_function
obj.__dict__), obj=obj)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 405, in save_reduce
self.memoize(obj)
File
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
line 244, in memoize
assert id(obj) not in self.memo
AssertionError{code}
> Timer parameter not supported in timer callback
> ---
>
> Key: BEAM-7127
> URL: https://issues.apache.org/jira/browse/BEAM-7127
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Priority: Major
> Labels: portability
>
> Referencing the timer in its on_timer callback produces a recursive pickle
> error.
> {code:java}
> @userstate.on_timer(timer_spec)
> def process_timer(self, timer_1=beam.DoFn.TimerParam(timer_spec)):
> {code}
> Unit test:
> [https://github.com/apache/beam/blob/cbe4dfbdbe5d0da5152568853ee5e17334dd1b54/sdks/python/apache_beam/transforms/userstate_test.py#L67]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7127) Timer parameter not supported in timer callback
Thomas Weise created BEAM-7127:
--
Summary: Timer parameter not supported in timer callback
Key: BEAM-7127
URL: https://issues.apache.org/jira/browse/BEAM-7127
Project: Beam
Issue Type: Bug
Components: sdk-py-harness
Affects Versions: 2.12.0
Reporter: Thomas Weise
Referencing the timer in its on_timer callback produces a recursive pickle
error.
{code:java}
@userstate.on_timer(timer_spec)
def process_timer(self, timer_1=beam.DoFn.TimerParam(timer_spec)):
{code}
Unit test:
[https://github.com/apache/beam/blob/cbe4dfbdbe5d0da5152568853ee5e17334dd1b54/sdks/python/apache_beam/transforms/userstate_test.py#L67]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6876) User state cleanup in portable Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822584#comment-16822584 ] Thomas Weise commented on BEAM-6876: Bumped the fix version since this is still broken in 2.12 > User state cleanup in portable Flink runner > --- > > Key: BEAM-6876 > URL: https://issues.apache.org/jira/browse/BEAM-6876 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.11.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability-flink, triaged > Fix For: 2.13.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > > State is currently not being cleaned up by the runner. > [https://lists.apache.org/thread.html/86f0809fbfa3da873051287b9ff249d6dd5a896b45409db1e484cf38@%3Cdev.beam.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6876) User state cleanup in portable Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-6876: --- Fix Version/s: (was: 2.12.0) 2.13.0 > User state cleanup in portable Flink runner > --- > > Key: BEAM-6876 > URL: https://issues.apache.org/jira/browse/BEAM-6876 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.11.0 >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability-flink, triaged > Fix For: 2.13.0 > > Time Spent: 7h 10m > Remaining Estimate: 0h > > State is currently not being cleaned up by the runner. > [https://lists.apache.org/thread.html/86f0809fbfa3da873051287b9ff249d6dd5a896b45409db1e484cf38@%3Cdev.beam.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7126) Double encoding of state keys in portable Flink runner
Thomas Weise created BEAM-7126: -- Summary: Double encoding of state keys in portable Flink runner Key: BEAM-7126 URL: https://issues.apache.org/jira/browse/BEAM-7126 Project: Beam Issue Type: Bug Components: runner-flink Affects Versions: 2.11.0 Reporter: Thomas Weise State keys currently need to be encoded as NESTED. My attempt to use the ByteString directly in BEAM-7112 caused checkpointing to fail. We should look into eliminating the redundant key encoding and adjusting StateRequestHandlers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7112) State cleanup interferes with user timer callback
Thomas Weise created BEAM-7112: -- Summary: State cleanup interferes with user timer callback Key: BEAM-7112 URL: https://issues.apache.org/jira/browse/BEAM-7112 Project: Beam Issue Type: Bug Components: runner-flink Affects Versions: 2.12.0 Reporter: Thomas Weise Assignee: Thomas Weise Cleanup timers and user timers are fired at the watermark. Processing of timers in the SDK worker is asynchronous, so it is possible that the state is already removed when the user timer callback executes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated BEAM-7035: --- Component/s: (was: beam-model) sdk-py-harness > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Fix For: 2.13.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
