[jira] [Created] (BEAM-10222) Java11 only detects 1 CPU when running on Dataflow
Mikhail Gryzykhin created BEAM-10222: Summary: Java11 only detects 1 CPU when running on Dataflow Key: BEAM-10222 URL: https://issues.apache.org/jira/browse/BEAM-10222 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Mikhail Gryzykhin When specifying the worker-type as "n1-highcpu-4" the resulting thread number is 4 when the pipeline is compiled by Java8, and only 1 when the pipeline is compiled by Java11. Specifying numberOfWorkerHarnessThreads does not change amount of detected threads. Amount of available CPUs is received via "Runtime.getRuntime().availableProcessors()" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9650) Add consistent slowly changing side inputs support
[ https://issues.apache.org/jira/browse/BEAM-9650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-9650: --- Assignee: Mikhail Gryzykhin > Add consistent slowly changing side inputs support > -- > > Key: BEAM-9650 > URL: https://issues.apache.org/jira/browse/BEAM-9650 > Project: Beam > Issue Type: Bug > Components: io-ideas >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > > Add implementation for slowly changing dimentions based on [design > doc](https://docs.google.com/document/d/1LDY_CtsOJ8Y_zNv1QtkP6AGFrtzkj1q5EW_gSChOIvg/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9650) Add consistent slowly changing side inputs support
Mikhail Gryzykhin created BEAM-9650: --- Summary: Add consistent slowly changing side inputs support Key: BEAM-9650 URL: https://issues.apache.org/jira/browse/BEAM-9650 Project: Beam Issue Type: Bug Components: io-ideas Reporter: Mikhail Gryzykhin Add implementation for slowly changing dimentions based on [design doc](https://docs.google.com/document/d/1LDY_CtsOJ8Y_zNv1QtkP6AGFrtzkj1q5EW_gSChOIvg/edit] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9481) Expansion-service fails when executed with --verify
[ https://issues.apache.org/jira/browse/BEAM-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-9481: Description: Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:330) at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:263) at java.util.jar.JarVerifier.processEntry(JarVerifier.java:275) at java.util.jar.JarVerifier.update(JarVerifier.java:230) at java.util.jar.JarFile.initializeVerifier(JarFile.java:385) at java.util.jar.JarFile.getInputStream(JarFile.java:452) at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:1252) at sun.misc.Resource.cachedInputStream(Resource.java:77) at sun.misc.Resource.getByteBuffer(Resource.java:160) at java.net.URLClassLoader.defineClass(URLClassLoader.java:454) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:350) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:491) was:Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:330) at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:263) at java.util.jar.JarVerifier.processEntry(JarVerifier.java:275) at java.util.jar.JarVerifier.update(JarVerifier.java:230) at java.util.jar.JarFile.initializeVerifier(JarFile.java:385) at java.util.jar.JarFile.getInputStream(JarFile.java:452) at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:1252) at sun.misc.Resource.cachedInputStream(Resource.java:77) at sun.misc.Resource.getByteBuffer(Resource.java:160) at java.net.URLClassLoader.defineClass(URLClassLoader.java:454) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:350) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:491) > Expansion-service fails when executed with --verify > --- > > Key: BEAM-9481 > URL: https://issues.apache.org/jira/browse/BEAM-9481 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Mikhail Gryzykhin >Assignee: Brian Hulette >Priority: Major > > > Error: A JNI error has occurred, please check your installation and try again > Exception in thread "main" java.lang.SecurityException: Invalid signature > file digest for Manifest main attributes > at > sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:330) > at > sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:263) > at java.util.jar.JarVerifier.processEntry(JarVerifier.java:275) > at java.util.jar.JarVerifier.update(JarVerifier.java:230) > at java.util.jar.JarFile.initializeVerifier(JarFile.java:385) > at java.util.jar.JarFile.getInputStream(JarFile.java:452) > at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:1252) > at sun.misc.Resource.cachedInputStream(Resource.java:77) > at sun.misc.Resource.getByteBuffer(Resource.java:160) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:454) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:350) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:491)
[jira] [Created] (BEAM-9481) Expansion-service fails when executed with --verify
Mikhail Gryzykhin created BEAM-9481: --- Summary: Expansion-service fails when executed with --verify Key: BEAM-9481 URL: https://issues.apache.org/jira/browse/BEAM-9481 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Mikhail Gryzykhin Assignee: Brian Hulette Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:330) at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:263) at java.util.jar.JarVerifier.processEntry(JarVerifier.java:275) at java.util.jar.JarVerifier.update(JarVerifier.java:230) at java.util.jar.JarFile.initializeVerifier(JarFile.java:385) at java.util.jar.JarFile.getInputStream(JarFile.java:452) at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:1252) at sun.misc.Resource.cachedInputStream(Resource.java:77) at sun.misc.Resource.getByteBuffer(Resource.java:160) at java.net.URLClassLoader.defineClass(URLClassLoader.java:454) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:350) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:491) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8919) Move JAVA_11_HOME and JAVA_8_HOME variables to Jenkins envs.
[ https://issues.apache.org/jira/browse/BEAM-8919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030072#comment-17030072 ] Mikhail Gryzykhin commented on BEAM-8919: - [~markliu] Can you take a look? > Move JAVA_11_HOME and JAVA_8_HOME variables to Jenkins envs. > > > Key: BEAM-8919 > URL: https://issues.apache.org/jira/browse/BEAM-8919 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Lukasz Gajowy >Priority: Minor > > Some tests that use different java versions rely on the following paths to > java home: > final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64' > final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64' > > The paths itself should be held as jenkins env variables. Benefits: > - easier to reuse > - no room for typo in the path > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8780) Improve publish_docker_images.sh
[ https://issues.apache.org/jira/browse/BEAM-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009217#comment-17009217 ] Mikhail Gryzykhin commented on BEAM-8780: - Good sources [https://dille.name/blog/2018/09/20/how-to-tag-docker-images-without-pulling-them/] [https://stackoverflow.com/questions/48363572/how-to-add-tag-alias-on-docker-hub-without-pulling-an-image] Rebuilding image is incorrect since version of compiler might change. > Improve publish_docker_images.sh > > > Key: BEAM-8780 > URL: https://issues.apache.org/jira/browse/BEAM-8780 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Mark Liu >Assignee: Hannah Jiang >Priority: Major > > Few improvement places: > * The script builds new image for publishing. Better to reuse existing > staged rc images. > * This script ask input that’s unrelated to publish (i.e rc #, gpg key...) > * This script first run unrelated staging steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8478) Create WatermarkEstimator for tracking watermark
[ https://issues.apache.org/jira/browse/BEAM-8478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-8478: Summary: Create WatermarkEstimator for tracking watermark (was: Create WatermarkEstomator for tracking watermark) > Create WatermarkEstimator for tracking watermark > > > Key: BEAM-8478 > URL: https://issues.apache.org/jira/browse/BEAM-8478 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, sdk-py-harness >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: Major > Fix For: 2.17.0 > > > Instead of tracking watermark in RestrictionTracker, use a WatermarkEstimator > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work stopped] (BEAM-8981) Test issue
[ https://issues.apache.org/jira/browse/BEAM-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-8981 stopped by Mikhail Gryzykhin. --- > Test issue > -- > > Key: BEAM-8981 > URL: https://issues.apache.org/jira/browse/BEAM-8981 > Project: Beam > Issue Type: Bug > Components: community-metrics >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-8981) Test issue
[ https://issues.apache.org/jira/browse/BEAM-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-8981 started by Mikhail Gryzykhin. --- > Test issue > -- > > Key: BEAM-8981 > URL: https://issues.apache.org/jira/browse/BEAM-8981 > Project: Beam > Issue Type: Bug > Components: community-metrics >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8981) Test issue
[ https://issues.apache.org/jira/browse/BEAM-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-8981: Issue Type: Bug (was: Improvement) > Test issue > -- > > Key: BEAM-8981 > URL: https://issues.apache.org/jira/browse/BEAM-8981 > Project: Beam > Issue Type: Bug > Components: community-metrics >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8981) Test issue
[ https://issues.apache.org/jira/browse/BEAM-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-8981: --- Assignee: Mikhail Gryzykhin > Test issue > -- > > Key: BEAM-8981 > URL: https://issues.apache.org/jira/browse/BEAM-8981 > Project: Beam > Issue Type: Improvement > Components: community-metrics >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8981) Test issue
[ https://issues.apache.org/jira/browse/BEAM-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-8981: Status: Open (was: Triage Needed) > Test issue > -- > > Key: BEAM-8981 > URL: https://issues.apache.org/jira/browse/BEAM-8981 > Project: Beam > Issue Type: Improvement > Components: community-metrics >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-8981) Test issue
[ https://issues.apache.org/jira/browse/BEAM-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-8981 started by Mikhail Gryzykhin. --- > Test issue > -- > > Key: BEAM-8981 > URL: https://issues.apache.org/jira/browse/BEAM-8981 > Project: Beam > Issue Type: Improvement > Components: community-metrics >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8981) Test issue
Mikhail Gryzykhin created BEAM-8981: --- Summary: Test issue Key: BEAM-8981 URL: https://issues.apache.org/jira/browse/BEAM-8981 Project: Beam Issue Type: Improvement Components: community-metrics Reporter: Mikhail Gryzykhin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8363) Nexmark regression in direct runner in streaming mode
[ https://issues.apache.org/jira/browse/BEAM-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986415#comment-16986415 ] Mikhail Gryzykhin commented on BEAM-8363: - Synced with Ken. It is worth moving this to 2.18+ > Nexmark regression in direct runner in streaming mode > - > > Key: BEAM-8363 > URL: https://issues.apache.org/jira/browse/BEAM-8363 > Project: Beam > Issue Type: Bug > Components: runner-direct >Affects Versions: 2.16.0 >Reporter: Mark Liu >Assignee: Kenneth Knowles >Priority: Major > Attachments: regression_screenshot.png > > > From Nexmark performance dashboard for direct runner: > https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424, > there is a regression for streaming mode happened on August 25. > The runtime increased about 25% in two days. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8835) Artifact retrieval fails with FlinkUberJarJobServer
[ https://issues.apache.org/jira/browse/BEAM-8835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986403#comment-16986403 ] Mikhail Gryzykhin commented on BEAM-8835: - Synced with Kyle. Lets move this feature to 2.18 that is close. > Artifact retrieval fails with FlinkUberJarJobServer > --- > > Key: BEAM-8835 > URL: https://issues.apache.org/jira/browse/BEAM-8835 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kyle Weaver >Priority: Major > Fix For: 2.17.0 > > > We seem to be able to stage artifacts and retrieve the manifest fine, but > retrieving the artifacts doesn't work. This happens on both my k8s Flink > cluster and on my local Flink cluster. At a quick glance the artifact is in > the jar where it should be. cc [~robertwb] > 2019-11-21 18:43:39,336 INFO > org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService > - GetArtifact name: "pickled_main_session" > retrieval_token: "BEAM-PIPELINE/pipeline/artifact-manifest.json" > failed > java.io.IOException: Unable to load > e1d24d848414cecf805a7b5c2b950c6430c20eb32875dac00b40f80f3c73a141/ea0d10d07f4601782ed647e8f6ba4a055be13674ab79fa0c6e2fa44917c5264c > with > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader@785297ac -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8812) [release 2.17.0 validation] org.apache.beam.sdk.transforms.ParDoTest$TimerTests.testEventTimeTimerOrderingWithCreate failure
[ https://issues.apache.org/jira/browse/BEAM-8812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin resolved BEAM-8812. - Resolution: Fixed > [release 2.17.0 validation] > org.apache.beam.sdk.transforms.ParDoTest$TimerTests.testEventTimeTimerOrderingWithCreate > failure > > > Key: BEAM-8812 > URL: https://issues.apache.org/jira/browse/BEAM-8812 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Assignee: Jan Lukavský >Priority: Major > Fix For: 2.17.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Job: > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow_PR/61/testReport/junit/org.apache.beam.sdk.transforms/ParDoTest$TimerTests/testEventTimeTimerOrderingWithCreate/] > Error from log: > INFO: Dataflow job 2019-11-21_13_37_48-7347702521640752127 threw exception. > Failure message was: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: Error received from SDK harness for instruction > -672: java.lang.IllegalArgumentException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8812) [release 2.17.0 validation] org.apache.beam.sdk.transforms.ParDoTest$TimerTests.testEventTimeTimerOrderingWithCreate failure
Mikhail Gryzykhin created BEAM-8812: --- Summary: [release 2.17.0 validation] org.apache.beam.sdk.transforms.ParDoTest$TimerTests.testEventTimeTimerOrderingWithCreate failure Key: BEAM-8812 URL: https://issues.apache.org/jira/browse/BEAM-8812 Project: Beam Issue Type: Bug Components: java-fn-execution Reporter: Mikhail Gryzykhin Fix For: 2.17.0 Job: [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow_PR/61/testReport/junit/org.apache.beam.sdk.transforms/ParDoTest$TimerTests/testEventTimeTimerOrderingWithCreate/] Error from log: INFO: Dataflow job 2019-11-21_13_37_48-7347702521640752127 threw exception. Failure message was: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error received from SDK harness for instruction -672: java.lang.IllegalArgumentException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8812) [release 2.17.0 validation] org.apache.beam.sdk.transforms.ParDoTest$TimerTests.testEventTimeTimerOrderingWithCreate failure
[ https://issues.apache.org/jira/browse/BEAM-8812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980546#comment-16980546 ] Mikhail Gryzykhin commented on BEAM-8812: - I'm trying to repro locally, but target fails to publish containers correctly. > [release 2.17.0 validation] > org.apache.beam.sdk.transforms.ParDoTest$TimerTests.testEventTimeTimerOrderingWithCreate > failure > > > Key: BEAM-8812 > URL: https://issues.apache.org/jira/browse/BEAM-8812 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Mikhail Gryzykhin >Priority: Major > Fix For: 2.17.0 > > > Job: > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow_PR/61/testReport/junit/org.apache.beam.sdk.transforms/ParDoTest$TimerTests/testEventTimeTimerOrderingWithCreate/] > Error from log: > INFO: Dataflow job 2019-11-21_13_37_48-7347702521640752127 threw exception. > Failure message was: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: Error received from SDK harness for instruction > -672: java.lang.IllegalArgumentException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8791) [2.17.0 Release Validation] Run Python PreCommit times out
Mikhail Gryzykhin created BEAM-8791: --- Summary: [2.17.0 Release Validation] Run Python PreCommit times out Key: BEAM-8791 URL: https://issues.apache.org/jira/browse/BEAM-8791 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Mikhail Gryzykhin Assignee: Valentyn Tymofieiev Fix For: 2.17.0 Code version: [https://github.com/apache/beam/pull/9884/commits/d52355c9712b5ed85900a941f50317c5ab9252cb] [Job: https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1051/|https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1051/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8788) [2.17.0 Release Validation] ParDoTest$TimerTests.testEventTimeTimerOrderingWithCreate Failed
Mikhail Gryzykhin created BEAM-8788: --- Summary: [2.17.0 Release Validation] ParDoTest$TimerTests.testEventTimeTimerOrderingWithCreate Failed Key: BEAM-8788 URL: https://issues.apache.org/jira/browse/BEAM-8788 Project: Beam Issue Type: Bug Components: runner-flink Reporter: Mikhail Gryzykhin Assignee: Maximilian Michels Fix For: 2.17.0 This blocks release 2.17.0 validation. Code version: [https://github.com/apache/beam/pull/9884/commits/d52355c9712b5ed85900a941f50317c5ab9252cb] Job: [https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch_PR/96/#showFailuresLink] Relevant logs: java.lang.AssertionError: Pipeline did not succeed. Expected: is but: was below in logs last entries are: INFO: Manifest at /tmp/beam-artifact-staging/pardotest0timertests0testpipelineoptionsparameterontimer-jenkins-1120162011-bb7dcc89_8b863b38-0c73-4cc7-b515-93055e0a76d0/MANIFEST has 207 artifact locations Nov 20, 2019 4:20:15 PM org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService removeArtifacts INFO: Removed dir /tmp/beam-artifact-staging/pardotest0timertests0testpipelineoptionsparameterontimer-jenkins-1120162011-bb7dcc89_8b863b38-0c73-4cc7-b515-93055e0a76d0/ Nov 20, 2019 4:20:22 PM org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver stop INFO: JobServer stopped on localhost:46137 Nov 20, 2019 4:20:22 PM org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver stop INFO: ArtifactStagingServer stopped on localhost:32991 Nov 20, 2019 4:20:22 PM org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver stop INFO: Expansion stopped on localhost:39239 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8506) Failing Dataflow job should explicitly state reason to be missing container.
Mikhail Gryzykhin created BEAM-8506: --- Summary: Failing Dataflow job should explicitly state reason to be missing container. Key: BEAM-8506 URL: https://issues.apache.org/jira/browse/BEAM-8506 Project: Beam Issue Type: Improvement Components: runner-dataflow Reporter: Mikhail Gryzykhin [https://builds.apache.org/job/beam_PreCommit_Python_Phrase/903/] Currenlty job timeouts or states that no values were returned. Job should fail with explicit reason of "container missing". Current relevant log is description of job containing: "experiments": [ "use_staged_dataflow_worker_jar", "use_fastavro", "runner_harness_container_image=gcr.io/cloud-dataflow/v1beta3/harness:2.17.0", "use_multiple_sdk_containers" ], Following container image path shows us that container is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8413) test_streaming_pipeline_returns_expected_user_metrics_fnapi_it failed on latest PostCommit Py36
[ https://issues.apache.org/jira/browse/BEAM-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-8413: Status: Open (was: Triage Needed) > test_streaming_pipeline_returns_expected_user_metrics_fnapi_it failed on > latest PostCommit Py36 > - > > Key: BEAM-8413 > URL: https://issues.apache.org/jira/browse/BEAM-8413 > Project: Beam > Issue Type: New Feature > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Ankur Goenka >Priority: Major > > https://builds.apache.org/job/beam_PostCommit_Python36/731/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8413) test_streaming_pipeline_returns_expected_user_metrics_fnapi_it failed on latest PostCommit Py36
[ https://issues.apache.org/jira/browse/BEAM-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954041#comment-16954041 ] Mikhail Gryzykhin commented on BEAM-8413: - Current suspect that we read more elements from pubsub than expected. Ankur was looking into this. > test_streaming_pipeline_returns_expected_user_metrics_fnapi_it failed on > latest PostCommit Py36 > - > > Key: BEAM-8413 > URL: https://issues.apache.org/jira/browse/BEAM-8413 > Project: Beam > Issue Type: New Feature > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Mikhail Gryzykhin >Priority: Major > > https://builds.apache.org/job/beam_PostCommit_Python36/731/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6816) More PR information in the code velocity dashboard?
[ https://issues.apache.org/jira/browse/BEAM-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-6816: Component/s: community-metrics > More PR information in the code velocity dashboard? > --- > > Key: BEAM-6816 > URL: https://issues.apache.org/jira/browse/BEAM-6816 > Project: Beam > Issue Type: Improvement > Components: community-metrics, website >Reporter: Pablo Estrada >Priority: Major > > I've been using this dashboard: > http://104.154.241.245/d/code_velocity/code-velocity?orgId=1 > To triage and get PRs reviewed. Some of them have DO NOT REVIEW / or special > instructions in the title. > Perhaps it would be nice to have more information displayed on the dashboard, > so we can skip PRs that don't need to be reviewed. > As you rightly mentioned, another idea is to exclude PRs that have a specific > tag - and I think that's a good idea in fact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6710) Add Landing page to community metrics dashboard
[ https://issues.apache.org/jira/browse/BEAM-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-6710: Component/s: community-metrics > Add Landing page to community metrics dashboard > > > Key: BEAM-6710 > URL: https://issues.apache.org/jira/browse/BEAM-6710 > Project: Beam > Issue Type: New Feature > Components: community-metrics, project-management >Reporter: Mikhail Gryzykhin >Priority: Major > > Community metrics dashboard sends user to list of recently opened dashboards, > that's empty. This confuses new users. > We want to add landing page with links to relevant dashboard. > Link: ttp://104.154.241.245/ > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5923) Utilize common .pylintrc with python sdk for .test-infra/metrics
[ https://issues.apache.org/jira/browse/BEAM-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-5923: Component/s: community-metrics > Utilize common .pylintrc with python sdk for .test-infra/metrics > > > Key: BEAM-5923 > URL: https://issues.apache.org/jira/browse/BEAM-5923 > Project: Beam > Issue Type: Sub-task > Components: community-metrics, project-management >Reporter: Mikhail Gryzykhin >Priority: Major > > Add common linter and formatter to metrics code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5863) Automate Community Metrics infrastructure deployment
[ https://issues.apache.org/jira/browse/BEAM-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-5863: Component/s: community-metrics > Automate Community Metrics infrastructure deployment > > > Key: BEAM-5863 > URL: https://issues.apache.org/jira/browse/BEAM-5863 > Project: Beam > Issue Type: Sub-task > Components: community-metrics, project-management >Reporter: Scott Wegner >Priority: Minor > Labels: community-metrics > > Currently the deployment process for the production Community Metrics stack > is manual (documented > [here|https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics]). > If we end up having to deploy more than a few times a year, it would be nice > to automate these steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8328) remove :beam-test-infra-metrics:test from build target.
[ https://issues.apache.org/jira/browse/BEAM-8328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-8328: Component/s: community-metrics > remove :beam-test-infra-metrics:test from build target. > --- > > Key: BEAM-8328 > URL: https://issues.apache.org/jira/browse/BEAM-8328 > Project: Beam > Issue Type: Bug > Components: community-metrics, project-management >Reporter: Mikhail Gryzykhin >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8327) beam_Prober_CommunityMetrics hits cache giving wrong results
[ https://issues.apache.org/jira/browse/BEAM-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-8327: Labels: Triaged (was: ) > beam_Prober_CommunityMetrics hits cache giving wrong results > > > Key: BEAM-8327 > URL: https://issues.apache.org/jira/browse/BEAM-8327 > Project: Beam > Issue Type: Bug > Components: community-metrics, project-management, testing >Reporter: Mikhail Gryzykhin >Priority: Major > Labels: Triaged > > We need to fix beam_Prober_CommunityMetrics target to not hit cache. It > always fetches fresh data from website even though binaries are the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-5840) Add unit tests for import scripts
[ https://issues.apache.org/jira/browse/BEAM-5840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-5840: --- Assignee: (was: Mikhail Gryzykhin) > Add unit tests for import scripts > - > > Key: BEAM-5840 > URL: https://issues.apache.org/jira/browse/BEAM-5840 > Project: Beam > Issue Type: Sub-task > Components: project-management >Reporter: Scott Wegner >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > For custom import scripts, we should refactor the logic to be testable and > add some basic unit tests. > At the very least, there should be some "smoke test" that the local > deployment works. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-5923) Utilize common .pylintrc with python sdk for .test-infra/metrics
[ https://issues.apache.org/jira/browse/BEAM-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-5923: --- Assignee: (was: Mikhail Gryzykhin) > Utilize common .pylintrc with python sdk for .test-infra/metrics > > > Key: BEAM-5923 > URL: https://issues.apache.org/jira/browse/BEAM-5923 > Project: Beam > Issue Type: Sub-task > Components: project-management >Reporter: Mikhail Gryzykhin >Priority: Major > > Add common linter and formatter to metrics code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6816) More PR information in the code velocity dashboard?
[ https://issues.apache.org/jira/browse/BEAM-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941314#comment-16941314 ] Mikhail Gryzykhin commented on BEAM-6816: - Unassigning since I'm not actively working on this task. > More PR information in the code velocity dashboard? > --- > > Key: BEAM-6816 > URL: https://issues.apache.org/jira/browse/BEAM-6816 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Pablo Estrada >Priority: Major > > I've been using this dashboard: > http://104.154.241.245/d/code_velocity/code-velocity?orgId=1 > To triage and get PRs reviewed. Some of them have DO NOT REVIEW / or special > instructions in the title. > Perhaps it would be nice to have more information displayed on the dashboard, > so we can skip PRs that don't need to be reviewed. > As you rightly mentioned, another idea is to exclude PRs that have a specific > tag - and I think that's a good idea in fact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-6816) More PR information in the code velocity dashboard?
[ https://issues.apache.org/jira/browse/BEAM-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-6816: --- Assignee: (was: Mikhail Gryzykhin) > More PR information in the code velocity dashboard? > --- > > Key: BEAM-6816 > URL: https://issues.apache.org/jira/browse/BEAM-6816 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Pablo Estrada >Priority: Major > > I've been using this dashboard: > http://104.154.241.245/d/code_velocity/code-velocity?orgId=1 > To triage and get PRs reviewed. Some of them have DO NOT REVIEW / or special > instructions in the title. > Perhaps it would be nice to have more information displayed on the dashboard, > so we can skip PRs that don't need to be reviewed. > As you rightly mentioned, another idea is to exclude PRs that have a specific > tag - and I think that's a good idea in fact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7969) Streaming Dataflow worker doesn't report FnAPI metrics.
[ https://issues.apache.org/jira/browse/BEAM-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin resolved BEAM-7969. - Fix Version/s: 2.17.0 Resolution: Fixed > Streaming Dataflow worker doesn't report FnAPI metrics. > --- > > Key: BEAM-7969 > URL: https://issues.apache.org/jira/browse/BEAM-7969 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-dataflow >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Fix For: 2.17.0 > > Time Spent: 7h > Remaining Estimate: 0h > > EOM -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-7902) :beam-test-infra-metrics:test is consistently failing
[
https://issues.apache.org/jira/browse/BEAM-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941312#comment-16941312
]
Mikhail Gryzykhin edited comment on BEAM-7902 at 9/30/19 8:34 PM:
--
Closing in favor of
https://issues.apache.org/jira/browse/BEAM-8327
and
https://issues.apache.org/jira/browse/BEAM-8328
was (Author: ardagan):
Closing in favor of https://issues.apache.org/jira/browse/BEAM-8327
> :beam-test-infra-metrics:test is consistently failing
> -
>
> Key: BEAM-7902
> URL: https://issues.apache.org/jira/browse/BEAM-7902
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mark Liu
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> A failure gradle run from my local machine:
> https://gradle.com/s/sqcwi7bo3a2nw. Stacktrace:
> {code}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> java.lang.AssertionError: Input data is stale! [id:1, dashboardId:5,
> dashboardUid:data-freshness, dashboardSlug:source-data-freshness, panelId:2,
> name:Source Data Freshness alert, state:alerting,
> newStateDate:2019-07-30T02:55:02Z, evalDate:0001-01-01T00:00:00Z,
> evalData:[evalMatches:[[metric:GitHub, tags:null,
> value:482.1661704504]]], executionError:,
> url:/d/data-freshness/source-data-freshness]
>See: http://104.154.241.245/d/data-freshness. Expression: (alert.state
> == ok)
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:406)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
> ...
> {code}
> From the stacktrace suggestion, I went to
> http://104.154.241.245/d/data-freshness and got Github source is older than a
> week, which I guess is potentially the failure reason. However, no further
> instruction on what to do next.
> After some investigations, I found this test `:beam-test-infra-metrics:test`
> is added to
> [beam_Prober_CommunityMetrics|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Prober_CommunityMetrics/]
> but never got passed (either failed or skipped due to cache).
> Is there anything wrong with Github data refresh? Should we reevaluate the
> alert on dashboard or modify the test?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-7902) :beam-test-infra-metrics:test is consistently failing
[
https://issues.apache.org/jira/browse/BEAM-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin resolved BEAM-7902.
-
Fix Version/s: Not applicable
Resolution: Fixed
> :beam-test-infra-metrics:test is consistently failing
> -
>
> Key: BEAM-7902
> URL: https://issues.apache.org/jira/browse/BEAM-7902
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mark Liu
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> A failure gradle run from my local machine:
> https://gradle.com/s/sqcwi7bo3a2nw. Stacktrace:
> {code}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> java.lang.AssertionError: Input data is stale! [id:1, dashboardId:5,
> dashboardUid:data-freshness, dashboardSlug:source-data-freshness, panelId:2,
> name:Source Data Freshness alert, state:alerting,
> newStateDate:2019-07-30T02:55:02Z, evalDate:0001-01-01T00:00:00Z,
> evalData:[evalMatches:[[metric:GitHub, tags:null,
> value:482.1661704504]]], executionError:,
> url:/d/data-freshness/source-data-freshness]
>See: http://104.154.241.245/d/data-freshness. Expression: (alert.state
> == ok)
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:406)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
> ...
> {code}
> From the stacktrace suggestion, I went to
> http://104.154.241.245/d/data-freshness and got Github source is older than a
> week, which I guess is potentially the failure reason. However, no further
> instruction on what to do next.
> After some investigations, I found this test `:beam-test-infra-metrics:test`
> is added to
> [beam_Prober_CommunityMetrics|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Prober_CommunityMetrics/]
> but never got passed (either failed or skipped due to cache).
> Is there anything wrong with Github data refresh? Should we reevaluate the
> alert on dashboard or modify the test?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7902) :beam-test-infra-metrics:test is consistently failing
[
https://issues.apache.org/jira/browse/BEAM-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941312#comment-16941312
]
Mikhail Gryzykhin commented on BEAM-7902:
-
Closing in favor of https://issues.apache.org/jira/browse/BEAM-8327
> :beam-test-infra-metrics:test is consistently failing
> -
>
> Key: BEAM-7902
> URL: https://issues.apache.org/jira/browse/BEAM-7902
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mark Liu
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> A failure gradle run from my local machine:
> https://gradle.com/s/sqcwi7bo3a2nw. Stacktrace:
> {code}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> java.lang.AssertionError: Input data is stale! [id:1, dashboardId:5,
> dashboardUid:data-freshness, dashboardSlug:source-data-freshness, panelId:2,
> name:Source Data Freshness alert, state:alerting,
> newStateDate:2019-07-30T02:55:02Z, evalDate:0001-01-01T00:00:00Z,
> evalData:[evalMatches:[[metric:GitHub, tags:null,
> value:482.1661704504]]], executionError:,
> url:/d/data-freshness/source-data-freshness]
>See: http://104.154.241.245/d/data-freshness. Expression: (alert.state
> == ok)
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:406)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
> ...
> {code}
> From the stacktrace suggestion, I went to
> http://104.154.241.245/d/data-freshness and got Github source is older than a
> week, which I guess is potentially the failure reason. However, no further
> instruction on what to do next.
> After some investigations, I found this test `:beam-test-infra-metrics:test`
> is added to
> [beam_Prober_CommunityMetrics|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Prober_CommunityMetrics/]
> but never got passed (either failed or skipped due to cache).
> Is there anything wrong with Github data refresh? Should we reevaluate the
> alert on dashboard or modify the test?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-8328) remove :beam-test-infra-metrics:test from build target.
Mikhail Gryzykhin created BEAM-8328: --- Summary: remove :beam-test-infra-metrics:test from build target. Key: BEAM-8328 URL: https://issues.apache.org/jira/browse/BEAM-8328 Project: Beam Issue Type: Bug Components: project-management Reporter: Mikhail Gryzykhin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8327) beam_Prober_CommunityMetrics hits cache giving wrong results
[ https://issues.apache.org/jira/browse/BEAM-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941311#comment-16941311 ] Mikhail Gryzykhin commented on BEAM-8327: - Came up from this failure. https://issues.apache.org/jira/browse/BEAM-7902 > beam_Prober_CommunityMetrics hits cache giving wrong results > > > Key: BEAM-8327 > URL: https://issues.apache.org/jira/browse/BEAM-8327 > Project: Beam > Issue Type: Bug > Components: project-management >Reporter: Mikhail Gryzykhin >Priority: Major > > We need to fix beam_Prober_CommunityMetrics target to not hit cache. It > always fetches fresh data from website even though binaries are the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8327) beam_Prober_CommunityMetrics hits cache giving wrong results
Mikhail Gryzykhin created BEAM-8327: --- Summary: beam_Prober_CommunityMetrics hits cache giving wrong results Key: BEAM-8327 URL: https://issues.apache.org/jira/browse/BEAM-8327 Project: Beam Issue Type: Bug Components: project-management Reporter: Mikhail Gryzykhin We need to fix beam_Prober_CommunityMetrics target to not hit cache. It always fetches fresh data from website even though binaries are the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7969) Streaming Dataflow worker doesn't report FnAPI metrics.
[ https://issues.apache.org/jira/browse/BEAM-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924461#comment-16924461 ] Mikhail Gryzykhin commented on BEAM-7969: - Initial PR double counted elements in GRPC PCollections > Streaming Dataflow worker doesn't report FnAPI metrics. > --- > > Key: BEAM-7969 > URL: https://issues.apache.org/jira/browse/BEAM-7969 > Project: Beam > Issue Type: Bug > Components: java-fn-execution, runner-dataflow >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > EOM -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7518) Protobuf Schema: Introduce logical type for Timestamp, Duration and other
[ https://issues.apache.org/jira/browse/BEAM-7518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917968#comment-16917968 ] Mikhail Gryzykhin commented on BEAM-7518: - Seem to be connected to BEAM-7711 BEAM-7554 > Protobuf Schema: Introduce logical type for Timestamp, Duration and other > - > > Key: BEAM-7518 > URL: https://issues.apache.org/jira/browse/BEAM-7518 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > > Protobuf Schema provider has some loosy conversion from some Proto types. > Introduce Logical Types for: > Timestamp, Duration and Unsigned Int64 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7711) Support DATETIME as a logical type in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-7711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917966#comment-16917966 ] Mikhail Gryzykhin commented on BEAM-7711: - partially duplicates BEAM-7554 > Support DATETIME as a logical type in BeamSQL > - > > Key: BEAM-7711 > URL: https://issues.apache.org/jira/browse/BEAM-7711 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > DATETIME as a type represents a year, month, day, hour, minute, second, and > subsecond(millis) > it ranges from 0001-01-01 00:00:00 to -12-31 23:59:59.999. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7554) datetime and decimal should be logical types
[ https://issues.apache.org/jira/browse/BEAM-7554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917967#comment-16917967 ] Mikhail Gryzykhin commented on BEAM-7554: - relates to BEAM-7711 > datetime and decimal should be logical types > > > Key: BEAM-7554 > URL: https://issues.apache.org/jira/browse/BEAM-7554 > Project: Beam > Issue Type: Improvement > Components: dsl-sql, sdk-java-core >Reporter: Brian Hulette >Priority: Major > > Currently datetime and decimal are implemented as primitive types in the Java > SDK. They should be logical types as documented in > https://s.apache.org/beam-schemas -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8036) [beam_PostCommit_SQL] [DataCatalogBigQueryIT > testReadWrite] No such method
Mikhail Gryzykhin created BEAM-8036: --- Summary: [beam_PostCommit_SQL] [DataCatalogBigQueryIT > testReadWrite] No such method Key: BEAM-8036 URL: https://issues.apache.org/jira/browse/BEAM-8036 Project: Beam Issue Type: Bug Components: test-failures Reporter: Mikhail Gryzykhin Assignee: Andrew Pilloud _Use this form to file an issue for test failure:_ * [Jenkins Job|[https://builds.apache.org/job/beam_PostCommit_SQL/2417/console]] * [Gradle Build Scan|TODO] * [Test source code|TODO] Initial investigation: *09:03:27* > *Task :sdks:java:extensions:sql:datacatalog:integrationTest* *09:03:27* *09:03:27* org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog.DataCatalogBigQueryIT > testReadWrite FAILED*09:03:27* java.lang.NoSuchMethodError at DataCatalogBigQueryIT.java:69*09:03:27* *09:03:27* 1 test completed, 1 failed*09:03:28* *09:03:28* > *Task :sdks:java:extensions:sql:datacatalog:integrationTest* FAILED*09:03:28* *09:03:28* FAILURE: Build failed with an exception. _After you've filled out the above details, please [assign the issue to an individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. Assignee should [treat test failures as high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], helping to fix the issue or find a more appropriate owner. See [Apache Beam Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8035) [beam_PreCommit_Java_Phrase] [WatchTest.testMultiplePollsWithManyResults] Flake: Outputs must be in timestamp order
Mikhail Gryzykhin created BEAM-8035: --- Summary: [beam_PreCommit_Java_Phrase] [WatchTest.testMultiplePollsWithManyResults] Flake: Outputs must be in timestamp order Key: BEAM-8035 URL: https://issues.apache.org/jira/browse/BEAM-8035 Project: Beam Issue Type: Bug Components: test-failures Reporter: Mikhail Gryzykhin Assignee: Luke Cwik _Use this form to file an issue for test failure:_ * [Jenkins Job|[https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1258/testReport/junit/org.apache.beam.sdk.transforms/WatchTest/testMultiplePollsWithManyResults/]] Initial investigation: java.lang.AssertionError: Drop timestamped input/Values/Map/ParMultiDo(Anonymous).output: Outputs must be in timestamp order _After you've filled out the above details, please [assign the issue to an individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. Assignee should [treat test failures as high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], helping to fix the issue or find a more appropriate owner. See [Apache Beam Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8026) [beam_PostCommit_Python_VR_Spark ] Multiple tests failing due to coders.
[ https://issues.apache.org/jira/browse/BEAM-8026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912552#comment-16912552 ] Mikhail Gryzykhin commented on BEAM-8026: - Assigning to Ismael since he seem to have open spark-related PRs. Forward as appropriate. > [beam_PostCommit_Python_VR_Spark ] Multiple tests failing due to coders. > > > Key: BEAM-8026 > URL: https://issues.apache.org/jira/browse/BEAM-8026 > Project: Beam > Issue Type: Bug > Components: runner-spark, test-failures >Reporter: Mikhail Gryzykhin >Assignee: Ismaël Mejía >Priority: Major > Labels: currently-failing > > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/719/]] > Initial investigation: > Multiple tests failing to set proper coder > RuntimeError: Pipeline > test_pardo_windowed_side_inputs_1566389313.07_875931ef-c02f-481c-89a2-2d9376c41ecc > failed in state FAILED: java.lang.ClassCastException: > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to > org.apache.beam.sdk.coders.KvCoder > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (BEAM-8026) [beam_PostCommit_Python_VR_Spark ] Multiple tests failing due to coders.
[ https://issues.apache.org/jira/browse/BEAM-8026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-8026: --- Assignee: Ismaël Mejía > [beam_PostCommit_Python_VR_Spark ] Multiple tests failing due to coders. > > > Key: BEAM-8026 > URL: https://issues.apache.org/jira/browse/BEAM-8026 > Project: Beam > Issue Type: Bug > Components: runner-spark, test-failures >Reporter: Mikhail Gryzykhin >Assignee: Ismaël Mejía >Priority: Major > Labels: currently-failing > > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/719/]] > Initial investigation: > Multiple tests failing to set proper coder > RuntimeError: Pipeline > test_pardo_windowed_side_inputs_1566389313.07_875931ef-c02f-481c-89a2-2d9376c41ecc > failed in state FAILED: java.lang.ClassCastException: > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to > org.apache.beam.sdk.coders.KvCoder > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8026) [beam_PostCommit_Python_VR_Spark ] Multiple tests failing due to coders.
Mikhail Gryzykhin created BEAM-8026: --- Summary: [beam_PostCommit_Python_VR_Spark ] Multiple tests failing due to coders. Key: BEAM-8026 URL: https://issues.apache.org/jira/browse/BEAM-8026 Project: Beam Issue Type: Bug Components: runner-spark, test-failures Reporter: Mikhail Gryzykhin * [Jenkins Job|[https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/719/]] Initial investigation: Multiple tests failing to set proper coder RuntimeError: Pipeline test_pardo_windowed_side_inputs_1566389313.07_875931ef-c02f-481c-89a2-2d9376c41ecc failed in state FAILED: java.lang.ClassCastException: org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to org.apache.beam.sdk.coders.KvCoder _After you've filled out the above details, please [assign the issue to an individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. Assignee should [treat test failures as high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], helping to fix the issue or find a more appropriate owner. See [Apache Beam Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-7993) portable python precommit is flaky
[
https://issues.apache.org/jira/browse/BEAM-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911712#comment-16911712
]
Mikhail Gryzykhin commented on BEAM-7993:
-
This seem to fail python precommits consistently lately
[https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/1048/]
> portable python precommit is flaky
> --
>
> Key: BEAM-7993
> URL: https://issues.apache.org/jira/browse/BEAM-7993
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures, testing
>Affects Versions: 2.15.0
>Reporter: Udi Meiri
>Assignee: Kyle Weaver
>Priority: Major
> Labels: currently-failing
> Fix For: 2.15.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I'm not sure what the root cause is here.
> Example log where
> :sdks:python:test-suites:portable:py35:portableWordCountBatch failed:
> {code}
> 11:51:22 [CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap
> (FlatMap at ExtractOutput[0]) (2/2)] ERROR
> org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN
> MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at
> ExtractOutput[0]) (2/2)
> 11:51:22 [CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap
> (FlatMap at ExtractOutput[0]) (1/2)] ERROR
> org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN
> MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at
> ExtractOutput[0]) (1/2)
> 11:51:22 [CHAIN MapPartition (MapPartition at
> [2]write/Write/WriteImpl/DoOnce/{FlatMap(),
> Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (2/2)] ERROR
> org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN
> MapPartition (MapPartition at
> [2]write/Write/WriteImpl/DoOnce/{FlatMap(),
> Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (2/2)
> 11:51:22 [CHAIN MapPartition (MapPartition at
> [2]write/Write/WriteImpl/DoOnce/{FlatMap(),
> Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)] ERROR
> org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN
> MapPartition (MapPartition at
> [2]write/Write/WriteImpl/DoOnce/{FlatMap(),
> Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)
> 11:51:22 java.lang.Exception: The user defined 'open()' method caused an
> exception: java.io.IOException: Received exit code 1 for command 'docker
> inspect -f {{.State.Running}}
> 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1'. stderr:
> Error: No such object:
> 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1
> 11:51:22 at
> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498)
> 11:51:22 at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
> 11:51:22 at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
> 11:51:22 at java.lang.Thread.run(Thread.java:748)
> 11:51:22 Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.io.IOException: Received exit code 1 for command 'docker inspect -f
> {{.State.Running}}
> 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1'. stderr:
> Error: No such object:
> 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1
> 11:51:22 at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4966)
> 11:51:22 at
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.(DefaultJobBundleFactory.java:211)
> 11:51:22 at
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.(DefaultJobBundleFactory.java:202)
> 11:51:22 at
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.forStage(DefaultJobBundleFactory.java:185)
> 11:51:22 at
> org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.getStageBundleFactory(FlinkDefaultExecutableStageContext.java:49)
> 11:51:22 at
> org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory$WrappedContext.getStageBundleFactory(ReferenceCountingFlinkExecutableStageContextFactory.java:203)
> 11:51:22 at
> org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.open(FlinkExecutableStageFunction.java:129)
> 11:51:22 at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> 11:51:22 at
> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494)
> 11:51:22 ... 3 more
> {code}
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/5512/consoleFull
--
This
[jira] [Updated] (BEAM-7993) portable python precommit is flaky
[
https://issues.apache.org/jira/browse/BEAM-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7993:
Labels: currently-failing (was: )
> portable python precommit is flaky
> --
>
> Key: BEAM-7993
> URL: https://issues.apache.org/jira/browse/BEAM-7993
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures, testing
>Affects Versions: 2.15.0
>Reporter: Udi Meiri
>Assignee: Kyle Weaver
>Priority: Major
> Labels: currently-failing
> Fix For: 2.15.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I'm not sure what the root cause is here.
> Example log where
> :sdks:python:test-suites:portable:py35:portableWordCountBatch failed:
> {code}
> 11:51:22 [CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap
> (FlatMap at ExtractOutput[0]) (2/2)] ERROR
> org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN
> MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at
> ExtractOutput[0]) (2/2)
> 11:51:22 [CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap
> (FlatMap at ExtractOutput[0]) (1/2)] ERROR
> org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN
> MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at
> ExtractOutput[0]) (1/2)
> 11:51:22 [CHAIN MapPartition (MapPartition at
> [2]write/Write/WriteImpl/DoOnce/{FlatMap(),
> Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (2/2)] ERROR
> org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN
> MapPartition (MapPartition at
> [2]write/Write/WriteImpl/DoOnce/{FlatMap(),
> Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (2/2)
> 11:51:22 [CHAIN MapPartition (MapPartition at
> [2]write/Write/WriteImpl/DoOnce/{FlatMap(),
> Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)] ERROR
> org.apache.flink.runtime.operators.BatchTask - Error in task code: CHAIN
> MapPartition (MapPartition at
> [2]write/Write/WriteImpl/DoOnce/{FlatMap(),
> Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/2)
> 11:51:22 java.lang.Exception: The user defined 'open()' method caused an
> exception: java.io.IOException: Received exit code 1 for command 'docker
> inspect -f {{.State.Running}}
> 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1'. stderr:
> Error: No such object:
> 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1
> 11:51:22 at
> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498)
> 11:51:22 at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
> 11:51:22 at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
> 11:51:22 at java.lang.Thread.run(Thread.java:748)
> 11:51:22 Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.io.IOException: Received exit code 1 for command 'docker inspect -f
> {{.State.Running}}
> 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1'. stderr:
> Error: No such object:
> 642c312c335d3881b885873c66917b536e79cff07503fdceaddee5fbeb10bfd1
> 11:51:22 at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4966)
> 11:51:22 at
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.(DefaultJobBundleFactory.java:211)
> 11:51:22 at
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.(DefaultJobBundleFactory.java:202)
> 11:51:22 at
> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.forStage(DefaultJobBundleFactory.java:185)
> 11:51:22 at
> org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.getStageBundleFactory(FlinkDefaultExecutableStageContext.java:49)
> 11:51:22 at
> org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory$WrappedContext.getStageBundleFactory(ReferenceCountingFlinkExecutableStageContextFactory.java:203)
> 11:51:22 at
> org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.open(FlinkExecutableStageFunction.java:129)
> 11:51:22 at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> 11:51:22 at
> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494)
> 11:51:22 ... 3 more
> {code}
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/5512/consoleFull
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (BEAM-7969) Streaming Dataflow worker doesn't report FnAPI metrics.
Mikhail Gryzykhin created BEAM-7969: --- Summary: Streaming Dataflow worker doesn't report FnAPI metrics. Key: BEAM-7969 URL: https://issues.apache.org/jira/browse/BEAM-7969 Project: Beam Issue Type: Bug Components: java-fn-execution, runner-dataflow Reporter: Mikhail Gryzykhin Assignee: Mikhail Gryzykhin EOM -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (BEAM-7902) :beam-test-infra-metrics:test is consistently failing
[
https://issues.apache.org/jira/browse/BEAM-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900566#comment-16900566
]
Mikhail Gryzykhin commented on BEAM-7902:
-
So far I patched issue with data freshness.
[~markflyhigh] do you know how to do 1 and 2?
# Integration test is regular test target in build.gradle. Is there a way to
make it trigger only when we call some specific task in gradle?
# How do I invalidate cache for specific project?
I'll continue searching tomorrow, but with lower priority.
> :beam-test-infra-metrics:test is consistently failing
> -
>
> Key: BEAM-7902
> URL: https://issues.apache.org/jira/browse/BEAM-7902
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mark Liu
>Assignee: Mikhail Gryzykhin
>Priority: Major
>
> A failure gradle run from my local machine:
> https://gradle.com/s/sqcwi7bo3a2nw. Stacktrace:
> {code}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> java.lang.AssertionError: Input data is stale! [id:1, dashboardId:5,
> dashboardUid:data-freshness, dashboardSlug:source-data-freshness, panelId:2,
> name:Source Data Freshness alert, state:alerting,
> newStateDate:2019-07-30T02:55:02Z, evalDate:0001-01-01T00:00:00Z,
> evalData:[evalMatches:[[metric:GitHub, tags:null,
> value:482.1661704504]]], executionError:,
> url:/d/data-freshness/source-data-freshness]
>See: http://104.154.241.245/d/data-freshness. Expression: (alert.state
> == ok)
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:406)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
> ...
> {code}
> From the stacktrace suggestion, I went to
> http://104.154.241.245/d/data-freshness and got Github source is older than a
> week, which I guess is potentially the failure reason. However, no further
> instruction on what to do next.
> After some investigations, I found this test `:beam-test-infra-metrics:test`
> is added to
> [beam_Prober_CommunityMetrics|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Prober_CommunityMetrics/]
> but never got passed (either failed or skipped due to cache).
> Is there anything wrong with Github data refresh? Should we reevaluate the
> alert on dashboard or modify the test?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (BEAM-7902) :beam-test-infra-metrics:test is consistently failing
[
https://issues.apache.org/jira/browse/BEAM-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7902:
Status: Open (was: Triage Needed)
> :beam-test-infra-metrics:test is consistently failing
> -
>
> Key: BEAM-7902
> URL: https://issues.apache.org/jira/browse/BEAM-7902
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mark Liu
>Assignee: Mikhail Gryzykhin
>Priority: Major
>
> A failure gradle run from my local machine:
> https://gradle.com/s/sqcwi7bo3a2nw. Stacktrace:
> {code}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> java.lang.AssertionError: Input data is stale! [id:1, dashboardId:5,
> dashboardUid:data-freshness, dashboardSlug:source-data-freshness, panelId:2,
> name:Source Data Freshness alert, state:alerting,
> newStateDate:2019-07-30T02:55:02Z, evalDate:0001-01-01T00:00:00Z,
> evalData:[evalMatches:[[metric:GitHub, tags:null,
> value:482.1661704504]]], executionError:,
> url:/d/data-freshness/source-data-freshness]
>See: http://104.154.241.245/d/data-freshness. Expression: (alert.state
> == ok)
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:406)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
> ...
> {code}
> From the stacktrace suggestion, I went to
> http://104.154.241.245/d/data-freshness and got Github source is older than a
> week, which I guess is potentially the failure reason. However, no further
> instruction on what to do next.
> After some investigations, I found this test `:beam-test-infra-metrics:test`
> is added to
> [beam_Prober_CommunityMetrics|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Prober_CommunityMetrics/]
> but never got passed (either failed or skipped due to cache).
> Is there anything wrong with Github data refresh? Should we reevaluate the
> alert on dashboard or modify the test?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (BEAM-7902) :beam-test-infra-metrics:test is consistently failing
[
https://issues.apache.org/jira/browse/BEAM-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900434#comment-16900434
]
Mikhail Gryzykhin commented on BEAM-7902:
-
This shows 3 issues:
# We should remove this test from build target, since it is an integration
test that shows the healthiness of metrics website.
# We need to fix beam_Prober_CommunityMetrics target to not hit cache. It
always fetches fresh data from website even though binaries are the same.
# There actually seem to be an issue with fetching latest data from github on
metrics site. As you noted it is week old and test catches the issue.
> :beam-test-infra-metrics:test is consistently failing
> -
>
> Key: BEAM-7902
> URL: https://issues.apache.org/jira/browse/BEAM-7902
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mark Liu
>Assignee: Mikhail Gryzykhin
>Priority: Major
>
> A failure gradle run from my local machine:
> https://gradle.com/s/sqcwi7bo3a2nw. Stacktrace:
> {code}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> java.lang.AssertionError: Input data is stale! [id:1, dashboardId:5,
> dashboardUid:data-freshness, dashboardSlug:source-data-freshness, panelId:2,
> name:Source Data Freshness alert, state:alerting,
> newStateDate:2019-07-30T02:55:02Z, evalDate:0001-01-01T00:00:00Z,
> evalData:[evalMatches:[[metric:GitHub, tags:null,
> value:482.1661704504]]], executionError:,
> url:/d/data-freshness/source-data-freshness]
>See: http://104.154.241.245/d/data-freshness. Expression: (alert.state
> == ok)
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:406)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
> ...
> {code}
> From the stacktrace suggestion, I went to
> http://104.154.241.245/d/data-freshness and got Github source is older than a
> week, which I guess is potentially the failure reason. However, no further
> instruction on what to do next.
> After some investigations, I found this test `:beam-test-infra-metrics:test`
> is added to
> [beam_Prober_CommunityMetrics|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Prober_CommunityMetrics/]
> but never got passed (either failed or skipped due to cache).
> Is there anything wrong with Github data refresh? Should we reevaluate the
> alert on dashboard or modify the test?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (BEAM-7902) :beam-test-infra-metrics:test is consistently failing
[
https://issues.apache.org/jira/browse/BEAM-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin reassigned BEAM-7902:
---
Assignee: Mikhail Gryzykhin
> :beam-test-infra-metrics:test is consistently failing
> -
>
> Key: BEAM-7902
> URL: https://issues.apache.org/jira/browse/BEAM-7902
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mark Liu
>Assignee: Mikhail Gryzykhin
>Priority: Major
>
> A failure gradle run from my local machine:
> https://gradle.com/s/sqcwi7bo3a2nw. Stacktrace:
> {code}
> org.apache.beam.testinfra.metrics.ProberTests > CheckGrafanaStalenessAlerts
> FAILED
> java.lang.AssertionError: Input data is stale! [id:1, dashboardId:5,
> dashboardUid:data-freshness, dashboardSlug:source-data-freshness, panelId:2,
> name:Source Data Freshness alert, state:alerting,
> newStateDate:2019-07-30T02:55:02Z, evalDate:0001-01-01T00:00:00Z,
> evalData:[evalMatches:[[metric:GitHub, tags:null,
> value:482.1661704504]]], executionError:,
> url:/d/data-freshness/source-data-freshness]
>See: http://104.154.241.245/d/data-freshness. Expression: (alert.state
> == ok)
> at
> org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:406)
> at
> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
> ...
> {code}
> From the stacktrace suggestion, I went to
> http://104.154.241.245/d/data-freshness and got Github source is older than a
> week, which I guess is potentially the failure reason. However, no further
> instruction on what to do next.
> After some investigations, I found this test `:beam-test-infra-metrics:test`
> is added to
> [beam_Prober_CommunityMetrics|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Prober_CommunityMetrics/]
> but never got passed (either failed or skipped due to cache).
> Is there anything wrong with Github data refresh? Should we reevaluate the
> alert on dashboard or modify the test?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (BEAM-7791) Grafana: update dashboards from git repo.
[ https://issues.apache.org/jira/browse/BEAM-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin resolved BEAM-7791. - Resolution: Fixed Fix Version/s: Not applicable > Grafana: update dashboards from git repo. > - > > Key: BEAM-7791 > URL: https://issues.apache.org/jira/browse/BEAM-7791 > Project: Beam > Issue Type: New Feature > Components: website >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Fix For: Not applicable > > Original Estimate: 24h > Time Spent: 1h 50m > Remaining Estimate: 22h 10m > > Currently, all grafana dashboards are created on Grafana host and might be > lost if service loses relevant docker volume. It also impossible to easily > replicate whole metrics service. > We want all Grafana dashboards to be initialized and later periodically > updated from git repo snapshots. This will add proper versioning and backup > for dashboards. Big benefit is being able to initialize whole service on dev > machine with less manual work. > References: > [https://grafana.com/docs/administration/provisioning/] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (BEAM-7791) Grafana: update dashboards from git repo.
[ https://issues.apache.org/jira/browse/BEAM-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-7791: Status: Open (was: Triage Needed) > Grafana: update dashboards from git repo. > - > > Key: BEAM-7791 > URL: https://issues.apache.org/jira/browse/BEAM-7791 > Project: Beam > Issue Type: New Feature > Components: website >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > Currently, all grafana dashboards are created on Grafana host and might be > lost if service loses relevant docker volume. It also impossible to easily > replicate whole metrics service. > We want all Grafana dashboards to be initialized and later periodically > updated from git repo snapshots. This will add proper versioning and backup > for dashboards. Big benefit is being able to initialize whole service on dev > machine with less manual work. > References: > [https://grafana.com/docs/administration/provisioning/] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (BEAM-7791) Grafana: update dashboards from git repo.
[ https://issues.apache.org/jira/browse/BEAM-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-7791: Description: Currently, all grafana dashboards are created on Grafana host and might be lost if service loses relevant docker volume. It also impossible to easily replicate whole metrics service. We want all Grafana dashboards to be initialized and later periodically updated from git repo snapshots. This will add proper versioning and backup for dashboards. Big benefit is being able to initialize whole service on dev machine with less manual work. References: [https://grafana.com/docs/administration/provisioning/] was: Currently, all grafana dashboards are created on Grafana host and might be lost if service loses relevant docker volume. It also impossible to easily replicate whole metrics service. We want all Grafana dashboards to be initialized and later periodically updated from git repo snapshots. This will add proper versioning and backup for dashboards. References: https://grafana.com/docs/administration/provisioning/ > Grafana: update dashboards from git repo. > - > > Key: BEAM-7791 > URL: https://issues.apache.org/jira/browse/BEAM-7791 > Project: Beam > Issue Type: New Feature > Components: website >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > Currently, all grafana dashboards are created on Grafana host and might be > lost if service loses relevant docker volume. It also impossible to easily > replicate whole metrics service. > We want all Grafana dashboards to be initialized and later periodically > updated from git repo snapshots. This will add proper versioning and backup > for dashboards. Big benefit is being able to initialize whole service on dev > machine with less manual work. > References: > [https://grafana.com/docs/administration/provisioning/] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (BEAM-7791) Grafana: update dashboards from git repo.
Mikhail Gryzykhin created BEAM-7791: --- Summary: Grafana: update dashboards from git repo. Key: BEAM-7791 URL: https://issues.apache.org/jira/browse/BEAM-7791 Project: Beam Issue Type: New Feature Components: website Reporter: Mikhail Gryzykhin Assignee: Mikhail Gryzykhin Currently, all grafana dashboards are created on Grafana host and might be lost if service loses relevant docker volume. It also impossible to easily replicate whole metrics service. We want all Grafana dashboards to be initialized and later periodically updated from git repo snapshots. This will add proper versioning and backup for dashboards. References: https://grafana.com/docs/administration/provisioning/ -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (BEAM-6458) Beam site links to grafana dashboard by ip
[ https://issues.apache.org/jira/browse/BEAM-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16889897#comment-16889897 ] Mikhail Gryzykhin commented on BEAM-6458: - [~pabloem] FYI: We have this ticket open. > Beam site links to grafana dashboard by ip > -- > > Key: BEAM-6458 > URL: https://issues.apache.org/jira/browse/BEAM-6458 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > > I would say this is not a great practice. As a stopgap use a shortlink at > https://s.apache.org. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (BEAM-1399) Code coverage numbers are not accurate
[ https://issues.apache.org/jira/browse/BEAM-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-1399: --- Assignee: (was: Mikhail Gryzykhin) > Code coverage numbers are not accurate > -- > > Key: BEAM-1399 > URL: https://issues.apache.org/jira/browse/BEAM-1399 > Project: Beam > Issue Type: Bug > Components: build-system, sdk-java-core, testing >Reporter: Daniel Halperin >Priority: Major > Labels: newbie, starter > > We've started adding Java Code Coverage numbers to PRs using the jacoco > gradle plugin. > When we ran this in Maven, we got very low coverage reported for things like > the Java SDK core. This was likely because we test the bulk of the SDK not in > the SDK module, but in fact in the DirectRunner and other similar modules. > This is the still the case in gradle. So we'll need to make sure to watch for > it. > This is a good "random improvement" issue for anyone to pick up. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (BEAM-1399) Code coverage numbers are not accurate
[ https://issues.apache.org/jira/browse/BEAM-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888381#comment-16888381 ] Mikhail Gryzykhin commented on BEAM-1399: - I'm not working on this atm, so will unassign so that anyone can pick it up. > Code coverage numbers are not accurate > -- > > Key: BEAM-1399 > URL: https://issues.apache.org/jira/browse/BEAM-1399 > Project: Beam > Issue Type: Bug > Components: build-system, sdk-java-core, testing >Reporter: Daniel Halperin >Assignee: Mikhail Gryzykhin >Priority: Major > Labels: newbie, starter > > We've started adding Java Code Coverage numbers to PRs using the jacoco > gradle plugin. > When we ran this in Maven, we got very low coverage reported for things like > the Java SDK core. This was likely because we test the bulk of the SDK not in > the SDK module, but in fact in the DirectRunner and other similar modules. > This is the still the case in gradle. So we'll need to make sure to watch for > it. > This is a good "random improvement" issue for anyone to pick up. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (BEAM-7642) (Dataflow) Python AfterProcessingTime fires before defined time
[ https://issues.apache.org/jira/browse/BEAM-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-7642: Summary: (Dataflow) Python AfterProcessingTime fires before defined time (was: (Dataflow) AfterProcessingTime fires before defined time) > (Dataflow) Python AfterProcessingTime fires before defined time > --- > > Key: BEAM-7642 > URL: https://issues.apache.org/jira/browse/BEAM-7642 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Mikhail Gryzykhin >Priority: Major > > [https://stackoverflow.com/questions/56700913/why-my-data-in-apache-beam-is-emitted-after-a-few-minutes-instead-of-10h] > > User on StackOverflow has a problem that AfterProcessingTime on global window > fires before allocated time (10h). It dumps events within first > minutes/seconds and then drops the rest due to window closed. > Code user provided seems valid. I tried to verify time units accepted by the > method, but I couldn't track it all the way through the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7642) (Dataflow) AfterProcessingTime fires before defined time
Mikhail Gryzykhin created BEAM-7642: --- Summary: (Dataflow) AfterProcessingTime fires before defined time Key: BEAM-7642 URL: https://issues.apache.org/jira/browse/BEAM-7642 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Mikhail Gryzykhin [https://stackoverflow.com/questions/56700913/why-my-data-in-apache-beam-is-emitted-after-a-few-minutes-instead-of-10h] User on StackOverflow has a problem that AfterProcessingTime on global window fires before allocated time (10h). It dumps events within first minutes/seconds and then drops the rest due to window closed. Code user provided seems valid. I tried to verify time units accepted by the method, but I couldn't track it all the way through the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
[ https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-7619: --- Assignee: Daniel Oliveira > [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] > java.lang.IllegalStateException: Already closed. > - > > Key: BEAM-7619 > URL: https://issues.apache.org/jira/browse/BEAM-7619 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Mikhail Gryzykhin >Assignee: Daniel Oliveira >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] > > Initial investigation: > 12:08:13 ERROR: test_wordcount_fnapi_it > (apache_beam.examples.wordcount_it_test.WordCountIT) > 12:08:13 > -- > 12:08:13 Traceback (most recent call last): > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 52, in test_wordcount_fnapi_it > 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 84, in _run_wordcount_it > 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts)) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", > line 114, in run > 12:08:13 result = p.run() > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 406, in run > 12:08:13 self._options).run(False) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 419, in run > 12:08:13 return self.runner.run_pipeline(self, self._options) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", > line 64, in run_pipeline > 12:08:13 self.result.wait_until_finish(duration=wait_duration) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", > line 1338, in wait_until_finish > 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self) > 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, > Error: > 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException: > Error received from SDK harness for instruction -129: Traceback (most recent > call last): > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 157, in _execute > 12:08:13 response = task() > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 190, in > 12:08:13 self._execute(lambda: worker.do_instruction(work), work) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 342, in do_instruction > 12:08:13 request.instruction_id) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 368, in process_bundle > 12:08:13 bundle_processor.process_bundle(instruction_id)) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 593, in process_bundle > 12:08:13 data.ptransform_id].process_encoded(data.data) > 12:08:13 KeyError: u'\n\x04-107\x12\x04-105' > 12:08:13 > 12:08:13 at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > 12:08:13 at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > 12:08:13 at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:412) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:381) > 12:08:13 at >
[jira] [Updated] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
[ https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-7619: Description: _Use this form to file an issue for test failure:_ * [Jenkins Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] Initial investigation: 12:08:13 ERROR: test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 12:08:13 -- 12:08:13 Traceback (most recent call last): 12:08:13 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", line 52, in test_wordcount_fnapi_it 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') 12:08:13 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", line 84, in _run_wordcount_it 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts)) 12:08:13 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", line 114, in run 12:08:13 result = p.run() 12:08:13 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", line 406, in run 12:08:13 self._options).run(False) 12:08:13 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", line 419, in run 12:08:13 return self.runner.run_pipeline(self, self._options) 12:08:13 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 64, in run_pipeline 12:08:13 self.result.wait_until_finish(duration=wait_duration) 12:08:13 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1338, in wait_until_finish 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self) 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error received from SDK harness for instruction -129: Traceback (most recent call last): 12:08:13 File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 157, in _execute 12:08:13 response = task() 12:08:13 File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 190, in 12:08:13 self._execute(lambda: worker.do_instruction(work), work) 12:08:13 File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 342, in do_instruction 12:08:13 request.instruction_id) 12:08:13 File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 368, in process_bundle 12:08:13 bundle_processor.process_bundle(instruction_id)) 12:08:13 File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 593, in process_bundle 12:08:13 data.ptransform_id].process_encoded(data.data) 12:08:13 KeyError: u'\n\x04-107\x12\x04-105' 12:08:13 12:08:13 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) 12:08:13 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) 12:08:13 at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) 12:08:13 at org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) 12:08:13 at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) 12:08:13 at org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) 12:08:13 at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:412) 12:08:13 at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:381) 12:08:13 at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306) 12:08:13 at org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195) 12:08:13 at org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123) 12:08:13 Suppressed: java.lang.IllegalStateException: Already closed. 12:08:13 at org.apache.beam.sdk.fn.data.BeamFnDataBufferingOutboundObserver.close(BeamFnDataBufferingOutboundObserver.java:95) 12:08:13 at org.apache.beam.runners.dataflow.worker.fn.data.RemoteGrpcPortWriteOperation.abort(RemoteGrpcPortWriteOperation.java:215) 12:08:13 at
[jira] [Created] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
Mikhail Gryzykhin created BEAM-7619: --- Summary: [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed. Key: BEAM-7619 URL: https://issues.apache.org/jira/browse/BEAM-7619 Project: Beam Issue Type: Bug Components: test-failures Reporter: Mikhail Gryzykhin _Use this form to file an issue for test failure:_ * [Jenkins Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] Initial investigation: *12:08:13* ERROR: test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT)*12:08:13* --*12:08:13* Traceback (most recent call last):*12:08:13* File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", line 52, in test_wordcount_fnapi_it*12:08:13* self._run_wordcount_it(wordcount.run, experiment='beam_fn_api')*12:08:13* File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", line 84, in _run_wordcount_it*12:08:13* run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts))*12:08:13* File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", line 114, in run*12:08:13* result = p.run()*12:08:13* File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", line 406, in run*12:08:13* self._options).run(False)*12:08:13* File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", line 419, in run*12:08:13* return self.runner.run_pipeline(self, self._options)*12:08:13* File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 64, in run_pipeline*12:08:13* self.result.wait_until_finish(duration=wait_duration)*12:08:13* File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1338, in wait_until_finish*12:08:13* (self.state, getattr(self._runner, 'last_error_msg', None)), self)*12:08:13* DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:*12:08:13* java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error received from SDK harness for instruction -129: Traceback (most recent call last):*12:08:13* File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 157, in _execute*12:08:13* response = task()*12:08:13* File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 190, in *12:08:13* self._execute(lambda: worker.do_instruction(work), work)*12:08:13* File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 342, in do_instruction*12:08:13* request.instruction_id)*12:08:13* File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 368, in process_bundle*12:08:13* bundle_processor.process_bundle(instruction_id))*12:08:13* File "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 593, in process_bundle*12:08:13* data.ptransform_id].process_encoded(data.data)*12:08:13* KeyError: u'\n\x04-107\x12\x04-105'*12:08:13* *12:08:13* at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)*12:08:13* at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)*12:08:13* at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)*12:08:13* at org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285)*12:08:13* at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)*12:08:13* at org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)*12:08:13* at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:412)*12:08:13* at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:381)*12:08:13* at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)*12:08:13* at org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.start(DataflowRunnerHarness.java:195)*12:08:13* at org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:123)*12:08:13* Suppressed:
[jira] [Updated] (BEAM-7225) [beam_PostCommit_Py_ValCont ] [test_metrics_fnapi_it] Unable to match metrics for matcher namespace
[ https://issues.apache.org/jira/browse/BEAM-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-7225: Priority: Major (was: Blocker) > [beam_PostCommit_Py_ValCont ] [test_metrics_fnapi_it] Unable to match metrics > for matcher namespace > > > Key: BEAM-7225 > URL: https://issues.apache.org/jira/browse/BEAM-7225 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ahmet Altay >Assignee: Alex Amato >Priority: Major > Labels: currently-failing > > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3115/console > 13:19:41 test_metrics_fnapi_it > (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest) > ... FAIL > 13:19:41 > 13:19:41 > == > 13:19:41 FAIL: test_metrics_fnapi_it > (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest) > 13:19:41 > -- > 13:19:41 Traceback (most recent call last): > 13:19:41 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py", > line 70, in test_metrics_fnapi_it > 13:19:41 self.assertFalse(errors, str(errors)) > 13:19:41 AssertionError: Unable to match metrics for matcher namespace: > 'apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline.UserMetricsDoFn' > name: 'total_values' step: 'metrics' attempted: <100> committed: <100> > 13:19:41 Actual MetricResults: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7225) [beam_PostCommit_Py_ValCont ] [test_metrics_fnapi_it] Unable to match metrics for matcher namespace
[ https://issues.apache.org/jira/browse/BEAM-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-7225: Priority: Blocker (was: Major) > [beam_PostCommit_Py_ValCont ] [test_metrics_fnapi_it] Unable to match metrics > for matcher namespace > > > Key: BEAM-7225 > URL: https://issues.apache.org/jira/browse/BEAM-7225 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ahmet Altay >Assignee: Alex Amato >Priority: Blocker > Labels: currently-failing > > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3115/console > 13:19:41 test_metrics_fnapi_it > (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest) > ... FAIL > 13:19:41 > 13:19:41 > == > 13:19:41 FAIL: test_metrics_fnapi_it > (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest) > 13:19:41 > -- > 13:19:41 Traceback (most recent call last): > 13:19:41 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py", > line 70, in test_metrics_fnapi_it > 13:19:41 self.assertFalse(errors, str(errors)) > 13:19:41 AssertionError: Unable to match metrics for matcher namespace: > 'apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline.UserMetricsDoFn' > name: 'total_values' step: 'metrics' attempted: <100> committed: <100> > 13:19:41 Actual MetricResults: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7463) Bigquery IO ITs are flaky: incorrect checksum
[
https://issues.apache.org/jira/browse/BEAM-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868810#comment-16868810
]
Mikhail Gryzykhin commented on BEAM-7463:
-
Sumilar repro, but "but" field is empty:
[https://builds.apache.org/job/beam_PostCommit_Python3_Verify/1189/consoleFull]
> Bigquery IO ITs are flaky: incorrect checksum
> -
>
> Key: BEAM-7463
> URL: https://issues.apache.org/jira/browse/BEAM-7463
> Project: Beam
> Issue Type: Bug
> Components: io-python-gcp
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> {noformat}
> 15:03:38 FAIL: test_big_query_new_types
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 15:03:38
> --
> 15:03:38 Traceback (most recent call last):
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
> line 211, in test_big_query_new_types
> 15:03:38 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
> line 82, in run_bq_pipeline
> 15:03:38 result = p.run()
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 15:03:38 else test_runner_api))
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 15:03:38 self._options).run(False)
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 15:03:38 return self.runner.run_pipeline(self, self._options)
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 51, in run_pipeline
> 15:03:38 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 15:03:38 AssertionError:
> 15:03:38 Expected: (Test pipeline expected terminated in state: DONE and
> Expected checksum is 24de460c4d344a4b77ccc4cc1acb7b7ffc11a214)
> 15:03:38 but: Expected checksum is
> 24de460c4d344a4b77ccc4cc1acb7b7ffc11a214 Actual checksum is
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> {noformat}
> [~Juta] could this be caused by changes to Bigquery matcher?
> https://github.com/apache/beam/pull/8621/files#diff-f1ec7e3a3e7e2e5082ddb7043954c108R134
>
> cc: [~pabloem] [~chamikara] [~apilloud]
> A recent postcommit run has BQ failures in other tests as well:
> https://builds.apache.org/job/beam_PostCommit_Python3_Verify/1000/consoleFull
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7463) Bigquery IO ITs are flaky: incorrect checksum
[
https://issues.apache.org/jira/browse/BEAM-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7463:
Labels: currently-failing (was: )
> Bigquery IO ITs are flaky: incorrect checksum
> -
>
> Key: BEAM-7463
> URL: https://issues.apache.org/jira/browse/BEAM-7463
> Project: Beam
> Issue Type: Bug
> Components: io-python-gcp
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
> Labels: currently-failing
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> {noformat}
> 15:03:38 FAIL: test_big_query_new_types
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 15:03:38
> --
> 15:03:38 Traceback (most recent call last):
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
> line 211, in test_big_query_new_types
> 15:03:38 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
> line 82, in run_bq_pipeline
> 15:03:38 result = p.run()
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 15:03:38 else test_runner_api))
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 15:03:38 self._options).run(False)
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 15:03:38 return self.runner.run_pipeline(self, self._options)
> 15:03:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 51, in run_pipeline
> 15:03:38 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 15:03:38 AssertionError:
> 15:03:38 Expected: (Test pipeline expected terminated in state: DONE and
> Expected checksum is 24de460c4d344a4b77ccc4cc1acb7b7ffc11a214)
> 15:03:38 but: Expected checksum is
> 24de460c4d344a4b77ccc4cc1acb7b7ffc11a214 Actual checksum is
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> {noformat}
> [~Juta] could this be caused by changes to Bigquery matcher?
> https://github.com/apache/beam/pull/8621/files#diff-f1ec7e3a3e7e2e5082ddb7043954c108R134
>
> cc: [~pabloem] [~chamikara] [~apilloud]
> A recent postcommit run has BQ failures in other tests as well:
> https://builds.apache.org/job/beam_PostCommit_Python3_Verify/1000/consoleFull
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7396) apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT.test_big_query_new_types is flaky
[
https://issues.apache.org/jira/browse/BEAM-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868809#comment-16868809
]
Mikhail Gryzykhin commented on BEAM-7396:
-
Another repro:
[https://builds.apache.org/job/beam_PostCommit_Python3_Verify/1189/consoleFull]
> apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT.test_big_query_new_types
> is flaky
>
>
> Key: BEAM-7396
> URL: https://issues.apache.org/jira/browse/BEAM-7396
> Project: Beam
> Issue Type: Bug
> Components: io-python-gcp, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
>
> cc: [~pabloem] [~chamikara] [~altay]
> {noformat}
> 16:23:37 test_big_query_new_types
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... FAIL
> 16:23:38
> ==
> 16:23:38 FAIL: test_big_query_new_types
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 16:23:38
> --
> 16:23:38 Traceback (most recent call last):
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
> line 217, in test_big_query_new_types
> 16:23:38 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
> line 82, in run_bq_pipeline
> 16:23:38 result = p.run()
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 16:23:38 else test_runner_api))
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 16:23:38 self._options).run(False)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 16:23:38 return self.runner.run_pipeline(self, self._options)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 51, in run_pipeline
> 16:23:38 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 16:23:38 AssertionError:
> 16:23:38 Expected: (Test pipeline expected terminated in state: DONE and
> Expected checksum is 1631ca7060b89a01760c81874b988c46156e18b5)
> 16:23:38 but:
> 16:23:38
> 16:23:38 >> begin captured logging <<
>
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: INFO: Running pipeline with DirectRunner.
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: INFO: Using location u'US' from table 16:23:38 datasetId: u'python_query_to_table_1558567400576'
> 16:23:38 projectId: u'apache-beam-testing'
> 16:23:38 tableId: u'python_new_types_table'> referenced by query SELECT
> bytes, date, time FROM
> [python_query_to_table_1558567400576.python_new_types_table]
> 16:23:38 root: WARNING: Dataset
> apache-beam-testing:temp_dataset_d291c6c6fe2942e999d849ae5f2539ec does not
> exist so we will create it as temporary with location=US
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 root: DEBUG: Creating or getting table 16:23:38 datasetId: 'python_query_to_table_1558567400576'
> 16:23:38 projectId: 'apache-beam-testing'
> 16:23:38 tableId: 'output_table'> with schema {'fields': [{'type': u'BYTES',
> 'name': u'bytes', 'mode': 'NULLABLE'}, {'type': u'DATE', 'name': u'date',
> 'mode': 'NULLABLE'}, {'type': u'TIME', 'name': u'time', 'mode': 'NULLABLE'}]}.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: DEBUG: Created the table with id output_table
> 16:23:38 root: INFO: Created table
> apache-beam-testing.python_query_to_table_1558567400576.output_table with
> schema 16:23:38 fields: [ 16:23:38 fields: []
> 16:23:38 mode: u'NULLABLE'
> 16:23:38 name: u'bytes'
> 16:23:38 type: u'BYTES'>, 16:23:38 fields: []
> 16:23:38 mode: u'NULLABLE'
> 16:23:38 name: u'date'
> 16:23:38 type: u'DATE'>,
[jira] [Updated] (BEAM-7396) apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT.test_big_query_new_types is flaky
[
https://issues.apache.org/jira/browse/BEAM-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7396:
Labels: currently-failing (was: )
> apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT.test_big_query_new_types
> is flaky
>
>
> Key: BEAM-7396
> URL: https://issues.apache.org/jira/browse/BEAM-7396
> Project: Beam
> Issue Type: Bug
> Components: io-python-gcp, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
> Labels: currently-failing
>
> cc: [~pabloem] [~chamikara] [~altay]
> {noformat}
> 16:23:37 test_big_query_new_types
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... FAIL
> 16:23:38
> ==
> 16:23:38 FAIL: test_big_query_new_types
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 16:23:38
> --
> 16:23:38 Traceback (most recent call last):
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
> line 217, in test_big_query_new_types
> 16:23:38 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
> line 82, in run_bq_pipeline
> 16:23:38 result = p.run()
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 16:23:38 else test_runner_api))
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 16:23:38 self._options).run(False)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 16:23:38 return self.runner.run_pipeline(self, self._options)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 51, in run_pipeline
> 16:23:38 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 16:23:38 AssertionError:
> 16:23:38 Expected: (Test pipeline expected terminated in state: DONE and
> Expected checksum is 1631ca7060b89a01760c81874b988c46156e18b5)
> 16:23:38 but:
> 16:23:38
> 16:23:38 >> begin captured logging <<
>
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: INFO: Running pipeline with DirectRunner.
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: INFO: Using location u'US' from table 16:23:38 datasetId: u'python_query_to_table_1558567400576'
> 16:23:38 projectId: u'apache-beam-testing'
> 16:23:38 tableId: u'python_new_types_table'> referenced by query SELECT
> bytes, date, time FROM
> [python_query_to_table_1558567400576.python_new_types_table]
> 16:23:38 root: WARNING: Dataset
> apache-beam-testing:temp_dataset_d291c6c6fe2942e999d849ae5f2539ec does not
> exist so we will create it as temporary with location=US
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 root: DEBUG: Creating or getting table 16:23:38 datasetId: 'python_query_to_table_1558567400576'
> 16:23:38 projectId: 'apache-beam-testing'
> 16:23:38 tableId: 'output_table'> with schema {'fields': [{'type': u'BYTES',
> 'name': u'bytes', 'mode': 'NULLABLE'}, {'type': u'DATE', 'name': u'date',
> 'mode': 'NULLABLE'}, {'type': u'TIME', 'name': u'time', 'mode': 'NULLABLE'}]}.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: DEBUG: Created the table with id output_table
> 16:23:38 root: INFO: Created table
> apache-beam-testing.python_query_to_table_1558567400576.output_table with
> schema 16:23:38 fields: [ 16:23:38 fields: []
> 16:23:38 mode: u'NULLABLE'
> 16:23:38 name: u'bytes'
> 16:23:38 type: u'BYTES'>, 16:23:38 fields: []
> 16:23:38 mode: u'NULLABLE'
> 16:23:38 name: u'date'
> 16:23:38 type: u'DATE'>, 16:23:38 fields: []
> 16:23:38 mode: u'NULLABLE'
> 16:23:38
[jira] [Resolved] (BEAM-7488) Grafana request: Add monthly aggregated view for "Precommit Test Latency - Time In Queue"
[ https://issues.apache.org/jira/browse/BEAM-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin resolved BEAM-7488. - Resolution: Fixed Fix Version/s: Not applicable > Grafana request: Add monthly aggregated view for "Precommit Test Latency - > Time In Queue" > - > > Key: BEAM-7488 > URL: https://issues.apache.org/jira/browse/BEAM-7488 > Project: Beam > Issue Type: New Feature > Components: project-management, testing >Reporter: Daniel Oliveira >Assignee: Mikhail Gryzykhin >Priority: Minor > Fix For: Not applicable > > > Similar to [BEAM-7259|https://issues.apache.org/jira/browse/BEAM-7259], was > hoping to get this metric in a view that's aggregated per month. In this case > averaging all the queue times seems like it would work well. Even better if > you could average only the 95th percentile. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7546) Portable WordCount-on-Flink Precommit is flaky - temporary folder not found.
[
https://issues.apache.org/jira/browse/BEAM-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868034#comment-16868034
]
Mikhail Gryzykhin commented on BEAM-7546:
-
Another repro:
[https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/801/consoleFull]
> Portable WordCount-on-Flink Precommit is flaky - temporary folder not found.
>
>
> Key: BEAM-7546
> URL: https://issues.apache.org/jira/browse/BEAM-7546
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Valentyn Tymofieiev
>Assignee: Ankur Goenka
>Priority: Major
>
> On a few occasions I see this test fail due to a temp directory being missing.
> Sample scan from
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/745/:
> https://scans.gradle.com/s/3ra4xw4hqvlyw/console-log?task=:sdks:python:portableWordCountBatch
> {noformat}
> [grpc-default-executor-0] ERROR sdk_worker._execute - Error processing
> instruction 8. Original traceback is
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 157, in _execute
> response = task()
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 190, in
> self._execute(lambda: worker.do_instruction(work), work)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 342, in do_instruction
> request.instruction_id)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 368, in process_bundle
> bundle_processor.process_bundle(instruction_id))
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 589, in process_bundle
> ].process_encoded(data.data)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 143, in process_encoded
> self.output(decoded_value)
> File "apache_beam/runners/worker/operations.py", line 255, in
> apache_beam.runners.worker.operations.Operation.output
> def output(self, windowed_value, output_index=0):
> File "apache_beam/runners/worker/operations.py", line 256, in
> apache_beam.runners.worker.operations.Operation.output
> cython.cast(Receiver,
> self.receivers[output_index]).receive(windowed_value)
> File "apache_beam/runners/worker/operations.py", line 143, in
> apache_beam.runners.worker.operations.SingletonConsumerSet.receive
> self.consumer.process(windowed_value)
> File "apache_beam/runners/worker/operations.py", line 593, in
> apache_beam.runners.worker.operations.DoOperation.process
> with self.scoped_process_state:
> File "apache_beam/runners/worker/operations.py", line 594, in
> apache_beam.runners.worker.operations.DoOperation.process
> delayed_application = self.dofn_receiver.receive(o)
> File "apache_beam/runners/common.py", line 778, in
> apache_beam.runners.common.DoFnRun
> ner.receive
> self.process(windowed_value)
> File "apache_beam/runners/common.py", line 784, in
> apache_beam.runners.common.DoFnRunner.process
> self._reraise_augmented(exn)
> File "apache_beam/runners/common.py", line 851, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
> raise_with_traceback(new_exn)
> File "apache_beam/runners/common.py", line 782, in
> apache_beam.runners.common.DoFnRunner.process
> return self.do_fn_invoker.invoke_process(windowed_value)
> File "apache_beam/runners/common.py", line 594, in
> apache_beam.runners.common.PerWindowInvoker.invoke_process
> self._invoke_process_per_window(
> File "apache_beam/runners/common.py", line 666, in
> apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
> windowed_value, self.process_method(*args_for_process))
> File "/usr/local/lib/python2.7/site-packages/apache_beam/io/iobase.py",
> line 1041, in process
> self.writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/options/value_provider.py",
> line 137, in _f
> return fnc(self, *args, **kwargs)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py",
> line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File
> "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsink.py",
> line 389, in __init__
> self.temp_handle = self.sink.open(temp_shard_path)
> File "/usr/local/lib/python2.7/site-packages/apache_beam/io/textio.py",
> line 391, in open
> file_handle = super(_TextSink, self).open(temp_path)
> File
>
[jira] [Resolved] (BEAM-7401) beam_PreCommit_CommunityMetrics_Cron consistently failing
[ https://issues.apache.org/jira/browse/BEAM-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin resolved BEAM-7401. - Resolution: Fixed Fix Version/s: Not applicable > beam_PreCommit_CommunityMetrics_Cron consistently failing > - > > Key: BEAM-7401 > URL: https://issues.apache.org/jira/browse/BEAM-7401 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Michael Luckey >Assignee: Mikhail Gryzykhin >Priority: Major > Fix For: Not applicable > > > Since a while, community metrics Jenkins job [1] is broken. > [1] > https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Cron/buildTimeTrend -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7401) beam_PreCommit_CommunityMetrics_Cron consistently failing
[ https://issues.apache.org/jira/browse/BEAM-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856003#comment-16856003 ] Mikhail Gryzykhin commented on BEAM-7401: - Missed this ticket. The issue was with docker credentials that was common across all our jenkins builds. Resolved by now. Relevant ticket: https://issues.apache.org/jira/browse/BEAM-7381 > beam_PreCommit_CommunityMetrics_Cron consistently failing > - > > Key: BEAM-7401 > URL: https://issues.apache.org/jira/browse/BEAM-7401 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Michael Luckey >Assignee: Mikhail Gryzykhin >Priority: Major > > Since a while, community metrics Jenkins job [1] is broken. > [1] > https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Cron/buildTimeTrend -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6284) [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with result UNKNOWN on succeeded job and checks passed
[ https://issues.apache.org/jira/browse/BEAM-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin resolved BEAM-6284. - Resolution: Fixed Fix Version/s: 2.13.0 > [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with > result UNKNOWN on succeeded job and checks passed > -- > > Key: BEAM-6284 > URL: https://issues.apache.org/jira/browse/BEAM-6284 > Project: Beam > Issue Type: Bug > Components: test-failures, testing >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Labels: currently-failing > Fix For: 2.13.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > _Use this form to file an issue for test failure:_ > * > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testWindowedSideInputFixedToGlobal/ > Initial investigation: > According to logs all test-relevant checks have passed and it seem to be > testing framework failure. > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7445) Add search field to Beam website
Mikhail Gryzykhin created BEAM-7445: --- Summary: Add search field to Beam website Key: BEAM-7445 URL: https://issues.apache.org/jira/browse/BEAM-7445 Project: Beam Issue Type: New Feature Components: website Reporter: Mikhail Gryzykhin Beam website has a lot of information and it is not always easy to find what you're looking for. In case of [documentation](https://beam.apache.org/documentation/) it often happens so that user knows what he's looking for, but can't can't find a way to navigate to relevant section. Adding a search bar to website will make it much easier to navigate and search for relevant information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7410) Tests fail with Quota issue.
[
https://issues.apache.org/jira/browse/BEAM-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7410:
Priority: Critical (was: Major)
> Tests fail with Quota issue.
>
>
> Key: BEAM-7410
> URL: https://issues.apache.org/jira/browse/BEAM-7410
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Critical
>
>
> {noformat}
> java.lang.RuntimeException: Workflow failed. Causes: Project
> apache-beam-testing has insufficient quota(s) to execute this workflow with 1
> instances in region us-central1. Quota summary (required/available): 1/11743
> instances, 1/170 CPUs, 250/278399 disk GB, 0/4046 SSD disk GB, 1/188 instance
> groups, 1/188 managed instance groups, 1/126 instance templates, 1/0 in-use
> IP addresses.{noformat}
>
> Sample jobs:
> [https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/440/testReport/junit/org.apache.beam.sdk.testing/PAssertTest/testGlobalWindowContainsInAnyOrder/]
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/67/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyIterableSideInput/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7410) Tests fail with Quota issue.
[
https://issues.apache.org/jira/browse/BEAM-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7410:
Labels: (was: Triage)
> Tests fail with Quota issue.
>
>
> Key: BEAM-7410
> URL: https://issues.apache.org/jira/browse/BEAM-7410
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>
>
> {noformat}
> java.lang.RuntimeException: Workflow failed. Causes: Project
> apache-beam-testing has insufficient quota(s) to execute this workflow with 1
> instances in region us-central1. Quota summary (required/available): 1/11743
> instances, 1/170 CPUs, 250/278399 disk GB, 0/4046 SSD disk GB, 1/188 instance
> groups, 1/188 managed instance groups, 1/126 instance templates, 1/0 in-use
> IP addresses.{noformat}
>
> Sample jobs:
> [https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/440/testReport/junit/org.apache.beam.sdk.testing/PAssertTest/testGlobalWindowContainsInAnyOrder/]
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/67/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyIterableSideInput/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7410) Tests fail with Quota issue.
[
https://issues.apache.org/jira/browse/BEAM-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7410:
Component/s: (was: io-python-gcp)
> Tests fail with Quota issue.
>
>
> Key: BEAM-7410
> URL: https://issues.apache.org/jira/browse/BEAM-7410
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>
>
> {noformat}
> java.lang.RuntimeException: Workflow failed. Causes: Project
> apache-beam-testing has insufficient quota(s) to execute this workflow with 1
> instances in region us-central1. Quota summary (required/available): 1/11743
> instances, 1/170 CPUs, 250/278399 disk GB, 0/4046 SSD disk GB, 1/188 instance
> groups, 1/188 managed instance groups, 1/126 instance templates, 1/0 in-use
> IP addresses.{noformat}
>
> Sample jobs:
> [https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/440/testReport/junit/org.apache.beam.sdk.testing/PAssertTest/testGlobalWindowContainsInAnyOrder/]
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/67/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyIterableSideInput/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-7410) Tests fail with Quota issue.
[
https://issues.apache.org/jira/browse/BEAM-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin reassigned BEAM-7410:
---
Assignee: (was: yifan zou)
> Tests fail with Quota issue.
>
>
> Key: BEAM-7410
> URL: https://issues.apache.org/jira/browse/BEAM-7410
> Project: Beam
> Issue Type: Bug
> Components: io-python-gcp, test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>
>
> {noformat}
> java.lang.RuntimeException: Workflow failed. Causes: Project
> apache-beam-testing has insufficient quota(s) to execute this workflow with 1
> instances in region us-central1. Quota summary (required/available): 1/11743
> instances, 1/170 CPUs, 250/278399 disk GB, 0/4046 SSD disk GB, 1/188 instance
> groups, 1/188 managed instance groups, 1/126 instance templates, 1/0 in-use
> IP addresses.{noformat}
>
> Sample jobs:
> [https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/440/testReport/junit/org.apache.beam.sdk.testing/PAssertTest/testGlobalWindowContainsInAnyOrder/]
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/67/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyIterableSideInput/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7410) Tests fail with Quota issue.
[
https://issues.apache.org/jira/browse/BEAM-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7410:
Description:
{noformat}
java.lang.RuntimeException: Workflow failed. Causes: Project
apache-beam-testing has insufficient quota(s) to execute this workflow with 1
instances in region us-central1. Quota summary (required/available): 1/11743
instances, 1/170 CPUs, 250/278399 disk GB, 0/4046 SSD disk GB, 1/188 instance
groups, 1/188 managed instance groups, 1/126 instance templates, 1/0 in-use IP
addresses.{noformat}
Sample jobs:
[https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/440/testReport/junit/org.apache.beam.sdk.testing/PAssertTest/testGlobalWindowContainsInAnyOrder/]
[https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/67/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyIterableSideInput/]
was:
cc: [~pabloem] [~chamikara] [~altay]
{noformat}
16:23:37 test_big_query_new_types
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
... FAIL
16:23:38 ==
16:23:38 FAIL: test_big_query_new_types
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
16:23:38 --
16:23:38 Traceback (most recent call last):
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
line 217, in test_big_query_new_types
16:23:38 big_query_query_to_table_pipeline.run_bq_pipeline(options)
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
line 82, in run_bq_pipeline
16:23:38 result = p.run()
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
line 107, in run
16:23:38 else test_runner_api))
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
line 406, in run
16:23:38 self._options).run(False)
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
line 419, in run
16:23:38 return self.runner.run_pipeline(self, self._options)
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
line 51, in run_pipeline
16:23:38 hc_assert_that(self.result, pickler.loads(on_success_matcher))
16:23:38 AssertionError:
16:23:38 Expected: (Test pipeline expected terminated in state: DONE and
Expected checksum is 1631ca7060b89a01760c81874b988c46156e18b5)
16:23:38 but:
16:23:38
16:23:38 >> begin captured logging <<
16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
access_token
16:23:38 root: INFO: Running pipeline with DirectRunner.
16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
access_token
16:23:38 root: INFO: Using location u'US' from table referenced by query SELECT bytes,
date, time FROM [python_query_to_table_1558567400576.python_new_types_table]
16:23:38 root: WARNING: Dataset
apache-beam-testing:temp_dataset_d291c6c6fe2942e999d849ae5f2539ec does not
exist so we will create it as temporary with location=US
16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
16:23:38 root: DEBUG: Creating or getting table with schema {'fields': [{'type': u'BYTES',
'name': u'bytes', 'mode': 'NULLABLE'}, {'type': u'DATE', 'name': u'date',
'mode': 'NULLABLE'}, {'type': u'TIME', 'name': u'time', 'mode': 'NULLABLE'}]}.
16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
access_token
16:23:38 root: DEBUG: Created the table with id output_table
16:23:38 root: INFO: Created table
apache-beam-testing.python_query_to_table_1558567400576.output_table with
schema , , ]>. Result: , , ]>
16:23:38 selfLink:
u'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_1558567400576/tables/output_table'
16:23:38 tableReference:
16:23:38 type: u'TABLE'>.
16:23:38 root: DEBUG: Attempting to flush to all destinations. Total buffered: 4
16:23:38 root: DEBUG: Flushing data to
apache-beam-testing:python_query_to_table_1558567400576.output_table. Total 4
rows.
16:23:38 root: DEBUG: Passed: True. Errors are []
16:23:38 root: INFO: Start verify Bigquery data.
16:23:38
[jira] [Assigned] (BEAM-7410) Tests fail with Quota issue.
[
https://issues.apache.org/jira/browse/BEAM-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin reassigned BEAM-7410:
---
Assignee: yifan zou
> Tests fail with Quota issue.
>
>
> Key: BEAM-7410
> URL: https://issues.apache.org/jira/browse/BEAM-7410
> Project: Beam
> Issue Type: Bug
> Components: io-python-gcp, test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: yifan zou
>Priority: Major
>
>
> {noformat}
> java.lang.RuntimeException: Workflow failed. Causes: Project
> apache-beam-testing has insufficient quota(s) to execute this workflow with 1
> instances in region us-central1. Quota summary (required/available): 1/11743
> instances, 1/170 CPUs, 250/278399 disk GB, 0/4046 SSD disk GB, 1/188 instance
> groups, 1/188 managed instance groups, 1/126 instance templates, 1/0 in-use
> IP addresses.{noformat}
>
> Sample jobs:
> [https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/440/testReport/junit/org.apache.beam.sdk.testing/PAssertTest/testGlobalWindowContainsInAnyOrder/]
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/67/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyIterableSideInput/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7410) Tests fail with Quota issue.
[
https://issues.apache.org/jira/browse/BEAM-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin updated BEAM-7410:
Summary: Tests fail with Quota issue. (was: CLONE -
apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT.test_big_query_new_types
is flaky)
> Tests fail with Quota issue.
>
>
> Key: BEAM-7410
> URL: https://issues.apache.org/jira/browse/BEAM-7410
> Project: Beam
> Issue Type: Bug
> Components: io-python-gcp, test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>
> cc: [~pabloem] [~chamikara] [~altay]
> {noformat}
> 16:23:37 test_big_query_new_types
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... FAIL
> 16:23:38
> ==
> 16:23:38 FAIL: test_big_query_new_types
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 16:23:38
> --
> 16:23:38 Traceback (most recent call last):
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
> line 217, in test_big_query_new_types
> 16:23:38 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
> line 82, in run_bq_pipeline
> 16:23:38 result = p.run()
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> 16:23:38 else test_runner_api))
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 16:23:38 self._options).run(False)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 16:23:38 return self.runner.run_pipeline(self, self._options)
> 16:23:38 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 51, in run_pipeline
> 16:23:38 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 16:23:38 AssertionError:
> 16:23:38 Expected: (Test pipeline expected terminated in state: DONE and
> Expected checksum is 1631ca7060b89a01760c81874b988c46156e18b5)
> 16:23:38 but:
> 16:23:38
> 16:23:38 >> begin captured logging <<
>
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: INFO: Running pipeline with DirectRunner.
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: INFO: Using location u'US' from table 16:23:38 datasetId: u'python_query_to_table_1558567400576'
> 16:23:38 projectId: u'apache-beam-testing'
> 16:23:38 tableId: u'python_new_types_table'> referenced by query SELECT
> bytes, date, time FROM
> [python_query_to_table_1558567400576.python_new_types_table]
> 16:23:38 root: WARNING: Dataset
> apache-beam-testing:temp_dataset_d291c6c6fe2942e999d849ae5f2539ec does not
> exist so we will create it as temporary with location=US
> 16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
> 16:23:38 root: DEBUG: Creating or getting table 16:23:38 datasetId: 'python_query_to_table_1558567400576'
> 16:23:38 projectId: 'apache-beam-testing'
> 16:23:38 tableId: 'output_table'> with schema {'fields': [{'type': u'BYTES',
> 'name': u'bytes', 'mode': 'NULLABLE'}, {'type': u'DATE', 'name': u'date',
> 'mode': 'NULLABLE'}, {'type': u'TIME', 'name': u'time', 'mode': 'NULLABLE'}]}.
> 16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 16:23:38 root: DEBUG: Created the table with id output_table
> 16:23:38 root: INFO: Created table
> apache-beam-testing.python_query_to_table_1558567400576.output_table with
> schema 16:23:38 fields: [ 16:23:38 fields: []
> 16:23:38 mode: u'NULLABLE'
> 16:23:38 name: u'bytes'
> 16:23:38 type: u'BYTES'>, 16:23:38 fields: []
> 16:23:38 mode: u'NULLABLE'
> 16:23:38 name: u'date'
> 16:23:38 type: u'DATE'>, 16:23:38 fields: []
> 16:23:38 mode: u'NULLABLE'
> 16:23:38 name: u'time'
> 16:23:38 type: u'TIME'>]>. Result: 16:23:38 creationTime: 1558567406362
> 16:23:38 etag:
[jira] [Created] (BEAM-7410) CLONE - apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT.test_big_query_new_types is flaky
Mikhail Gryzykhin created BEAM-7410:
---
Summary: CLONE -
apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT.test_big_query_new_types
is flaky
Key: BEAM-7410
URL: https://issues.apache.org/jira/browse/BEAM-7410
Project: Beam
Issue Type: Bug
Components: io-python-gcp, test-failures
Reporter: Mikhail Gryzykhin
cc: [~pabloem] [~chamikara] [~altay]
{noformat}
16:23:37 test_big_query_new_types
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
... FAIL
16:23:38 ==
16:23:38 FAIL: test_big_query_new_types
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
16:23:38 --
16:23:38 Traceback (most recent call last):
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
line 217, in test_big_query_new_types
16:23:38 big_query_query_to_table_pipeline.run_bq_pipeline(options)
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
line 82, in run_bq_pipeline
16:23:38 result = p.run()
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
line 107, in run
16:23:38 else test_runner_api))
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
line 406, in run
16:23:38 self._options).run(False)
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
line 419, in run
16:23:38 return self.runner.run_pipeline(self, self._options)
16:23:38 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
line 51, in run_pipeline
16:23:38 hc_assert_that(self.result, pickler.loads(on_success_matcher))
16:23:38 AssertionError:
16:23:38 Expected: (Test pipeline expected terminated in state: DONE and
Expected checksum is 1631ca7060b89a01760c81874b988c46156e18b5)
16:23:38 but:
16:23:38
16:23:38 >> begin captured logging <<
16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
access_token
16:23:38 root: INFO: Running pipeline with DirectRunner.
16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
access_token
16:23:38 root: INFO: Using location u'US' from table referenced by query SELECT bytes,
date, time FROM [python_query_to_table_1558567400576.python_new_types_table]
16:23:38 root: WARNING: Dataset
apache-beam-testing:temp_dataset_d291c6c6fe2942e999d849ae5f2539ec does not
exist so we will create it as temporary with location=US
16:23:38 root: DEBUG: Connecting using Google Application Default Credentials.
16:23:38 root: DEBUG: Creating or getting table with schema {'fields': [{'type': u'BYTES',
'name': u'bytes', 'mode': 'NULLABLE'}, {'type': u'DATE', 'name': u'date',
'mode': 'NULLABLE'}, {'type': u'TIME', 'name': u'time', 'mode': 'NULLABLE'}]}.
16:23:38 oauth2client.transport: INFO: Attempting refresh to obtain initial
access_token
16:23:38 root: DEBUG: Created the table with id output_table
16:23:38 root: INFO: Created table
apache-beam-testing.python_query_to_table_1558567400576.output_table with
schema , , ]>. Result: , , ]>
16:23:38 selfLink:
u'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_1558567400576/tables/output_table'
16:23:38 tableReference:
16:23:38 type: u'TABLE'>.
16:23:38 root: DEBUG: Attempting to flush to all destinations. Total buffered: 4
16:23:38 root: DEBUG: Flushing data to
apache-beam-testing:python_query_to_table_1558567400576.output_table. Total 4
rows.
16:23:38 root: DEBUG: Passed: True. Errors are []
16:23:38 root: INFO: Start verify Bigquery data.
16:23:38 google.auth.transport._http_client: DEBUG: Making request: GET
http://169.254.169.254
16:23:38 google.auth.transport._http_client: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/project/project-id
16:23:38 urllib3.util.retry: DEBUG: Converted retries value: 3 ->
Retry(total=3, connect=None, read=None, redirect=None, status=None)
16:23:38 google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
[jira] [Commented] (BEAM-6284) [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with result UNKNOWN on succeeded job and checks passed
[ https://issues.apache.org/jira/browse/BEAM-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845306#comment-16845306 ] Mikhail Gryzykhin commented on BEAM-6284: - Seems that the problem is within waitUntilFinish. It treats UNKNOWN state as error and exits after MAX_RETRIES. I got PR out to treat UNKNOWN as non-terminal state, but still confirming whether UNKNOWN 'CAN' be terminal. If it can, then I'll look for alternative way to handle the issue. > [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with > result UNKNOWN on succeeded job and checks passed > -- > > Key: BEAM-6284 > URL: https://issues.apache.org/jira/browse/BEAM-6284 > Project: Beam > Issue Type: Bug > Components: test-failures, testing >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testWindowedSideInputFixedToGlobal/ > Initial investigation: > According to logs all test-relevant checks have passed and it seem to be > testing framework failure. > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6284) [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with result UNKNOWN on succeeded job and checks passed
[ https://issues.apache.org/jira/browse/BEAM-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-6284: --- Assignee: Mikhail Gryzykhin (was: Daniel Oliveira) > [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with > result UNKNOWN on succeeded job and checks passed > -- > > Key: BEAM-6284 > URL: https://issues.apache.org/jira/browse/BEAM-6284 > Project: Beam > Issue Type: Bug > Components: test-failures, testing >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testWindowedSideInputFixedToGlobal/ > Initial investigation: > According to logs all test-relevant checks have passed and it seem to be > testing framework failure. > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7088) Utilize Name and Namespace labels instead of urn parts for MonitoringInfos
[
https://issues.apache.org/jira/browse/BEAM-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Gryzykhin resolved BEAM-7088.
-
Resolution: Fixed
Fix Version/s: 2.13.0
> Utilize Name and Namespace labels instead of urn parts for MonitoringInfos
> --
>
> Key: BEAM-7088
> URL: https://issues.apache.org/jira/browse/BEAM-7088
> Project: Beam
> Issue Type: Improvement
> Components: runner-core
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
> Fix For: 2.13.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Existing state:
> Counter Name and namespace are part of urn: "::"
> Target state:
> urn: , labels \{"NAMESPACE":, "NAME":}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-6458) Beam site links to grafana dashboard by ip
[ https://issues.apache.org/jira/browse/BEAM-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844291#comment-16844291 ] Mikhail Gryzykhin commented on BEAM-6458: - We already have short link: [https://s.apache.org/beam-community-metrics] Does this mitigate this ticket? > Beam site links to grafana dashboard by ip > -- > > Key: BEAM-6458 > URL: https://issues.apache.org/jira/browse/BEAM-6458 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Kenneth Knowles >Assignee: Mikhail Gryzykhin >Priority: Major > > I would say this is not a great practice. As a stopgap use a shortlink at > https://s.apache.org. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6834) Extract RegisterAndProcessBundleOperation constructor parameters into a single class.
[ https://issues.apache.org/jira/browse/BEAM-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin reassigned BEAM-6834: --- Assignee: (was: Mikhail Gryzykhin) > Extract RegisterAndProcessBundleOperation constructor parameters into a > single class. > - > > Key: BEAM-6834 > URL: https://issues.apache.org/jira/browse/BEAM-6834 > Project: Beam > Issue Type: New Feature > Components: runner-dataflow >Reporter: Mikhail Gryzykhin >Priority: Minor > > [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java#L117] > > These constructor parameters play similar role and it might be worth > extracting them to a single class that can be passed around. > > Map ptransformIdToSystemStepContext, > Map ptransformIdToSideInputReader, Map SideInputReader> ptransformIdToSideInputReader, > Table> > ptransformIdToSideInputIdToPCollectionView, Table PCollectionView> ptransformIdToSideInputIdToPCollectionView, > Map sdkToDfePCollectionMapping, -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7001) Remove MonitoringInfoUrns proto from FnApi metrics.proto
[ https://issues.apache.org/jira/browse/BEAM-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin resolved BEAM-7001. - Resolution: Fixed Fix Version/s: Not applicable > Remove MonitoringInfoUrns proto from FnApi metrics.proto > > > Key: BEAM-7001 > URL: https://issues.apache.org/jira/browse/BEAM-7001 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 10m > Remaining Estimate: 0h > > All urns are defined in MonitoringInfoSpecs. We want to remove duplicate > definition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7088) Utilize Name and Namespace labes instead of urn parts for MonitoringInfos
Mikhail Gryzykhin created BEAM-7088:
---
Summary: Utilize Name and Namespace labes instead of urn parts for
MonitoringInfos
Key: BEAM-7088
URL: https://issues.apache.org/jira/browse/BEAM-7088
Project: Beam
Issue Type: Improvement
Components: runner-core
Reporter: Mikhail Gryzykhin
Assignee: Mikhail Gryzykhin
Existing state:
Counter Name and namespace are part of urn: "::"
Target state:
urn: , labels \{"NAMESPACE":, "NAME":}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-6974) [beam_PreCommit_Java_Cron] [:beam-sdks-java-io-cassandra:test][Flake] Test fails in native code
[ https://issues.apache.org/jira/browse/BEAM-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin updated BEAM-6974: Description: This test fails often during last several days. Suspicion is that same error also fails :beam-sdks-java-io-hadoop-input-format:test that flakes as well. [https://builds.apache.org/job/beam_PreCommit_Java_Cron/1136/consoleFull] [https://builds.apache.org/job/beam_PreCommit_Java_Cron/1133/consoleFull] Initial investigation: Jenkins doesn't give any error output. Local run fails gradle task stating that there's error in native code. I was reproing it on version before any of PRs mentioned below (8178 & 8182) as well. So there might be some independent issue. Dev list thread: [https://lists.apache.org/thread.html/80258cf7dcdf404b3de686a5ffdc13a4319c7d2e11698747d62cf5ad@%3Cdev.beam.apache.org%3E] Relevant PRs that might have influenced the issue: [https://github.com/apache/beam/pull/8178] [https://github.com/apache/beam/pull/8182] _After you've filled out the above details, please [assign the issue to an individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. Assignee should [treat test failures as high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], helping to fix the issue or find a more appropriate owner. See [Apache Beam Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._ was: This test fails often during last several days. Suspicion is that same error also fails :beam-sdks-java-io-hadoop-input-format:test that flakes as well. [https://builds.apache.org/job/beam_PreCommit_Java_Cron/1136/consoleFull] [https://builds.apache.org/job/beam_PreCommit_Java_Cron/1133/consoleFull] Initial investigation: Jenkins doesn't give any error output. Local run fails gradle task stating that there's error in native code. Dev list thread: [https://lists.apache.org/thread.html/80258cf7dcdf404b3de686a5ffdc13a4319c7d2e11698747d62cf5ad@%3Cdev.beam.apache.org%3E] Relevant PRs that might have influenced the issue: [https://github.com/apache/beam/pull/8178] [https://github.com/apache/beam/pull/8182] _After you've filled out the above details, please [assign the issue to an individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. Assignee should [treat test failures as high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], helping to fix the issue or find a more appropriate owner. See [Apache Beam Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._ > [beam_PreCommit_Java_Cron] [:beam-sdks-java-io-cassandra:test][Flake] Test > fails in native code > --- > > Key: BEAM-6974 > URL: https://issues.apache.org/jira/browse/BEAM-6974 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Mikhail Gryzykhin >Priority: Blocker > Labels: currently-failing > > This test fails often during last several days. > Suspicion is that same error also fails > :beam-sdks-java-io-hadoop-input-format:test that flakes as well. > [https://builds.apache.org/job/beam_PreCommit_Java_Cron/1136/consoleFull] > [https://builds.apache.org/job/beam_PreCommit_Java_Cron/1133/consoleFull] > Initial investigation: > Jenkins doesn't give any error output. Local run fails gradle task stating > that there's error in native code. > I was reproing it on version before any of PRs mentioned below (8178 & 8182) > as well. So there might be some independent issue. > Dev list thread: > [https://lists.apache.org/thread.html/80258cf7dcdf404b3de686a5ffdc13a4319c7d2e11698747d62cf5ad@%3Cdev.beam.apache.org%3E] > > Relevant PRs that might have influenced the issue: > [https://github.com/apache/beam/pull/8178] > [https://github.com/apache/beam/pull/8182] > > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7001) Remove MonitoringInfoUrns proto from FnApi metrics.proto
Mikhail Gryzykhin created BEAM-7001: --- Summary: Remove MonitoringInfoUrns proto from FnApi metrics.proto Key: BEAM-7001 URL: https://issues.apache.org/jira/browse/BEAM-7001 Project: Beam Issue Type: Improvement Components: runner-dataflow Reporter: Mikhail Gryzykhin Assignee: Mikhail Gryzykhin All urns are defined in MonitoringInfoSpecs. We want to remove duplicate definition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6824) Plumb FnApi ElementCount metrics in Dataflow Runner.
[ https://issues.apache.org/jira/browse/BEAM-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Gryzykhin resolved BEAM-6824. - Resolution: Fixed Fix Version/s: 2.13.0 > Plumb FnApi ElementCount metrics in Dataflow Runner. > > > Key: BEAM-6824 > URL: https://issues.apache.org/jira/browse/BEAM-6824 > Project: Beam > Issue Type: New Feature > Components: runner-dataflow >Reporter: Mikhail Gryzykhin >Assignee: Mikhail Gryzykhin >Priority: Major > Fix For: 2.13.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > We want to provide FnApi metrics to Dataflow Users. > This bug covers ElementCount metric. > Current approach utilizes mapping of PCollectionID based on WorkItem and > ProcessBundle graphs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
