[jira] [Created] (BEAM-10228) Reshuffle seems to be dropping elements in streaming Dataflow pipeline.
Daniel Oliveira created BEAM-10228: -- Summary: Reshuffle seems to be dropping elements in streaming Dataflow pipeline. Key: BEAM-10228 URL: https://issues.apache.org/jira/browse/BEAM-10228 Project: Beam Issue Type: Bug Components: runner-dataflow, sdk-java-harness Reporter: Daniel Oliveira https://stackoverflow.com/questions/62017572/beam-java-dataflow-bigquery-streaming-insert-groupbykey-reducing-elements This showed up on SO and it shows a GBK in a reshuffle actually dropping elements. I asked the user if they've tried it on other runners, and from what I understood they tried with the Direct runner but never got to the point in the pipeline where the original issue occurred, so it's hard to tell if this is a runner or SDK issue. The user has also contacted Dataflow support. Not sure if this is high priority or not or who would be an appropriate assignee, so I'll just leave the bug unassigned for future reference, so if this pops up again there's evidence that it's been a problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9978) Add offset range restrictions to the Go SDK.
[ https://issues.apache.org/jira/browse/BEAM-9978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9978: -- Summary: Add offset range restrictions to the Go SDK. (was: Add offset range restrictions to the Go SDK proper.) > Add offset range restrictions to the Go SDK. > > > Key: BEAM-9978 > URL: https://issues.apache.org/jira/browse/BEAM-9978 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > Currently these are part of the stringsplit example. but they should probably > be generalized and in the actual SDK, and should have adequate testing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9978) Add offset range restrictions to the Go SDK proper.
Daniel Oliveira created BEAM-9978: - Summary: Add offset range restrictions to the Go SDK proper. Key: BEAM-9978 URL: https://issues.apache.org/jira/browse/BEAM-9978 Project: Beam Issue Type: Improvement Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira Currently these are part of the stringsplit example. but they should probably be generalized and in the actual SDK, and should have adequate testing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9775) Add capability to run SDFs on Flink Runner and Python FnApiRunner.
[ https://issues.apache.org/jira/browse/BEAM-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9775. --- Fix Version/s: Not applicable Resolution: Fixed This seems complete now. We have an example SDF and it runs on both those runners. > Add capability to run SDFs on Flink Runner and Python FnApiRunner. > -- > > Key: BEAM-9775 > URL: https://issues.apache.org/jira/browse/BEAM-9775 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Fix For: Not applicable > > Time Spent: 2h 50m > Remaining Estimate: 0h > > This is pretty simple: Add any missing requirements for being able to > actually execute SDFs, and _maybe_ an example SDF or something that actually > works. This can be marked completed when we can run a simple SDF with the Go > SDK. > I'm still hesitant on the example SDF because it may imply that the feature > is ready for general usage, so it might need to be bundled with the > documentation PR, but we'll see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9799) Add automated RTracker validation to Go SDF
[ https://issues.apache.org/jira/browse/BEAM-9799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9799. --- Fix Version/s: Not applicable Resolution: Fixed > Add automated RTracker validation to Go SDF > --- > > Key: BEAM-9799 > URL: https://issues.apache.org/jira/browse/BEAM-9799 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > After finishing executing an SDF we should be validating the restriction > trackers to make sure they succeeded without errors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9951) Create Go SDK synthetic sources.
Daniel Oliveira created BEAM-9951: - Summary: Create Go SDK synthetic sources. Key: BEAM-9951 URL: https://issues.apache.org/jira/browse/BEAM-9951 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira Create synthetic sources for the Go SDK like [Java|https://github.com/apache/beam/tree/master/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic] and [Python|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/synthetic_pipeline.py] have. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9882) Go SplittableDoFn testing
Daniel Oliveira created BEAM-9882: - Summary: Go SplittableDoFn testing Key: BEAM-9882 URL: https://issues.apache.org/jira/browse/BEAM-9882 Project: Beam Issue Type: Improvement Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira This is a catch-all jira for all tasks needed to fully test SplittableDoFns in Go. For progress on SplittableDoFns themselves, see BEAM-3301 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9799) Add automated RTracker validation to Go SDF
Daniel Oliveira created BEAM-9799: - Summary: Add automated RTracker validation to Go SDF Key: BEAM-9799 URL: https://issues.apache.org/jira/browse/BEAM-9799 Project: Beam Issue Type: Improvement Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira After finishing executing an SDF we should be validating the restriction trackers to make sure they succeeded without errors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9798) Go SplittableDoFn performance testing and optimizations
[ https://issues.apache.org/jira/browse/BEAM-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9798: -- Status: Open (was: Triage Needed) > Go SplittableDoFn performance testing and optimizations > --- > > Key: BEAM-9798 > URL: https://issues.apache.org/jira/browse/BEAM-9798 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > Add SDF benchmarks and implement performance optimizations to speed up SDFs. > Pretty straightforward. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9798) Go SplittableDoFn performance testing and optimizations
Daniel Oliveira created BEAM-9798: - Summary: Go SplittableDoFn performance testing and optimizations Key: BEAM-9798 URL: https://issues.apache.org/jira/browse/BEAM-9798 Project: Beam Issue Type: Improvement Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira Add SDF benchmarks and implement performance optimizations to speed up SDFs. Pretty straightforward. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9788) Allow Go pipelines to use RetainDockerContainers runner option.
[ https://issues.apache.org/jira/browse/BEAM-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9788. --- Fix Version/s: Not applicable Resolution: Fixed > Allow Go pipelines to use RetainDockerContainers runner option. > --- > > Key: BEAM-9788 > URL: https://issues.apache.org/jira/browse/BEAM-9788 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > The Go SDK currently has no mechanism to pass the RetainDockerContainers flag > to runners. This flag is nearly essential for debugging pipelines on the > Flink runner, especially for devs who are testing new features on it (like me > :) ). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9788) Allow Go pipelines to use RetainDockerContainers runner option.
[ https://issues.apache.org/jira/browse/BEAM-9788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9788 started by Daniel Oliveira. - > Allow Go pipelines to use RetainDockerContainers runner option. > --- > > Key: BEAM-9788 > URL: https://issues.apache.org/jira/browse/BEAM-9788 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > The Go SDK currently has no mechanism to pass the RetainDockerContainers flag > to runners. This flag is nearly essential for debugging pipelines on the > Flink runner, especially for devs who are testing new features on it (like me > :) ). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9788) Allow Go pipelines to use RetainDockerContainers runner option.
Daniel Oliveira created BEAM-9788: - Summary: Allow Go pipelines to use RetainDockerContainers runner option. Key: BEAM-9788 URL: https://issues.apache.org/jira/browse/BEAM-9788 Project: Beam Issue Type: Improvement Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira The Go SDK currently has no mechanism to pass the RetainDockerContainers flag to runners. This flag is nearly essential for debugging pipelines on the Flink runner, especially for devs who are testing new features on it (like me :) ). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9775) Add capability to run SDFs on Flink Runner and Python FnApiRunner.
Daniel Oliveira created BEAM-9775: - Summary: Add capability to run SDFs on Flink Runner and Python FnApiRunner. Key: BEAM-9775 URL: https://issues.apache.org/jira/browse/BEAM-9775 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira This is pretty simple: Add any missing requirements for being able to actually execute SDFs, and _maybe_ an example SDF or something that actually works. This can be marked completed when we can run a simple SDF with the Go SDK. I'm still hesitant on the example SDF because it may imply that the feature is ready for general usage, so it might need to be bundled with the documentation PR, but we'll see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9775) Add capability to run SDFs on Flink Runner and Python FnApiRunner.
[ https://issues.apache.org/jira/browse/BEAM-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9775: -- Status: Open (was: Triage Needed) > Add capability to run SDFs on Flink Runner and Python FnApiRunner. > -- > > Key: BEAM-9775 > URL: https://issues.apache.org/jira/browse/BEAM-9775 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > This is pretty simple: Add any missing requirements for being able to > actually execute SDFs, and _maybe_ an example SDF or something that actually > works. This can be marked completed when we can run a simple SDF with the Go > SDK. > I'm still hesitant on the example SDF because it may imply that the feature > is ready for general usage, so it might need to be bundled with the > documentation PR, but we'll see. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9753) Use cmp in fullvalue_test.go
Daniel Oliveira created BEAM-9753: - Summary: Use cmp in fullvalue_test.go Key: BEAM-9753 URL: https://issues.apache.org/jira/browse/BEAM-9753 Project: Beam Issue Type: Improvement Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira We could probably update the comparison helpers in [fullvalue_test.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go] to use cmp options and Transformers instead which would make things much clearer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9753) Use cmp in fullvalue_test.go
[ https://issues.apache.org/jira/browse/BEAM-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9753: -- Status: Open (was: Triage Needed) > Use cmp in fullvalue_test.go > > > Key: BEAM-9753 > URL: https://issues.apache.org/jira/browse/BEAM-9753 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > > We could probably update the comparison helpers in > [fullvalue_test.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go] > to use cmp options and Transformers instead which would make things much > clearer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira reassigned BEAM-9745:
-
Assignee: Kyle Weaver (was: Daniel Oliveira)
Handing it to Kyle who is likely going to be rebuilding the worker in the next
few days. Feel free to hand it back to me if that doesn't fix it.
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kyle Weaver
>Priority: Blocker
> Labels: currently-failing
> Fix For: 2.21.0
>
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
>
[jira] [Commented] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082722#comment-17082722
]
Daniel Oliveira commented on BEAM-9745:
---
Did some asking around. Looks like this may be related to
https://github.com/apache/beam/commit/0cd2fb6633a9d3d9183fc0532336501c3a56406c#diff-ecb570b49f9b4854404be5fbd74b0f22
And it will probably be fixed once the worker is rebuilt.
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Blocker
> Labels: currently-failing
> Fix For: 2.21.0
>
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See
[jira] [Updated] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-9745:
--
Priority: Blocker (was: Major)
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Blocker
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-9745:
--
Description:
_Use this form to file an issue for test failure:_
* [Jenkins
Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
* [Gradle Build Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
Initial investigation:
The bug appears to be popping up on BigQuery tests mostly, but also a BigTable
and a Datastore test.
Here's an example stacktrace of the two errors, showing _only_ the error
messages themselves. Source:
[https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
{noformat}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
received from SDK harness for instruction -191:
java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
Execution Info
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
...
Caused by: java.lang.RuntimeException: Error received from SDK harness for
instruction -191: java.lang.IllegalArgumentException: unable to deserialize
Custom DoFn With Execution Info
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
received from SDK harness for instruction -206:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
Coder Bytes
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
...
Caused by: java.lang.RuntimeException: Error received from SDK harness for
instruction -206:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
Coder Bytes
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
...
{noformat}
Update: Looks like this has been failing as far back as [Apr
4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
after a long period where the test was consistently timing out since [Mar
31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/]. So
it's hard to narrow down what commit may have caused this. Plus, the test was
failing due to a completely different BigQuery failure before anyway, so it
seems like this test will need to be completely fixed from scratch, instead of
tracking down a specific breaking change.
_After you've filled out the above details, please [assign the issue to an
individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
Assignee should [treat test failures as
high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
helping to fix the issue or find a more appropriate owner. See [Apache Beam
Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._
was:
_Use this form to file an issue for test failure:_
* [Jenkins
Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
* [Gradle Build Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
Initial investigation:
The bug appears to be popping up on BigQuery tests mostly, but also a BigTable
and a Datastore test.
Here's an example stacktrace of the two errors, showing _only_ the error
messages themselves. Source:
[https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
{noformat}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
received from SDK harness for instruction -191:
java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
Execution Info
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
...
Caused by: java.lang.RuntimeException: Error received from SDK harness for
instruction -191:
[jira] [Created] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
Daniel Oliveira created BEAM-9745:
-
Summary: [beam_PostCommit_Java_PortabilityApi] Various GCP IO
tests failing, unable to deserialize Custom DoFns and Custom Coders.
Key: BEAM-9745
URL: https://issues.apache.org/jira/browse/BEAM-9745
Project: Beam
Issue Type: Bug
Components: io-java-gcp, java-fn-execution, sdk-java-harness,
test-failures
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira
_Use this form to file an issue for test failure:_
* [Jenkins
Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
* [Gradle Build Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
Initial investigation:
The bug appears to be popping up on BigQuery tests mostly, but also a BigTable
and a Datastore test.
Here's an example stacktrace of the two errors, showing _only_ the error
messages themselves. Source:
[https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
{noformat}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
received from SDK harness for instruction -191:
java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
Execution Info
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
...
Caused by: java.lang.RuntimeException: Error received from SDK harness for
instruction -191: java.lang.IllegalArgumentException: unable to deserialize
Custom DoFn With Execution Info
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
received from SDK harness for instruction -206:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
Coder Bytes
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
...
Caused by: java.lang.RuntimeException: Error received from SDK harness for
instruction -206:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by:
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
...
Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
Coder Bytes
...
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
...
{noformat}
_After you've filled out the above details, please [assign the issue to an
individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
Assignee should [treat test failures as
high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
helping to fix the issue or find a more appropriate owner. See [Apache Beam
Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9721) Dataflow-based Jenkins jobs are failing due to lack of --region option
[ https://issues.apache.org/jira/browse/BEAM-9721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080150#comment-17080150 ] Daniel Oliveira commented on BEAM-9721: --- Looks like this is also affecting beam_PerformanceTests_WordCountIT jobs ([example|https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/1314/]). > Dataflow-based Jenkins jobs are failing due to lack of --region option > -- > > Key: BEAM-9721 > URL: https://issues.apache.org/jira/browse/BEAM-9721 > Project: Beam > Issue Type: Bug > Components: test-failures, testing >Reporter: Michał Walenia >Assignee: Michał Walenia >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9644) Add SDF validation code
[ https://issues.apache.org/jira/browse/BEAM-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9644. --- Fix Version/s: Not applicable Resolution: Fixed > Add SDF validation code > --- > > Key: BEAM-9644 > URL: https://issues.apache.org/jira/browse/BEAM-9644 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Fix For: Not applicable > > > Add code to allow DoFns with SDF methods implemented to pass DoFn validation > and have DoFn objects representing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9644) Add SDF validation code
[ https://issues.apache.org/jira/browse/BEAM-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9644: -- Status: Open (was: Triage Needed) > Add SDF validation code > --- > > Key: BEAM-9644 > URL: https://issues.apache.org/jira/browse/BEAM-9644 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > Add code to allow DoFns with SDF methods implemented to pass DoFn validation > and have DoFn objects representing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9644) Add SDF validation code
Daniel Oliveira created BEAM-9644: - Summary: Add SDF validation code Key: BEAM-9644 URL: https://issues.apache.org/jira/browse/BEAM-9644 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira Add code to allow DoFns with SDF methods implemented to pass DoFn validation and have DoFn objects representing them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9643) Add user-facing Go SDF documentation.
[ https://issues.apache.org/jira/browse/BEAM-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9643: -- Status: Open (was: Triage Needed) > Add user-facing Go SDF documentation. > - > > Key: BEAM-9643 > URL: https://issues.apache.org/jira/browse/BEAM-9643 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > This means adding the documentation about how to use SDFs and the contracts > of all the SDF methods to the Go SDK code, as well as updating the Go SDF > design doc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9643) Add user-facing Go SDF documentation.
Daniel Oliveira created BEAM-9643: - Summary: Add user-facing Go SDF documentation. Key: BEAM-9643 URL: https://issues.apache.org/jira/browse/BEAM-9643 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira This means adding the documentation about how to use SDFs and the contracts of all the SDF methods to the Go SDK code, as well as updating the Go SDF design doc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9642) Add SDF execution-time runners
Daniel Oliveira created BEAM-9642: - Summary: Add SDF execution-time runners Key: BEAM-9642 URL: https://issues.apache.org/jira/browse/BEAM-9642 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira Adds execution-time SDF runner units to the exec package, and any unit tests + helpers required. This is needed to get the expanded SDF URNs to execute in the runner harness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9642) Add SDF execution-time runners
[ https://issues.apache.org/jira/browse/BEAM-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9642: -- Status: Open (was: Triage Needed) > Add SDF execution-time runners > -- > > Key: BEAM-9642 > URL: https://issues.apache.org/jira/browse/BEAM-9642 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > Adds execution-time SDF runner units to the exec package, and any unit tests > + helpers required. > This is needed to get the expanded SDF URNs to execute in the runner harness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9611) Add a test for Iter/ReIter parameters in ParDos (such as after a GBK).
Daniel Oliveira created BEAM-9611: - Summary: Add a test for Iter/ReIter parameters in ParDos (such as after a GBK). Key: BEAM-9611 URL: https://issues.apache.org/jira/browse/BEAM-9611 Project: Beam Issue Type: Test Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira Some bugs popped up recently because DoFn validation caused an error with ParDos that had Iter/ReIter elements. That could've been caught if a test existed that checked the creation of ParDos with Iter/ReIters, so that should be added, either as a new test or as part of some existing integration test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
[ https://issues.apache.org/jira/browse/BEAM-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-9526. --- Fix Version/s: Not applicable Resolution: Fixed > Bug in top.LargestPerKey, disappearing elements. > > > Key: BEAM-9526 > URL: https://issues.apache.org/jira/browse/BEAM-9526 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > There's a bug in top.LargestPerKey that appears when running it on a runner > that has combiner lifting. Since it's similar to a GBK, we should be > inputting a K, V and receiving a K, []V as output. When the runner has > combiner lifting, the []V always contains an empty slice. > > I already found the fix for this, just logging this to attach it to the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
[ https://issues.apache.org/jira/browse/BEAM-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-9526: -- Status: Open (was: Triage Needed) > Bug in top.LargestPerKey, disappearing elements. > > > Key: BEAM-9526 > URL: https://issues.apache.org/jira/browse/BEAM-9526 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > > There's a bug in top.LargestPerKey that appears when running it on a runner > that has combiner lifting. Since it's similar to a GBK, we should be > inputting a K, V and receiving a K, []V as output. When the runner has > combiner lifting, the []V always contains an empty slice. > > I already found the fix for this, just logging this to attach it to the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
[ https://issues.apache.org/jira/browse/BEAM-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-9526 started by Daniel Oliveira. - > Bug in top.LargestPerKey, disappearing elements. > > > Key: BEAM-9526 > URL: https://issues.apache.org/jira/browse/BEAM-9526 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > > There's a bug in top.LargestPerKey that appears when running it on a runner > that has combiner lifting. Since it's similar to a GBK, we should be > inputting a K, V and receiving a K, []V as output. When the runner has > combiner lifting, the []V always contains an empty slice. > > I already found the fix for this, just logging this to attach it to the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9526) Bug in top.LargestPerKey, disappearing elements.
Daniel Oliveira created BEAM-9526: - Summary: Bug in top.LargestPerKey, disappearing elements. Key: BEAM-9526 URL: https://issues.apache.org/jira/browse/BEAM-9526 Project: Beam Issue Type: Bug Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira There's a bug in top.LargestPerKey that appears when running it on a runner that has combiner lifting. Since it's similar to a GBK, we should be inputting a K, V and receiving a K, []V as output. When the runner has combiner lifting, the []V always contains an empty slice. I already found the fix for this, just logging this to attach it to the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9459) Go Postcommit failing at GBK
[
https://issues.apache.org/jira/browse/BEAM-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-9459:
--
Description:
Example: [https://builds.apache.org/job/beam_PostCommit_Go_PR/106/]
[https://scans.gradle.com/s/es67rfaomu26m]
{noformat}
2020/03/06 00:47:41 Submitted job: 2020-03-05_16_47_40-13139296997856231782
2020/03/06 00:47:41 Submitted job: 2020-03-05_16_47_40-13139296997856231782
2020/03/06 00:47:41 Console:
https://console.cloud.google.com/dataflow/job/2020-03-05_16_47_40-13139296997856231782?project=apache-beam-testing
2020/03/06 00:47:41 Logs:
https://console.cloud.google.com/logs/viewer?project=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-03-05_16_47_40-13139296997856231782
...
2020/03/06 00:50:41 Test cogbk:cogbk failed: job
2020-03-05_16_47_40-13139296997856231782 failed{noformat}
And then in the console logs:
[https://pantheon.corp.google.com/logs/viewer?project=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-03-05_16_47_40-13139296997856231782=500=false=2020-03-06T01:01:14.21000Z==true=2020-03-06T00:01:14.460Z=2020-03-06T01:01:14.460Z=PT1H=2020-03-06T00:49:14.413355915Z]
{code:java}
exception: "java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Error received from SDK harness for instruction
-165: process bundle failed for instruction -165 using plan -122 : panic:
Unexpected coder:
CoGBK goroutine 81
[running]:
runtime/debug.Stack(0xc001103970, 0xd2c5e0, 0xc000bd7f40)
/usr/lib/go-1.12/src/runtime/debug/stack.go:24 +0x9d
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.callNoPanic.func1(0xc001103b90)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:40
+0x60
panic(0xd2c5e0, 0xc000bd7f40)
/usr/lib/go-1.12/src/runtime/panic.go:522 +0x1b5
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.MakeElementEncoder(0xc000b99cc0,
0xc000aa4930, 0xc000b64a00)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/coder.go:91
+0x479
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.(*PCollection).Up(0xc000af3dd0,
0x10018e0, 0xc000b57f80, 0x0, 0xc000346b50)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/pcollection.go:59
+0xfe
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.callNoPanic(0x10018e0,
0xc000b57f80, 0xc000346c28, 0x0, 0x0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:43
+0x6c
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.(*Plan).Execute(0xc0002623f0,
0x10018e0, 0xc000b57f80, 0xc0002365a0, 0x4, 0xff0340, 0xc000aa4750, 0xff0380,
0xc000b57fc0, 0xc000346de0, ...)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/plan.go:93
+0xdf
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.(*control).handleInstruction(0xc0001f4680,
0x10017a0, 0xc0001bafc0, 0xc000b57dc0, 0xc0001bafc0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:211
+0xa34
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.Main.func2(0x10017a0,
0xc0001bafc0, 0xc000b57dc0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:118
+0x1cf
created by
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.Main
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:131
+0x6e8
at
[jira] [Updated] (BEAM-9459) Go Postcommit failing at GBK
[
https://issues.apache.org/jira/browse/BEAM-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-9459:
--
Description:
Example: [https://builds.apache.org/job/beam_PostCommit_Go_PR/106/]
[https://scans.gradle.com/s/es67rfaomu26m]
{noformat}
2020/03/06 00:47:41 Submitted job: 2020-03-05_16_47_40-13139296997856231782
2020/03/06 00:47:41 Submitted job: 2020-03-05_16_47_40-13139296997856231782
2020/03/06 00:47:41 Console:
https://console.cloud.google.com/dataflow/job/2020-03-05_16_47_40-13139296997856231782?project=apache-beam-testing
2020/03/06 00:47:41 Logs:
https://console.cloud.google.com/logs/viewer?project=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-03-05_16_47_40-13139296997856231782
...
2020/03/06 00:50:41 Test cogbk:cogbk failed: job
2020-03-05_16_47_40-13139296997856231782 failed{noformat}
And then in the console logs:
[https://pantheon.corp.google.com/logs/viewer?project=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-03-05_16_47_40-13139296997856231782=500=false=2020-03-06T01:01:14.21000Z==true=2020-03-06T00:01:14.460Z=2020-03-06T01:01:14.460Z=PT1H=2020-03-06T00:49:14.413355915Z]
{code:java}
Error message from worker: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Error received from SDK harness for instruction
-489: process bundle failed for instruction -489 using plan -446 : panic:
Unexpected coder:
CoGBK goroutine 87
[running]: runtime/debug.Stack(0xc0010ff970, 0xd2c5e0, 0xc00022e3d0)
/usr/lib/go-1.12/src/runtime/debug/stack.go:24 +0x9d
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.callNoPanic.func1(0xc0010ffb90)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:40
+0x60 panic(0xd2c5e0, 0xc00022e3d0) /usr/lib/go-1.12/src/runtime/panic.go:522
+0x1b5
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.MakeElementEncoder(0xc0013dc460,
0xc0002466c0, 0xc000166000)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/coder.go:91
+0x479
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.(*PCollection).Up(0xc001313dd0,
0x10018e0, 0xc000268080, 0x0, 0xc0013f3b50)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/pcollection.go:59
+0xfe
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.callNoPanic(0x10018e0,
0xc000268080, 0xc0013f3c28, 0x0, 0x0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:43
+0x6c
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.(*Plan).Execute(0xc0013e8000,
0x10018e0, 0xc000268080, 0xc000d14008, 0x4, 0xff0340, 0xc0002461e0, 0xff0380,
0xc0002680c0, 0xc0013f3de0, ...)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/plan.go:93
+0xdf
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.(*control).handleInstruction(0xc0001f4680,
0x10017a0, 0xc0001bafc0, 0xc00136d9c0, 0xc0001bafc0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:211
+0xa34
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.Main.func2(0x10017a0,
0xc0001bafc0, 0xc00136d9c0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:118
+0x1cf created by
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.Main
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:131
+0x6e8
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
[jira] [Created] (BEAM-9459) Go Postcommit failing at GBK
Daniel Oliveira created BEAM-9459:
-
Summary: Go Postcommit failing at GBK
Key: BEAM-9459
URL: https://issues.apache.org/jira/browse/BEAM-9459
Project: Beam
Issue Type: Bug
Components: sdk-go, test-failures
Reporter: Daniel Oliveira
Assignee: Robert Burke
Example: [https://builds.apache.org/job/beam_PostCommit_Go_PR/106/]
[https://scans.gradle.com/s/es67rfaomu26m]
{noformat}
2020/03/06 00:47:41 Submitted job: 2020-03-05_16_47_40-13139296997856231782
2020/03/06 00:47:41 Submitted job: 2020-03-05_16_47_40-13139296997856231782
2020/03/06 00:47:41 Console:
https://console.cloud.google.com/dataflow/job/2020-03-05_16_47_40-13139296997856231782?project=apache-beam-testing
2020/03/06 00:47:41 Logs:
https://console.cloud.google.com/logs/viewer?project=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-03-05_16_47_40-13139296997856231782
...
2020/03/06 00:50:41 Test cogbk:cogbk failed: job
2020-03-05_16_47_40-13139296997856231782 failed{noformat}
And then in the console logs:
[https://pantheon.corp.google.com/logs/viewer?project=apache-beam-testing=dataflow_step%2Fjob_id%2F2020-03-05_16_47_40-13139296997856231782=500=false=2020-03-06T01:01:14.21000Z==true=2020-03-06T00:01:14.460Z=2020-03-06T01:01:14.460Z=PT1H=2020-03-06T00:49:14.413355915Z]
{noformat}
Error message from worker: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Error received from SDK harness for instruction
-489: process bundle failed for instruction -489 using plan -446 : panic:
Unexpected coder:
CoGBK goroutine 87
[running]: runtime/debug.Stack(0xc0010ff970, 0xd2c5e0, 0xc00022e3d0)
/usr/lib/go-1.12/src/runtime/debug/stack.go:24 +0x9d
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.callNoPanic.func1(0xc0010ffb90)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:40
+0x60 panic(0xd2c5e0, 0xc00022e3d0) /usr/lib/go-1.12/src/runtime/panic.go:522
+0x1b5
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.MakeElementEncoder(0xc0013dc460,
0xc0002466c0, 0xc000166000)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/coder.go:91
+0x479
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.(*PCollection).Up(0xc001313dd0,
0x10018e0, 0xc000268080, 0x0, 0xc0013f3b50)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/pcollection.go:59
+0xfe
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.callNoPanic(0x10018e0,
0xc000268080, 0xc0013f3c28, 0x0, 0x0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/util.go:43
+0x6c
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec.(*Plan).Execute(0xc0013e8000,
0x10018e0, 0xc000268080, 0xc000d14008, 0x4, 0xff0340, 0xc0002461e0, 0xff0380,
0xc0002680c0, 0xc0013f3de0, ...)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/plan.go:93
+0xdf
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.(*control).handleInstruction(0xc0001f4680,
0x10017a0, 0xc0001bafc0, 0xc00136d9c0, 0xc0001bafc0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:211
+0xa34
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.Main.func2(0x10017a0,
0xc0001bafc0, 0xc00136d9c0)
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_PR/src/sdks/go/test/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go:118
+0x1cf created by
github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness.Main
[jira] [Resolved] (BEAM-7970) Regenerate Go SDK proto files in correct version
[
https://issues.apache.org/jira/browse/BEAM-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira resolved BEAM-7970.
---
Resolution: Fixed
Think it's really fixed this time. Go protos have been regenerated with a newer
version of the proto compiler and golang/protobuf.
> Regenerate Go SDK proto files in correct version
>
>
> Key: BEAM-7970
> URL: https://issues.apache.org/jira/browse/BEAM-7970
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Generated proto files in the Go SDK currently include this bit:
> {{// This is a compile-time assertion to ensure that this generated file}}
> {{// is compatible with the proto package it is being compiled against.}}
> {{// A compilation error at this line likely means your copy of the}}
> {{// proto package needs to be updated.}}
> {{const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package}}
>
> This indicates that the protos are being generated as proto v2 for whatever
> reason. Most likely, as mentioned by this post with someone with a similar
> issue, because the proto generation binary needs to be rebuilt before
> generating the files again:
> [https://github.com/golang/protobuf/issues/449#issuecomment-340884839]
> This hasn't caused any errors so far, but might eventually cause errors if we
> hit version differences between the v2 and v3 protos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7970) Regenerate Go SDK proto files in correct version
[
https://issues.apache.org/jira/browse/BEAM-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7970:
--
Status: Open (was: Triage Needed)
> Regenerate Go SDK proto files in correct version
>
>
> Key: BEAM-7970
> URL: https://issues.apache.org/jira/browse/BEAM-7970
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Generated proto files in the Go SDK currently include this bit:
> {{// This is a compile-time assertion to ensure that this generated file}}
> {{// is compatible with the proto package it is being compiled against.}}
> {{// A compilation error at this line likely means your copy of the}}
> {{// proto package needs to be updated.}}
> {{const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package}}
>
> This indicates that the protos are being generated as proto v2 for whatever
> reason. Most likely, as mentioned by this post with someone with a similar
> issue, because the proto generation binary needs to be rebuilt before
> generating the files again:
> [https://github.com/golang/protobuf/issues/449#issuecomment-340884839]
> This hasn't caused any errors so far, but might eventually cause errors if we
> hit version differences between the v2 and v3 protos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (BEAM-7970) Regenerate Go SDK proto files in correct version
[
https://issues.apache.org/jira/browse/BEAM-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira reopened BEAM-7970:
---
Old explanation I wrote is unclear, and also I found a way to actually handle
this bug, so I'm reopening this.
> Regenerate Go SDK proto files in correct version
>
>
> Key: BEAM-7970
> URL: https://issues.apache.org/jira/browse/BEAM-7970
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Generated proto files in the Go SDK currently include this bit:
> {{// This is a compile-time assertion to ensure that this generated file}}
> {{// is compatible with the proto package it is being compiled against.}}
> {{// A compilation error at this line likely means your copy of the}}
> {{// proto package needs to be updated.}}
> {{const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package}}
>
> This indicates that the protos are being generated as proto v2 for whatever
> reason. Most likely, as mentioned by this post with someone with a similar
> issue, because the proto generation binary needs to be rebuilt before
> generating the files again:
> [https://github.com/golang/protobuf/issues/449#issuecomment-340884839]
> This hasn't caused any errors so far, but might eventually cause errors if we
> hit version differences between the v2 and v3 protos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-7629) Improve DoFn method validation in core/graph/fn.go
[ https://issues.apache.org/jira/browse/BEAM-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-7629. --- Fix Version/s: Not applicable Resolution: Fixed > Improve DoFn method validation in core/graph/fn.go > -- > > Key: BEAM-7629 > URL: https://issues.apache.org/jira/browse/BEAM-7629 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Fix For: Not applicable > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Various improvements can be made to validating the signatures and type usages > in DoFns. Some things that should probably be checked: > * Check that StartBundle and FinishBundle contain any emit parameters and > side inputs present in ProcessElement > * Check that any side inputs/emits have correctly matching types between > Start/FinishBundle and ProcessElement > * Check that parameters and return values for the various methods are valid > (for ex. Teardown/Setup should have no params) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8493) Add standard double coder to Go SDK
[ https://issues.apache.org/jira/browse/BEAM-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-8493. --- Fix Version/s: Not applicable Resolution: Fixed > Add standard double coder to Go SDK > --- > > Key: BEAM-8493 > URL: https://issues.apache.org/jira/browse/BEAM-8493 > Project: Beam > Issue Type: New Feature > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > For upcoming features, we need to add support for the standard double coder: > [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L573] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (BEAM-8493) Add standard double coder to Go SDK
[ https://issues.apache.org/jira/browse/BEAM-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-8493 started by Daniel Oliveira. - > Add standard double coder to Go SDK > --- > > Key: BEAM-8493 > URL: https://issues.apache.org/jira/browse/BEAM-8493 > Project: Beam > Issue Type: New Feature > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > > For upcoming features, we need to add support for the standard double coder: > [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L573] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8493) Add standard double coder to Go SDK
[ https://issues.apache.org/jira/browse/BEAM-8493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-8493: -- Status: Open (was: Triage Needed) > Add standard double coder to Go SDK > --- > > Key: BEAM-8493 > URL: https://issues.apache.org/jira/browse/BEAM-8493 > Project: Beam > Issue Type: New Feature > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > > For upcoming features, we need to add support for the standard double coder: > [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L573] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8493) Add standard double coder to Go SDK
Daniel Oliveira created BEAM-8493: - Summary: Add standard double coder to Go SDK Key: BEAM-8493 URL: https://issues.apache.org/jira/browse/BEAM-8493 Project: Beam Issue Type: New Feature Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira For upcoming features, we need to add support for the standard double coder: [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L573] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-7970) Regenerate Go SDK proto files in correct version
[
https://issues.apache.org/jira/browse/BEAM-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira resolved BEAM-7970.
---
Fix Version/s: Not applicable
Resolution: Won't Fix
> Regenerate Go SDK proto files in correct version
>
>
> Key: BEAM-7970
> URL: https://issues.apache.org/jira/browse/BEAM-7970
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Generated proto files in the Go SDK currently include this bit:
> {{// This is a compile-time assertion to ensure that this generated file}}
> {{// is compatible with the proto package it is being compiled against.}}
> {{// A compilation error at this line likely means your copy of the}}
> {{// proto package needs to be updated.}}
> {{const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package}}
>
> This indicates that the protos are being generated as proto v2 for whatever
> reason. Most likely, as mentioned by this post with someone with a similar
> issue, because the proto generation binary needs to be rebuilt before
> generating the files again:
> [https://github.com/golang/protobuf/issues/449#issuecomment-340884839]
> This hasn't caused any errors so far, but might eventually cause errors if we
> hit version differences between the v2 and v3 protos.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8167) Replace old Postcommit_Python_Verify test with new python postcommits in Grafana.
[ https://issues.apache.org/jira/browse/BEAM-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-8167. --- Fix Version/s: Not applicable Resolution: Fixed > Replace old Postcommit_Python_Verify test with new python postcommits in > Grafana. > - > > Key: BEAM-8167 > URL: https://issues.apache.org/jira/browse/BEAM-8167 > Project: Beam > Issue Type: Bug > Components: project-management, testing >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Fix For: Not applicable > > Time Spent: 50m > Remaining Estimate: 0h > > The old postcommit was replaced with multiple new postcommits for each > supported python version (Python2, Python35, Python36, Python37). These > aren't being reflected on the Grafana dashboards yet, so that should be fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7970) Regenerate Go SDK proto files in correct version
[
https://issues.apache.org/jira/browse/BEAM-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924656#comment-16924656
]
Daniel Oliveira commented on BEAM-7970:
---
>From my PR attempt:
{quote}Closing because it turns out attempting to regen protos with an updated
version of the proto compiler also requires updating a bunch of the Go SDK's
dependencies, which is a huge can of worms.
And it's mostly unnecessary anyway. The real way to avoid issues is to use an
older version of the proto compiler, so the change that really needs to be made
is updating the documentation to mention which version to use.
{quote}
So I'll be making a PR modifying the message instead and closing this for now.
> Regenerate Go SDK proto files in correct version
>
>
> Key: BEAM-7970
> URL: https://issues.apache.org/jira/browse/BEAM-7970
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Generated proto files in the Go SDK currently include this bit:
> {{// This is a compile-time assertion to ensure that this generated file}}
> {{// is compatible with the proto package it is being compiled against.}}
> {{// A compilation error at this line likely means your copy of the}}
> {{// proto package needs to be updated.}}
> {{const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package}}
>
> This indicates that the protos are being generated as proto v2 for whatever
> reason. Most likely, as mentioned by this post with someone with a similar
> issue, because the proto generation binary needs to be rebuilt before
> generating the files again:
> [https://github.com/golang/protobuf/issues/449#issuecomment-340884839]
> This hasn't caused any errors so far, but might eventually cause errors if we
> hit version differences between the v2 and v3 protos.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Closed] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] KeyError
[ https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira closed BEAM-7619. - Fix Version/s: Not applicable Resolution: Cannot Reproduce Looks like this specific bug was fixed once Dataflow service releases caught up. But the root cause (the blocking bug) is still unresolved, so the issue could pop back up in the future. > [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] KeyError > - > > Key: BEAM-7619 > URL: https://issues.apache.org/jira/browse/BEAM-7619 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Mikhail Gryzykhin >Assignee: Valentyn Tymofieiev >Priority: Major > Labels: currently-failing > Fix For: Not applicable > > > _Use this form to file an issue for test failure:_ > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] > > Initial investigation: > 12:08:13 ERROR: test_wordcount_fnapi_it > (apache_beam.examples.wordcount_it_test.WordCountIT) > 12:08:13 > -- > 12:08:13 Traceback (most recent call last): > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 52, in test_wordcount_fnapi_it > 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 84, in _run_wordcount_it > 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts)) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", > line 114, in run > 12:08:13 result = p.run() > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 406, in run > 12:08:13 self._options).run(False) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 419, in run > 12:08:13 return self.runner.run_pipeline(self, self._options) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", > line 64, in run_pipeline > 12:08:13 self.result.wait_until_finish(duration=wait_duration) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", > line 1338, in wait_until_finish > 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self) > 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, > Error: > 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException: > Error received from SDK harness for instruction -129: Traceback (most recent > call last): > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 157, in _execute > 12:08:13 response = task() > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 190, in > 12:08:13 self._execute(lambda: worker.do_instruction(work), work) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 342, in do_instruction > 12:08:13 request.instruction_id) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 368, in process_bundle > 12:08:13 bundle_processor.process_bundle(instruction_id)) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 593, in process_bundle > 12:08:13 data.ptransform_id].process_encoded(data.data) > 12:08:13 KeyError: u'\n\x04-107\x12\x04-105' > 12:08:13 > 12:08:13 at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > 12:08:13 at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > 12:08:13 at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > 12:08:13 at >
[jira] [Closed] (BEAM-7225) [beam_PostCommit_Py_ValCont ] [test_metrics_fnapi_it] Unable to match metrics for matcher namespace
[ https://issues.apache.org/jira/browse/BEAM-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira closed BEAM-7225. - Fix Version/s: Not applicable Resolution: Cannot Reproduce Looks like this bug is obsolete now. Failure hasn't been popping up. > [beam_PostCommit_Py_ValCont ] [test_metrics_fnapi_it] Unable to match metrics > for matcher namespace > > > Key: BEAM-7225 > URL: https://issues.apache.org/jira/browse/BEAM-7225 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Ahmet Altay >Assignee: Alex Amato >Priority: Major > Fix For: Not applicable > > > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3115/console > 13:19:41 test_metrics_fnapi_it > (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest) > ... FAIL > 13:19:41 > 13:19:41 > == > 13:19:41 FAIL: test_metrics_fnapi_it > (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest) > 13:19:41 > -- > 13:19:41 Traceback (most recent call last): > 13:19:41 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py", > line 70, in test_metrics_fnapi_it > 13:19:41 self.assertFalse(errors, str(errors)) > 13:19:41 AssertionError: Unable to match metrics for matcher namespace: > 'apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline.UserMetricsDoFn' > name: 'total_values' step: 'metrics' attempted: <100> committed: <100> > 13:19:41 Actual MetricResults: -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work started] (BEAM-8167) Replace old Postcommit_Python_Verify test with new python postcommits in Grafana.
[ https://issues.apache.org/jira/browse/BEAM-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-8167 started by Daniel Oliveira. - > Replace old Postcommit_Python_Verify test with new python postcommits in > Grafana. > - > > Key: BEAM-8167 > URL: https://issues.apache.org/jira/browse/BEAM-8167 > Project: Beam > Issue Type: Bug > Components: project-management, testing >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > The old postcommit was replaced with multiple new postcommits for each > supported python version (Python2, Python35, Python36, Python37). These > aren't being reflected on the Grafana dashboards yet, so that should be fixed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (BEAM-8167) Replace old Postcommit_Python_Verify test with new python postcommits in Grafana.
[ https://issues.apache.org/jira/browse/BEAM-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-8167: -- Status: Open (was: Triage Needed) > Replace old Postcommit_Python_Verify test with new python postcommits in > Grafana. > - > > Key: BEAM-8167 > URL: https://issues.apache.org/jira/browse/BEAM-8167 > Project: Beam > Issue Type: Bug > Components: project-management, testing >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > The old postcommit was replaced with multiple new postcommits for each > supported python version (Python2, Python35, Python36, Python37). These > aren't being reflected on the Grafana dashboards yet, so that should be fixed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (BEAM-8167) Replace old Postcommit_Python_Verify test with new python postcommits in Grafana.
Daniel Oliveira created BEAM-8167: - Summary: Replace old Postcommit_Python_Verify test with new python postcommits in Grafana. Key: BEAM-8167 URL: https://issues.apache.org/jira/browse/BEAM-8167 Project: Beam Issue Type: Bug Components: project-management, testing Reporter: Daniel Oliveira Assignee: Daniel Oliveira The old postcommit was replaced with multiple new postcommits for each supported python version (Python2, Python35, Python36, Python37). These aren't being reflected on the Grafana dashboards yet, so that should be fixed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8096) Allow runner to configure "subnetwork"
[ https://issues.apache.org/jira/browse/BEAM-8096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919759#comment-16919759 ] Daniel Oliveira commented on BEAM-8096: --- The change looks good to me. You're right that there don't seem to be any tests for this yet. Feel free to make write the PR and add me or [~lostluck] as a reviewer to it (I believe he'll be back sometime next week). > Allow runner to configure "subnetwork" > -- > > Key: BEAM-8096 > URL: https://issues.apache.org/jira/browse/BEAM-8096 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Affects Versions: 2.15.0 >Reporter: Jack Whelpton >Assignee: Robert Burke >Priority: Major > > When running a Dataflow job, the network can be specified using the --network > flag; however, there is no support for doing the same for the subnetwork. > This would be the go equivalent of the following Java code: > [https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/options/DataflowPipelineWorkerPoolOptions.html#getSubnetwork--|https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java#L151] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (BEAM-7354) Starcgen tool not working when no identifiers specified
[ https://issues.apache.org/jira/browse/BEAM-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-7354. --- Fix Version/s: Not applicable Resolution: Fixed > Starcgen tool not working when no identifiers specified > --- > > Key: BEAM-7354 > URL: https://issues.apache.org/jira/browse/BEAM-7354 > Project: Beam > Issue Type: Bug > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > Fix For: Not applicable > > Time Spent: 50m > Remaining Estimate: 0h > > Stumbled onto this bug, starcgen tool is currently used only with identifiers > specified so this was missed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (BEAM-5709) Tests in BeamFnControlServiceTest are flaky.
[ https://issues.apache.org/jira/browse/BEAM-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-5709. --- Fix Version/s: (was: Not applicable) 2.14.0 Resolution: Fixed My bad, completely forgot to close this. > Tests in BeamFnControlServiceTest are flaky. > > > Key: BEAM-5709 > URL: https://issues.apache.org/jira/browse/BEAM-5709 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Critical > Fix For: 2.14.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlServiceTest.java > Tests for BeamFnControlService are currently flaky. The test attempts to > verify that onCompleted was called on the mock streams, but that function > gets called on a separate thread, so occasionally the function will not have > been called yet, despite the server being shut down. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (BEAM-8096) Allow runner to configure "subnetwork"
[ https://issues.apache.org/jira/browse/BEAM-8096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918769#comment-16918769 ] Daniel Oliveira commented on BEAM-8096: --- Just an update: I took a cursory look through the bug & PR, but I've been busy and probably won't have a chance to give this a real review until tomorrow. > Allow runner to configure "subnetwork" > -- > > Key: BEAM-8096 > URL: https://issues.apache.org/jira/browse/BEAM-8096 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Affects Versions: 2.15.0 >Reporter: Jack Whelpton >Assignee: Robert Burke >Priority: Major > > When running a Dataflow job, the network can be specified using the --network > flag; however, there is no support for doing the same for the subnetwork. > This would be the go equivalent of the following Java code: > [https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/options/DataflowPipelineWorkerPoolOptions.html#getSubnetwork--|https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java#L151] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work started] (BEAM-7970) Regenerate Go SDK proto files in correct version
[
https://issues.apache.org/jira/browse/BEAM-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-7970 started by Daniel Oliveira.
-
> Regenerate Go SDK proto files in correct version
>
>
> Key: BEAM-7970
> URL: https://issues.apache.org/jira/browse/BEAM-7970
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> Generated proto files in the Go SDK currently include this bit:
> {{// This is a compile-time assertion to ensure that this generated file}}
> {{// is compatible with the proto package it is being compiled against.}}
> {{// A compilation error at this line likely means your copy of the}}
> {{// proto package needs to be updated.}}
> {{const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package}}
>
> This indicates that the protos are being generated as proto v2 for whatever
> reason. Most likely, as mentioned by this post with someone with a similar
> issue, because the proto generation binary needs to be rebuilt before
> generating the files again:
> [https://github.com/golang/protobuf/issues/449#issuecomment-340884839]
> This hasn't caused any errors so far, but might eventually cause errors if we
> hit version differences between the v2 and v3 protos.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (BEAM-7970) Regenerate Go SDK proto files in correct version
[
https://issues.apache.org/jira/browse/BEAM-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7970:
--
Status: Open (was: Triage Needed)
> Regenerate Go SDK proto files in correct version
>
>
> Key: BEAM-7970
> URL: https://issues.apache.org/jira/browse/BEAM-7970
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> Generated proto files in the Go SDK currently include this bit:
> {{// This is a compile-time assertion to ensure that this generated file}}
> {{// is compatible with the proto package it is being compiled against.}}
> {{// A compilation error at this line likely means your copy of the}}
> {{// proto package needs to be updated.}}
> {{const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package}}
>
> This indicates that the protos are being generated as proto v2 for whatever
> reason. Most likely, as mentioned by this post with someone with a similar
> issue, because the proto generation binary needs to be rebuilt before
> generating the files again:
> [https://github.com/golang/protobuf/issues/449#issuecomment-340884839]
> This hasn't caused any errors so far, but might eventually cause errors if we
> hit version differences between the v2 and v3 protos.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (BEAM-7970) Regenerate Go SDK proto files in correct version
Daniel Oliveira created BEAM-7970:
-
Summary: Regenerate Go SDK proto files in correct version
Key: BEAM-7970
URL: https://issues.apache.org/jira/browse/BEAM-7970
Project: Beam
Issue Type: Bug
Components: sdk-go
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira
Generated proto files in the Go SDK currently include this bit:
{{// This is a compile-time assertion to ensure that this generated file}}
{{// is compatible with the proto package it is being compiled against.}}
{{// A compilation error at this line likely means your copy of the}}
{{// proto package needs to be updated.}}
{{const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package}}
This indicates that the protos are being generated as proto v2 for whatever
reason. Most likely, as mentioned by this post with someone with a similar
issue, because the proto generation binary needs to be rebuilt before
generating the files again:
[https://github.com/golang/protobuf/issues/449#issuecomment-340884839]
This hasn't caused any errors so far, but might eventually cause errors if we
hit version differences between the v2 and v3 protos.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (BEAM-6825) Improve pipeline construction time error messages in Go SDK.
[ https://issues.apache.org/jira/browse/BEAM-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-6825. --- Resolution: Fixed Fix Version/s: Not applicable Forgot to update this bug, but this effort has been done for about a month. > Improve pipeline construction time error messages in Go SDK. > > > Key: BEAM-6825 > URL: https://issues.apache.org/jira/browse/BEAM-6825 > Project: Beam > Issue Type: New Feature > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > Fix For: Not applicable > > Time Spent: 1.5h > Remaining Estimate: 0h > > Many error messages for common pipeline construction mistakes are unclear and > unhelpful. They need to be improved to provide more context, especially for > newer users. This bug tracks these error message improvements. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (BEAM-7669) [beam_PostCommit_Python3_Verify] [test_big_query_write_new_types] AssertionError, expected data but got empty result
[
https://issues.apache.org/jira/browse/BEAM-7669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876524#comment-16876524
]
Daniel Oliveira commented on BEAM-7669:
---
Valentyn, assigning this to you for triaging since you're working on Python 3
support.
> [beam_PostCommit_Python3_Verify] [test_big_query_write_new_types]
> AssertionError, expected data but got empty result
>
>
> Key: BEAM-7669
> URL: https://issues.apache.org/jira/browse/BEAM-7669
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Python3_Verify/1269/]
> * [Gradle Build Scan|https://scans.gradle.com/s/tkx67xvesxujq]
> * [Test source
> code|https://github.com/apache/beam/blob/07190ee6063743ce20bda1e14c314c5f9f987f50/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py#L163]
> Initial investigation:
> This seems to be the cause of the error according to the Gradle build scan.
> {noformat}
> Traceback (most recent call last):
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam
> /io/gcp/bigquery_write_it_test.py", line 206, in
> test_big_query_write_new_types
> write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 426, in __exit__
> self.run().wait_until_finish()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 51, in run_pipeline
> hc_assert_that(self.result, pickler.loads(on_success_matcher))
> AssertionError:
> Expected: (Expected data is [(b'xyw', datetime.date(2011, 1, 1),
> datetime.time(23, 59, 59, 99)), (b'abc', datetime.date(2000, 1, 1),
> datetime.time(0, 0)), (b'\xe4\xbd\xa0\xe5\xa5\xbd', datetime.date(3000, 12,
> 31), datetime.time(23, 59, 59)), (b'\xab\xac\xad', datetime.date(2000, 1, 1),
> datetime.time(0, 0))])
> but:
> {noformat}
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7669) [beam_PostCommit_Python3_Verify] [test_big_query_write_new_types] AssertionError, expected data but got empty result
Daniel Oliveira created BEAM-7669:
-
Summary: [beam_PostCommit_Python3_Verify]
[test_big_query_write_new_types] AssertionError, expected data but got empty
result
Key: BEAM-7669
URL: https://issues.apache.org/jira/browse/BEAM-7669
Project: Beam
Issue Type: Bug
Components: test-failures
Reporter: Daniel Oliveira
Assignee: Valentyn Tymofieiev
_Use this form to file an issue for test failure:_
* [Jenkins
Job|https://builds.apache.org/job/beam_PostCommit_Python3_Verify/1269/]
* [Gradle Build Scan|https://scans.gradle.com/s/tkx67xvesxujq]
* [Test source
code|https://github.com/apache/beam/blob/07190ee6063743ce20bda1e14c314c5f9f987f50/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py#L163]
Initial investigation:
This seems to be the cause of the error according to the Gradle build scan.
{noformat}
Traceback (most recent call last):
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam
/io/gcp/bigquery_write_it_test.py", line 206, in test_big_query_write_new_types
write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY))
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
line 426, in __exit__
self.run().wait_until_finish()
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
line 406, in run
self._options).run(False)
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/pipeline.py",
line 419, in run
return self.runner.run_pipeline(self, self._options)
File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
line 51, in run_pipeline
hc_assert_that(self.result, pickler.loads(on_success_matcher))
AssertionError:
Expected: (Expected data is [(b'xyw', datetime.date(2011, 1, 1),
datetime.time(23, 59, 59, 99)), (b'abc', datetime.date(2000, 1, 1),
datetime.time(0, 0)), (b'\xe4\xbd\xa0\xe5\xa5\xbd', datetime.date(3000, 12,
31), datetime.time(23, 59, 59)), (b'\xab\xac\xad', datetime.date(2000, 1, 1),
datetime.time(0, 0))])
but:
{noformat}
_After you've filled out the above details, please [assign the issue to an
individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
Assignee should [treat test failures as
high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
helping to fix the issue or find a more appropriate owner. See [Apache Beam
Post-Commit Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] KeyError
[ https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-7619: -- Summary: [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] KeyError (was: [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.) > [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] KeyError > - > > Key: BEAM-7619 > URL: https://issues.apache.org/jira/browse/BEAM-7619 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Mikhail Gryzykhin >Assignee: Valentyn Tymofieiev >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] > > Initial investigation: > 12:08:13 ERROR: test_wordcount_fnapi_it > (apache_beam.examples.wordcount_it_test.WordCountIT) > 12:08:13 > -- > 12:08:13 Traceback (most recent call last): > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 52, in test_wordcount_fnapi_it > 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 84, in _run_wordcount_it > 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts)) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", > line 114, in run > 12:08:13 result = p.run() > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 406, in run > 12:08:13 self._options).run(False) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 419, in run > 12:08:13 return self.runner.run_pipeline(self, self._options) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", > line 64, in run_pipeline > 12:08:13 self.result.wait_until_finish(duration=wait_duration) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", > line 1338, in wait_until_finish > 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self) > 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, > Error: > 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException: > Error received from SDK harness for instruction -129: Traceback (most recent > call last): > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 157, in _execute > 12:08:13 response = task() > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 190, in > 12:08:13 self._execute(lambda: worker.do_instruction(work), work) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 342, in do_instruction > 12:08:13 request.instruction_id) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 368, in process_bundle > 12:08:13 bundle_processor.process_bundle(instruction_id)) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 593, in process_bundle > 12:08:13 data.ptransform_id].process_encoded(data.data) > 12:08:13 KeyError: u'\n\x04-107\x12\x04-105' > 12:08:13 > 12:08:13 at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > 12:08:13 at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > 12:08:13 at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:412) > 12:08:13 at >
[jira] [Comment Edited] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874577#comment-16874577
]
Daniel Oliveira edited comment on BEAM-7648 at 6/28/19 12:37 AM:
-
Actually I may have spoke too soon. The errors seem to be due to beam-15 being
down. I'll leave this bug open for now just in case, but it may be a separate
issue.
*17:02:02* ERROR: apache-beam-jenkins-15 is offline; cannot locate JDK 1.8
(latest)
was (Author: danoliveira):
Actually I may have spoke too soon. The errors seem to be due to beam-15 being
down. I'll leave this bug open for now just in case, but it may be a separate
issue.
*17:02:02* ERROR: apache-beam-jenkins-15 is offline; cannot locate JDK 1.8
(latest)
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
>
[jira] [Updated] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7648:
--
Component/s: test-failures
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: test-failures, testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
> 18:11:49 at
[jira] [Commented] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874577#comment-16874577
]
Daniel Oliveira commented on BEAM-7648:
---
Actually I may have spoke too soon. The errors seem to be due to beam-15 being
down. I'll leave this bug open for now just in case, but it may be a separate
issue.
*17:02:02* ERROR: apache-beam-jenkins-15 is offline; cannot locate JDK 1.8
(latest)
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
>
[jira] [Updated] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7648:
--
Labels: currently-failing (was: )
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: test-failures, testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Labels: currently-failing
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
> 18:11:49 at
>
[jira] [Reopened] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira reopened BEAM-7648:
---
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at com.sun.proxy.$Proxy114.setRemoteUrl(Unknown Source)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
> 18:11:49 at
[jira] [Commented] (BEAM-7648) apache-beam-jenkins-15 out of disk
[
https://issues.apache.org/jira/browse/BEAM-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874576#comment-16874576
]
Daniel Oliveira commented on BEAM-7648:
---
Looks like this is back:
[https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron/946/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3670/]
(and many others)
> apache-beam-jenkins-15 out of disk
> --
>
> Key: BEAM-7648
> URL: https://issues.apache.org/jira/browse/BEAM-7648
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.15.0
>
>
> getting error like these:
> {code:java}
> 18:11:49 ERROR: Error fetching remote repo 'origin'
> 18:11:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 18:11:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 18:11:49 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
> 18:11:49 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
> 18:11:49 at hudson.scm.SCM.checkout(SCM.java:504)
> 18:11:49 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 18:11:49 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 18:11:49 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 18:11:49 at hudson.model.Run.execute(Run.java:1810)
> 18:11:49 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 18:11:49 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 18:11:49 at hudson.model.Executor.run(Executor.java:429)
> 18:11:49 Caused by: hudson.plugins.git.GitException: Command "git config
> remote.origin.url https://github.com/apache/beam.git; returned status code 4:
> 18:11:49 stdout:
> 18:11:49 stderr: error: failed to write new configuration file
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/.git/config.lock
> 18:11:49
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2010)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2006)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1638)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1650)
> 18:11:49 at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1284)
> 18:11:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
> 18:11:49 at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
> 18:11:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 18:11:49 at java.lang.reflect.Method.invoke(Method.java:498)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 18:11:49 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 18:11:49 at hudson.remoting.Request$2.run(Request.java:369)
> 18:11:49 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 18:11:49 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:11:49 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:11:49 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 18:11:49 at java.lang.Thread.run(Thread.java:748)
> 18:11:49 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote
> call to JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 18:11:49 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 18:11:49 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 18:11:49 at hudson.remoting.Channel.call(Channel.java:955)
> 18:11:49 at
> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
> 18:11:49 at
[jira] [Closed] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira closed BEAM-7651.
-
Resolution: Duplicate
Fix Version/s: Not applicable
> Failures in various pre and post-commits: "No space left on device"
> ---
>
> Key: BEAM-7651
> URL: https://issues.apache.org/jira/browse/BEAM-7651
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Mark Liu
>Priority: Major
> Labels: currently-failing
> Fix For: Not applicable
>
>
> I've seen this popping up in several pre and post-commits today. I'll list
> one example with a full message and then several tests where I've seen it
> happen.
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
>
> {noformat}
> 11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
> JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 11:09:46 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 11:09:46 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 11:09:46 at hudson.remoting.Channel.call(Channel.java:955)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1072)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1061)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:1246)
> 11:09:46 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 11:09:46 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 11:09:46 at hudson.model.Run.execute(Run.java:1810)
> 11:09:46 at
> hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 11:09:46 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 11:09:46 at hudson.model.Executor.run(Executor.java:429)
> 11:09:46 java.nio.file.FileSystemException:
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
> No space left on device
> 11:09:46 at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 11:09:46 at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> 11:09:46 at java.nio.file.Files.createDirectory(Files.java:674)
> 11:09:46 at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
> 11:09:46 at java.nio.file.Files.createDirectories(Files.java:767)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:3273)
> 11:09:46 at hudson.FilePath.access$1300(FilePath.java:213)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
> 11:09:46 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 11:09:46 at hudson.remoting.Request$2.run(Request.java:369)
> 11:09:46 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 11:09:46 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 11:09:46 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 11:09:46 at java.lang.Thread.run(Thread.java:748){noformat}
>
> [https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
> [https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
> [https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
>
> Edit:
> This other failure in several post-commits seems related:
> [https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/1054/]
> {noformat}
> 11:04:49 ERROR: Error fetching remote repo 'origin'
> 11:04:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 11:04:49 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
> 11:04:49 at
[jira] [Commented] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874561#comment-16874561
]
Daniel Oliveira commented on BEAM-7651:
---
Looks like this is a duplicate of
[https://issues.apache.org/jira/browse/BEAM-7648], so I'll close this.
> Failures in various pre and post-commits: "No space left on device"
> ---
>
> Key: BEAM-7651
> URL: https://issues.apache.org/jira/browse/BEAM-7651
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Mark Liu
>Priority: Major
> Labels: currently-failing
>
> I've seen this popping up in several pre and post-commits today. I'll list
> one example with a full message and then several tests where I've seen it
> happen.
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
>
> {noformat}
> 11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
> JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 11:09:46 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 11:09:46 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 11:09:46 at hudson.remoting.Channel.call(Channel.java:955)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1072)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1061)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:1246)
> 11:09:46 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 11:09:46 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 11:09:46 at hudson.model.Run.execute(Run.java:1810)
> 11:09:46 at
> hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 11:09:46 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 11:09:46 at hudson.model.Executor.run(Executor.java:429)
> 11:09:46 java.nio.file.FileSystemException:
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
> No space left on device
> 11:09:46 at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 11:09:46 at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> 11:09:46 at java.nio.file.Files.createDirectory(Files.java:674)
> 11:09:46 at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
> 11:09:46 at java.nio.file.Files.createDirectories(Files.java:767)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:3273)
> 11:09:46 at hudson.FilePath.access$1300(FilePath.java:213)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
> 11:09:46 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 11:09:46 at hudson.remoting.Request$2.run(Request.java:369)
> 11:09:46 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 11:09:46 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 11:09:46 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 11:09:46 at java.lang.Thread.run(Thread.java:748){noformat}
>
> [https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
> [https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
> [https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
>
> Edit:
> This other failure in several post-commits seems related:
> [https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/1054/]
> {noformat}
> 11:04:49 ERROR: Error fetching remote repo 'origin'
> 11:04:49 hudson.plugins.git.GitException: Failed to fetch from
> https://github.com/apache/beam.git
> 11:04:49 at
[jira] [Updated] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7651:
--
Description:
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:09:46at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:09:46at hudson.model.Run.execute(Run.java:1810)
11:09:46at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:09:46at
hudson.model.ResourceController.execute(ResourceController.java:97)
11:09:46at hudson.model.Executor.run(Executor.java:429)
11:09:46 java.nio.file.FileSystemException:
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
No space left on device
11:09:46at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
11:09:46at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
11:09:46at java.nio.file.Files.createDirectory(Files.java:674)
11:09:46at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
11:09:46at java.nio.file.Files.createDirectories(Files.java:767)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:3273)
11:09:46at hudson.FilePath.access$1300(FilePath.java:213)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
11:09:46at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:212)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:54)
11:09:46at hudson.remoting.Request$2.run(Request.java:369)
11:09:46at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:09:46at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11:09:46at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
11:09:46at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
11:09:46at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
11:09:46at java.lang.Thread.run(Thread.java:748){noformat}
[https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
[https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
Edit:
This other failure in several post-commits seems related:
[https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/1054/]
{noformat}
11:04:49 ERROR: Error fetching remote repo 'origin'
11:04:49 hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/beam.git
11:04:49at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
11:04:49at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
11:04:49at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
11:04:49at hudson.scm.SCM.checkout(SCM.java:504)
11:04:49at
hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
11:04:49at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:04:49at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:04:49at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:04:49at hudson.model.Run.execute(Run.java:1810)
11:04:49at
[jira] [Updated] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7651:
--
Description:
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:09:46at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:09:46at hudson.model.Run.execute(Run.java:1810)
11:09:46at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:09:46at
hudson.model.ResourceController.execute(ResourceController.java:97)
11:09:46at hudson.model.Executor.run(Executor.java:429)
11:09:46 java.nio.file.FileSystemException:
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
No space left on device
11:09:46at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
11:09:46at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
11:09:46at java.nio.file.Files.createDirectory(Files.java:674)
11:09:46at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
11:09:46at java.nio.file.Files.createDirectories(Files.java:767)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:3273)
11:09:46at hudson.FilePath.access$1300(FilePath.java:213)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
11:09:46at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:212)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:54)
11:09:46at hudson.remoting.Request$2.run(Request.java:369)
11:09:46at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:09:46at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11:09:46at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
11:09:46at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
11:09:46at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
11:09:46at java.lang.Thread.run(Thread.java:748){noformat}
[https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
[https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
Edit:
This other failure in several post-commits seems related:
[https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Cron/1054/]
{noformat}
11:04:49 ERROR: Error fetching remote repo 'origin'
11:04:49 hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/beam.git
11:04:49at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
11:04:49at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
11:04:49at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
11:04:49at hudson.scm.SCM.checkout(SCM.java:504)
11:04:49at
hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
11:04:49at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:04:49at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:04:49at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:04:49at hudson.model.Run.execute(Run.java:1810)
11:04:49at
[jira] [Commented] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874520#comment-16874520
]
Daniel Oliveira commented on BEAM-7651:
---
Mark, this sounds like some sort of quota issue maybe. Could you take look?
> Failures in various pre and post-commits: "No space left on device"
> ---
>
> Key: BEAM-7651
> URL: https://issues.apache.org/jira/browse/BEAM-7651
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Mark Liu
>Priority: Major
> Labels: currently-failing
>
> I've seen this popping up in several pre and post-commits today. I'll list
> one example with a full message and then several tests where I've seen it
> happen.
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
>
> {noformat}
> 11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
> JNLP4-connect connection from
> 103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
> 11:09:46 at
> hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
> 11:09:46 at
> hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
> 11:09:46 at hudson.remoting.Channel.call(Channel.java:955)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1072)
> 11:09:46 at hudson.FilePath.act(FilePath.java:1061)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:1246)
> 11:09:46 at
> hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
> 11:09:46 at
> jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> 11:09:46 at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
> 11:09:46 at hudson.model.Run.execute(Run.java:1810)
> 11:09:46 at
> hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> 11:09:46 at
> hudson.model.ResourceController.execute(ResourceController.java:97)
> 11:09:46 at hudson.model.Executor.run(Executor.java:429)
> 11:09:46 java.nio.file.FileSystemException:
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
> No space left on device
> 11:09:46 at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 11:09:46 at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 11:09:46 at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> 11:09:46 at java.nio.file.Files.createDirectory(Files.java:674)
> 11:09:46 at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
> 11:09:46 at java.nio.file.Files.createDirectories(Files.java:767)
> 11:09:46 at hudson.FilePath.mkdirs(FilePath.java:3273)
> 11:09:46 at hudson.FilePath.access$1300(FilePath.java:213)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
> 11:09:46 at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
> 11:09:46 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
> 11:09:46 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
> 11:09:46 at hudson.remoting.Request$2.run(Request.java:369)
> 11:09:46 at
> hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
> 11:09:46 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 11:09:46 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 11:09:46 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
> 11:09:46 at java.lang.Thread.run(Thread.java:748){noformat}
>
> [https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
> [https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
> [https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
> [https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
[
https://issues.apache.org/jira/browse/BEAM-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7651:
--
Description:
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:09:46at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:09:46at hudson.model.Run.execute(Run.java:1810)
11:09:46at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:09:46at
hudson.model.ResourceController.execute(ResourceController.java:97)
11:09:46at hudson.model.Executor.run(Executor.java:429)
11:09:46 java.nio.file.FileSystemException:
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
No space left on device
11:09:46at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
11:09:46at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
11:09:46at java.nio.file.Files.createDirectory(Files.java:674)
11:09:46at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
11:09:46at java.nio.file.Files.createDirectories(Files.java:767)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:3273)
11:09:46at hudson.FilePath.access$1300(FilePath.java:213)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
11:09:46at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:212)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:54)
11:09:46at hudson.remoting.Request$2.run(Request.java:369)
11:09:46at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:09:46at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11:09:46at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
11:09:46at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
11:09:46at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
11:09:46at java.lang.Thread.run(Thread.java:748){noformat}
[https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
[https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
was:
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
[jira] [Created] (BEAM-7651) Failures in various pre and post-commits: "No space left on device"
Daniel Oliveira created BEAM-7651:
-
Summary: Failures in various pre and post-commits: "No space left
on device"
Key: BEAM-7651
URL: https://issues.apache.org/jira/browse/BEAM-7651
Project: Beam
Issue Type: Bug
Components: test-failures
Reporter: Daniel Oliveira
Assignee: Mark Liu
I've seen this popping up in several pre and post-commits today. I'll list one
example with a full message and then several tests where I've seen it happen.
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
{noformat}
11:09:46 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to
JNLP4-connect connection from
103.55.66.34.bc.googleusercontent.com/34.66.55.103:46040
11:09:46at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
11:09:46at
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
11:09:46at hudson.remoting.Channel.call(Channel.java:955)
11:09:46at hudson.FilePath.act(FilePath.java:1072)
11:09:46at hudson.FilePath.act(FilePath.java:1061)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:1246)
11:09:46at
hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
11:09:46at
jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
11:09:46at
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
11:09:46at hudson.model.Run.execute(Run.java:1810)
11:09:46at
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:09:46at
hudson.model.ResourceController.execute(ResourceController.java:97)
11:09:46at hudson.model.Executor.run(Executor.java:429)
11:09:46 java.nio.file.FileSystemException:
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py27:
No space left on device
11:09:46at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
11:09:46at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
11:09:46at
sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
11:09:46at java.nio.file.Files.createDirectory(Files.java:674)
11:09:46at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
11:09:46at java.nio.file.Files.createDirectories(Files.java:767)
11:09:46at hudson.FilePath.mkdirs(FilePath.java:3273)
11:09:46at hudson.FilePath.access$1300(FilePath.java:213)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1254)
11:09:46at hudson.FilePath$Mkdirs.invoke(FilePath.java:1250)
11:09:46at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:212)
11:09:46at hudson.remoting.UserRequest.perform(UserRequest.java:54)
11:09:46at hudson.remoting.Request$2.run(Request.java:369)
11:09:46at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:09:46at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11:09:46at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
11:09:46at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
11:09:46at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
11:09:46at java.lang.Thread.run(Thread.java:748){noformat}
[https://builds.apache.org/job/beam_PreCommit_RAT_Cron/978/]
[https://builds.apache.org/job/beam_PreCommit_Website_Cron/1196/]
[https://builds.apache.org/job/beam_PreCommit_Python_Cron/1482/]
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3667/]
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/176/]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-7644) Python PerformanceTests failing to build: "Project not found in root project beam"
[
https://issues.apache.org/jira/browse/BEAM-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873701#comment-16873701
]
Daniel Oliveira edited comment on BEAM-7644 at 6/26/19 11:27 PM:
-
Valentyn, you're working on Python 3 correct? This seems like it could be
related. Could you triage this?
was (Author: danoliveira):
Valentyn, you're working on Python 3 correct? Could you triage this?
> Python PerformanceTests failing to build: "Project not found in root project
> beam"
> --
>
> Key: BEAM-7644
> URL: https://issues.apache.org/jira/browse/BEAM-7644
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> The following four tests are failing:
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/170/]
> [https://scans.gradle.com/s/3zkscha7sgook]
> {noformat}
> Project 'sdks/python' not found in root project 'beam'.
> {noformat}
>
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py35/167/]
> [https://scans.gradle.com/s/op6i7nkofdq2q]
> {noformat}
> Project 'sdks/python/test-suites/dataflow/py35' not found in root project
> 'beam'.
> {noformat}
>
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/144/]
> [https://scans.gradle.com/s/5uwtzzkjpnj7k]
> {noformat}
> Project 'sdks/python/test-suites/dataflow/py36' not found in root project
> 'beam'.
> {noformat}
>
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/145/]
> [https://scans.gradle.com/s/7avntl75tgpys]
> {noformat}
> Project 'sdks/python/test-suites/dataflow/py37' not found in root project
> 'beam'.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7644) Python PerformanceTests failing to build: "Project not found in root project beam"
[
https://issues.apache.org/jira/browse/BEAM-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873701#comment-16873701
]
Daniel Oliveira commented on BEAM-7644:
---
Valentyn, you're working on Python 3 correct? Could you triage this?
> Python PerformanceTests failing to build: "Project not found in root project
> beam"
> --
>
> Key: BEAM-7644
> URL: https://issues.apache.org/jira/browse/BEAM-7644
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Daniel Oliveira
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> The following four tests are failing:
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/170/]
> [https://scans.gradle.com/s/3zkscha7sgook]
> {noformat}
> Project 'sdks/python' not found in root project 'beam'.
> {noformat}
>
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py35/167/]
> [https://scans.gradle.com/s/op6i7nkofdq2q]
> {noformat}
> Project 'sdks/python/test-suites/dataflow/py35' not found in root project
> 'beam'.
> {noformat}
>
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/144/]
> [https://scans.gradle.com/s/5uwtzzkjpnj7k]
> {noformat}
> Project 'sdks/python/test-suites/dataflow/py36' not found in root project
> 'beam'.
> {noformat}
>
> [https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/145/]
> [https://scans.gradle.com/s/7avntl75tgpys]
> {noformat}
> Project 'sdks/python/test-suites/dataflow/py37' not found in root project
> 'beam'.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7644) Python PerformanceTests failing to build: "Project not found in root project beam"
Daniel Oliveira created BEAM-7644:
-
Summary: Python PerformanceTests failing to build: "Project not
found in root project beam"
Key: BEAM-7644
URL: https://issues.apache.org/jira/browse/BEAM-7644
Project: Beam
Issue Type: Bug
Components: test-failures
Reporter: Daniel Oliveira
Assignee: Valentyn Tymofieiev
The following four tests are failing:
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/170/]
[https://scans.gradle.com/s/3zkscha7sgook]
{noformat}
Project 'sdks/python' not found in root project 'beam'.
{noformat}
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py35/167/]
[https://scans.gradle.com/s/op6i7nkofdq2q]
{noformat}
Project 'sdks/python/test-suites/dataflow/py35' not found in root project
'beam'.
{noformat}
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/144/]
[https://scans.gradle.com/s/5uwtzzkjpnj7k]
{noformat}
Project 'sdks/python/test-suites/dataflow/py36' not found in root project
'beam'.
{noformat}
[https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/145/]
[https://scans.gradle.com/s/7avntl75tgpys]
{noformat}
Project 'sdks/python/test-suites/dataflow/py37' not found in root project
'beam'.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7643) Nearly all PostCommits failing due to Google Cloud issues
Daniel Oliveira created BEAM-7643:
-
Summary: Nearly all PostCommits failing due to Google Cloud issues
Key: BEAM-7643
URL: https://issues.apache.org/jira/browse/BEAM-7643
Project: Beam
Issue Type: Bug
Components: test-failures
Reporter: Daniel Oliveira
Assignee: Mark Liu
Multiple Beam PostCommits seem to be failing due to a variety of errors, such
as "503 Service unavailable", "429 Too Many Requests", or "404 Not Found".
It's hard to add a complete description of the issue here since there are too
many tests failing. Also made worse because some other flakes and issues seem
to be masking these failures. But I'll try to find a few examples:
[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3661/]
{noformat}
11:17:01 BadStatusCodeError: HttpError accessing
:
response: <{'status': '429', 'content-length': '598', 'x-xss-protection': '0',
'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary':
'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip',
'cache-control': 'private', 'date': 'Wed, 26 Jun 2019 18:17:01 GMT',
'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json;
charset=UTF-8'}>, content <{
11:17:01 "error": {
11:17:01 "code": 429,
11:17:01 "message": "Quota exceeded for quota metric
'dataflow.googleapis.com/create_requests' and limit
'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for
consumer 'project_number:844138762903'.",
11:17:01 "status": "RESOURCE_EXHAUSTED",
11:17:01 "details": [
11:17:01 {
11:17:01 "@type": "type.googleapis.com/google.rpc.Help",
11:17:01 "links": [
11:17:01 {
11:17:01 "description": "Google developer console API key",
11:17:01 "url":
"https://console.developers.google.com/project/844138762903/apiui/credential;
11:17:01 }
11:17:01 ]
11:17:01 }
11:17:01 ]
11:17:01 }
11:17:01 }{noformat}
[https://builds.apache.org/job/beam_PostCommit_Python_Verify/8598/]
{noformat}
BadStatusCodeError: HttpError accessing
:
response: <{'status': '429', 'content-length': '598', 'x-xss-protection': '0',
'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary':
'Origin, X-Origin, Referer', 'se
rver': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date':
'Wed, 26 Jun 2019 18:10:42 GMT', 'x-frame-options': 'SAMEORIGIN',
'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 429,
"message": "Quota exceeded for quota metric
'dataflow.googleapis.com/create_requests' and limit
'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for
consumer 'project_number:844138762903'.",
"status": "RESOURCE_EXHAUSTED",
"details": [
{
"@type": "type.googleapis.com/google.rpc.Help",
"links": [
{
"description": "Google developer console API key",
"url":
"https://console.developers.google.com/project/844138762903/apiui/credential;
}
{noformat}
[https://builds.apache.org/job/beam_PostCommit_Java/3617/]
{noformat}
java.lang.RuntimeException : Failed to create a workflow job: The service is
currently unavailable.
Caused by:
com.google.api.client.googleapis.json.GoogleJsonResponseException:
503 Service Unavailable
{
"code" : 503,
"errors" : [ {
"domain" : "global",
"message" : "The service is currently unavailable.",
"reason" : "backendError"
} ],
"message" : "The service is currently unavailable.",
"status" : "UNAVAILABLE"
}
Dataflow SDK version: 2.15.0-SNAPSHOT{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7629) Improve DoFn method validation in core/graph/fn.go
Daniel Oliveira created BEAM-7629: - Summary: Improve DoFn method validation in core/graph/fn.go Key: BEAM-7629 URL: https://issues.apache.org/jira/browse/BEAM-7629 Project: Beam Issue Type: Improvement Components: sdk-go Reporter: Daniel Oliveira Assignee: Daniel Oliveira Various improvements can be made to validating the signatures and type usages in DoFns. Some things that should probably be checked: * Check that StartBundle and FinishBundle contain any emit parameters and side inputs present in ProcessElement * Check that any side inputs/emits have correctly matching types between Start/FinishBundle and ProcessElement * Check that parameters and return values for the various methods are valid (for ex. Teardown/Setup should have no params) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7629) Improve DoFn method validation in core/graph/fn.go
[ https://issues.apache.org/jira/browse/BEAM-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-7629: -- Status: Open (was: Triage Needed) > Improve DoFn method validation in core/graph/fn.go > -- > > Key: BEAM-7629 > URL: https://issues.apache.org/jira/browse/BEAM-7629 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Major > > Various improvements can be made to validating the signatures and type usages > in DoFns. Some things that should probably be checked: > * Check that StartBundle and FinishBundle contain any emit parameters and > side inputs present in ProcessElement > * Check that any side inputs/emits have correctly matching types between > Start/FinishBundle and ProcessElement > * Check that parameters and return values for the various methods are valid > (for ex. Teardown/Setup should have no params) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
[ https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira reassigned BEAM-7619: - Assignee: Valentyn Tymofieiev (was: Daniel Oliveira) > [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] > java.lang.IllegalStateException: Already closed. > - > > Key: BEAM-7619 > URL: https://issues.apache.org/jira/browse/BEAM-7619 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Mikhail Gryzykhin >Assignee: Valentyn Tymofieiev >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] > > Initial investigation: > 12:08:13 ERROR: test_wordcount_fnapi_it > (apache_beam.examples.wordcount_it_test.WordCountIT) > 12:08:13 > -- > 12:08:13 Traceback (most recent call last): > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 52, in test_wordcount_fnapi_it > 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 84, in _run_wordcount_it > 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts)) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", > line 114, in run > 12:08:13 result = p.run() > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 406, in run > 12:08:13 self._options).run(False) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 419, in run > 12:08:13 return self.runner.run_pipeline(self, self._options) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", > line 64, in run_pipeline > 12:08:13 self.result.wait_until_finish(duration=wait_duration) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", > line 1338, in wait_until_finish > 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self) > 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, > Error: > 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException: > Error received from SDK harness for instruction -129: Traceback (most recent > call last): > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 157, in _execute > 12:08:13 response = task() > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 190, in > 12:08:13 self._execute(lambda: worker.do_instruction(work), work) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 342, in do_instruction > 12:08:13 request.instruction_id) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 368, in process_bundle > 12:08:13 bundle_processor.process_bundle(instruction_id)) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 593, in process_bundle > 12:08:13 data.ptransform_id].process_encoded(data.data) > 12:08:13 KeyError: u'\n\x04-107\x12\x04-105' > 12:08:13 > 12:08:13 at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > 12:08:13 at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > 12:08:13 at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:412) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:381) > 12:08:13 at >
[jira] [Commented] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
[ https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869936#comment-16869936 ] Daniel Oliveira commented on BEAM-7619: --- Marking this as blocked by the probable cause of this bug, but leaving it open otherwise to avoid this bug getting filed again. > [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] > java.lang.IllegalStateException: Already closed. > - > > Key: BEAM-7619 > URL: https://issues.apache.org/jira/browse/BEAM-7619 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Mikhail Gryzykhin >Assignee: Daniel Oliveira >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] > > Initial investigation: > 12:08:13 ERROR: test_wordcount_fnapi_it > (apache_beam.examples.wordcount_it_test.WordCountIT) > 12:08:13 > -- > 12:08:13 Traceback (most recent call last): > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 52, in test_wordcount_fnapi_it > 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 84, in _run_wordcount_it > 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts)) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", > line 114, in run > 12:08:13 result = p.run() > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 406, in run > 12:08:13 self._options).run(False) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 419, in run > 12:08:13 return self.runner.run_pipeline(self, self._options) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", > line 64, in run_pipeline > 12:08:13 self.result.wait_until_finish(duration=wait_duration) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", > line 1338, in wait_until_finish > 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self) > 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, > Error: > 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException: > Error received from SDK harness for instruction -129: Traceback (most recent > call last): > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 157, in _execute > 12:08:13 response = task() > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 190, in > 12:08:13 self._execute(lambda: worker.do_instruction(work), work) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 342, in do_instruction > 12:08:13 request.instruction_id) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 368, in process_bundle > 12:08:13 bundle_processor.process_bundle(instruction_id)) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 593, in process_bundle > 12:08:13 data.ptransform_id].process_encoded(data.data) > 12:08:13 KeyError: u'\n\x04-107\x12\x04-105' > 12:08:13 > 12:08:13 at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > 12:08:13 at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > 12:08:13 at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:412) > 12:08:13 at >
[jira] [Updated] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
[ https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-7619: -- Status: Open (was: Triage Needed) > [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] > java.lang.IllegalStateException: Already closed. > - > > Key: BEAM-7619 > URL: https://issues.apache.org/jira/browse/BEAM-7619 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Mikhail Gryzykhin >Assignee: Daniel Oliveira >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] > > Initial investigation: > 12:08:13 ERROR: test_wordcount_fnapi_it > (apache_beam.examples.wordcount_it_test.WordCountIT) > 12:08:13 > -- > 12:08:13 Traceback (most recent call last): > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 52, in test_wordcount_fnapi_it > 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 84, in _run_wordcount_it > 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts)) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", > line 114, in run > 12:08:13 result = p.run() > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 406, in run > 12:08:13 self._options).run(False) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 419, in run > 12:08:13 return self.runner.run_pipeline(self, self._options) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", > line 64, in run_pipeline > 12:08:13 self.result.wait_until_finish(duration=wait_duration) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", > line 1338, in wait_until_finish > 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self) > 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, > Error: > 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException: > Error received from SDK harness for instruction -129: Traceback (most recent > call last): > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 157, in _execute > 12:08:13 response = task() > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 190, in > 12:08:13 self._execute(lambda: worker.do_instruction(work), work) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 342, in do_instruction > 12:08:13 request.instruction_id) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 368, in process_bundle > 12:08:13 bundle_processor.process_bundle(instruction_id)) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 593, in process_bundle > 12:08:13 data.ptransform_id].process_encoded(data.data) > 12:08:13 KeyError: u'\n\x04-107\x12\x04-105' > 12:08:13 > 12:08:13 at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > 12:08:13 at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > 12:08:13 at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:412) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:381) > 12:08:13 at >
[jira] [Commented] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
[ https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869926#comment-16869926 ] Daniel Oliveira commented on BEAM-7619: --- After some discussion with [~lcwik], sounds like this bug is likely another instance of a known bug where some tests are using the incorrect container image. This bug started happening around May 24 when that bug was revealed, and since the test is validating containers it sounds very likely that it would be hitting that bug. > [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] > java.lang.IllegalStateException: Already closed. > - > > Key: BEAM-7619 > URL: https://issues.apache.org/jira/browse/BEAM-7619 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Mikhail Gryzykhin >Assignee: Daniel Oliveira >Priority: Major > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * [Jenkins > Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]] > > Initial investigation: > 12:08:13 ERROR: test_wordcount_fnapi_it > (apache_beam.examples.wordcount_it_test.WordCountIT) > 12:08:13 > -- > 12:08:13 Traceback (most recent call last): > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 52, in test_wordcount_fnapi_it > 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py", > line 84, in _run_wordcount_it > 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts)) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py", > line 114, in run > 12:08:13 result = p.run() > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 406, in run > 12:08:13 self._options).run(False) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py", > line 419, in run > 12:08:13 return self.runner.run_pipeline(self, self._options) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", > line 64, in run_pipeline > 12:08:13 self.result.wait_until_finish(duration=wait_duration) > 12:08:13 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", > line 1338, in wait_until_finish > 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self) > 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, > Error: > 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException: > Error received from SDK harness for instruction -129: Traceback (most recent > call last): > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 157, in _execute > 12:08:13 response = task() > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 190, in > 12:08:13 self._execute(lambda: worker.do_instruction(work), work) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 342, in do_instruction > 12:08:13 request.instruction_id) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 368, in process_bundle > 12:08:13 bundle_processor.process_bundle(instruction_id)) > 12:08:13 File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 593, in process_bundle > 12:08:13 data.ptransform_id].process_encoded(data.data) > 12:08:13 KeyError: u'\n\x04-107\x12\x04-105' > 12:08:13 > 12:08:13 at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > 12:08:13 at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > 12:08:13 at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) > 12:08:13 at > org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) > 12:08:13 at >
[jira] [Comment Edited] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
[
https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869911#comment-16869911
]
Daniel Oliveira edited comment on BEAM-7619 at 6/21/19 10:01 PM:
-
Looks like there's an error occurring in the SDK Harness:
{noformat}
12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException:
Error received from SDK harness for instruction -129: Traceback (most recent
call last):
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 157, in _execute
12:08:13 response = task()
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 190, in
12:08:13 self._execute(lambda: worker.do_instruction(work), work)
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 342, in do_instruction
12:08:13 request.instruction_id)
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 368, in process_bundle
12:08:13 bundle_processor.process_bundle(instruction_id))
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 593, in process_bundle
12:08:13 data.ptransform_id].process_encoded(data.data)
12:08:13 KeyError: u'\n\x04-107\x12\x04-105'
{noformat}
Seems like in [bundle_processor.py line
593|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L593]
we use a ptransform_id that isn't present in the dict for whatever reason.
I'll try to look into any changes that were made back when the test started
failing, or try to find people familiar with this code.
was (Author: danoliveira):
Looks like there's an error occurring in the SDK Harness:
{noformat}
12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException:
Error received from SDK harness for instruction -129: Traceback (most recent
call last):
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 157, in _execute
12:08:13 response = task()
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 190, in
12:08:13 self._execute(lambda: worker.do_instruction(work), work)
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 342, in do_instruction
12:08:13 request.instruction_id)
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 368, in process_bundle
12:08:13 bundle_processor.process_bundle(instruction_id))
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 593, in process_bundle
12:08:13 data.ptransform_id].process_encoded(data.data)
12:08:13 KeyError: u'\n\x04-107\x12\x04-105'
{noformat}
Seems like in [bundle_processor.py line
593|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L593]
we use a `ptransform_id` that isn't present in the dict for whatever reason.
I'll try to look into any changes that were made back when the test started
failing, or try to find people familiar with this code.
> [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it]
> java.lang.IllegalStateException: Already closed.
> -
>
> Key: BEAM-7619
> URL: https://issues.apache.org/jira/browse/BEAM-7619
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Daniel Oliveira
>Priority: Major
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]]
>
> Initial investigation:
> 12:08:13 ERROR: test_wordcount_fnapi_it
> (apache_beam.examples.wordcount_it_test.WordCountIT)
> 12:08:13
> --
> 12:08:13 Traceback (most recent call last):
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
> line 52, in test_wordcount_fnapi_it
> 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api')
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
> line 84, in _run_wordcount_it
> 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts))
> 12:08:13 File
>
[jira] [Commented] (BEAM-7619) [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it] java.lang.IllegalStateException: Already closed.
[
https://issues.apache.org/jira/browse/BEAM-7619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869911#comment-16869911
]
Daniel Oliveira commented on BEAM-7619:
---
Looks like there's an error occurring in the SDK Harness:
{noformat}
12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException:
Error received from SDK harness for instruction -129: Traceback (most recent
call last):
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 157, in _execute
12:08:13 response = task()
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 190, in
12:08:13 self._execute(lambda: worker.do_instruction(work), work)
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 342, in do_instruction
12:08:13 request.instruction_id)
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 368, in process_bundle
12:08:13 bundle_processor.process_bundle(instruction_id))
12:08:13 File
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 593, in process_bundle
12:08:13 data.ptransform_id].process_encoded(data.data)
12:08:13 KeyError: u'\n\x04-107\x12\x04-105'
{noformat}
Seems like in [bundle_processor.py line
593|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L593]
we use a `ptransform_id` that isn't present in the dict for whatever reason.
I'll try to look into any changes that were made back when the test started
failing, or try to find people familiar with this code.
> [beam_PostCommit_Py_ValCont] [test_metrics_fnapi_it]
> java.lang.IllegalStateException: Already closed.
> -
>
> Key: BEAM-7619
> URL: https://issues.apache.org/jira/browse/BEAM-7619
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Daniel Oliveira
>Priority: Major
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|[https://builds.apache.org/job/beam_PostCommit_Py_ValCont/3625/consoleFull]]
>
> Initial investigation:
> 12:08:13 ERROR: test_wordcount_fnapi_it
> (apache_beam.examples.wordcount_it_test.WordCountIT)
> 12:08:13
> --
> 12:08:13 Traceback (most recent call last):
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
> line 52, in test_wordcount_fnapi_it
> 12:08:13 self._run_wordcount_it(wordcount.run, experiment='beam_fn_api')
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount_it_test.py",
> line 84, in _run_wordcount_it
> 12:08:13 run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts))
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/examples/wordcount.py",
> line 114, in run
> 12:08:13 result = p.run()
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 12:08:13 self._options).run(False)
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 12:08:13 return self.runner.run_pipeline(self, self._options)
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
> line 64, in run_pipeline
> 12:08:13 self.result.wait_until_finish(duration=wait_duration)
> 12:08:13 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_ValCont/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 1338, in wait_until_finish
> 12:08:13 (self.state, getattr(self._runner, 'last_error_msg', None)), self)
> 12:08:13 DataflowRuntimeException: Dataflow pipeline failed. State: FAILED,
> Error:
> 12:08:13 java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> Error received from SDK harness for instruction -129: Traceback (most recent
> call last):
> 12:08:13 File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 157, in _execute
> 12:08:13 response = task()
> 12:08:13 File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 190, in
> 12:08:13 self._execute(lambda: worker.do_instruction(work), work)
> 12:08:13 File
>
[jira] [Updated] (BEAM-7259) Add monthly aggregated view for "Stability critical jobs status - Greenness per week"
[ https://issues.apache.org/jira/browse/BEAM-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-7259: -- Status: Open (was: Triage Needed) > Add monthly aggregated view for "Stability critical jobs status - Greenness > per week" > - > > Key: BEAM-7259 > URL: https://issues.apache.org/jira/browse/BEAM-7259 > Project: Beam > Issue Type: New Feature > Components: project-management, testing >Reporter: Daniel Oliveira >Assignee: Mikhail Gryzykhin >Priority: Minor > > Hey Mikhail, could you create a view of the Greenness metric in Grafana > ([Stability Critical Jobs > Status|http://104.154.241.245/d/McTAiu0ik/stability-critical-jobs-status?orgId=1]) > that's aggregated per month? Specifically, as an average greenness for an > entire calendar month (ex. April 1 - April 30). > I need to track this metric and seeing the weekly greenness makes it hard to > average over a month, especially because different weeks may have different > numbers of runs, and they don't line up perfectly over a calendar month. I > already tried changing the timespan in the upper right, but when I set it to > a month range the graph's points are still one point per week instead of a > monthly average. > Also just to be clear, this shouldn't replace the existing graph, just be a > new one placed wherever makes the most sense. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7259) Add monthly aggregated view for "Stability critical jobs status - Greenness per week"
[ https://issues.apache.org/jira/browse/BEAM-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira resolved BEAM-7259. --- Resolution: Implemented Fix Version/s: Not applicable > Add monthly aggregated view for "Stability critical jobs status - Greenness > per week" > - > > Key: BEAM-7259 > URL: https://issues.apache.org/jira/browse/BEAM-7259 > Project: Beam > Issue Type: New Feature > Components: project-management, testing >Reporter: Daniel Oliveira >Assignee: Mikhail Gryzykhin >Priority: Minor > Fix For: Not applicable > > > Hey Mikhail, could you create a view of the Greenness metric in Grafana > ([Stability Critical Jobs > Status|http://104.154.241.245/d/McTAiu0ik/stability-critical-jobs-status?orgId=1]) > that's aggregated per month? Specifically, as an average greenness for an > entire calendar month (ex. April 1 - April 30). > I need to track this metric and seeing the weekly greenness makes it hard to > average over a month, especially because different weeks may have different > numbers of runs, and they don't line up perfectly over a calendar month. I > already tried changing the timespan in the upper right, but when I set it to > a month range the graph's points are still one point per week instead of a > monthly average. > Also just to be clear, this shouldn't replace the existing graph, just be a > new one placed wherever makes the most sense. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7259) Add monthly aggregated view for "Stability critical jobs status - Greenness per week"
[ https://issues.apache.org/jira/browse/BEAM-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855920#comment-16855920 ] Daniel Oliveira commented on BEAM-7259: --- This was added: http://104.154.241.245/d/McTAiu0ik/stability-critical-jobs-status?orgId=1 https://github.com/apache/beam/pull/8630 Forgot to update this bug. I'll mark it closed. > Add monthly aggregated view for "Stability critical jobs status - Greenness > per week" > - > > Key: BEAM-7259 > URL: https://issues.apache.org/jira/browse/BEAM-7259 > Project: Beam > Issue Type: New Feature > Components: project-management, testing >Reporter: Daniel Oliveira >Assignee: Mikhail Gryzykhin >Priority: Minor > > Hey Mikhail, could you create a view of the Greenness metric in Grafana > ([Stability Critical Jobs > Status|http://104.154.241.245/d/McTAiu0ik/stability-critical-jobs-status?orgId=1]) > that's aggregated per month? Specifically, as an average greenness for an > entire calendar month (ex. April 1 - April 30). > I need to track this metric and seeing the weekly greenness makes it hard to > average over a month, especially because different weeks may have different > numbers of runs, and they don't line up perfectly over a calendar month. I > already tried changing the timespan in the upper right, but when I set it to > a month range the graph's points are still one point per week instead of a > monthly average. > Also just to be clear, this shouldn't replace the existing graph, just be a > new one placed wherever makes the most sense. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7488) Grafana request: Add monthly aggregated view for "Precommit Test Latency - Time In Queue"
Daniel Oliveira created BEAM-7488: - Summary: Grafana request: Add monthly aggregated view for "Precommit Test Latency - Time In Queue" Key: BEAM-7488 URL: https://issues.apache.org/jira/browse/BEAM-7488 Project: Beam Issue Type: New Feature Components: project-management, testing Reporter: Daniel Oliveira Assignee: Mikhail Gryzykhin Similar to [BEAM-7259|https://issues.apache.org/jira/browse/BEAM-7259], was hoping to get this metric in a view that's aggregated per month. In this case averaging all the queue times seems like it would work well. Even better if you could average only the 95th percentile. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5709) Tests in BeamFnControlServiceTest are flaky.
[ https://issues.apache.org/jira/browse/BEAM-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855163#comment-16855163 ] Daniel Oliveira commented on BEAM-5709: --- I did another fix to this and it seems to be working so far (I haven't seen any flakes yet). I'll close this bug in a few days if no one reports any flakes. > Tests in BeamFnControlServiceTest are flaky. > > > Key: BEAM-5709 > URL: https://issues.apache.org/jira/browse/BEAM-5709 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Critical > Fix For: Not applicable > > Time Spent: 4h 20m > Remaining Estimate: 0h > > https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlServiceTest.java > Tests for BeamFnControlService are currently flaky. The test attempts to > verify that onCompleted was called on the mock streams, but that function > gets called on a separate thread, so occasionally the function will not have > been called yet, despite the server being shut down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-3301) Go SplittableDoFn support
[ https://issues.apache.org/jira/browse/BEAM-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-3301 started by Daniel Oliveira. - > Go SplittableDoFn support > - > > Key: BEAM-3301 > URL: https://issues.apache.org/jira/browse/BEAM-3301 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Henning Rohde >Assignee: Daniel Oliveira >Priority: Major > > SDFs will be the only way to add streaming and liquid sharded IO for Go. > Design doc: https://s.apache.org/splittable-do-fn -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3301) Go SplittableDoFn support
[ https://issues.apache.org/jira/browse/BEAM-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Oliveira updated BEAM-3301: -- Priority: Major (was: Minor) > Go SplittableDoFn support > - > > Key: BEAM-3301 > URL: https://issues.apache.org/jira/browse/BEAM-3301 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Henning Rohde >Assignee: Daniel Oliveira >Priority: Major > > SDFs will be the only way to add streaming and liquid sharded IO for Go. > Design doc: https://s.apache.org/splittable-do-fn -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-7086) Reduce repetition of context in edge.go
[ https://issues.apache.org/jira/browse/BEAM-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7086 started by Daniel Oliveira. - > Reduce repetition of context in edge.go > --- > > Key: BEAM-7086 > URL: https://issues.apache.org/jira/browse/BEAM-7086 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Daniel Oliveira >Assignee: Daniel Oliveira >Priority: Minor > > A reminder bug to reduce/eliminate the large amounts of copy/pasting in > edge.go once it's reasonable. (i.e. after new errors package has been added) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7393) Misc. improvements to error formatting.
[
https://issues.apache.org/jira/browse/BEAM-7393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira resolved BEAM-7393.
---
Resolution: Fixed
Fix Version/s: Not applicable
> Misc. improvements to error formatting.
> ---
>
> Key: BEAM-7393
> URL: https://issues.apache.org/jira/browse/BEAM-7393
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Just need a second pass on error formatting. Two things are bothering me:
> # Default top-level messages are a bit too repetitive and obtuse. Right now
> if there's any kind of context or wrapping the top-level message is just the
> base error and this is usually confusing.
> # When an error is wrapped you have a line for "Caused by:" and it can be
> hard to read. Example:
> {noformat}
> error number 4
> caused by:
> some context:
> error number 3
> caused by:
> error number 2
> caused by:
> error number 1{noformat}
> Gotta make this example easier to read.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-7393) Misc. improvements to error formatting.
[
https://issues.apache.org/jira/browse/BEAM-7393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-7393:
--
Description:
Just need a second pass on error formatting. Two things are bothering me:
# Default top-level messages are a bit too repetitive and obtuse. Right now if
there's any kind of context or wrapping the top-level message is just the base
error and this is usually confusing.
# When an error is wrapped you have a line for "Caused by:" and it can be hard
to read. Example:
{noformat}
error number 4
caused by:
some context:
error number 3
caused by:
error number 2
caused by:
error number 1{noformat}
Gotta make this example easier to read.
was:
Just need a second pass on error formatting. Two things are bothering me:
# Default top-level messages are a bit too repetitive and obtuse. Right now if
there's any kind of context or wrapping the top-level message is just the base
error and this is usually confusing.
# When an error is wrapped you have a line for "Caused by:" and it can be hard
to read. Example:
{noformat}
error number 4
caused by:
encoding struct proto.marshalFieldInfo:
error number 3
caused by:
error number 2
caused by:
error number 1{noformat}
Gotta make this example easier to read.
> Misc. improvements to error formatting.
> ---
>
> Key: BEAM-7393
> URL: https://issues.apache.org/jira/browse/BEAM-7393
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
>
> Just need a second pass on error formatting. Two things are bothering me:
> # Default top-level messages are a bit too repetitive and obtuse. Right now
> if there's any kind of context or wrapping the top-level message is just the
> base error and this is usually confusing.
> # When an error is wrapped you have a line for "Caused by:" and it can be
> hard to read. Example:
> {noformat}
> error number 4
> caused by:
> some context:
> error number 3
> caused by:
> error number 2
> caused by:
> error number 1{noformat}
> Gotta make this example easier to read.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
