See
<https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/4696/display/redirect?page=changes>
Changes:
[chamikaramj] Fixes a case where BigQuery sink swallows HttpErrors preventing
retries.
[Boyuan Zhang] [BEAM-10670] Use non-SDF based translation for Read by default
on all
[noreply] [BEAM-9547] Add support for drop_duplicates and duplicated (#14786)
[noreply] Don't compile with 8 if flag is set (#14838)
[noreply] [BEAM-11644] Remove pack_all_combiners optimizer phase (#14647)
[noreply] [BEAM-12341] Enable Portable job submission for Dataflow (#14822)
[noreply] [BEAM-9615] Enable Schemas for struct coding. (#14826)
[noreply] [BEAM-7372] cleanup py2 codepath from combiners.py (#14634)
------------------------------------------
[...truncated 564.64 KB...]
coders: <
key: "c3"
value: <
spec: <
urn: "beam:coder:kv:v1"
>
component_coder_ids: "c0"
component_coder_ids: "c2"
>
>
coders: <
key: "c4"
value: <
spec: <
urn: "beam:coder:global_window:v1"
>
>
>
coders: <
key: "c5"
value: <
spec: <
urn: "beam:coder:bytes:v1"
>
>
>
coders: <
key: "c6"
value: <
spec: <
urn: "beam:coder:iterable:v1"
>
component_coder_ids: "c2"
>
>
coders: <
key: "c7"
value: <
spec: <
urn: "beam:coder:kv:v1"
>
component_coder_ids: "c0"
component_coder_ids: "c6"
>
>
coders: <
key: "c8"
value: <
spec: <
urn: "beam:coder:kv:v1"
>
component_coder_ids: "c2"
component_coder_ids: "c0"
>
>
coders: <
key: "c9"
value: <
spec: <
urn: "beam:coder:iterable:v1"
>
component_coder_ids: "c0"
>
>
environments: <
key: "go"
value: <
urn: "beam:env:docker:v1"
payload: "\n\026apache/beam_go_sdk:dev"
capabilities: "beam:protocol:progress_reporting:v0"
capabilities: "beam:protocol:multi_core_bundle_processing:v1"
capabilities: "beam:version:sdk_base:go"
capabilities: "beam:coder:bytes:v1"
capabilities: "beam:coder:bool:v1"
capabilities: "beam:coder:varint:v1"
capabilities: "beam:coder:double:v1"
capabilities: "beam:coder:string_utf8:v1"
capabilities: "beam:coder:length_prefix:v1"
capabilities: "beam:coder:kv:v1"
capabilities: "beam:coder:iterable:v1"
capabilities: "beam:coder:state_backed_iterable:v1"
capabilities: "beam:coder:windowed_value:v1"
capabilities: "beam:coder:global_window:v1"
capabilities: "beam:coder:interval_window:v1"
dependencies: <
type_urn: "beam:artifact:type:go_worker_binary:v1"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\006worker"
>
>
>
>
root_transform_ids: "s1"
root_transform_ids: "s2"
root_transform_ids: "e9"
root_transform_ids: "s6"
2021/05/20 00:16:28 Prepared job with id:
go-job-5-1621469788637603180_17358b51-e248-4920-a031-0ca87bd6e14a and staging
token: go-job-5-1621469788637603180_17358b51-e248-4920-a031-0ca87bd6e14a
2021/05/20 00:16:28 Cross-compiling
<https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/ws/temp_gopath/src/github.com/apache/beam/sdks/go/test/integration/wordcount/wordcount_test.go>
as /tmp/worker-5-1621469788642730249
21/05/20 00:16:34 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Staging
artifacts for go-job-5-1621469788637603180_17358b51-e248-4920-a031-0ca87bd6e14a.
21/05/20 00:16:34 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Resolving
artifacts for
go-job-5-1621469788637603180_17358b51-e248-4920-a031-0ca87bd6e14a.go.
21/05/20 00:16:34 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Getting 1
artifacts for
go-job-5-1621469788637603180_17358b51-e248-4920-a031-0ca87bd6e14a.null.
21/05/20 00:16:34 INFO
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Artifacts
fully staged for
go-job-5-1621469788637603180_17358b51-e248-4920-a031-0ca87bd6e14a.
2021/05/20 00:16:34 Staged binary artifact with token:
21/05/20 00:16:34 INFO org.apache.beam.runners.spark.SparkJobInvoker: Invoking
job
go0job0501621469788637603180-jenkins-0520001634-f7baac0a_d9b3607f-5e7c-4ec9-be42-09c9f66883a5
21/05/20 00:16:34 INFO org.apache.beam.runners.jobsubmission.JobInvocation:
Starting job invocation
go0job0501621469788637603180-jenkins-0520001634-f7baac0a_d9b3607f-5e7c-4ec9-be42-09c9f66883a5
2021/05/20 00:16:34 Submitted job:
go0job0501621469788637603180-jenkins-0520001634-f7baac0a_d9b3607f-5e7c-4ec9-be42-09c9f66883a5
2021/05/20 00:16:34 Job state: STOPPED
2021/05/20 00:16:34 Job state: STARTING
2021/05/20 00:16:34 Job state: RUNNING
21/05/20 00:16:34 INFO
org.apache.beam.runners.spark.translation.SparkContextFactory: Creating a brand
new Spark Context.
21/05/20 00:16:35 INFO org.apache.beam.runners.spark.SparkPipelineRunner:
Running job
go0job0501621469788637603180-jenkins-0520001634-f7baac0a_d9b3607f-5e7c-4ec9-be42-09c9f66883a5
on Spark master local
21/05/20 00:16:35 WARN
org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions:
Either coder LengthPrefixCoder(ByteArrayCoder) or GlobalWindow$Coder is not
consistent with equals. That might cause issues on some runners.
21/05/20 00:16:35 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job
go0job0501621469788637603180-jenkins-0520001634-f7baac0a_d9b3607f-5e7c-4ec9-be42-09c9f66883a5:
Pipeline translated successfully. Computing outputs
21/05/20 00:16:36 WARN
org.apache.beam.runners.fnexecution.environment.DockerCommand: Unable to pull
docker image apache/beam_go_sdk:dev, cause: Received exit code 1 for command
'docker pull apache/beam_go_sdk:dev'. stderr: Error response from daemon:
manifest for apache/beam_go_sdk:dev not found
21/05/20 00:16:39 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: Beam Fn Logging
client connected.
21/05/20 00:16:39 INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService: Beam
Fn Control client connected with id 38-1
21/05/20 00:16:39 INFO
<https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/ws/temp_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go>:504:
Connecting via grpc @ localhost:43869 ...
21/05/20 00:16:39 INFO
<https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/ws/temp_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go>:504:
Connecting via grpc @ localhost:33657 ...
21/05/20 00:16:39 INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService:
getProcessBundleDescriptor request with id 38-2
21/05/20 00:16:39 INFO
<https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/ws/temp_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/harness.go>:504:
Connecting via grpc @ localhost:34967 ...
21/05/20 00:16:39 INFO
org.apache.beam.runners.fnexecution.data.GrpcDataService: Beam Fn Data client
connected.
21/05/20 00:16:39 INFO
<https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/ws/temp_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/datasource.go>:232:
DataSource: 1 elements in 13690418 ns
21/05/20 00:16:39 INFO
<https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/ws/temp_gopath/src/github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec/datasink.go>:84:
DataSink: 0 elements in 5540085 ns
21/05/20 00:16:39 INFO
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory: Closing
environment urn: "beam:env:docker:v1"
payload: "\n\026apache/beam_go_sdk:dev"
capabilities: "beam:protocol:progress_reporting:v0"
capabilities: "beam:protocol:multi_core_bundle_processing:v1"
capabilities: "beam:version:sdk_base:go"
capabilities: "beam:coder:bytes:v1"
capabilities: "beam:coder:bool:v1"
capabilities: "beam:coder:varint:v1"
capabilities: "beam:coder:double:v1"
capabilities: "beam:coder:string_utf8:v1"
capabilities: "beam:coder:length_prefix:v1"
capabilities: "beam:coder:kv:v1"
capabilities: "beam:coder:iterable:v1"
capabilities: "beam:coder:state_backed_iterable:v1"
capabilities: "beam:coder:windowed_value:v1"
capabilities: "beam:coder:global_window:v1"
capabilities: "beam:coder:interval_window:v1"
dependencies {
type_urn: "beam:artifact:type:file:v1"
type_payload:
"\ng/tmp/beam-artifact-staging/28e0ec831d95a6cc3ced14afa342fdbab7fa19d2e005eda161a5dabaa3831b08/1-go-worker"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload: "\n\006worker"
}
21/05/20 00:16:39 INFO
org.apache.beam.runners.fnexecution.logging.GrpcLoggingService: 1 Beam Fn
Logging clients still connected during shutdown.
21/05/20 00:16:39 WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer:
Hanged up for unknown endpoint.
21/05/20 00:16:39 INFO
org.apache.beam.runners.fnexecution.environment.DockerContainerEnvironment:
Closing Docker container
20be5e2a9ee1d11ad97ad5999a17bb70aea154d675fd0e6d378874e5ae246175. Logs:
2021/05/20 00:16:38 Provision info:
pipeline_options:{fields:{key:"beam:option:app_name:v1"
value:{string_value:"go-job-5-1621469788637603180"}}
fields:{key:"beam:option:experiments:v1"
value:{list_value:{values:{string_value:"beam_fn_api"}}}}
fields:{key:"beam:option:go_options:v1"
value:{struct_value:{fields:{key:"options"
value:{struct_value:{fields:{key:"hooks" value:{string_value:"{}"}}}}}}}}
fields:{key:"beam:option:job_name:v1"
value:{string_value:"go0job0501621469788637603180-jenkins-0520001634-f7baac0a"}}
fields:{key:"beam:option:options_id:v1" value:{number_value:13}}
fields:{key:"beam:option:parallelism:v1" value:{number_value:-1}}
fields:{key:"beam:option:retain_docker_containers:v1" value:{bool_value:false}}
fields:{key:"beam:option:runner:v1" value:{null_value:NULL_VALUE}}
fields:{key:"beam:option:spark_master:v1" value:{string_value:"local"}}}
logging_endpoint:{url:"localhost:33657"}
artifact_endpoint:{url:"localhost:33567"}
control_endpoint:{url:"localhost:43869"}
dependencies:{type_urn:"beam:artifact:type:file:v1"
type_payload:"\ng/tmp/beam-artifact-staging/28e0ec831d95a6cc3ced14afa342fdbab7fa19d2e005eda161a5dabaa3831b08/1-go-worker"
role_urn:"beam:artifact:role:staging_to:v1" role_payload:"\n\x06worker"}
2021/05/20 00:16:38 Initializing Go harness: /opt/apache/beam/boot --id=38-1
--provision_endpoint=localhost:38273
2021/05/20 00:16:38 Downloaded: /tmp/staged/worker (sha256:
a60ca6db8d0402976b4bd40906afeb33d133e2391d644bd6fe5384471a5449e2, size:
27921326)
2021/05/20 00:16:39 WARNING: proto: file "v1.proto" is already registered
previously from:
"github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
currently from:
"github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio/v1"
A future release will panic on registration conflicts. See:
https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict
Worker exited successfully!
2021/05/20 00:16:39 control response channel closed
21/05/20 00:16:41 INFO org.apache.beam.runners.spark.SparkPipelineRunner: Job
go0job0501621469788637603180-jenkins-0520001634-f7baac0a_d9b3607f-5e7c-4ec9-be42-09c9f66883a5
finished.
2021/05/20 00:16:41 Job state: DONE
21/05/20 00:16:41 INFO
org.apache.beam.runners.jobsubmission.InMemoryJobService: Getting job metrics
for
go0job0501621469788637603180-jenkins-0520001634-f7baac0a_d9b3607f-5e7c-4ec9-be42-09c9f66883a5
21/05/20 00:16:41 INFO
org.apache.beam.runners.jobsubmission.InMemoryJobService: Finished getting job
metrics for
go0job0501621469788637603180-jenkins-0520001634-f7baac0a_d9b3607f-5e7c-4ec9-be42-09c9f66883a5
2021/05/20 00:16:41 Failed to deduce Step from MonitoringInfo:
urn:"beam:metric:element_count:v1" type:"beam:metrics:sum_int64:v1"
payload:"\x01" labels:{key:"PCOLLECTION" value:"n1"}
--- PASS: TestWordCount (52.86s)
PASS
ok github.com/apache/beam/sdks/go/test/integration/wordcount 52.909s
2021/05/20 00:16:46 WARNING: proto: file "v1.proto" is already registered
previously from:
"github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
currently from:
"github.com/apache/beam/sdks/go/test/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio/v1"
A future release will panic on registration conflicts. See:
https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict
=== RUN TestXLang_Prefix
--- SKIP: TestXLang_Prefix (0.00s)
integration.go:134: Test TestXLang_Prefix is currently filtered for runner
spark
=== RUN TestXLang_CoGroupBy
--- SKIP: TestXLang_CoGroupBy (0.00s)
integration.go:134: Test TestXLang_CoGroupBy is currently filtered for
runner spark
=== RUN TestXLang_Combine
--- SKIP: TestXLang_Combine (0.00s)
integration.go:134: Test TestXLang_Combine is currently filtered for runner
spark
=== RUN TestXLang_CombineGlobally
--- SKIP: TestXLang_CombineGlobally (0.00s)
integration.go:134: Test TestXLang_CombineGlobally is currently filtered
for runner spark
=== RUN TestXLang_Flatten
--- SKIP: TestXLang_Flatten (0.00s)
integration.go:134: Test TestXLang_Flatten is currently filtered for runner
spark
=== RUN TestXLang_GroupBy
--- SKIP: TestXLang_GroupBy (0.00s)
integration.go:134: Test TestXLang_GroupBy is currently filtered for runner
spark
=== RUN TestXLang_Multi
--- SKIP: TestXLang_Multi (0.00s)
integration.go:134: Test TestXLang_Multi is currently filtered for runner
spark
=== RUN TestXLang_Partition
--- SKIP: TestXLang_Partition (0.00s)
integration.go:134: Test TestXLang_Partition is currently filtered for
runner spark
PASS
ok github.com/apache/beam/sdks/go/test/integration/xlang 0.060s
if [[ "$RUNNER" == "dataflow" ]]; then
# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG || echo "Failed to remove container"
gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to
delete container"
if [[ -n "$EXPANSION_ADDR" ]]; then
# Delete the java cross-language container locally and remotely
docker rmi $JAVA_CONTAINER:$JAVA_TAG || echo "Failed to remove container"
gcloud --quiet container images delete $JAVA_CONTAINER:$JAVA_TAG || echo
"Failed to delete container"
fi
# Clean up tempdir
rm -rf $TMPDIR
fi
exit $TEST_EXIT_CODE
exit_background_processes
> Task :sdks:go:test:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file
'<https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/ws/src/sdks/go/test/build.gradle'>
line: 114
* What went wrong:
Execution failed for task ':sdks:go:test:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 16m 20s
79 actionable tasks: 57 executed, 22 from cache
Publishing build scan...
https://gradle.com/s/ei5zl2m5coryg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
