[
https://issues.apache.org/jira/browse/BEAM-3287?focusedWorklogId=82116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82116
]
ASF GitHub Bot logged work on BEAM-3287:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Mar/18 01:24
Start Date: 20/Mar/18 01:24
Worklog Time Spent: 10m
Work Description: herohde commented on issue #4888: [BEAM-3287] Add Go
support for universal runners, incl Flink
URL: https://github.com/apache/beam/pull/4888#issuecomment-374440447
@aljoscha Thanks! Yes! That is how I'm testing this stuff. I'm currently
hitting an issue with the artifacts on the Flink side:
```
$ go run examples/wordcount/wordcount.go --runner=flink --output=/tmp/foo
--endpoint=localhost:3000
[{6: KV<string,int>/GW/KV<bytes,int[varintz]>}]
[{10: KV<int,string>/GW/KV<int[varintz],bytes>}]
2018/03/19 18:19:09 No container image specified. Using dev image:
'herohde-docker-apache.bintray.io/beam/go:latest'
2018/03/19 18:19:09 Prepared job with id:
go-job-1521508749032988000_2100233503
2018/03/19 18:19:09 Cross-compiling
/Users/herohde/go/src/github.com/apache/beam/sdks/go/examples/wordcount/wordcount.go
as /var/folders/s2/97strbs55_353t_t7r24yf94009w4s/T/beam-go-1521508749041446000
2018/03/19 18:19:25 Staged binary artifact with token:
/private/tmp/beam-artifact-staging/go-job-1521508749032988000_2100233503
2018/03/19 18:19:25 Submitted job:
go-job-1521508749032988000_2100233503_-2090543569
2018/03/19 18:19:25 Job state: RUNNING
2018/03/19 18:19:25 Job state: FAILED
2018/03/19 18:19:25 Failed to execute job: job
go-job-1521508749032988000_2100233503_-2090543569 failed
exit status 1
```
Flink runner logs (info):
```
[...]
[grpc-default-executor-1] INFO org.apache.beam.runners.flink.FlinkRunner -
PipelineOptions.filesToStage was not specified. Defaulting to files from the
classpath: will stage 1 files. Enable logging at DEBUG level to see which files
will be staged.
[flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner -
Executing pipeline using FlinkRunner.
[flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner -
Translating pipeline to Flink program.
[grpc-default-executor-2] WARN
org.apache.beam.runners.flink.FlinkJobInvocation - addMessageObserver() not yet
implemented.
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment - Using
portability layer
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment - Creating the
required Batch Execution Environment.
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator -
enterCompositeTransform-
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform- Impulse
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform-
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform- CoGBK
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform-
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator - |
visitPrimitiveTransform- CoGBK
[flink-runner-job-server] INFO
org.apache.beam.runners.flink.FlinkBatchPipelineTranslator -
leaveCompositeTransform-
[flink-runner-job-server] INFO org.apache.beam.runners.flink.FlinkRunner -
Registering pipeline artifacts in Flink program.
[flink-runner-job-server] ERROR org.apache.beam.runners.flink.FlinkRunner -
Artifact registration failed
java.lang.IllegalMonitorStateException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:350)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:121)
at
org.apache.beam.runners.flink.FlinkJobInvocation.lambda$start$0(FlinkJobInvocation.java:61)
at
org.apache.beam.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at
org.apache.beam.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at
org.apache.beam.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[flink-runner-job-server] ERROR
org.apache.beam.runners.flink.FlinkJobInvocation - Error during job invocation
go-job-1521508749032988000_2100233503_-2090543569.
java.lang.RuntimeException: Artifact registration failed
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:124)
at
org.apache.beam.runners.flink.FlinkJobInvocation.lambda$start$0(FlinkJobInvocation.java:61)
at
org.apache.beam.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at
org.apache.beam.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at
org.apache.beam.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalMonitorStateException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.loadStagedArtifacts(FlinkPipelineExecutionEnvironment.java:350)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:121)
... 7 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 82116)
Time Spent: 1h 20m (was: 1h 10m)
> Go SDK support for portable pipelines
> -------------------------------------
>
> Key: BEAM-3287
> URL: https://issues.apache.org/jira/browse/BEAM-3287
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Henning Rohde
> Assignee: Henning Rohde
> Priority: Major
> Labels: portability
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The Go SDK should participate in the portability framework, incl. job
> submission w/ a docker container image.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
