Hi Beamers,
I am running the `wordcount` example, but following example from
https://beam.apache.org/documentation/runners/flink/, that is, I change the
pipeline initialization as follows.
```
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
options = PipelineOptions(["--runner=FlinkRunner", "--flink_version=1.8",
"--flink_master_url=localhost:8081"])
with beam.Pipeline(options) as p:
…
```
Running against my minikube cluster (), I get:
“RuntimeError: Pipeline
BeamApp-mpatterson-0911164258-7ef8768c_71984a02-5036-421e-9754-b57dbc628d3f
failed in state FAILED: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
”
Any ideas?
Thanks,
Matt
(minikube version: v1.3.1
commit: ca60a424ce69a4d79f502650199ca2b52f29e631
bash-3.2$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6",
GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean",
BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc",
Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2",
GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean",
BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc",
Platform:"linux/amd64"}
Flink job- and task-manager containers both built from flink:1.8
)
Full output
>>>
/anaconda3/envs/aws/bin/python
/Users/mpatterson/Library/Preferences/PyCharmCE2019.2/scratches/beam_me_up.py
/Users/mpatterson/dev/beam/sdks/python/apache_beam/__init__.py:84: UserWarning:
Some syntactic constructs of Python 3 are not yet fully supported by Apache
Beam.
'Some syntactic constructs of Python 3 are not yet fully supported by '
executable: /anaconda3/envs/aws/bin/python
beam.__version__: 2.15.0
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.7 interpreter. See also: BEAM-7474.
INFO:root:Using latest locally built Python SDK docker image:
mpatterson-docker-apache.bintray.io/beam/python3:latest.
INFO:root:==================== <function lift_combiners at 0x11850b200>
====================
INFO:root:==================== <function expand_sdf at 0x11850b290>
====================
WARNING:root:Downloading job server jar from
https://repo.maven.apache.org/maven2/org/apache/beam/beam-runners-flink-1.8-job-server/2.15.0/beam-runners-flink-1.8-job-server-2.15.0.jar
[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
ArtifactStagingService started on localhost:57443
[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
Java ExpansionService started on localhost:57444
[main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
JobService started on localhost:57439
[grpc-default-executor-0] ERROR
org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService -
Encountered Unexpected Exception for Invocation
job_9443af24-5ef1-4a65-8952-67b4f13eb5cb
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusException: NOT_FOUND
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status.asException(Status.java:541)
at
org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.getInvocation(InMemoryJobService.java:364)
at
org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.getStateStream(InMemoryJobService.java:285)
at
org.apache.beam.model.jobmanagement.v1.JobServiceGrpc$MethodHandlers.invoke(JobServiceGrpc.java:896)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:322)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:762)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[grpc-default-executor-0] INFO org.apache.beam.runners.flink.FlinkJobInvoker -
Invoking job
BeamApp-mpatterson-0911164258-7ef8768c_71984a02-5036-421e-9754-b57dbc628d3f
[grpc-default-executor-0] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation - Starting job
invocation
BeamApp-mpatterson-0911164258-7ef8768c_71984a02-5036-421e-9754-b57dbc628d3f
INFO:root:Job state changed to RUNNING
[flink-runner-job-invoker] INFO
org.apache.beam.runners.flink.FlinkPipelineRunner - Translating pipeline to
Flink program.
[flink-runner-job-invoker] INFO
org.apache.beam.runners.flink.FlinkExecutionEnvironments - Creating a Batch
Execution Environment.
[flink-runner-job-invoker] INFO
org.apache.beam.runners.flink.FlinkExecutionEnvironments - Using Flink Master
URL localhost:8081.
[flink-runner-job-invoker] WARN
org.apache.beam.runners.flink.FlinkExecutionEnvironments - No default
parallelism could be found. Defaulting to parallelism 1. Please set an explicit
parallelism with --parallelism
[flink-runner-job-invoker] ERROR
org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation - Error during
job invocation
BeamApp-mpatterson-0911164258-7ef8768c_71984a02-5036-421e-9754-b57dbc628d3f.
java.lang.IllegalArgumentException: unable to deserialize BoundedSource
at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
at
org.apache.beam.runners.core.construction.ReadTranslation.boundedSourceFromProto(ReadTranslation.java:94)
at
org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translateRead(FlinkBatchPortablePipelineTranslator.java:573)
at
org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:278)
at
org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:120)
at
org.apache.beam.runners.flink.FlinkPipelineRunner.runPipelineWithTranslator(FlinkPipelineRunner.java:84)
at
org.apache.beam.runners.flink.FlinkPipelineRunner.run(FlinkPipelineRunner.java:63)
at
org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:74)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at
org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:98)
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:474)
at org.xerial.snappy.Snappy.uncompress(Snappy.java:513)
at
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:147)
at
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:99)
at
org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:59)
at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:68)
... 13 more
ERROR:root:java.io.IOException: FAILED_TO_UNCOMPRESS(5)
[flink-runner-job-invoker] INFO
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService
- Manifest at
/var/folders/fm/pg9v9trs1g9g0w9k6tbrkn_xtksx7d/T/artifactsa25mrn59/job_9443af24-5ef1-4a65-8952-67b4f13eb5cb/MANIFEST
has 1 artifact locations
[flink-runner-job-invoker] INFO
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService
- Removed dir
/var/folders/fm/pg9v9trs1g9g0w9k6tbrkn_xtksx7d/T/artifactsa25mrn59/job_9443af24-5ef1-4a65-8952-67b4f13eb5cb/
INFO:root:Job state changed to FAILED
Traceback (most recent call last):
File
"/Users/mpatterson/Library/Preferences/PyCharmCE2019.2/scratches/beam_me_up.py",
line 111, in <module>
run()
File
"/Users/mpatterson/Library/Preferences/PyCharmCE2019.2/scratches/beam_me_up.py",
line 88, in run
result.wait_until_finish()
File
"/Users/mpatterson/dev/beam/sdks/python/apache_beam/runners/portability/portable_runner.py",
line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline
BeamApp-mpatterson-0911164258-7ef8768c_71984a02-5036-421e-9754-b57dbc628d3f
failed in state FAILED: java.io.IOException: FAILED_TO_UNCOMPRESS(5)
Process finished with exit code 1
