Hello folks,
Day 3 of trying to get the basics going with Python & Spark 2.2.3.
I’ve downgraded the spark version to 2.2.3 in the cradle build so that I
can run jobs against it.
I’ve then written this:
options = PipelineOptions(["--runner=PortableRunner",
"--job_endpoint=localhost:8099", "--environment_type=LOOPBACK"])
with beam.Pipeline(options=options) as p:
print("hello")
lines = p | 'Create' >> beam.Create(['hello', 'world'])
Which in turn explodes with:
Caused by:
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException:
UNIMPLEMENTED: Method not found!
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141)
at
org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.startWorker(BeamFnExternalWorkerPoolGrpc.java:226)
When run against python 2 and
AttributeError: module 'apache_beam.coders.coders' has no
attribute ‘VarIntCoder'
When run with python 3.5.
Should I just give up trying the portable runner at the moment or am I
doing something dumb/incompatible? I’m stuck on Spark 2.2.3 so upgrading
isn’t currently an option.
Tom
--
Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.
All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.
