Hi Matthias,
This is already reflected in the compatibility matrix, if you look under SDF.
There is no UnboundedSource interface for portable pipelines. That's a legacy
abstraction that will be replaced with SDF.
Fortunately, there is already a pending PR for cross-language pipelines which
will allow us to use Java IO like PubSub in Python jobs.
Thanks,
Max
On 31.01.19 12:06, Matthias Baetens wrote:
Hey Ankur,
Thanks for the swift reply. Should I change this in the capability matrix
<https://s.apache.org/apache-beam-portability-support-table> then?
Many thanks.
Best,
Matthias
On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goe...@google.com
<mailto:goe...@google.com>> wrote:
Hi Matthias,
Unfortunately, unbounded reads including pubsub are not yet supported for
portable runners.
Thanks,
Ankur
On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatth...@gmail.com
<mailto:baetensmatth...@gmail.com>> wrote:
Hi everyone,
Last few days I have been trying to run a streaming pipeline (code on
Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
I am running a Flink cluster locally (v1.5.6
<https://flink.apache.org/downloads.html>)
I have built the SDK Harness Container: /./gradlew
:beam-sdks-python-container:docker/
and started the JobServer: /./gradlew
:beam-runners-flink_2.11-job-server:runShadow
-PflinkMasterUrl=localhost:8081./
I run my pipeline with: /env/bin/python streaming_pipeline.py
--runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
--input_subscription xxx --output_subscription xxx/
/
/
All this is running inside a Ubuntu (Bionic) in a Virtualbox.
The job submits fine, but unfortunately fails after a few seconds with
the error attached.
Anything I am missing or doing wrong?
Many thanks.
Best,
Matthias
