Re: Flink Runner logging FAILED_TO_UNCOMPRESS

Kyle Thank you for the assistance. By specifying "experiments" in PipelineOptions , == options = PipelineOptions([ "--runner=FlinkRunner", "--flink_version=1.8",

Re: Flink Runner logging FAILED_TO_UNCOMPRESS

Try adding "--experiments=beam_fn_api" to your pipeline options. (This is a known issue with Beam 2.15 that will be fixed in 2.16.) Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com On Sat, Sep 14, 2019 at 12:52 AM Yu Watanabe wrote: > Hello. > > I am trying to spin up

Flink Runner logging FAILED_TO_UNCOMPRESS

Hello. I am trying to spin up the flink runner but looks like data serialization is failing. I would like to ask for help to get over with this error. [flink-runner-job-invoker] ERROR

Re: How do you write portable runner pipeline on separate python code ?

Lukasz Thank you for the reply. > * Using a "remote" filesystem such as HDFS/S3/GCS/... > * Mounting an external directory into the container so that any "local" writes appear outside the container > * Using a non-docker environment such as external or process. Understood. Thanks, Yu

Re: How do you write portable runner pipeline on separate python code ?

Kyle Thank you for the advice. > For example, Yu's pipeline errored here because the expected Docker container wasn't built before running. I was able to spin up the harness container and submit job to the job service by preparing the container properly. I needed to do extra steps in the