Kyle. Thank you for the reply.
Error had disappeared after creating sdk and job-server with correct apache beam branch (2.16.0 in this case). Thanks, Yu Watanabe On Tue, Dec 31, 2019 at 2:30 AM Kyle Weaver <[email protected]> wrote: > This error can happen when the job server and sdk versions are mismatched > (due to protobuf incompatibilities). The sdk and job server containers > should use the same beam version. > > On Mon, Dec 30, 2019 at 11:47 AM Yu Watanabe <[email protected]> > wrote: > >> Hello. >> >> I would like to get help with issue having in job-server. >> >> I have set up flink (session cluster) and job server everything in >> docker , however, jobserver seems to reject requests from the beam >> client as described in below error. >> >> ================================================== >> jobserver_1 | [grpc-default-executor-11] WARN >> org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService - >> Encountered Unexpected Exception during validation >> jobserver_1 | java.lang.RuntimeException: Failed to validate >> transform ref_AppliedPTransform_Create/FlatMap(<lambda at >> core.py:2468>)_4 >> jobserver_1 | at >> >> org.apache.beam.runners.core.construction.graph.PipelineValidator.validateTransform(PipelineValidator.java:203) >> jobserver_1 | at >> >> org.apache.beam.runners.core.construction.graph.PipelineValidator.validateComponents(PipelineValidator.java:112) >> <TRIMMED> >> jobserver_1 | Caused by: >> >> org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException: >> Protocol message had invalid UTF-8. >> jobserver_1 | at >> >> org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:141) >> jobserver_1 | at >> >> org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Utf8$DecodeUtil.handleTwoBytes(Utf8.java:1909) >> jobserver_1 | at >> >> org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Utf8$DecodeUtil.access$700(Utf8.java:1883) >> <TRIMMED> >> ================================================== >> >> Below is description about my environment. >> >> 1. Combination of versions of each containers are, >> >> apache beam: 2.16.0 >> sdk: python3.7_sdk: 2.19.0.dev >> flink: flink1.9_job_server:latest >> >> 2. Set up containers using beam repository. >> >> https://github.com/yuwtennis/beam-deployment/wiki/Setting-up-flink-runner >> >> 3. Set up docker-compose as below. >> >> >> https://github.com/yuwtennis/beam-deployment/blob/master/flink-session-cluster/docker-compose.yml >> >> 4. Kept source code very simple. >> >> >> https://github.com/yuwtennis/beam-deployment/blob/master/flink-session-cluster/samples/src/sample.py >> >> Do I need any other options in job server to properly >> serialize/deserialize the incoming message from beam client ? >> >> My current options are below. >> ====================================================== >> command: [ "--artifacts-dir", "${ARTIFACTS_DIR}", "--flink-master-url", >> "${FLINK_MASTER_URL}", "--job-host", "${JOB_HOST}", "--job-port", >> "${JOB_PORT}" ] >> ====================================================== >> >> I appreciate if I could get some help. >> >> Thanks, >> Yu Watanabe >> >> -- >> Yu Watanabe >> [email protected] >> > -- Yu Watanabe [email protected] [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image: Twitter icon] <https://twitter.com/yuwtennis>
