[
https://issues.apache.org/jira/browse/BEAM-11433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath updated BEAM-11433:
-------------------------------------------------
Issue Type: Bug (was: Improvement)
> Dataflow Multi-language pipelines broken for Java 11 users
> ----------------------------------------------------------
>
> Key: BEAM-11433
> URL: https://issues.apache.org/jira/browse/BEAM-11433
> Project: Beam
> Issue Type: Bug
> Components: cross-language, runner-dataflow
> Reporter: Chamikara Madhusanka Jayalath
> Priority: P1
> Fix For: 2.27.0
>
>
> I get [1] for Dataflow multi-language Kafka example [2] when running the
> pipeline with Java11 installed in the computer and with Java11 containers.
> Due to recent changes related to JDK specific container naming, Beam
> environments default to Java 11 containers. This means that Dataflow
> multi-language pipelines will end up using Java 11 containers for users that
> use Java 11. Seems like these containers are currently broken for Dataflow.
>
> See [2] for instructions for reproducing (run with Java11 containers instead
> of Java8 containers)
> [~emilymye], what would be a good solution for Dataflow till we support Java
> 11 ? I suspect we'll run into similar failures if we just run regular Java
> pipelines on UW with Java 11 containers.
> This should be a blocker for Beam 2.27.0 since this breaks Dataflow
> multi-language pipelines for Java 11 users.
> cc: [~pabloem] [~robertwb] [~kenn]
> [1]
> Error message from worker: generic::unknown:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
>
> org.apache.beam.runners.core.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:706)
>
> org.apache.beam.runners.core.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:360)
> org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:346)
> org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:178)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:152)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:239)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:505)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:279)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:567)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:274)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157)
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> java.base/java.lang.Thread.run(Thread.java:834) Caused by:
> java.io.InvalidClassException:
> org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn; local class
> incompatible: stream classdesc serialVersionUID = -1752193696517031629, local
> class serialVersionUID = 3454873399160940695
> java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2012)
>
> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1862)
>
> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2169)
> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
> java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2464)
>
> java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2358)
>
> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:493)
> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:451)
> org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
> ... 18 more passed through: ==>
> dist_proc/dax/workflow/worker/fnapi_service.cc:632
> [2]
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/kafkataxi/README.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
