[
https://issues.apache.org/jira/browse/BEAM-11433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath updated BEAM-11433:
-------------------------------------------------
Description:
I get [1] for Dataflow multi-language Kafka example [2] when running the
pipeline with Java11 installed in the computer and with Java11 containers.
Due to recent changes related to JDK specific container naming, Beam Docker
environments default to using Java version specific containers. This means that
Dataflow multi-language pipelines will end up using Java 11 containers for
users that use Java 11. Seems like these containers are currently broken for
Dataflow.
See [2] for instructions for reproducing (run with Java11 containers instead of
Java8 containers)
[~emilymye], what would be a good solution for Dataflow till we support Java 11
? I suspect we'll run into similar failures if we just run regular Java
pipelines on UW with Java 11 containers.
This should be a blocker for Beam 2.27.0 since this breaks Dataflow
multi-language pipelines for Java 11 users.
cc: [~pabloem] [~robertwb] [~kenn]
[1]
Error message from worker: generic::unknown:
java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
Execution Info
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
org.apache.beam.runners.core.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:706)
org.apache.beam.runners.core.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:360)
org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:346)
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:178)
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:152)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:239)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:505)
org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:279)
org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:567)
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:274)
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173)
org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834) Caused by:
java.io.InvalidClassException:
org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn; local class
incompatible: stream classdesc serialVersionUID = -1752193696517031629, local
class serialVersionUID = 3454873399160940695
java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2012)
java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1862)
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2169)
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2464)
java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2358)
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
[2]
[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/kafkataxi/README.md]
was:
I get [1] for Dataflow multi-language Kafka example [2] when running the
pipeline with Java11 installed in the computer and with Java11 containers.
Due to recent changes related to JDK specific container naming, Beam
environments default to Java 11 containers. This means that Dataflow
multi-language pipelines will end up using Java 11 containers for users that
use Java 11. Seems like these containers are currently broken for Dataflow.
See [2] for instructions for reproducing (run with Java11 containers instead of
Java8 containers)
[~emilymye], what would be a good solution for Dataflow till we support Java 11
? I suspect we'll run into similar failures if we just run regular Java
pipelines on UW with Java 11 containers.
This should be a blocker for Beam 2.27.0 since this breaks Dataflow
multi-language pipelines for Java 11 users.
cc: [~pabloem] [~robertwb] [~kenn]
[1]
Error message from worker: generic::unknown:
java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
Execution Info
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
org.apache.beam.runners.core.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:706)
org.apache.beam.runners.core.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:360)
org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:346)
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:178)
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:152)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:239)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:505)
org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:279)
org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:567)
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:274)
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173)
org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834) Caused by:
java.io.InvalidClassException:
org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn; local class
incompatible: stream classdesc serialVersionUID = -1752193696517031629, local
class serialVersionUID = 3454873399160940695
java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2012)
java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1862)
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2169)
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2464)
java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2358)
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:493)
java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:451)
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
... 18 more passed through: ==>
dist_proc/dax/workflow/worker/fnapi_service.cc:632
[2]
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/kafkataxi/README.md
> Dataflow Multi-language pipelines broken for Java 11 users
> ----------------------------------------------------------
>
> Key: BEAM-11433
> URL: https://issues.apache.org/jira/browse/BEAM-11433
> Project: Beam
> Issue Type: Bug
> Components: cross-language, runner-dataflow
> Reporter: Chamikara Madhusanka Jayalath
> Priority: P1
> Fix For: 2.27.0
>
>
> I get [1] for Dataflow multi-language Kafka example [2] when running the
> pipeline with Java11 installed in the computer and with Java11 containers.
> Due to recent changes related to JDK specific container naming, Beam Docker
> environments default to using Java version specific containers. This means
> that Dataflow multi-language pipelines will end up using Java 11 containers
> for users that use Java 11. Seems like these containers are currently broken
> for Dataflow.
>
> See [2] for instructions for reproducing (run with Java11 containers instead
> of Java8 containers)
> [~emilymye], what would be a good solution for Dataflow till we support Java
> 11 ? I suspect we'll run into similar failures if we just run regular Java
> pipelines on UW with Java 11 containers.
> This should be a blocker for Beam 2.27.0 since this breaks Dataflow
> multi-language pipelines for Java 11 users.
> cc: [~pabloem] [~robertwb] [~kenn]
> [1]
> Error message from worker: generic::unknown:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
>
> org.apache.beam.runners.core.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:706)
>
> org.apache.beam.runners.core.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:360)
> org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:346)
> org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:178)
>
> org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:152)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:239)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:505)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:279)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:567)
>
> org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:274)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173)
>
> org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157)
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> java.base/java.lang.Thread.run(Thread.java:834) Caused by:
> java.io.InvalidClassException:
> org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn; local class
> incompatible: stream classdesc serialVersionUID = -1752193696517031629, local
> class serialVersionUID = 3454873399160940695
> java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2012)
>
> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1862)
>
> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2169)
> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
> java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2464)
>
> java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2358)
>
> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196)
> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
> [2]
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/kafkataxi/README.md]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
