Kyle Weaver created BEAM-12762:
----------------------------------
Summary: java.io.InvalidClassException with Spark 3.1.2
Key: BEAM-12762
URL: https://issues.apache.org/jira/browse/BEAM-12762
Project: Beam
Issue Type: Bug
Components: runner-spark
Reporter: Kyle Weaver
This was reported on the mailing list.
----
Using spark downloaded from below link,
[https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz]
I get below error when submitting a pipeline.
Full error is on
[https://gist.github.com/yuwtennis/7b0c1dc0dcf98297af1e3179852ca693].
------------------------------------------------------------------------------------------------------------------
21/08/16 01:10:26 WARN TransportChannelHandler: Exception in connection from
/[192.168.11.2:35601|http://192.168.11.2:35601/]
java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef;
local class incompatible: stream classdesc serialVersionUID =
3456489343829468865, local class serialVersionUID = 1028182004549731694
at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
...
------------------------------------------------------------------------------------------------------------------
SDK Harness and Job service are deployed as below.
1. SDK Harness
sudo docker run --net=host apache/beam_spark3_job_server:2.31.0
--spark-master-url=spark://localhost:7077 --clean-artifacts-per-job true
2. Job service
sudo docker run --net=host apache/beam_python3.8_sdk:2.31.0 --worker_pool
* apache/beam_spark_job_server:2.31.0 for spark 2.4.8
3. SDK client code
[https://gist.github.com/yuwtennis/2e4c13c79f71e8f713e947955115b3e2]
Spark 2.4.8 succeeded without any errors using above components.
[https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
