[
https://issues.apache.org/jira/browse/BEAM-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17496642#comment-17496642
]
Yu Watanabe edited comment on BEAM-12762 at 2/23/22, 10:34 AM:
---------------------------------------------------------------
[~syronanm] No. For [release-2.33.0|#L165]] , it depends on
[spark-core_2.12-3.1.2|https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.1.2]
and it uses {_}2.12.10{_}. This can be confirmed by check mark not marked on
_Updates_ . (See
[this|https://stackoverflow.com/questions/35354334/maven-dependencies-version-vs-updates]
for detail)
This goes all the way to [master
branch|https://github.com/apache/beam/blob/master/runners/spark/3/build.gradle]
(commit 60ea50f6b7fff7d482890b9006050b7ab4ac4157 at its HEAD) .
[spark-core-2_12-3.2.1|https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.2.1]
uses _scala 2.12.15,_ so I tried compiling but looks like the code base is not
ready yet.
I tweaked
[build.gradle|https://gist.github.com/yuwtennis/b675609826d43b4a2590336dc3072f74]
and used above master branch,
{code:java}
./gradlew :runners:spark:3:job-server:container:docker
-Pdocker-tag=spark2_12.3.2.1
...
> Task :runners:spark:3:compileJava
/root/beam/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java:61:
error: EncodeUsingBeamCoder is not abstract and does not override abstract
method withNewChildrenInternal(IndexedSeq<Expression>) in TreeNode
public static class EncodeUsingBeamCoder<T> extends UnaryExpression
^
/root/beam/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java:168:
error: DecodeUsingBeamCoder is not abstract and does not override abstract
method withNewChildrenInternal(IndexedSeq<Expression>) in TreeNode
public static class DecodeUsingBeamCoder<T> extends UnaryExpression {code}
was (Author: y-watanabe):
No. For
[release-2.33.0|[https://github.com/apache/beam/blob/v2.33.0/runners/spark/spark_runner.gradle#L165]]
, it depends on
[spark-core_2.12-3.1.2|https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.1.2]
and it uses {_}2.12.10{_}. This can be confirmed by check mark not marked on
_Updates_ . (See
[this|https://stackoverflow.com/questions/35354334/maven-dependencies-version-vs-updates]
for detail)
This goes all the way to [master
branch|https://github.com/apache/beam/blob/master/runners/spark/3/build.gradle]
(commit 60ea50f6b7fff7d482890b9006050b7ab4ac4157 at its HEAD) .
[spark-core-2_12-3.2.1|https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.2.1]
uses _scala 2.12.15,_ so I tried compiling but looks like the code base is not
ready yet.
I tweaked
[build.gradle|https://gist.github.com/yuwtennis/b675609826d43b4a2590336dc3072f74]
and used above master branch,
{code:java}
./gradlew :runners:spark:3:job-server:container:docker
-Pdocker-tag=spark2_12.3.2.1
...
> Task :runners:spark:3:compileJava
/root/beam/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java:61:
error: EncodeUsingBeamCoder is not abstract and does not override abstract
method withNewChildrenInternal(IndexedSeq<Expression>) in TreeNode
public static class EncodeUsingBeamCoder<T> extends UnaryExpression
^
/root/beam/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java:168:
error: DecodeUsingBeamCoder is not abstract and does not override abstract
method withNewChildrenInternal(IndexedSeq<Expression>) in TreeNode
public static class DecodeUsingBeamCoder<T> extends UnaryExpression {code}
> java.io.InvalidClassException with Spark 3.1.2
> ----------------------------------------------
>
> Key: BEAM-12762
> URL: https://issues.apache.org/jira/browse/BEAM-12762
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Reporter: Kyle Weaver
> Priority: P3
>
> This was reported on the mailing list.
>
> ----
>
> Using spark downloaded from below link,
>
> [https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz]
>
> I get below error when submitting a pipeline.
> Full error is on
> [https://gist.github.com/yuwtennis/7b0c1dc0dcf98297af1e3179852ca693].
>
> ------------------------------------------------------------------------------------------------------------------
> 21/08/16 01:10:26 WARN TransportChannelHandler: Exception in connection from
> /[192.168.11.2:35601|http://192.168.11.2:35601/]
> java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef;
> local class incompatible: stream classdesc serialVersionUID =
> 3456489343829468865, local class serialVersionUID = 1028182004549731694
> at
> java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
> ...
> ------------------------------------------------------------------------------------------------------------------
>
> SDK Harness and Job service are deployed as below.
>
> 1. SDK Harness
>
> sudo docker run --net=host apache/beam_spark3_job_server:2.31.0
> --spark-master-url=spark://localhost:7077 --clean-artifacts-per-job true
>
> 2. Job service
>
> sudo docker run --net=host apache/beam_python3.8_sdk:2.31.0 --worker_pool
>
> * apache/beam_spark_job_server:2.31.0 for spark 2.4.8
>
> 3. SDK client code
>
> [https://gist.github.com/yuwtennis/2e4c13c79f71e8f713e947955115b3e2]
> Spark 2.4.8 succeeded without any errors using above components.
>
> [https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
