[
https://issues.apache.org/jira/browse/BEAM-7092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated BEAM-7092:
-------------------------------
Description:
Since Spark 3 is relying on Scala 2.12 and a different set of dependencies
probably we need to divide the project and reuse most of the code as the Flink
runner does to support multiple major versions. (Notice that minor versions are
provided in the case of the Spark runner).
Probably the best (and simplest) path forward is to continue with Scala 2.11 as
the default for the current runner (based on Spark 2). And make the Scala 2.12
set of dependencies the default for Spark 3.
was:
Since Spark 3 is relying on Scala 2.12 and a different set of dependencies
probably we need to divide the project and reuse most of the code as Flink
runner does to support multiple versions.
Probably the best (and simplest) path forward is to support Scala 2.11
Spark is deprecating Scala 2.11 support starting with the next release (3.0.0),
so it is probably a good idea to move the runner to use the already available
dependencies based on Scala 2.12.
> Update Spark runner support multiple versions of Spark/Scala
> ------------------------------------------------------------
>
> Key: BEAM-7092
> URL: https://issues.apache.org/jira/browse/BEAM-7092
> Project: Beam
> Issue Type: Sub-task
> Components: runner-spark
> Reporter: Ismaël Mejía
> Priority: Minor
>
> Since Spark 3 is relying on Scala 2.12 and a different set of dependencies
> probably we need to divide the project and reuse most of the code as the
> Flink runner does to support multiple major versions. (Notice that minor
> versions are provided in the case of the Spark runner).
> Probably the best (and simplest) path forward is to continue with Scala 2.11
> as the default for the current runner (based on Spark 2). And make the Scala
> 2.12 set of dependencies the default for Spark 3.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
