[
https://issues.apache.org/jira/browse/SPARK-27911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854151#comment-16854151
]
Hyukjin Kwon commented on SPARK-27911:
--------------------------------------
Hi [~marmbrus],
the issues looks because users used Scala 2.11 complied Spark (2.4.2).
{code}
pyspark --packages io.delta:delta-core_2.12:0.1.0
{code}
Does this mean we should replace {{io.delta:delta-core_2.12:0.1.0}} to
{{io.delta:delta-core_2.11:0.1.0}} automatically? If so, I am wondering if this
is something that should be done in Scala side so that Scala shell can detect
the right version.
https://github.com/delta-io/delta/issues/6 this issue seems describing Scala
side.
Would you mind if I ask to elaborate why in this JIRA? it looks a general
problem but wondering why this targets PySpark only.
> PySpark Packages should automatically choose correct scala version
> ------------------------------------------------------------------
>
> Key: SPARK-27911
> URL: https://issues.apache.org/jira/browse/SPARK-27911
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
> Affects Versions: 2.4.3
> Reporter: Michael Armbrust
> Priority: Major
>
> Today, users of pyspark (and Scala) need to manually specify the version of
> Scala that their Spark installation is using when adding a Spark package to
> their application. This extra configuration is confusing to users who may not
> even know which version of Scala they are using (for example, if they
> installed Spark using {{pip}}). The confusion here is exacerbated by releases
> in Spark that have changed the default from {{2.11}} -> {{2.12}} -> {{2.11}}.
> https://spark.apache.org/releases/spark-release-2-4-2.html
> https://spark.apache.org/releases/spark-release-2-4-3.html
> Since Spark can know which version of Scala it was compiled for, we should
> give users the option to automatically choose the correct version. This
> could be as simple as a substitution for {{$scalaVersion}} or something when
> resolving a package (similar to SBTs support for automatically handling scala
> dependencies).
> Here are some concrete examples of users getting it wrong and getting
> confused:
> https://github.com/delta-io/delta/issues/6
> https://github.com/delta-io/delta/issues/63
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
