sarutak commented on a change in pull request #31880:
URL: https://github.com/apache/spark/pull/31880#discussion_r598073022
##########
File path: dev/change-scala-version.sh
##########
@@ -60,14 +60,13 @@ BASEDIR=$(dirname $0)/..
find "$BASEDIR" -name 'pom.xml' -not -path '*target*' -print \
-exec bash -c "sed_i 's/\(artifactId.*\)_'$FROM_VERSION'/\1_'$TO_VERSION'/g'
{}" \;
-# dependency:get is workaround for SPARK-34762 to download the JAR file of
commons-cli.
-# Without this, build with Scala 2.13 using SBT will fail because the help
plugin used below downloads only the POM file.
-COMMONS_CLI_VERSION=`build/mvn help:evaluate -Dexpression=commons-cli.version
-q -DforceStdout`
-build/mvn dependency:get
-Dartifact=commons-cli:commons-cli:${COMMONS_CLI_VERSION} -q
-
# Update <scala.version> in parent POM
# First find the right full version from the profile's build
-SCALA_VERSION=`build/mvn help:evaluate -Pscala-${TO_VERSION}
-Dexpression=scala.version -q -DforceStdout`
+# NOTE: We used mvn help:evaluate to fetch the value of scala.version before
but sed is used now.
+# This is a workaround for SPARK-34762.
+ESCAPED_TO_VERSION=$(echo $TO_VERSION | sed -n "s/\./\\\\./gp")
+SCALA_VERSION=$(sed -n "/<id>scala-$ESCAPED_TO_VERSION<\/id>/,/<\/profile>/ \
+ s;^.*<scala\.version>\(.*\)</scala\.version>.*$;\1;p" pom.xml)
Review comment:
I don't think we can fix this by using newer Maven or newer plugins.
Spark needs to use newer version of dependencies than what plugins use.
This problem can happen if all the following condition is true.
* Spark and a plugin have a common direct/indirect dependcy.
* A plugin uses newer or the same version of the dependency.
* The plugin downloads `pom` but not `jar` for the dependency.
* Build with `sbt` (or may be the case `courier` is used) under the
condition that the pom is present but the jar is absent..
One example is `dependency-plugin` and `commons-io`. Both Spark and
`dependency-plugin` depends on `commons-io` (`dependency-plugin` seems to
depend on it indirectly).
And `branch-3.1` depends on `commons-io:2.4`, while `dependency-plugin`
depends on newer `commons-io:2.6`.
When `mvn dependency:get` runs, pom is downloaded but doesn't jar for
`commons-io:2.6`.
Under this condition, if we build with `sbt`, `sbt` or `courier` doesn't
download the dependent jar, leading this issue.
Newer Maven and newer plugins can depends on newer version of the common
dependency than what Spark depends on. So I don't think we can't fix this issue
using newer Maven or newer plugins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
