sarutak commented on a change in pull request #31880:
URL: https://github.com/apache/spark/pull/31880#discussion_r597490256
##########
File path: dev/change-scala-version.sh
##########
@@ -60,14 +60,13 @@ BASEDIR=$(dirname $0)/..
find "$BASEDIR" -name 'pom.xml' -not -path '*target*' -print \
-exec bash -c "sed_i 's/\(artifactId.*\)_'$FROM_VERSION'/\1_'$TO_VERSION'/g'
{}" \;
-# dependency:get is workaround for SPARK-34762 to download the JAR file of
commons-cli.
-# Without this, build with Scala 2.13 using SBT will fail because the help
plugin used below downloads only the POM file.
-COMMONS_CLI_VERSION=`build/mvn help:evaluate -Dexpression=commons-cli.version
-q -DforceStdout`
-build/mvn dependency:get
-Dartifact=commons-cli:commons-cli:${COMMONS_CLI_VERSION} -q
-
# Update <scala.version> in parent POM
# First find the right full version from the profile's build
-SCALA_VERSION=`build/mvn help:evaluate -Pscala-${TO_VERSION}
-Dexpression=scala.version -q -DforceStdout`
+# NOTE: We used mvn help:evaluate to fetch the value of scala.version before
but sed is used now.
+# This is a workaround for SPARK-34762.
+ESCAPED_TO_VERSION=$(echo $TO_VERSION | sed -n "s/\./\\\\./gp")
+SCALA_VERSION=$(sed -n "/<id>scala-$ESCAPED_TO_VERSION<\/id>/,/<\/profile>/ \
+ s;^.*<scala\.version>\(.*\)</scala\.version>.*$;\1;p" pom.xml)
Review comment:
sbt (or courier?) seems to fail to resolve dependency if the pom file
for a dependency is in `~/.m2` but jar file is not. I don't know the reason.
For `master` branch, before #31862, `change-scala-version.sh` run `mvn
help:evaluate` and it downloads `commons-cli-1.2.pom` but doesn't
`commons-cli-1.2.jar`.
```
$ ls -R ~/.m2/repository/commons-cli/commons-cli/
/home/kou/.m2/repository/commons-cli/commons-cli/:
1.2
/home/kou/.m2/repository/commons-cli/commons-cli/1.2:
_remote.repositories commons-cli-1.2.pom commons-cli-1.2.pom.sha1
```
You can also confirm with `mvn -X help:evaluate`.
So, I resolved by getting `commons-cli-1.2.jar`using `mvn dependency:get` in
#31862.
For `branch-3.1`, `mvn help:evaluate` also downloads `commons-cli-1.2.jar`
but it's resolved this part by #31862.
But `mvn dependency:get` downloads `commons-io-2.6.pom` though it doesn't
download `commons-io-2.6.jar`.
```
ls -R ~/.m2/repository/commons-io/commons-io/
/home/kou/.m2/repository/commons-io/commons-io/:
2.4 2.5 2.6
/home/kou/.m2/repository/commons-io/commons-io/2.4:
_remote.repositories commons-io-2.4.pom commons-io-2.4.pom.sha1
/home/kou/.m2/repository/commons-io/commons-io/2.5:
_remote.repositories commons-io-2.5.jar commons-io-2.5.jar.sha1
commons-io-2.5.pom commons-io-2.5.pom.sha1
/home/kou/.m2/repository/commons-io/commons-io/2.6:
_remote.repositories commons-io-2.6.pom commons-io-2.6.pom.sha1
```
I understand `branch-3.1` depends on `commons-io-2.5` but, in fact, if we
manually delete `commons-io/comons-io-2.6` before `sbt`, build successfully
finishes.
It's also true for `master` that `commons-io-2.6.pom` is present but
`commons-io-2.6.jar` is absent.
But there is one difference between `master` and `branch-3.1`.
`master` depends on `commons-io-2.8` which is newer version than
`commons-io-2.6` while `branch-3.1` depends on `commons-io-2.5` which is older
than `commons-io-2.6`.
So I guess this affects build failure for `branch-3.1` while it succeeds for
`master`.
Anyway, if we don't use maven plugins in `change-scala-version.sh`, this
problem can be resolved easily.
Or, do you have a better solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
