Repository: spark
Updated Branches:
refs/heads/master 6364735bc -> f5b6dc5e3
[SPARK-8401] [BUILD] Scala version switching build enhancements
These commits address a few minor issues in the Scala cross-version support in
the build:
1. Correct two missing `${scala.binary.version}` pom file substitutions.
2. Don't update `scala.binary.version` in parent POM. This property is set
through profiles.
3. Update the source of the generated scaladocs in
`docs/_plugins/copy_api_dirs.rb`.
4. Factor common code out of `dev/change-version-to-*.sh` and add some
validation. We also test `sed` to see if it's GNU sed and try `gsed` as an
alternative if not. This prevents the script from running with a non-GNU sed.
This is my original work and I license this work to the Spark project under the
Apache License.
Author: Michael Allman <mich...@videoamp.com>
Closes #6832 from mallman/scala-versions and squashes the following commits:
cde2f17 [Michael Allman] Delete dev/change-version-to-*.sh, replacing them with
single dev/change-scala-version.sh script that takes a version as argument
02296f2 [Michael Allman] Make the scala version change scripts cross-platform
by restricting ourselves to POSIX sed syntax instead of looking for GNU sed
ad9b40a [Michael Allman] Factor change-scala-version.sh out of
change-version-to-*.sh, adding command line argument validation and testing for
GNU sed
bdd20bf [Michael Allman] Update source of scaladocs when changing Scala version
475088e [Michael Allman] Replace jackson-module-scala_2.10 with
jackson-module-scala_${scala.binary.version}
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f5b6dc5e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f5b6dc5e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f5b6dc5e
Branch: refs/heads/master
Commit: f5b6dc5e3e7e3b586096b71164f052318b840e8a
Parents: 6364735
Author: Michael Allman <mich...@videoamp.com>
Authored: Tue Jul 21 11:14:31 2015 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Tue Jul 21 11:14:31 2015 +0100
----------------------------------------------------------------------
core/pom.xml | 2 +-
dev/change-scala-version.sh | 66 +++++++++++++++++++++++++++++++
dev/change-version-to-2.10.sh | 26 ------------
dev/change-version-to-2.11.sh | 26 ------------
dev/create-release/create-release.sh | 6 +--
docs/building-spark.md | 2 +-
pom.xml | 2 +-
7 files changed, 72 insertions(+), 58 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/f5b6dc5e/core/pom.xml
----------------------------------------------------------------------
diff --git a/core/pom.xml b/core/pom.xml
index 73f7a75..95f36eb 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -261,7 +261,7 @@
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
- <artifactId>jackson-module-scala_2.10</artifactId>
+ <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
http://git-wip-us.apache.org/repos/asf/spark/blob/f5b6dc5e/dev/change-scala-version.sh
----------------------------------------------------------------------
diff --git a/dev/change-scala-version.sh b/dev/change-scala-version.sh
new file mode 100755
index 0000000..b81c00c
--- /dev/null
+++ b/dev/change-scala-version.sh
@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+
+usage() {
+ echo "Usage: $(basename $0) <version>" 1>&2
+ exit 1
+}
+
+if [ $# -ne 1 ]; then
+ usage
+fi
+
+TO_VERSION=$1
+
+VALID_VERSIONS=( 2.10 2.11 )
+
+check_scala_version() {
+ for i in ${VALID_VERSIONS[*]}; do [ $i = "$1" ] && return 0; done
+ echo "Invalid Scala version: $1. Valid versions: ${VALID_VERSIONS[*]}" 1>&2
+ exit 1
+}
+
+check_scala_version "$TO_VERSION"
+
+if [ $TO_VERSION = "2.11" ]; then
+ FROM_VERSION="2.10"
+else
+ FROM_VERSION="2.11"
+fi
+
+sed_i() {
+ sed -e "$1" "$2" > "$2.tmp" && mv "$2.tmp" "$2"
+}
+
+export -f sed_i
+
+BASEDIR=$(dirname $0)/..
+find "$BASEDIR" -name 'pom.xml' -not -path '*target*' -print \
+ -exec bash -c "sed_i 's/\(artifactId.*\)_'$FROM_VERSION'/\1_'$TO_VERSION'/g'
{}" \;
+
+# Also update <scala.binary.version> in parent POM
+# Match any scala binary version to ensure idempotency
+sed_i
'1,/<scala\.binary\.version>[0-9]*\.[0-9]*</s/<scala\.binary\.version>[0-9]*\.[0-9]*</<scala.binary.version>'$TO_VERSION'</'
\
+ "$BASEDIR/pom.xml"
+
+# Update source of scaladocs
+echo "$BASEDIR/docs/_plugins/copy_api_dirs.rb"
+sed_i 's/scala\-'$FROM_VERSION'/scala\-'$TO_VERSION'/'
"$BASEDIR/docs/_plugins/copy_api_dirs.rb"
http://git-wip-us.apache.org/repos/asf/spark/blob/f5b6dc5e/dev/change-version-to-2.10.sh
----------------------------------------------------------------------
diff --git a/dev/change-version-to-2.10.sh b/dev/change-version-to-2.10.sh
deleted file mode 100755
index c4adb1f..0000000
--- a/dev/change-version-to-2.10.sh
+++ /dev/null
@@ -1,26 +0,0 @@
-#!/usr/bin/env bash
-
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Note that this will not necessarily work as intended with non-GNU sed (e.g.
OS X)
-BASEDIR=$(dirname $0)/..
-find $BASEDIR -name 'pom.xml' | grep -v target \
- | xargs -I {} sed -i -e 's/\(artifactId.*\)_2.11/\1_2.10/g' {}
-
-# Also update <scala.binary.version> in parent POM
-sed -i -e '0,/<scala\.binary\.version>2.11</s//<scala.binary.version>2.10</'
$BASEDIR/pom.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/f5b6dc5e/dev/change-version-to-2.11.sh
----------------------------------------------------------------------
diff --git a/dev/change-version-to-2.11.sh b/dev/change-version-to-2.11.sh
deleted file mode 100755
index d370019..0000000
--- a/dev/change-version-to-2.11.sh
+++ /dev/null
@@ -1,26 +0,0 @@
-#!/usr/bin/env bash
-
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Note that this will not necessarily work as intended with non-GNU sed (e.g.
OS X)
-BASEDIR=$(dirname $0)/..
-find $BASEDIR -name 'pom.xml' | grep -v target \
- | xargs -I {} sed -i -e 's/\(artifactId.*\)_2.10/\1_2.11/g' {}
-
-# Also update <scala.binary.version> in parent POM
-sed -i -e '0,/<scala\.binary\.version>2.10</s//<scala.binary.version>2.11</'
$BASEDIR/pom.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/f5b6dc5e/dev/create-release/create-release.sh
----------------------------------------------------------------------
diff --git a/dev/create-release/create-release.sh
b/dev/create-release/create-release.sh
index 30190dc..86a7a40 100755
--- a/dev/create-release/create-release.sh
+++ b/dev/create-release/create-release.sh
@@ -122,13 +122,13 @@ if [[ ! "$@" =~ --skip-publish ]]; then
-Phive-thriftserver -Phadoop-2.2 -Pspark-ganglia-lgpl -Pkinesis-asl \
clean install
- ./dev/change-version-to-2.11.sh
+ ./dev/change-scala-version.sh 2.11
build/mvn -DskipTests -Pyarn -Phive -Prelease\
-Dscala-2.11 -Phadoop-2.2 -Pspark-ganglia-lgpl -Pkinesis-asl \
clean install
- ./dev/change-version-to-2.10.sh
+ ./dev/change-scala-version.sh 2.10
pushd $SPARK_REPO
@@ -205,7 +205,7 @@ if [[ ! "$@" =~ --skip-package ]]; then
# TODO There should probably be a flag to make-distribution to allow 2.11
support
if [[ $FLAGS == *scala-2.11* ]]; then
- ./dev/change-version-to-2.11.sh
+ ./dev/change-scala-version.sh 2.11
fi
export ZINC_PORT=$ZINC_PORT
http://git-wip-us.apache.org/repos/asf/spark/blob/f5b6dc5e/docs/building-spark.md
----------------------------------------------------------------------
diff --git a/docs/building-spark.md b/docs/building-spark.md
index 2128fdf..a5da3b3 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -124,7 +124,7 @@ mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive
-Phive-thriftserver -Dskip
# Building for Scala 2.11
To produce a Spark package compiled with Scala 2.11, use the `-Dscala-2.11`
property:
- dev/change-version-to-2.11.sh
+ dev/change-scala-version.sh 2.11
mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
Spark does not yet support its JDBC component for Scala 2.11.
http://git-wip-us.apache.org/repos/asf/spark/blob/f5b6dc5e/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 2de0c35..1f44dc8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -614,7 +614,7 @@
15.0, which causes runtime incompatibility issues. -->
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
- <artifactId>jackson-module-scala_2.10</artifactId>
+ <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
<version>${fasterxml.jackson.version}</version>
<exclusions>
<exclusion>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
