This is an automated email from the ASF dual-hosted git repository.
lixiao pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new e417168 [SPARK-26048][SPARK-24530][2.4] Cherrypick all the missing
commits to 2.4 release script
e417168 is described below
commit e417168ed012190db66a21e626b2b8d2332d6c01
Author: Wenchen Fan <[email protected]>
AuthorDate: Wed May 1 07:36:00 2019 -0700
[SPARK-26048][SPARK-24530][2.4] Cherrypick all the missing commits to 2.4
release script
## What changes were proposed in this pull request?
This PR is to cherry-pick all the missing and relevant commits that were
merged to master but not to branch-2.4.
Previously, dbtsai used the release script in the branch 2.4 to release
2.4.1.
After more investigation, I found it is risky to make a 2.4 release by
using the release script in the master branch since the release script has
various changes. It could easily introduce unnoticeable issues, like what we
did for 2.4.2.
Thus, I would cherry-pick all the missing fixes and use the updated release
script to release 2.4.3
## How was this patch tested?
N/A
Closes #24503 from gatorsmile/upgradeReleaseScript.
Lead-authored-by: Wenchen Fan <[email protected]>
Co-authored-by: gatorsmile <[email protected]>
Co-authored-by: wright <[email protected]>
Signed-off-by: gatorsmile <[email protected]>
---
dev/create-release/do-release-docker.sh | 3 +++
dev/create-release/release-build.sh | 7 +++++--
dev/create-release/releaseutils.py | 2 +-
dev/create-release/spark-rm/Dockerfile | 4 ++--
4 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/dev/create-release/do-release-docker.sh
b/dev/create-release/do-release-docker.sh
index fa7b73c..c1a122e 100755
--- a/dev/create-release/do-release-docker.sh
+++ b/dev/create-release/do-release-docker.sh
@@ -135,6 +135,9 @@ if [ -n "$JAVA" ]; then
JAVA_VOL="--volume $JAVA:/opt/spark-java"
fi
+# SPARK-24530: Sphinx must work with python 3 to generate doc correctly.
+echo "SPHINXPYTHON=/opt/p35/bin/python" >> $ENVFILE
+
echo "Building $RELEASE_TAG; output will be at $WORKDIR/output"
docker run -ti \
--env-file "$ENVFILE" \
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
index 5e65d99..affb4dc 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -122,6 +122,9 @@ fi
PUBLISH_SCALA_2_12=0
SCALA_2_12_PROFILES="-Pscala-2.12"
+if [[ $SPARK_VERSION < "3.0." ]]; then
+ SCALA_2_12_PROFILES="-Pscala-2.12 -Pflume"
+fi
if [[ $SPARK_VERSION > "2.4" ]]; then
PUBLISH_SCALA_2_12=1
fi
@@ -327,7 +330,7 @@ if [[ "$1" == "package" ]]; then
svn add "svn-spark/${DEST_DIR_NAME}-bin"
cd svn-spark
- svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Apache Spark
$SPARK_PACKAGE_VERSION"
+ svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Apache Spark
$SPARK_PACKAGE_VERSION" --no-auth-cache
cd ..
rm -rf svn-spark
fi
@@ -355,7 +358,7 @@ if [[ "$1" == "docs" ]]; then
svn add "svn-spark/${DEST_DIR_NAME}-docs"
cd svn-spark
- svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Apache Spark
$SPARK_PACKAGE_VERSION docs"
+ svn ci --username $ASF_USERNAME --password "$ASF_PASSWORD" -m"Apache Spark
$SPARK_PACKAGE_VERSION docs" --no-auth-cache
cd ..
rm -rf svn-spark
fi
diff --git a/dev/create-release/releaseutils.py
b/dev/create-release/releaseutils.py
index f273b33..a5a26ae 100755
--- a/dev/create-release/releaseutils.py
+++ b/dev/create-release/releaseutils.py
@@ -236,7 +236,7 @@ def translate_component(component, commit_hash, warnings):
# The returned components are already filtered and translated
def find_components(commit, commit_hash):
components = re.findall(r"\[\w*\]", commit.lower())
- components = [translate_component(c, commit_hash)
+ components = [translate_component(c, commit_hash, [])
for c in components if c in known_components]
return components
diff --git a/dev/create-release/spark-rm/Dockerfile
b/dev/create-release/spark-rm/Dockerfile
index 15f831c..4231544 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -62,8 +62,8 @@ RUN echo 'deb http://cran.cnr.Berkeley.edu/bin/linux/ubuntu
xenial/' >> /etc/apt
pip install $BASE_PIP_PKGS && \
pip install $PIP_PKGS && \
cd && \
- virtualenv -p python3 p35 && \
- . p35/bin/activate && \
+ virtualenv -p python3 /opt/p35 && \
+ . /opt/p35/bin/activate && \
pip install $BASE_PIP_PKGS && \
pip install $PIP_PKGS && \
# Install R packages and dependencies used when building.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
