This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3ef18e2d00f [SPARK-45546][BUILD][INFRA] Make `publish-snapshot`
support `package` first then `deploy`
3ef18e2d00f is described below
commit 3ef18e2d00f386196292f0c768816626bc903d47
Author: yangjie01 <[email protected]>
AuthorDate: Wed Oct 18 10:15:27 2023 +0800
[SPARK-45546][BUILD][INFRA] Make `publish-snapshot` support `package` first
then `deploy`
### What changes were proposed in this pull request?
This pr adds an environment variable `PACKAGE_BEFORE_DEPLOY` to the
`publish-snapshot` process. When `PACKAGE_BEFORE_DEPLOY` is true, the publish
process will be split into two steps: the first step is to package with mvn
package, and the second step is to deploy the packaged jar.
At the same time, this PR sets `PACKAGE_BEFORE_DEPLOY` to true in the
`publish_snapshot.yml` configuration.
### Why are the changes needed?
Make the `publish-snapshot` task in GitHub Action to be divided into two
steps, which can alleviate the resource pressure brought by direct deploy.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GitHub Actions
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43378 from LuciferYang/no-doc-deploy.
Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
.github/workflows/publish_snapshot.yml | 4 ++++
dev/create-release/release-build.sh | 14 ++++++++++++--
pom.xml | 7 +++++++
3 files changed, 23 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/publish_snapshot.yml
b/.github/workflows/publish_snapshot.yml
index 7ed836f016b..476d41d0cf1 100644
--- a/.github/workflows/publish_snapshot.yml
+++ b/.github/workflows/publish_snapshot.yml
@@ -66,4 +66,8 @@ jobs:
GPG_KEY: "not_used"
GPG_PASSPHRASE: "not_used"
GIT_REF: ${{ matrix.branch }}
+ # SPARK-45546 adds this environment variable to split the publish
snapshot process into two steps:
+ # first package, then deploy. This is intended to reduce the resource
pressure of deploy.
+ # When PACKAGE_BEFORE_DEPLOY is not set to true, it will revert to the
one-step deploy method.
+ PACKAGE_BEFORE_DEPLOY: true
run: ./dev/create-release/release-build.sh publish-snapshot
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
index f3571c4e48c..3776c64e31e 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -432,14 +432,24 @@ if [[ "$1" == "publish-snapshot" ]]; then
echo "</server></servers></settings>" >> $tmp_settings
if [[ $PUBLISH_SCALA_2_12 = 1 ]]; then
- $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES
$PUBLISH_PROFILES clean deploy
+ if [ "$PACKAGE_BEFORE_DEPLOY" = "true" ]; then
+ $MVN -DskipTests $SCALA_2_12_PROFILES $PUBLISH_PROFILES clean package
+ $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES
$PUBLISH_PROFILES deploy
+ else
+ $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES
$PUBLISH_PROFILES clean deploy
+ fi
fi
if [[ $PUBLISH_SCALA_2_13 = 1 ]]; then
if [[ $SPARK_VERSION < "4.0" ]]; then
./dev/change-scala-version.sh 2.13
fi
- $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES
$PUBLISH_PROFILES clean deploy
+ if [ "$PACKAGE_BEFORE_DEPLOY" = "true" ]; then
+ $MVN -DskipTests $SCALA_2_13_PROFILES $PUBLISH_PROFILES clean package
+ $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES
$PUBLISH_PROFILES deploy
+ else
+ $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES
$PUBLISH_PROFILES clean deploy
+ fi
fi
rm $tmp_settings
diff --git a/pom.xml b/pom.xml
index 824ae49f6da..ade6537c2a1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -3840,4 +3840,11 @@
</build>
</profile>
</profiles>
+ <distributionManagement>
+ <snapshotRepository>
+ <id>internal.snapshot</id>
+ <name>Internal Snapshot Repository</name>
+ <url>http://localhost:8081/repository/maven-snapshots/</url>
+ </snapshotRepository>
+ </distributionManagement>
</project>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
