This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a89536e5f [SPARK-39512] Document docker image release steps (#400)
a89536e5f is described below
commit a89536e5fc4498945149de3e0fb3ec8dc456b908
Author: Holden Karau <[email protected]>
AuthorDate: Mon Jul 18 18:27:13 2022 -0700
[SPARK-39512] Document docker image release steps (#400)
Document the docker image release steps for the release manager to follow
when finalizing the release.
---
release-process.md | 18 ++++++++++++++++--
site/release-process.html | 15 +++++++++++++--
2 files changed, 29 insertions(+), 4 deletions(-)
diff --git a/release-process.md b/release-process.md
index 7c1e9ff2d..e8bba053f 100644
--- a/release-process.md
+++ b/release-process.md
@@ -35,6 +35,9 @@ If you are a new Release Manager, you can read up on the
process from the follow
- gpg for signing https://www.apache.org/dev/openpgp.html
- svn https://www.apache.org/dev/version-control.html#https-svn
+
+You should also get access to the ASF Dockerhub. You can get access by filing
a INFRA JIRA ticket (see an example ticket
https://issues.apache.org/jira/browse/INFRA-21282 ).
+
<h3>Preparing gpg key</h3>
You can skip this section if you have already uploaded your key.
@@ -175,10 +178,11 @@ To cut a release candidate, there are 4 steps:
1. Package the release binaries & sources, and upload them to the Apache
staging SVN repo.
1. Create the release docs, and upload them to the Apache staging SVN repo.
1. Publish a snapshot to the Apache staging Maven repo.
+1. Create a RC docker image tag (e.g. `3.4.0-rc1`)
-The process of cutting a release candidate has been automated via the
`dev/create-release/do-release-docker.sh` script.
+The process of cutting a release candidate has been mostly automated via the
`dev/create-release/do-release-docker.sh` script.
Run this script, type information it requires, and wait until it finishes. You
can also do a single step via the `-s` option.
-Please run `do-release-docker.sh -h` and see more details.
+Please run `do-release-docker.sh -h` and see more details. It does not
currently generate the RC docker image tag.
<h3>Call a vote on the release candidate</h3>
@@ -387,6 +391,16 @@ $ git shortlog v1.1.1 --grep "$EXPR" > contrib.txt
$ git log v1.1.1 --grep "$expr" --shortstat --oneline | grep -B 1 -e
"[3-9][0-9][0-9] insert" -e "[1-9][1-9][1-9][1-9] insert" | grep SPARK >
large-patches.txt
```
+<h4>Create and upload Spark Docker Images</h4>
+
+The Spark docker images are created using the `./bin/docker-image-tool.sh`
that is included in the release artifacts.
+
+
+You should install `docker buildx` so that you can cross-compile for multiple
archs as ARM is becoming increasing popular. If you have access to both an ARM
and an x86 machine you should set up a [remote builder as described
here](https://scalingpythonml.com/2020/12/11/some-sharp-corners-with-docker-buildx.html),
but if you only have one [docker buildx with QEMU works fine as we don't use
cgo](https://docs.docker.com/buildx/working-with-buildx/).
+
+
+Once you have your cross-platform docker build environment setup, extract the
build artifact (e.g. `tar -xvf spark-3.3.0-bin-hadoop3.tgz`), go into the
directory (e.g. `cd spark-3.3.0-bin-hadoop3`) and build the containers and
publish them to the Spark dockerhub (e.g. `./bin/docker-image-tool.sh -r
docker.io/apache -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile
-t v3.3.0 -X -b java_image_tag=11-jre-slim build`)
+
<h4>Create an announcement</h4>
Once everything is working (website docs, website changes) create an
announcement on the website
diff --git a/site/release-process.html b/site/release-process.html
index 236dd4631..dbecf8ff3 100644
--- a/site/release-process.html
+++ b/site/release-process.html
@@ -163,6 +163,8 @@
<li>svn https://www.apache.org/dev/version-control.html#https-svn</li>
</ul>
+<p>You should also get access to the ASF Dockerhub. You can get access by
filing a INFRA JIRA ticket (see an example ticket
https://issues.apache.org/jira/browse/INFRA-21282 ).</p>
+
<h3>Preparing gpg key</h3>
<p>You can skip this section if you have already uploaded your key.</p>
@@ -295,11 +297,12 @@ Note that not all permutations are run on PR therefore it
is important to check
<li>Package the release binaries & sources, and upload them to the
Apache staging SVN repo.</li>
<li>Create the release docs, and upload them to the Apache staging SVN
repo.</li>
<li>Publish a snapshot to the Apache staging Maven repo.</li>
+ <li>Create a RC docker image tag (e.g. <code class="language-plaintext
highlighter-rouge">3.4.0-rc1</code>)</li>
</ol>
-<p>The process of cutting a release candidate has been automated via the <code
class="language-plaintext
highlighter-rouge">dev/create-release/do-release-docker.sh</code> script.
+<p>The process of cutting a release candidate has been mostly automated via
the <code class="language-plaintext
highlighter-rouge">dev/create-release/do-release-docker.sh</code> script.
Run this script, type information it requires, and wait until it finishes. You
can also do a single step via the <code class="language-plaintext
highlighter-rouge">-s</code> option.
-Please run <code class="language-plaintext
highlighter-rouge">do-release-docker.sh -h</code> and see more details.</p>
+Please run <code class="language-plaintext
highlighter-rouge">do-release-docker.sh -h</code> and see more details. It does
not currently generate the RC docker image tag.</p>
<h3>Call a vote on the release candidate</h3>
@@ -497,6 +500,14 @@ $ git shortlog v1.1.1 --grep "$EXPR" > contrib.txt
$ git log v1.1.1 --grep "$expr" --shortstat --oneline | grep -B 1 -e
"[3-9][0-9][0-9] insert" -e "[1-9][1-9][1-9][1-9] insert" | grep SPARK >
large-patches.txt
</code></pre></div></div>
+<h4>Create and upload Spark Docker Images</h4>
+
+<p>The Spark docker images are created using the <code
class="language-plaintext highlighter-rouge">./bin/docker-image-tool.sh</code>
that is included in the release artifacts.</p>
+
+<p>You should install <code class="language-plaintext
highlighter-rouge">docker buildx</code> so that you can cross-compile for
multiple archs as ARM is becoming increasing popular. If you have access to
both an ARM and an x86 machine you should set up a <a
href="https://scalingpythonml.com/2020/12/11/some-sharp-corners-with-docker-buildx.html";>remote
builder as described here</a>, but if you only have one <a
href="https://docs.docker.com/buildx/working-with-buildx/";>docker buildx with Q
[...]
+
+<p>Once you have your cross-platform docker build environment setup, extract
the build artifact (e.g. <code class="language-plaintext highlighter-rouge">tar
-xvf spark-3.3.0-bin-hadoop3.tgz</code>), go into the directory (e.g. <code
class="language-plaintext highlighter-rouge">cd spark-3.3.0-bin-hadoop3</code>)
and build the containers and publish them to the Spark dockerhub (e.g. <code
class="language-plaintext highlighter-rouge">./bin/docker-image-tool.sh -r
docker.io/apache -p ./kuber [...]
+
<h4>Create an announcement</h4>
<p>Once everything is working (website docs, website changes) create an
announcement on the website
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
