wgtmac commented on code in PR #1096:
URL: https://github.com/apache/iceberg-cpp/pull/1096#discussion_r4653223520


##########
dev/release/test_release_rc.py:
##########
@@ -0,0 +1,218 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import gzip
+import hashlib
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+
+
+MOCK_TOOL = r'''
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+
+tool = Path(sys.argv[0]).name
+args = sys.argv[1:]
+if tool in ("git", "gzip"):
+    result = subprocess.run([os.environ["REAL_" + tool.upper()], *args])
+    if tool == "git" and "archive" in args and os.environ.get("FAIL_ARCHIVE"):
+        sys.exit(1)
+    sys.exit(result.returncode)
+elif tool == "gh" and args[:2] == ["run", "watch"]:
+    pass
+elif tool == "gh" and args[:2] == ["release", "download"]:
+    destination = Path(args[args.index("--dir") + 1])
+    for index, arg in enumerate(args):
+        if arg == "--pattern":
+            filename = args[index + 1]
+            target = destination / filename
+            if target.exists() and "--skip-existing" in args:
+                continue
+            shutil.copyfile(Path(os.environ["ASSETS"]) / filename, target)
+elif tool == "gpg":
+    Path(args[args.index("--output") + 1]).write_text("test signature")
+elif tool == "svn" and args[0] == "import":
+    pass
+elif tool != "gh" or args[:2] != ["release", "upload"]:
+    sys.exit("Unexpected command: " + tool + " " + " ".join(args))
+
+if tool in ("gpg", "svn") or (tool == "gh" and args[:2] == ["release", 
"upload"]):
+    with open(os.environ["CALL_LOG"], "a") as log:
+        log.write(tool + " " + " ".join(args) + "\n")
+'''
+
+
+class ReleaseRcTest(unittest.TestCase):

Review Comment:
   Please drop this file from this PR. The CLI stubs have to track command 
arguments and error messages. Each runner also builds its own fixture, so the 
test misses the CI-to-signer handoff.



##########
.github/workflows/release_test.yml:
##########
@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Release Script Tests

Review Comment:
   Please drop this workflow with the Python harness. The existing RC workflow 
already builds and tests the tagged archive.



##########
dev/release/release_rc.sh:
##########
@@ -19,13 +19,22 @@
 
 set -eu
 
-for cmd in git gh gpg svn; do
+for cmd in git gh gpg svn gzip cmp mktemp; do
   if ! command -v ${cmd} &> /dev/null; then
     echo "This script requires '${cmd}' but it's not installed. Aborting."
     exit 1
   fi
 done
 
+if command -v shasum > /dev/null 2>&1; then
+  sha512_verify=(shasum -a 512 -c)
+elif command -v sha512sum > /dev/null 2>&1; then
+  sha512_verify=(sha512sum -c)
+else
+  echo "This script requires 'shasum' or 'sha512sum'. Aborting."
+  exit 1
+fi

Review Comment:
   Please keep the existing startup checks.
   
   ```suggestion
   for cmd in git gh gpg svn; do
     if ! command -v ${cmd} &> /dev/null; then
       echo "This script requires '${cmd}' but it's not installed. Aborting."
       exit 1
     fi
   done
   ```
   
   Add this block after the `RELEASE_*` defaults. It checks the new tools 
before we push the tag. An SVN-only retry can skip them.
   
   ```bash
   if [ "${RELEASE_SIGN}" -gt 0 ]; then
     for cmd in gzip cmp mktemp; do
       if ! command -v "${cmd}" > /dev/null 2>&1; then
         echo "The signing step requires '${cmd}'."
         exit 1
       fi
     done
   
     if command -v shasum > /dev/null 2>&1; then
       sha512_sum=(shasum -a 512 -t)
     elif command -v sha512sum > /dev/null 2>&1; then
       sha512_sum=(sha512sum -t)
     else
       echo "The signing step requires 'shasum' or 'sha512sum'."
       exit 1
     fi
   fi
   ```



##########
dev/release/release_rc.sh:
##########
@@ -78,6 +87,11 @@ rc_id="${id}-rc${rc}"
 tar_gz="${id}.tar.gz"
 
 if [ "${RELEASE_SIGN}" -gt 0 ]; then
+  if [ -e "${rc_id}" ] || [ -L "${rc_id}" ]; then
+    echo "Release directory ${rc_id} already exists. Move it aside before 
retrying."
+    exit 1
+  fi

Review Comment:
   Please allow retries to reuse the output directory. The new download already 
goes to a fresh temporary directory.
   
   ```suggestion
   ```



##########
dev/release/README.md:
##########
@@ -50,6 +50,13 @@ Requirements to run `release_rc.sh`:
     * You must be an Apache Iceberg committer or PMC member
     * You must prepare your PGP key for signing
 
+The signing step also requires `gzip`, `cmp`, `mktemp`, and either `shasum` or
+`sha512sum`. It verifies the downloaded checksum and compares the uncompressed
+archive against `git archive` of the RC commit before signing. Both CI and 
local
+verification pin line endings and archive permissions. Differences in Git's tar
+output across versions can still cause verification to fail.
+An existing RC output directory must be moved aside before retrying.

Review Comment:
   Update the retry instructions to match the signing flow.
   
   ```suggestion
   Each signing retry downloads and verifies the archive again.
   ```



##########
dev/release/README.md:
##########
@@ -108,3 +115,10 @@ $ dev/release/verify_rc.sh ${VERSION} ${RC}
 ```
 
 If the verification is successful, the message `RC looks good!` is shown.
+
+The release script's regression tests run in CI on Ubuntu and macOS with
+simulated signing and upload commands. To run them locally:
+
+```console
+$ python3 -B -m unittest discover -s dev/release -p 'test_release_rc.py' -v
+```

Review Comment:
   Please remove these instructions along with the test harness.
   
   ```suggestion
   ```



##########
dev/release/release_rc.sh:
##########
@@ -126,15 +140,38 @@ if [ "${RELEASE_SIGN}" -gt 0 ]; then
     done
   fi
 
-  mkdir -p "${rc_id}"
+  verify_dir="$(mktemp -d "${TMPDIR:-/tmp}/iceberg-cpp-rc.XXXXXX")"
+  trap 'rm -rf "${verify_dir}"' EXIT
+  mkdir "${verify_dir}/artifacts"
 
   echo "Downloading .tar.gz and .sha512 from GitHub Releases"
   gh release download "${rc_tag}" \
-    --dir "${rc_id}" \
+    --dir "${verify_dir}/artifacts" \
     --pattern "${tar_gz}" \
     --pattern "${tar_gz}.sha512" \
-    --repo "${repository}" \
-    --skip-existing
+    --repo "${repository}"
+
+  echo "Verifying downloaded tar.gz before signing"
+  if ! (cd "${verify_dir}/artifacts" && "${sha512_verify[@]}" 
"${tar_gz}.sha512"); then
+    echo "Downloaded ${tar_gz} failed SHA-512 verification. Not signing."
+    exit 1
+  fi
+  if ! gzip -dc "${verify_dir}/artifacts/${tar_gz}" > 
"${verify_dir}/downloaded.tar"; then
+    echo "Failed to decompress ${tar_gz}. Not signing."
+    exit 1
+  fi
+  if ! git -c core.autocrlf=false -c core.eol=lf -c tar.umask=0002 \
+      archive "${rc_hash}" --format=tar --prefix "${id}/" > 
"${verify_dir}/expected.tar"; then
+    echo "Failed to archive RC commit ${rc_hash}. Not signing."
+    exit 1
+  fi
+  # The checksum can be replaced along with the asset; anchor it to the RC 
commit.
+  if ! cmp -s "${verify_dir}/expected.tar" "${verify_dir}/downloaded.tar"; then
+    echo "Downloaded ${tar_gz} does not match RC commit ${rc_hash}. Not 
signing."
+    exit 1
+  fi
+
+  mv "${verify_dir}/artifacts" "${rc_id}"

Review Comment:
   Please replace lines 174–184 with this block. It signs the verified file in 
the temporary directory. Then it moves the three release files into the output 
directory before uploading the signature.
   
   ```bash
     echo "Signing tar.gz"
     gpg --armor --output "${verify_dir}/artifacts/${tar_gz}.asc" \
       --detach-sig "${verify_dir}/artifacts/${tar_gz}"
   
     mkdir -p "${rc_id}"
     mv -f \
       "${verify_dir}/artifacts/${tar_gz}" \
       "${verify_dir}/artifacts/${tar_gz}.sha512" \
       "${verify_dir}/artifacts/${tar_gz}.asc" \
       "${rc_id}/"
   
     echo "Add signature to GitHub release"
     gh release upload "${rc_tag}" \
       --clobber \
       --repo "${repository}" \
       "${rc_id}/${tar_gz}.asc"
   ```



##########
dev/release/release_rc.sh:
##########
@@ -126,15 +140,38 @@ if [ "${RELEASE_SIGN}" -gt 0 ]; then
     done
   fi
 
-  mkdir -p "${rc_id}"
+  verify_dir="$(mktemp -d "${TMPDIR:-/tmp}/iceberg-cpp-rc.XXXXXX")"
+  trap 'rm -rf "${verify_dir}"' EXIT
+  mkdir "${verify_dir}/artifacts"
 
   echo "Downloading .tar.gz and .sha512 from GitHub Releases"
   gh release download "${rc_tag}" \
-    --dir "${rc_id}" \
+    --dir "${verify_dir}/artifacts" \
     --pattern "${tar_gz}" \
     --pattern "${tar_gz}.sha512" \
-    --repo "${repository}" \
-    --skip-existing
+    --repo "${repository}"
+
+  echo "Verifying downloaded tar.gz before signing"
+  if ! (cd "${verify_dir}/artifacts" && "${sha512_verify[@]}" 
"${tar_gz}.sha512"); then
+    echo "Downloaded ${tar_gz} failed SHA-512 verification. Not signing."
+    exit 1
+  fi

Review Comment:
   Please compute SHA-512 from the exact archive we will sign. Compare the full 
record from `rc.yml` so both the filename and digest have to match. This uses 
`sha512_sum` from the tool-check comment above.
   
   ```suggestion
     if ! (cd "${verify_dir}/artifacts" && "${sha512_sum[@]}" "${tar_gz}") \
         > "${verify_dir}/expected.sha512" ||
       ! cmp -s "${verify_dir}/expected.sha512" 
"${verify_dir}/artifacts/${tar_gz}.sha512"; then
       echo "Downloaded ${tar_gz} failed SHA-512 verification. Not signing."
       exit 1
     fi
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to