This is an automated email from the ASF dual-hosted git repository.
weibin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-graphar.git
The following commit(s) were added to refs/heads/main by this push:
new f4da959d feat(dev): Add release and verify scripts (#507)
f4da959d is described below
commit f4da959dc1b2d7f57bdb929bb7324fe67e62b495
Author: Weibin Zeng <[email protected]>
AuthorDate: Fri Jun 7 17:42:59 2024 +0800
feat(dev): Add release and verify scripts (#507)
Reason for this PR
Add scripts for developer or release manager to easily release version or
verify a version.
What changes are included in this PR?
Add release and verify scripts
related document is updated to website, see Update the release and verify
document, and add development document incubator-graphar-website#18
Are these changes tested?
yes
Are there any user-facing changes?
no
---------
Signed-off-by: acezen <[email protected]>
---
.devcontainer/graphar-dev.Dockerfile | 6 ++
LICENSE | 16 +++-
NOTICE | 8 ++
dev/download_test_data.sh | 32 +++++++
dev/release/conda_env_cpp.txt | 22 +++++
dev/release/conda_env_scala.txt | 19 ++++
dev/release/release.py | 119 ++++++++++++++++++++++++
dev/release/setup-ubuntu.sh | 52 +++++++++++
dev/release/verify.py | 174 +++++++++++++++++++++++++++++++++++
maven-projects/spark/README.md | 2 +-
10 files changed, 446 insertions(+), 4 deletions(-)
diff --git a/.devcontainer/graphar-dev.Dockerfile
b/.devcontainer/graphar-dev.Dockerfile
index 2bdd07a2..1c910d6f 100644
--- a/.devcontainer/graphar-dev.Dockerfile
+++ b/.devcontainer/graphar-dev.Dockerfile
@@ -40,6 +40,12 @@ RUN git clone --branch v1.8.3
https://github.com/google/benchmark.git /tmp/bench
&& make install \
&& rm -rf /tmp/benchmark
+RUN git clone --branch v3.6.0 https://github.com/catchorg/Catch2.git
/tmp/catch2 --depth 1 \
+ && cd /tmp/catch2 \
+ && cmake -Bbuild -H. -DBUILD_TESTING=OFF \
+ && cmake --build build/ --target install \
+ && rm -rf /tmp/catch2
+
ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib:/usr/local/lib64
ENV JAVA_HOME=/usr/lib/jvm/default-java
diff --git a/LICENSE b/LICENSE
index d1d8cf77..b9617232 100644
--- a/LICENSE
+++ b/LICENSE
@@ -212,7 +212,7 @@ Apache-2.0 licenses
The following components are provided under the Apache-2.0 License. See
project link for details.
The text of each license is the standard Apache 2.0 license.
-* spark 3.1.1 and 3.3.4 (https://github.com/apache/spark)
+* Apache Spark 3.1.1 and 3.3.4 (https://github.com/apache/spark)
Files:
maven-projects/spark/datasourcs-32/src/main/scala/org/apache/graphar/datasources/GarCommitProtocol.scala
maven-projects/spark/datasourcs-32/src/main/scala/org/apache/graphar/datasources/GarDataSource.scala
@@ -234,9 +234,13 @@ The text of each license is the standard Apache 2.0
license.
maven-projects/spark/datasourcs-33/src/main/scala/org/apache/graphar/datasources/orc/ORCOutputWriter.scala
maven-projects/spark/datasourcs-33/src/main/scala/org/apache/graphar/datasources/orc/ORCWriteBuilder.scala
maven-projects/spark/datasourcs-33/src/main/scala/org/apache/graphar/datasources/parquet/ParquetWriteBuilder.scala
- are modified from spark.
+ are modified from Apache Spark.
+
+* Apache Arrow 12.0.0 (https://github.com/apache/arrow)
+ Files:
+ dev/release/setup-ubuntu.sh
+ are modified from Apache Arrow.
-* arrow 12.0.0 (https://github.com/apache/arrow)
* fastFFI v0.1.2 (https://github.com/alibaba/fastFFI)
Files:
maven-projects/java/src/main/java/org/apache/graphar/stdcxx/StdString.java
@@ -251,6 +255,12 @@ The text of each license is the standard Apache 2.0
license.
maven-projects/java/src/main/java/org/apache/graphar/stdcxx/StdUnorderedMap.java
are modified from GraphScope.
+* Apache OpenDAL v0.45.1 (https://github.com/apache/opendal)
+ Files:
+ dev/release/release.py
+ dev/release/verify.py
+ are modified from OpenDAL.
+
================================================================
MIT licenses
================================================================
diff --git a/NOTICE b/NOTICE
index cb5fbb1b..4dc3200b 100644
--- a/NOTICE
+++ b/NOTICE
@@ -31,3 +31,11 @@ which includes the following in its NOTICE file:
fastFFI
Copyright 1999-2021 Alibaba Group Holding Ltd.
+
+--------------------------------------------------------------------------------
+
+This product includes code from Apache OpenDAL, which includes the following in
+its NOTICE file:
+
+ Apache OpenDAL
+ Copyright 2022 and onwards The Apache Software Foundation.
diff --git a/dev/download_test_data.sh b/dev/download_test_data.sh
new file mode 100755
index 00000000..83555be3
--- /dev/null
+++ b/dev/download_test_data.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# A script to download test data for GraphAr
+
+if [ -n "${GAR_TEST_DATA}" ]; then
+ if [[ ! -d "$GAR_TEST_DATA" ]]; then
+ echo "GAR_TEST_DATA is set but the directory does not exist, cloning
the test data to $GAR_TEST_DATA"
+ git clone https://github.com/apache/incubator-graphar-testing.git
"$GAR_TEST_DATA" --depth 1 || true
+ fi
+else
+ echo "GAR_TEST_DATA is not set, cloning the test data to
/tmp/graphar-testing"
+ git clone https://github.com/apache/incubator-graphar-testing.git
/tmp/graphar-testing --depth 1 || true
+ echo "Test data has been cloned to /tmp/graphar-testing, please run"
+ echo " export GAR_TEST_DATA=/tmp/graphar-testing"
+fi
diff --git a/dev/release/conda_env_cpp.txt b/dev/release/conda_env_cpp.txt
new file mode 100644
index 00000000..c0025b04
--- /dev/null
+++ b/dev/release/conda_env_cpp.txt
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+cmake
+conda-forge::arrow-cpp=13.0.0
+make
+clangxx_linux-64
+conda-forge::catch2=3.6.0
diff --git a/dev/release/conda_env_scala.txt b/dev/release/conda_env_scala.txt
new file mode 100644
index 00000000..c63df3f9
--- /dev/null
+++ b/dev/release/conda_env_scala.txt
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+maven
+openjdk=11.0.13
\ No newline at end of file
diff --git a/dev/release/release.py b/dev/release/release.py
new file mode 100644
index 00000000..366ddeab
--- /dev/null
+++ b/dev/release/release.py
@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Derived from Apache OpenDAL v0.45.1
+# https://github.com/apache/opendal/blob/5079125/scripts/release.py
+
+import re
+import subprocess
+from pathlib import Path
+
+ROOT_DIR = Path(__file__).parent.parent.parent
+
+def get_package_version():
+ major_version = None
+ minor_version = None
+ patch_version = None
+ major_pattern =
re.compile(r'set\s*\(\s*GRAPHAR_MAJOR_VERSION\s+(\d+)\s*\)', re.IGNORECASE)
+ minor_pattern =
re.compile(r'set\s*\(\s*GRAPHAR_MINOR_VERSION\s+(\d+)\s*\)', re.IGNORECASE)
+ patch_pattern =
re.compile(r'set\s*\(\s*GRAPHAR_PATCH_VERSION\s+(\d+)\s*\)', re.IGNORECASE)
+
+ file_path = ROOT_DIR / "cpp/CMakeLists.txt"
+ with open(file_path, 'r') as file:
+ for line in file:
+ major_match = major_pattern.search(line)
+ minor_match = minor_pattern.search(line)
+ patch_match = patch_pattern.search(line)
+
+ if major_match:
+ major_version = major_match.group(1)
+ if minor_match:
+ minor_version = minor_match.group(1)
+ if patch_match:
+ patch_version = patch_match.group(1)
+
+ if major_version and minor_version and patch_version:
+ return f"{major_version}.{minor_version}.{patch_version}"
+ else:
+ return None
+
+def archive_source_package():
+ print(f"Archive source package started")
+
+ version = get_package_version()
+ assert version, "Failed to get the package version"
+ name = f"apache-graphar-{version}-incubating-src"
+
+ archive_command = [
+ "git",
+ "archive",
+ "--prefix",
+ f"apache-graphar-{version}-incubating-src/",
+ "-o",
+ f"{ROOT_DIR}/dist/{name}.tar.gz",
+ "HEAD",
+ ]
+ subprocess.run(
+ archive_command,
+ cwd=ROOT_DIR,
+ check=True,
+ )
+
+ print(f"Archive source package to dist/{name}.tar.gz")
+
+
+def generate_signature():
+ for i in Path(ROOT_DIR / "dist").glob("*.tar.gz"):
+ print(f"Generate signature for {i}")
+ subprocess.run(
+ ["gpg", "--yes", "--armor", "--output", f"{i}.asc",
"--detach-sig", str(i)],
+ cwd=ROOT_DIR / "dist",
+ check=True,
+ )
+
+ for i in Path(ROOT_DIR / "dist").glob("*.tar.gz"):
+ print(f"Check signature for {i}")
+ subprocess.run(
+ ["gpg", "--verify", f"{i}.asc", str(i)], cwd=ROOT_DIR / "dist",
check=True
+ )
+
+
+def generate_checksum():
+ for i in Path(ROOT_DIR / "dist").glob("*.tar.gz"):
+ print(f"Generate checksum for {i}")
+ subprocess.run(
+ ["sha512sum", str(i.relative_to(ROOT_DIR / "dist"))],
+ stdout=open(f"{i}.sha512", "w"),
+ cwd=ROOT_DIR / "dist",
+ check=True,
+ )
+
+ for i in Path(ROOT_DIR / "dist").glob("*.tar.gz"):
+ print(f"Check checksum for {i}")
+ subprocess.run(
+ ["sha512sum", "--check", f"{str(i.relative_to(ROOT_DIR /
'dist'))}.sha512"],
+ cwd=ROOT_DIR / "dist",
+ check=True,
+ )
+
+
+if __name__ == "__main__":
+ (ROOT_DIR / "dist").mkdir(exist_ok=True)
+ archive_source_package()
+ generate_signature()
+ generate_checksum()
diff --git a/dev/release/setup-ubuntu.sh b/dev/release/setup-ubuntu.sh
new file mode 100644
index 00000000..6e74b3fc
--- /dev/null
+++ b/dev/release/setup-ubuntu.sh
@@ -0,0 +1,52 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Derived from Apache Arrow 12.0.0
+# https://github.com/apache/arrow/blob/9736dde/dev/release/setup-ubuntu.sh
+
+# A script to install dependencies required for release
+# verification on Ubuntu.
+
+set -exu
+
+codename=$(. /etc/os-release && echo ${UBUNTU_CODENAME})
+id=$(. /etc/os-release && echo ${ID})
+
+apt-get install -y -q --no-install-recommends \
+ build-essential \
+ cmake \
+ git \
+ gnupg \
+ libcurl4-openssl-dev \
+ maven \
+ openjdk-11-jdk \
+ wget \
+ pkg-config \
+ tzdata \
+ subversion
+
+wget -c
https://apache.jfrog.io/artifactory/arrow/${id}/apache-arrow-apt-source-latest-${codename}.deb
\
+ -P /tmp/
+apt-get install -y -q /tmp/apache-arrow-apt-source-latest-${codename}.deb
+apt-get update -y -q
+apt-get install -y -q --no-install-recommends \
+ libarrow-dev \
+ libarrow-dataset-dev \
+ libarrow-acero-dev \
+ libparquet-dev
diff --git a/dev/release/verify.py b/dev/release/verify.py
new file mode 100644
index 00000000..fe9c46ce
--- /dev/null
+++ b/dev/release/verify.py
@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Derived from Apache OpenDAL v0.45.1
+# https://github.com/apache/opendal/blob//5079125/scripts/verify.py
+
+import subprocess
+import os
+from pathlib import Path
+
+BASE_DIR = Path(os.getcwd())
+
+# Define colors for output
+YELLOW = "\033[37;1m"
+GREEN = "\033[32;1m"
+ENDCOLOR = "\033[0m"
+
+
+def check_signature(pkg):
+ """Check the GPG signature of the package."""
+ try:
+ subprocess.check_call(["gpg", "--verify", f"{pkg}.asc", pkg])
+ print(f"{GREEN}> Success to verify the gpg sign for {pkg}{ENDCOLOR}")
+ except subprocess.CalledProcessError:
+ print(f"{YELLOW}> Failed to verify the gpg sign for {pkg}{ENDCOLOR}")
+
+
+def check_sha512sum(pkg):
+ """Check the sha512 checksum of the package."""
+ try:
+ subprocess.check_call(["sha512sum", "--check", f"{pkg}.sha512"])
+ print(f"{GREEN}> Success to verify the checksum for {pkg}{ENDCOLOR}")
+ except subprocess.CalledProcessError:
+ print(f"{YELLOW}> Failed to verify the checksum for {pkg}{ENDCOLOR}")
+
+
+def extract_packages():
+ for file in BASE_DIR.glob("*.tar.gz"):
+ subprocess.run(["tar", "-xzf", file], check=True)
+
+
+def check_license(dir):
+ print(f"> Start checking LICENSE file in {dir}")
+ if not (dir / "LICENSE").exists():
+ raise f"{YELLOW}> LICENSE file is not found{ENDCOLOR}"
+ print(f"{GREEN}> LICENSE file exists in {dir}{ENDCOLOR}")
+
+
+def check_notice(dir):
+ print(f"> Start checking NOTICE file in {dir}")
+ if not (dir / "NOTICE").exists():
+ raise f"{YELLOW}> NOTICE file is not found{ENDCOLOR}"
+ print(f"{GREEN}> NOTICE file exists in {dir}{ENDCOLOR}")
+
+
+def install_conda():
+ print("Start installing conda")
+ subprocess.run(["wget",
"https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh";],
check=True)
+ subprocess.run(["bash", "Miniconda3-latest-Linux-x86_64.sh", "-b"],
check=True)
+ print(f"{GREEN}Success to install conda{ENDCOLOR}")
+
+
+def maybe_setup_conda(dependencies):
+ # Optionally setup a conda environment with the given dependencies
+ if ("USE_CONDA" in os.environ) and (os.environ["USE_CONDA"] > 0):
+ print("Configuring conda environment...")
+ subprocess.run(["conda", "deactivate"], check=False,
stderr=subprocess.STDOUT)
+ create_env_command = ["conda", "create", "--name", "graphar", "--yes",
"python=3.8"]
+ subprocess.run(create_env_command, check=True,
stderr=subprocess.STDOUT)
+ install_deps_command = ["conda", "install", "--name", "graphar",
"--yes"] + dependencies
+ subprocess.run(install_deps_command, check=True,
stderr=subprocess.STDOUT)
+ subprocess.run(["conda", "activate", "graphar"], check=True,
stderr=subprocess.STDOUT, shell=True)
+
+
+def build_and_test_cpp(dir):
+ print("Start building, install and test C++ library")
+
+ maybe_setup_conda(["--file", f"{dir}/dev/release/conda_env_cpp.txt"])
+
+ cmake_command = ["cmake", ".", "-DBUILD_TESTS=ON"]
+ subprocess.run(
+ cmake_command,
+ cwd=dir / "cpp",
+ check=True,
+ stderr=subprocess.STDOUT,
+ )
+ build_and_install_command = [
+ "cmake",
+ "--build",
+ ".",
+ "--target",
+ "install",
+ ]
+ subprocess.run(
+ build_and_install_command,
+ cwd=dir / "cpp",
+ check=True,
+ stderr=subprocess.STDOUT,
+ )
+ test_command = [
+ "ctest",
+ "--output-on-failure",
+ "--timeout",
+ "300",
+ "-VV"
+ ]
+ subprocess.run(
+ test_command,
+ cwd=dir / "cpp",
+ check=True,
+ stderr=subprocess.STDOUT,
+ )
+ print(f"{GREEN}Success to build graphar c++{ENDCOLOR}")
+
+
+def build_and_test_scala(dir):
+ print("Start building, install and test Scala with Spark library")
+
+ maybe_setup_conda(["--file", f"{dir}/dev/release/conda_env_scala.txt"])
+
+ build_command_32=["mvn", "clean", "package", "-P", "datasource32"]
+ subprocess.run(
+ build_command_32,
+ cwd=dir / "maven-projects/spark",
+ check=True,
+ stderr=subprocess.STDOUT,
+ )
+ build_command_33=["mvn", "clean", "package", "-P", "datasource33"]
+ subprocess.run(
+ build_command_33,
+ cwd=dir / "maven-projects/spark",
+ check=True,
+ stderr=subprocess.STDOUT,
+ )
+
+ print(f"{GREEN}Success to build graphar scala{ENDCOLOR}")
+
+if __name__ == "__main__":
+ # Get a list of all files in the current directory
+ files = [f for f in os.listdir(".") if os.path.isfile(f)]
+
+ for pkg in files:
+ # Skip files that don't have a corresponding .asc or .sha512 file
+ if not os.path.exists(f"{pkg}.asc") or not
os.path.exists(f"{pkg}.sha512"):
+ continue
+
+ print(f"> Checking {pkg}")
+
+ # Perform the checks
+ check_signature(pkg)
+ check_sha512sum(pkg)
+
+ extract_packages()
+
+ for dir in BASE_DIR.glob("apache-graphar-*-src/"):
+ check_license(dir)
+ check_notice(dir)
+ build_and_test_cpp(dir)
+ build_and_test_scala(dir)
diff --git a/maven-projects/spark/README.md b/maven-projects/spark/README.md
index a0967ca0..a7d5ad2e 100644
--- a/maven-projects/spark/README.md
+++ b/maven-projects/spark/README.md
@@ -22,7 +22,7 @@ repository and navigated to the ``spark`` subdirectory:
$ git clone https://github.com/apache/incubator-graphar.git
$ cd incubator-graphar
$ git submodule update --init
- $ cd mavens-projects/spark
+ $ cd maven-projects/spark
```
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
