This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ray.git
The following commit(s) were added to refs/heads/main by this push:
new 3273663 Add source release scripts (#40)
3273663 is described below
commit 3273663449be6737c45b48288d520edd7307abb9
Author: Andy Grove <[email protected]>
AuthorDate: Sun Nov 10 10:35:56 2024 -0700
Add source release scripts (#40)
* Add release scripts copied from datafusion-python
* remove GH_TOKEN check:
* replace datafusion-python with datafusion-ray
* update changelog generator based on latest version in datafusion
* update release instructions
* update scripts
---
dev/release/README.md | 251 ++++++++++++++++++++++++++++++++
dev/release/check-rat-report.py | 60 ++++++++
dev/release/create-tarball.sh | 133 +++++++++++++++++
dev/release/generate-changelog.py | 164 +++++++++++++++++++++
dev/release/rat_exclude_files.txt | 48 ++++++
dev/release/release-tarball.sh | 74 ++++++++++
dev/release/run-rat.sh | 43 ++++++
dev/release/verify-release-candidate.sh | 155 ++++++++++++++++++++
8 files changed, 928 insertions(+)
diff --git a/dev/release/README.md b/dev/release/README.md
new file mode 100644
index 0000000..e73858a
--- /dev/null
+++ b/dev/release/README.md
@@ -0,0 +1,251 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# DataFusion Ray Release Process
+
+Development happens on the `main` branch, and most of the time, we depend on
DataFusion using GitHub dependencies
+rather than using an official release from crates.io. This allows us to pick
up new features and bug fixes frequently
+by creating PRs to move to a later revision of the code. It also means we can
incrementally make updates that are
+required due to changes in DataFusion rather than having a large amount of
work to do when the next official release
+is available.
+
+When there is a new official release of DataFusion, we update the `main`
branch to point to that, update the version
+number, and create a new release branch, such as `branch-0.2`. Once this
branch is created, we switch the `main` branch
+back to using GitHub dependencies. The release activity (such as generating
the changelog) can then happen on the
+release branch without blocking ongoing development in the `main` branch.
+
+We can cherry-pick commits from the `main` branch into `branch-0.2` as needed
and then create new patch releases
+from that branch.
+
+## Detailed Guide
+
+### Pre-requisites
+
+Releases can currently only be created by PMC members due to the permissions
needed.
+
+You will need a GitHub Personal Access Token. Follow
+[these
instructions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
+to generate one if you do not already have one.
+
+You will need a PyPI API token. Create one at
https://test.pypi.org/manage/account/#api-tokens, setting the “Scope” to
+“Entire account”.
+
+You will also need access to the
[datafusion-ray](https://test.pypi.org/project/datafusion-ray/) project on
testpypi.
+
+### Preparing the `main` Branch
+
+Before creating a new release:
+
+- We need to ensure that the main branch does not have any GitHub dependencies
+- a PR should be created and merged to update the major version number of the
project
+- A new release branch should be created, such as `branch-0.2`
+
+### Change Log
+
+We maintain a `CHANGELOG.md` so our users know what has been changed between
releases.
+
+The changelog is generated using a Python script:
+
+```bash
+$ GITHUB_TOKEN=<TOKEN> ./dev/release/generate-changelog.py 0.1.0 HEAD 0.2.0 >
dev/changelog/0.2.0.md
+```
+
+This script creates a changelog from GitHub PRs based on the labels associated
with them as well as looking for
+titles starting with `feat:`, `fix:`, or `docs:` . The script will produce
output similar to:
+
+```
+Fetching list of commits between 24.0.0 and HEAD
+Fetching pull requests
+Categorizing pull requests
+Generating changelog content
+```
+
+### Preparing a Release Candidate
+
+### Tag the Repository
+
+```bash
+git tag 0.2.0-rc1
+git push apache 0.2.0-rc1
+```
+
+### Create a source release
+
+```bash
+./dev/release/create-tarball.sh 0.2.0 1
+```
+
+This will also create the email template to send to the mailing list.
+
+Create a draft email using this content, but do not send until after
completing the next step.
+
+### Publish Python Artifacts to testpypi
+
+This section assumes some familiarity with publishing Python packages to PyPi.
For more information, refer to \
+[this
tutorial](https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives).
+
+#### Publish Python Wheels to testpypi
+
+Pushing an `rc` tag to the release branch will cause a GitHub Workflow to run
that will build the Python wheels.
+
+Go to https://github.com/apache/datafusion-ray/actions and look for an action
named "Python Release Build"
+that has run against the pushed tag.
+
+Click on the action and scroll down to the bottom of the page titled
"Artifacts". Download `dist.zip`. It should
+contain files such as:
+
+```text
+datafusion-ray-0.2.0-cp37-abi3-macosx_10_7_x86_64.whl
+datafusion-ray-0.2.0-cp37-abi3-macosx_11_0_arm64.whl
+datafusion-ray-0.2.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
+datafusion-ray-0.2.0-cp37-abi3-win_amd64.whl
+```
+
+Upload the wheels to testpypi.
+
+```bash
+unzip dist.zip
+python3 -m pip install --upgrade setuptools twine build
+python3 -m twine upload --repository testpypi
datafusion-ray-0.2.0-cp37-abi3-*.whl
+```
+
+When prompted for username, enter `__token__`. When prompted for a password,
enter a valid GitHub Personal Access Token
+
+#### Publish Python Source Distribution to testpypi
+
+Download the source tarball created in the previous step, untar it, and run:
+
+```bash
+maturin sdist
+```
+
+This will create a file named `dist/datafusion-ray-0.2.0.tar.gz`. Upload this
to testpypi:
+
+```bash
+python3 -m twine upload --repository testpypi dist/datafusion-ray-0.2.0.tar.gz
+```
+
+### Send the Email
+
+Send the email to start the vote.
+
+## Verifying a Release
+
+Running the unit tests against a testpypi release candidate:
+
+```bash
+# clone a fresh repo
+git clone https://github.com/apache/datafusion-ray.git
+cd datafusion-ray
+
+# checkout the release commit
+git fetch --tags
+git checkout 0.2.0-rc1
+
+# create the env
+python3 -m venv venv
+source venv/bin/activate
+
+# install release candidate
+pip install --extra-index-url https://test.pypi.org/simple/
datafusion-ray==0.2.0
+
+# only dep needed to run tests is pytest
+pip install pytest
+
+# run the tests
+pytest --import-mode=importlib python/tests
+```
+
+Try running one of the examples from the top-level README, or write some
custom Python code to query some available
+data files.
+
+## Publishing a Release
+
+### Publishing Apache Source Release
+
+Once the vote passes, we can publish the release.
+
+Create the source release tarball:
+
+```bash
+./dev/release/release-tarball.sh 0.2.0 1
+```
+
+### Publishing Python Artifacts to PyPi
+
+Go to the Test PyPI page of Datafusion, and download
+[all published artifacts](https://test.pypi.org/project/datafusion-ray/#files)
under `dist-release/` directory. Then proceed
+uploading them using `twine`:
+
+```bash
+twine upload --repository pypi dist-release/*
+```
+
+### Push the Release Tag
+
+```bash
+git checkout 0.2.0-rc1
+git tag 0.2.0
+git push apache 0.2.0
+```
+
+### Add the release to Apache Reporter
+
+Add the release to https://reporter.apache.org/addrelease.html?datafusion with
a version name prefixed with `DATAFUSION-RAY`,
+for example `DATAFUSION-RAY-0.2.0`.
+
+The release information is used to generate a template for a board report (see
example from Apache Arrow
+[here](https://github.com/apache/arrow/pull/14357)).
+
+### Delete old RCs and Releases
+
+See the ASF documentation on [when to
archive](https://www.apache.org/legal/release-policy.html#when-to-archive)
+for more information.
+
+#### Deleting old release candidates from `dev` svn
+
+Release candidates should be deleted once the release is published.
+
+Get a list of DataFusion release candidates:
+
+```bash
+svn ls https://dist.apache.org/repos/dist/dev/datafusion | grep datafusion-ray
+```
+
+Delete a release candidate:
+
+```bash
+svn delete -m "delete old DataFusion RC"
https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-ray-0.1.0-rc1/
+```
+
+#### Deleting old releases from `release` svn
+
+Only the latest release should be available. Delete old releases after
publishing the new release.
+
+Get a list of DataFusion releases:
+
+```bash
+svn ls https://dist.apache.org/repos/dist/release/datafusion | grep
datafusion-ray
+```
+
+Delete a release:
+
+```bash
+svn delete -m "delete old DataFusion release"
https://dist.apache.org/repos/dist/release/datafusion/datafusion-ray-0.1.0
+```
diff --git a/dev/release/check-rat-report.py b/dev/release/check-rat-report.py
new file mode 100644
index 0000000..d3dd7c5
--- /dev/null
+++ b/dev/release/check-rat-report.py
@@ -0,0 +1,60 @@
+#!/usr/bin/python
+##############################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+##############################################################################
+import fnmatch
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+if len(sys.argv) != 3:
+ sys.stderr.write("Usage: %s exclude_globs.lst rat_report.xml\n" %
sys.argv[0])
+ sys.exit(1)
+
+exclude_globs_filename = sys.argv[1]
+xml_filename = sys.argv[2]
+
+globs = [line.strip() for line in open(exclude_globs_filename, "r")]
+
+tree = ET.parse(xml_filename)
+root = tree.getroot()
+resources = root.findall("resource")
+
+all_ok = True
+for r in resources:
+ approvals = r.findall("license-approval")
+ if not approvals or approvals[0].attrib["name"] == "true":
+ continue
+ clean_name = re.sub("^[^/]+/", "", r.attrib["name"])
+ excluded = False
+ for g in globs:
+ if fnmatch.fnmatch(clean_name, g):
+ excluded = True
+ break
+ if not excluded:
+ sys.stdout.write(
+ "NOT APPROVED: %s (%s): %s\n"
+ % (clean_name, r.attrib["name"], approvals[0].attrib["name"])
+ )
+ all_ok = False
+
+if not all_ok:
+ sys.exit(1)
+
+print("OK")
+sys.exit(0)
diff --git a/dev/release/create-tarball.sh b/dev/release/create-tarball.sh
new file mode 100755
index 0000000..778743c
--- /dev/null
+++ b/dev/release/create-tarball.sh
@@ -0,0 +1,133 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Adapted from
https://github.com/apache/arrow-rs/tree/master/dev/release/create-tarball.sh
+
+# This script creates a signed tarball in
+# dev/dist/apache-datafusion-ray-<version>-<sha>.tar.gz and uploads it to
+# the "dev" area of the dist.apache.arrow repository and prepares an
+# email for sending to the [email protected] list for a formal
+# vote.
+#
+# See release/README.md for full release instructions
+#
+# Requirements:
+#
+# 1. gpg setup for signing and have uploaded your public
+# signature to https://pgp.mit.edu/
+#
+# 2. Logged into the apache svn server with the appropriate
+# credentials
+#
+# 3. Install the requests python package
+#
+#
+# Based in part on 02-source.sh from apache/arrow
+#
+
+set -e
+
+SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SOURCE_TOP_DIR="$(cd "${SOURCE_DIR}/../../" && pwd)"
+
+if [ "$#" -ne 2 ]; then
+ echo "Usage: $0 <version> <rc>"
+ echo "ex. $0 4.1.0 2"
+ exit
+fi
+
+version=$1
+rc=$2
+tag="${version}-rc${rc}"
+
+echo "Attempting to create ${tarball} from tag ${tag}"
+release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag})
+
+release=apache-datafusion-ray-${version}
+distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc}
+tarname=${release}.tar.gz
+tarball=${distdir}/${tarname}
+url="https://dist.apache.org/repos/dist/dev/datafusion/${release}-rc${rc}";
+
+if [ -z "$release_hash" ]; then
+ echo "Cannot continue: unknown git tag: ${tag}"
+fi
+
+echo "Draft email for [email protected] mailing list"
+echo ""
+echo "---------------------------------------------------------"
+cat <<MAIL
+To: [email protected]
+Subject: [VOTE] Release DataFusion Ray ${version} RC${rc}
+Hi,
+
+I would like to propose a release of the Apache DataFusion Ray subproject,
+version ${version}.
+
+This release candidate is based on commit: ${release_hash} [1]
+The proposed release tarball and signatures are hosted at [2].
+The changelog is located at [3].
+The Python wheels are located at [4].
+
+Please download, verify checksums and signatures, run the unit tests, and vote
+on the release. The vote will be open for at least 72 hours.
+
+Only votes from PMC members are binding, but all members of the community are
+encouraged to test the release and vote with "(non-binding)".
+
+The standard verification procedure is documented at
https://github.com/apache/datafusion-ray/blob/main/dev/release/README.md#verifying-release-candidates.
+
+[ ] +1 Release this as Apache DataFusion Ray ${version}
+[ ] +0
+[ ] -1 Do not release this as Apache DataFusion Ray ${version} because...
+
+Here is my vote:
+
++1
+
+[1]: https://github.com/apache/datafusion-ray/tree/${release_hash}
+[2]: ${url}
+[3]: https://github.com/apache/datafusion-ray/blob/${release_hash}/CHANGELOG.md
+[4]: https://test.pypi.org/project/datafusion-ray/${version}/
+MAIL
+echo "---------------------------------------------------------"
+
+
+# create <tarball> containing the files in git at $release_hash
+# the files in the tarball are prefixed with {version} (e.g. 4.0.1)
+mkdir -p ${distdir}
+(cd "${SOURCE_TOP_DIR}" && git archive ${release_hash} --prefix ${release}/ |
gzip > ${tarball})
+
+echo "Running rat license checker on ${tarball}"
+${SOURCE_DIR}/run-rat.sh ${tarball}
+
+echo "Signing tarball and creating checksums"
+gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
+# create signing with relative path of tarball
+# so that they can be verified with a command such as
+# shasum --check apache-datafusion-ray-4.1.0-rc2.tar.gz.sha512
+(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256
+(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512
+
+
+echo "Uploading to datafusion dist/dev to ${url}"
+svn co --depth=empty https://dist.apache.org/repos/dist/dev/datafusion
${SOURCE_TOP_DIR}/dev/dist
+svn add ${distdir}
+svn ci -m "Apache DataFusion Ray ${version} ${rc}" ${distdir}
diff --git a/dev/release/generate-changelog.py
b/dev/release/generate-changelog.py
new file mode 100755
index 0000000..f8cf67b
--- /dev/null
+++ b/dev/release/generate-changelog.py
@@ -0,0 +1,164 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import sys
+from github import Github
+import os
+import re
+import subprocess
+
+def print_pulls(repo_name, title, pulls):
+ if len(pulls) > 0:
+ print("**{}:**".format(title))
+ print()
+ for (pull, commit) in pulls:
+ url = "https://github.com/{}/pull/{}".format(repo_name,
pull.number)
+ print("- {} [#{}]({}) ({})".format(pull.title, pull.number, url,
commit.author.login))
+ print()
+
+
+def generate_changelog(repo, repo_name, tag1, tag2, version):
+
+ # get a list of commits between two tags
+ print(f"Fetching list of commits between {tag1} and {tag2}",
file=sys.stderr)
+ comparison = repo.compare(tag1, tag2)
+
+ # get the pull requests for these commits
+ print("Fetching pull requests", file=sys.stderr)
+ unique_pulls = []
+ all_pulls = []
+ for commit in comparison.commits:
+ pulls = commit.get_pulls()
+ for pull in pulls:
+ # there can be multiple commits per PR if squash merge is not
being used and
+ # in this case we should get all the author names, but for now
just pick one
+ if pull.number not in unique_pulls:
+ unique_pulls.append(pull.number)
+ all_pulls.append((pull, commit))
+
+ # we split the pulls into categories
+ breaking = []
+ bugs = []
+ docs = []
+ enhancements = []
+ performance = []
+ other = []
+
+ # categorize the pull requests based on GitHub labels
+ print("Categorizing pull requests", file=sys.stderr)
+ for (pull, commit) in all_pulls:
+
+ # see if PR title uses Conventional Commits
+ cc_type = ''
+ cc_scope = ''
+ cc_breaking = ''
+ parts = re.findall(r'^([a-z]+)(\([a-z]+\))?(!)?:', pull.title)
+ if len(parts) == 1:
+ parts_tuple = parts[0]
+ cc_type = parts_tuple[0] # fix, feat, docs, chore
+ cc_scope = parts_tuple[1] # component within project
+ cc_breaking = parts_tuple[2] == '!'
+
+ labels = [label.name for label in pull.labels]
+ if 'api change' in labels or cc_breaking:
+ breaking.append((pull, commit))
+ elif 'bug' in labels or cc_type == 'fix':
+ bugs.append((pull, commit))
+ elif 'performance' in labels or cc_type == 'perf':
+ performance.append((pull, commit))
+ elif 'enhancement' in labels or cc_type == 'feat':
+ enhancements.append((pull, commit))
+ elif 'documentation' in labels or cc_type == 'docs' or cc_type ==
'doc':
+ docs.append((pull, commit))
+ else:
+ other.append((pull, commit))
+
+ # produce the changelog content
+ print("Generating changelog content", file=sys.stderr)
+
+ # ASF header
+ print("""<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->\n""")
+
+ print(f"# Apache DataFusion Ray {version} Changelog\n")
+
+ # get the number of commits
+ commit_count = subprocess.check_output(f"git log --pretty=oneline
{tag1}..{tag2} | wc -l", shell=True, text=True).strip()
+
+ # get number of contributors
+ contributor_count = subprocess.check_output(f"git shortlog -sn
{tag1}..{tag2} | wc -l", shell=True, text=True).strip()
+
+ print(f"This release consists of {commit_count} commits from
{contributor_count} contributors. "
+ f"See credits at the end of this changelog for more information.\n")
+
+ print_pulls(repo_name, "Breaking changes", breaking)
+ print_pulls(repo_name, "Performance related", performance)
+ print_pulls(repo_name, "Implemented enhancements", enhancements)
+ print_pulls(repo_name, "Fixed bugs", bugs)
+ print_pulls(repo_name, "Documentation updates", docs)
+ print_pulls(repo_name, "Other", other)
+
+ # show code contributions
+ credits = subprocess.check_output(f"git shortlog -sn {tag1}..{tag2}",
shell=True, text=True).rstrip()
+
+ print("## Credits\n")
+ print("Thank you to everyone who contributed to this release. Here is a
breakdown of commits (PRs merged) "
+ "per contributor.\n")
+ print("```")
+ print(credits)
+ print("```\n")
+
+ print("Thank you also to everyone who contributed in other ways such as
filing issues, reviewing "
+ "PRs, and providing feedback on this release.\n")
+
+def cli(args=None):
+ """Process command line arguments."""
+ if not args:
+ args = sys.argv[1:]
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument("tag1", help="The previous commit or tag (e.g. 0.1.0)")
+ parser.add_argument("tag2", help="The current commit or tag (e.g. HEAD)")
+ parser.add_argument("version", help="The version number to include in the
changelog")
+ args = parser.parse_args()
+
+ token = os.getenv("GITHUB_TOKEN")
+ project = "apache/datafusion-ray"
+
+ g = Github(token)
+ repo = g.get_repo(project)
+ generate_changelog(repo, project, args.tag1, args.tag2, args.version)
+
+if __name__ == "__main__":
+ cli()
\ No newline at end of file
diff --git a/dev/release/rat_exclude_files.txt
b/dev/release/rat_exclude_files.txt
new file mode 100644
index 0000000..f65ddd0
--- /dev/null
+++ b/dev/release/rat_exclude_files.txt
@@ -0,0 +1,48 @@
+*.npmrc
+*.gitignore
+*.dockerignore
+.gitmodules
+*_generated.h
+*_generated.js
+*_generated.ts
+*.csv
+*.json
+*.snap
+.github/ISSUE_TEMPLATE/*.md
+.github/pull_request_template.md
+CHANGELOG.md
+dev/release/rat_exclude_files.txt
+MANIFEST.in
+__init__.pxd
+__init__.py
+*.html
+*.sgml
+*.css
+*.png
+*.ico
+*.svg
+*.devhelp2
+*.scss
+.gitattributes
+requirements.txt
+*requirements*.txt
+**/testdata/*
+ci/*
+**/*.svg
+**/*.csv
+**/*.json
+**/*.sql
+venv/*
+parquet/*
+testing/*
+target/*
+**/target/*
+Cargo.lock
+**/Cargo.lock
+.history
+*rat.txt
+*/.git
+.github/*
+benchmarks/tpch/queries/q*.sql
+benchmarks/tpch/create_tables.sql
+.cargo/config.toml
\ No newline at end of file
diff --git a/dev/release/release-tarball.sh b/dev/release/release-tarball.sh
new file mode 100755
index 0000000..0225a37
--- /dev/null
+++ b/dev/release/release-tarball.sh
@@ -0,0 +1,74 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Adapted from
https://github.com/apache/arrow-rs/tree/master/dev/release/release-tarball.sh
+
+# This script copies a tarball from the "dev" area of the
+# dist.apache.arrow repository to the "release" area
+#
+# This script should only be run after the release has been approved
+# by the arrow PMC committee.
+#
+# See release/README.md for full release instructions
+#
+# Based in part on post-01-upload.sh from apache/arrow
+
+
+set -e
+set -u
+
+if [ "$#" -ne 2 ]; then
+ echo "Usage: $0 <version> <rc-num>"
+ echo "ex. $0 4.1.0 2"
+ exit
+fi
+
+version=$1
+rc=$2
+
+tmp_dir=tmp-apache-datafusion-ray-dist
+
+echo "Recreate temporary directory: ${tmp_dir}"
+rm -rf ${tmp_dir}
+mkdir -p ${tmp_dir}
+
+echo "Clone dev dist repository"
+svn \
+ co \
+
https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-ray-${version}-rc${rc}
\
+ ${tmp_dir}/dev
+
+echo "Clone release dist repository"
+svn co https://dist.apache.org/repos/dist/release/datafusion ${tmp_dir}/release
+
+echo "Copy ${version}-rc${rc} to release working copy"
+release_version=datafusion-ray-${version}
+mkdir -p ${tmp_dir}/release/${release_version}
+cp -r ${tmp_dir}/dev/* ${tmp_dir}/release/${release_version}/
+svn add ${tmp_dir}/release/${release_version}
+
+echo "Commit release"
+svn ci -m "Apache DataFusion Ray ${version}" ${tmp_dir}/release
+
+echo "Clean up"
+rm -rf ${tmp_dir}
+
+echo "Success! The release is available here:"
+echo "
https://dist.apache.org/repos/dist/release/datafusion/${release_version}";
diff --git a/dev/release/run-rat.sh b/dev/release/run-rat.sh
new file mode 100755
index 0000000..94fa55f
--- /dev/null
+++ b/dev/release/run-rat.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+RAT_VERSION=0.13
+
+# download apache rat
+if [ ! -f apache-rat-${RAT_VERSION}.jar ]; then
+ curl -s
https://repo1.maven.org/maven2/org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar
> apache-rat-${RAT_VERSION}.jar
+fi
+
+RAT="java -jar apache-rat-${RAT_VERSION}.jar -x "
+
+RELEASE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
+
+# generate the rat report
+$RAT $1 > rat.txt
+python $RELEASE_DIR/check-rat-report.py $RELEASE_DIR/rat_exclude_files.txt
rat.txt > filtered_rat.txt
+cat filtered_rat.txt
+UNAPPROVED=`cat filtered_rat.txt | grep "NOT APPROVED" | wc -l`
+
+if [ "0" -eq "${UNAPPROVED}" ]; then
+ echo "No unapproved licenses"
+else
+ echo "${UNAPPROVED} unapproved licences. Check rat report: rat.txt"
+ exit 1
+fi
diff --git a/dev/release/verify-release-candidate.sh
b/dev/release/verify-release-candidate.sh
new file mode 100755
index 0000000..0ef01f0
--- /dev/null
+++ b/dev/release/verify-release-candidate.sh
@@ -0,0 +1,155 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+case $# in
+ 2) VERSION="$1"
+ RC_NUMBER="$2"
+ ;;
+ *) echo "Usage: $0 X.Y.Z RC_NUMBER"
+ exit 1
+ ;;
+esac
+
+set -e
+set -x
+set -o pipefail
+
+SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
+DATAFUSION_RAY_DIR="$(dirname $(dirname ${SOURCE_DIR}))"
+DATAFUSION_RAY_DIST_URL='https://dist.apache.org/repos/dist/dev/datafusion'
+
+download_dist_file() {
+ curl \
+ --silent \
+ --show-error \
+ --fail \
+ --location \
+ --remote-name $DATAFUSION_RAY_DIST_URL/$1
+}
+
+download_rc_file() {
+ download_dist_file apache-datafusion-ray-${VERSION}-rc${RC_NUMBER}/$1
+}
+
+import_gpg_keys() {
+ download_dist_file KEYS
+ gpg --import KEYS
+}
+
+if type shasum >/dev/null 2>&1; then
+ sha256_verify="shasum -a 256 -c"
+ sha512_verify="shasum -a 512 -c"
+else
+ sha256_verify="sha256sum -c"
+ sha512_verify="sha512sum -c"
+fi
+
+fetch_archive() {
+ local dist_name=$1
+ download_rc_file ${dist_name}.tar.gz
+ download_rc_file ${dist_name}.tar.gz.asc
+ download_rc_file ${dist_name}.tar.gz.sha256
+ download_rc_file ${dist_name}.tar.gz.sha512
+ verify_dir_artifact_signatures
+}
+
+verify_dir_artifact_signatures() {
+ # verify the signature and the checksums of each artifact
+ find . -name '*.asc' | while read sigfile; do
+ artifact=${sigfile/.asc/}
+ gpg --verify $sigfile $artifact || exit 1
+
+ # go into the directory because the checksum files contain only the
+ # basename of the artifact
+ pushd $(dirname $artifact)
+ base_artifact=$(basename $artifact)
+ ${sha256_verify} $base_artifact.sha256 || exit 1
+ ${sha512_verify} $base_artifact.sha512 || exit 1
+ popd
+ done
+}
+
+setup_tempdir() {
+ cleanup() {
+ if [ "${TEST_SUCCESS}" = "yes" ]; then
+ rm -fr "${DATAFUSION_RAY_TMPDIR}"
+ else
+ echo "Failed to verify release candidate. See ${DATAFUSION_RAY_TMPDIR}
for details."
+ fi
+ }
+
+ if [ -z "${DATAFUSION_RAY_TMPDIR}" ]; then
+ # clean up automatically if DATAFUSION_RAY_TMPDIR is not defined
+ DATAFUSION_RAY_TMPDIR=$(mktemp -d -t "$1.XXXXX")
+ trap cleanup EXIT
+ else
+ # don't clean up automatically
+ mkdir -p "${DATAFUSION_RAY_TMPDIR}"
+ fi
+}
+
+test_source_distribution() {
+ # install rust toolchain in a similar fashion like test-miniconda
+ export RUSTUP_HOME=$PWD/test-rustup
+ export CARGO_HOME=$PWD/test-rustup
+
+ curl https://sh.rustup.rs -sSf | sh -s -- -y --no-modify-path
+
+ export PATH=$RUSTUP_HOME/bin:$PATH
+ source $RUSTUP_HOME/env
+
+ # build and test rust
+
+ # raises on any formatting errors
+ rustup component add rustfmt --toolchain stable
+ cargo fmt --all -- --check
+
+ python3 -m venv venv
+ source venv/bin/activate
+ python3 -m pip install -U pip
+ python3 -m pip install -r requirements-in.txt
+ maturin develop
+
+ #TODO: we should really run tests here as well
+ #python3 -m pytest
+
+ if ( find -iname 'Cargo.toml' | xargs grep SNAPSHOT ); then
+ echo "Cargo.toml version should not contain SNAPSHOT for releases"
+ exit 1
+ fi
+}
+
+TEST_SUCCESS=no
+
+setup_tempdir "datafusion-ray-${VERSION}"
+echo "Working in sandbox ${DATAFUSION_RAY_TMPDIR}"
+cd ${DATAFUSION_RAY_TMPDIR}
+
+dist_name="apache-datafusion-ray-${VERSION}"
+import_gpg_keys
+fetch_archive ${dist_name}
+tar xf ${dist_name}.tar.gz
+pushd ${dist_name}
+ test_source_distribution
+popd
+
+TEST_SUCCESS=yes
+echo 'Release candidate looks good!'
+exit 0
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
