andygrove commented on code in PR #1610: URL: https://github.com/apache/datafusion-ballista/pull/1610#discussion_r3293252415
########## dev/release/README.md: ########## @@ -281,6 +282,158 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg (cd ballista-cli && cargo publish) ``` +### Publish Python Wheels to PyPI + +Only approved releases of the tarball should be published to PyPI, in order to +conform to Apache Software Foundation governance standards. The Python wheels +that get uploaded must be the same artifacts that the community voted on — they +are downloaded from the release candidate's CI run, not rebuilt. + +#### Prerequisites + +A DataFusion committer can publish the [`ballista` package on +PyPI](https://pypi.org/project/ballista/) after an official project release has +been made. One-time setup: + +- Create accounts on [pypi.org](https://pypi.org) and + [test.pypi.org](https://test.pypi.org) (separate accounts). +- Ask an existing maintainer of the `ballista` PyPI project — listed on the + project page — to add you as a maintainer. The request should be made on the + dev mailing list so it is publicly tracked. +- Generate project-scoped API tokens for both PyPI and TestPyPI. +- Configure `~/.pypirc`: + + ```ini + [distutils] + index-servers = + pypi + testpypi + + [pypi] + username = __token__ + password = pypi-... + + [testpypi] + repository = https://test.pypi.org/legacy/ + username = __token__ + password = pypi-... + ``` + +- Install `twine`: + + ```bash + pip install twine Review Comment: Added, the install step now does `pip install twine requests`. ########## dev/release/README.md: ########## @@ -281,6 +282,158 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg (cd ballista-cli && cargo publish) ``` +### Publish Python Wheels to PyPI + +Only approved releases of the tarball should be published to PyPI, in order to +conform to Apache Software Foundation governance standards. The Python wheels +that get uploaded must be the same artifacts that the community voted on — they +are downloaded from the release candidate's CI run, not rebuilt. + +#### Prerequisites + +A DataFusion committer can publish the [`ballista` package on +PyPI](https://pypi.org/project/ballista/) after an official project release has +been made. One-time setup: + +- Create accounts on [pypi.org](https://pypi.org) and + [test.pypi.org](https://test.pypi.org) (separate accounts). +- Ask an existing maintainer of the `ballista` PyPI project — listed on the + project page — to add you as a maintainer. The request should be made on the + dev mailing list so it is publicly tracked. +- Generate project-scoped API tokens for both PyPI and TestPyPI. +- Configure `~/.pypirc`: + + ```ini + [distutils] + index-servers = + pypi + testpypi + + [pypi] + username = __token__ + password = pypi-... + + [testpypi] + repository = https://test.pypi.org/legacy/ + username = __token__ + password = pypi-... + ``` + +- Install `twine`: + + ```bash + pip install twine + ``` + +#### Download the Voted-On Wheels + +Once the vote passes and the final tag has been created from the RC commit, +download the same wheels that were voted on from the RC's CI run. Retagging the +RC commit does not trigger a fresh build, so the RC artifacts remain the +canonical source. + +```bash +export GH_TOKEN=... # GitHub PAT with read access to actions +mkdir ballista-pypi-<version> && cd ballista-pypi-<version> +python ../dev/release/download-python-wheels.py <version>-rc<N> +ls *.whl *.tar.gz # confirm filenames carry the right version +``` + +The merged artifact should contain one of each of the following platform wheels +(file naming uses [PEP 425](https://peps.python.org/pep-0425/) tags): + +- `ballista-<version>-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl` +- `ballista-<version>-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl` +- `ballista-<version>-cp310-abi3-macosx_*_arm64.whl` +- `ballista-<version>-cp310-abi3-win_amd64.whl` +- `ballista-<version>.tar.gz` (sdist) + +> **Known CI caveat:** the merged artifact currently contains the macOS arm64 +> wheel twice (jobs `build-python-mac-win`'s macOS leg and `build-macos-x86_64` +> both run on `macos-latest`, which is now arm64) and **no** macOS x86_64 wheel. +> Keep one copy of the arm64 wheel and delete any duplicate before upload. +> Tracked in [#1608](https://github.com/apache/datafusion-ballista/issues/1608). Review Comment: You were right that #1608 is closed (by #1612 dropping Intel Mac), so the specific arm64-duplicate note is gone. While testing today though I hit a different problem with the same `merge-build-artifacts` job: the merged `dist` artifact was missing the manylinux aarch64 wheel and the sdist, even though the per-platform jobs all succeeded. I replaced the note with a broader caveat that calls out checking every expected file and falling back to the per-platform `dist-*` artifacts if any are missing. Also added a side note that `build-sdist` currently has no upload-artifact step at all, so the sdist may be absent regardless. ########## dev/release/README.md: ########## @@ -281,6 +282,158 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg (cd ballista-cli && cargo publish) ``` +### Publish Python Wheels to PyPI + +Only approved releases of the tarball should be published to PyPI, in order to +conform to Apache Software Foundation governance standards. The Python wheels +that get uploaded must be the same artifacts that the community voted on — they +are downloaded from the release candidate's CI run, not rebuilt. + +#### Prerequisites + +A DataFusion committer can publish the [`ballista` package on +PyPI](https://pypi.org/project/ballista/) after an official project release has +been made. One-time setup: + +- Create accounts on [pypi.org](https://pypi.org) and + [test.pypi.org](https://test.pypi.org) (separate accounts). +- Ask an existing maintainer of the `ballista` PyPI project — listed on the + project page — to add you as a maintainer. The request should be made on the + dev mailing list so it is publicly tracked. +- Generate project-scoped API tokens for both PyPI and TestPyPI. +- Configure `~/.pypirc`: + + ```ini + [distutils] + index-servers = + pypi + testpypi + + [pypi] + username = __token__ + password = pypi-... + + [testpypi] + repository = https://test.pypi.org/legacy/ + username = __token__ + password = pypi-... + ``` + +- Install `twine`: + + ```bash + pip install twine + ``` + +#### Download the Voted-On Wheels + +Once the vote passes and the final tag has been created from the RC commit, +download the same wheels that were voted on from the RC's CI run. Retagging the +RC commit does not trigger a fresh build, so the RC artifacts remain the +canonical source. + +```bash +export GH_TOKEN=... # GitHub PAT with read access to actions +mkdir ballista-pypi-<version> && cd ballista-pypi-<version> +python ../dev/release/download-python-wheels.py <version>-rc<N> +ls *.whl *.tar.gz # confirm filenames carry the right version +``` + +The merged artifact should contain one of each of the following platform wheels +(file naming uses [PEP 425](https://peps.python.org/pep-0425/) tags): + +- `ballista-<version>-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl` +- `ballista-<version>-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl` +- `ballista-<version>-cp310-abi3-macosx_*_arm64.whl` +- `ballista-<version>-cp310-abi3-win_amd64.whl` +- `ballista-<version>.tar.gz` (sdist) + +> **Known CI caveat:** the merged artifact currently contains the macOS arm64 +> wheel twice (jobs `build-python-mac-win`'s macOS leg and `build-macos-x86_64` +> both run on `macos-latest`, which is now arm64) and **no** macOS x86_64 wheel. +> Keep one copy of the arm64 wheel and delete any duplicate before upload. +> Tracked in [#1608](https://github.com/apache/datafusion-ballista/issues/1608). + +#### Validate the Artifacts + +```bash +twine check *.whl *.tar.gz +``` + +The `download-python-wheels.py` script also writes `.asc` GPG signatures and +`.sha256` / `.sha512` checksum files alongside each artifact. Those are for ASF +SVN — PyPI rejects them. Pass explicit globs to `twine` so only the wheels and +sdist are considered. + +#### TestPyPI Dry-Run + +PyPI uploads are immutable: once a version is published it cannot be replaced +or re-uploaded, only yanked. A TestPyPI dry-run takes a few minutes and catches +the common ways a release goes wrong. + +```bash +twine upload --repository testpypi *.whl *.tar.gz + +python -m venv /tmp/ballista-pypi-smoke +source /tmp/ballista-pypi-smoke/bin/activate +pip install -i https://test.pypi.org/simple/ \ + --extra-index-url https://pypi.org/simple/ \ + ballista==<version> +python -c "from ballista import BallistaSessionContext; print('ok')" +deactivate +``` + +`--extra-index-url` is required because TestPyPI does not mirror dependencies +like `pyarrow` and `datafusion`. + +#### Upload to PyPI + +```bash +twine upload *.whl *.tar.gz +``` + +If the upload fails partway through, re-run with `--skip-existing` to retry only +the files that did not get through. + +#### Verify + +Confirm the new version appears at +`https://pypi.org/project/ballista/<version>/`. Then in another fresh +virtual environment: + +```bash +python -m venv /tmp/ballista-pypi-verify +source /tmp/ballista-pypi-verify/bin/activate +pip install ballista==<version> Review Comment: Done. Introduced `BALLISTA_VERSION` and `BALLISTA_RC_NUM` at the top of the section and replaced every `<version>` in the PyPI publish section with `${BALLISTA_VERSION}`. Pre-existing `<version>` placeholders in earlier sections of this doc are left as-is so the convention there stays consistent. ########## dev/release/README.md: ########## @@ -281,6 +282,166 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg (cd ballista-cli && cargo publish) ``` +### Publish Python Wheels to PyPI + +Only approved releases of the tarball should be published to PyPI, in order to +conform to Apache Software Foundation governance standards. The Python wheels +that get uploaded must be the same artifacts that the community voted on — they +are downloaded from the release candidate's CI run, not rebuilt. + +#### Prerequisites + +A DataFusion committer can publish the [`ballista` package on +PyPI](https://pypi.org/project/ballista/) after an official project release has +been made. One-time setup: + +- Create accounts on [pypi.org](https://pypi.org) and + [test.pypi.org](https://test.pypi.org) (separate accounts). +- Ask an existing maintainer of the `ballista` PyPI project — listed on the + project page — to add you as a maintainer. The request should be made on the + dev mailing list so it is publicly tracked. +- Generate project-scoped API tokens for both PyPI and TestPyPI. +- Configure `~/.pypirc`: + + ```ini + [distutils] + index-servers = + pypi + testpypi + + [pypi] + username = __token__ + password = pypi-... + + [testpypi] + repository = https://test.pypi.org/legacy/ + username = __token__ + password = pypi-... + ``` Review Comment: Added as a step right after the `~/.pypirc` configuration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
