kevinjqliu commented on code in PR #3493: URL: https://github.com/apache/iceberg-rust/pull/3493#discussion_r4351549774
########## .github/actions/generate-python-license/action.yml: ########## @@ -0,0 +1,36 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: Generate wheel third-party licenses +description: 'Replaces bindings/python/LICENSE and NOTICE with the root files plus the licenses and notices of the Rust dependencies bundled in the wheels. The sdist keeps the root files.' +runs: + using: "composite" + steps: + - name: Install cargo-about + uses: taiki-e/install-action@4cef1412cce204788f482e778a0b9187f9626a29 # v2.87.21 + with: + tool: [email protected] + + - name: Generate LICENSE and NOTICE + shell: bash + working-directory: bindings/python + run: | + cargo about generate -c about.toml --locked --fail --manifest-path Cargo.toml about.hbs > third-party-licenses.txt + # Both are symlinks to the root files, remove them so the redirects don't write through. + rm LICENSE NOTICE + cat ../../LICENSE third-party-licenses.txt > LICENSE + { cat ../../NOTICE; echo; cat third-party-notices.txt; } > NOTICE Review Comment: final NOTICE is root NOTICE + our hand written one in `bindings/python/third-party-notices.txt` ########## .github/actions/generate-python-license/action.yml: ########## @@ -0,0 +1,36 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: Generate wheel third-party licenses +description: 'Replaces bindings/python/LICENSE and NOTICE with the root files plus the licenses and notices of the Rust dependencies bundled in the wheels. The sdist keeps the root files.' +runs: + using: "composite" + steps: + - name: Install cargo-about + uses: taiki-e/install-action@4cef1412cce204788f482e778a0b9187f9626a29 # v2.87.21 + with: + tool: [email protected] + + - name: Generate LICENSE and NOTICE + shell: bash + working-directory: bindings/python + run: | + cargo about generate -c about.toml --locked --fail --manifest-path Cargo.toml about.hbs > third-party-licenses.txt + # Both are symlinks to the root files, remove them so the redirects don't write through. + rm LICENSE NOTICE Review Comment: both [`bindings/python/LICENSE`](https://github.com/apache/iceberg-rust/blob/ca21f1587c6caaeb88dd3a9e1cc245747e1aba08/bindings/python/LICENSE) and https://github.com/apache/iceberg-rust/blob/ca21f1587c6caaeb88dd3a9e1cc245747e1aba08/bindings/python/NOTICE already exist and symlinks to the root LICENSE/NOTICE. they are needed by the sdist. During generation we need to remove them first in order to regenerate these files ########## bindings/python/third-party-notices.txt: ########## @@ -0,0 +1,30 @@ +This package bundles the following Rust dependencies that ship their own NOTICE. Review Comment: as mentioned in the pr description, this is manually maintained. i checked with my agent a couple of times. i think this is good enough for now to unblock 0.11 release ########## .github/workflows/release_python.yml: ########## @@ -184,6 +185,7 @@ jobs: uses: $/.github/actions/setup-builder with: rust-version: ${{ steps.get-msrv.outputs.msrv }} + - uses: $/.github/actions/generate-python-license Review Comment: only run when building wheels ########## .github/actions/generate-python-license/action.yml: ########## @@ -0,0 +1,36 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: Generate wheel third-party licenses +description: 'Replaces bindings/python/LICENSE and NOTICE with the root files plus the licenses and notices of the Rust dependencies bundled in the wheels. The sdist keeps the root files.' +runs: + using: "composite" + steps: + - name: Install cargo-about + uses: taiki-e/install-action@4cef1412cce204788f482e778a0b9187f9626a29 # v2.87.21 + with: + tool: [email protected] + + - name: Generate LICENSE and NOTICE + shell: bash + working-directory: bindings/python + run: | + cargo about generate -c about.toml --locked --fail --manifest-path Cargo.toml about.hbs > cargo-about-output.txt + # Both are symlinks to the root files, remove them so the redirects don't write through. + rm LICENSE NOTICE + cat ../../LICENSE cargo-about-output.txt third-party-licenses.txt > LICENSE Review Comment: final LICENSE is root LICENSE + `cargo about` output + manual entries (`third-party-licenses.txt`) ########## bindings/python/LICENSING.md: ########## @@ -0,0 +1,72 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +# LICENSE and NOTICE for pyiceberg-core + +The `pyiceberg-core` wheels are a convenience artifact. We build them and upload them to PyPI as part of every release. + +The Rust crates that `pyiceberg-core` depends on are statically linked into each wheel. So the `LICENSE` and `NOTICE` inside a wheel must cover those crates as well as iceberg-rust itself. + +The sdist only contains iceberg-rust's own source, so it ships the root `LICENSE` and `NOTICE` unchanged. + +## How it works + +`LICENSE` and `NOTICE` are generated at build time, before `maturin` builds and packages each wheel. + +The [`generate-python-license`](../../.github/actions/generate-python-license/action.yml) action regenerates the wheel's `LICENSE` and `NOTICE` in `bindings/python/`: + +- `LICENSE`: the root `LICENSE`, then the [`cargo-about`](https://github.com/EmbarkStudios/cargo-about) output, then the manual entries in `third-party-licenses.txt`. +- `NOTICE`: the root `NOTICE`, then the manual entries in `third-party-notices.txt`. + +The action is used in these workflows: + +- [`release_python.yml`](../../.github/workflows/release_python.yml): the release wheels published to PyPI. +- [`release_python_nightly.yml`](../../.github/workflows/release_python_nightly.yml): the nightly wheels published to TestPyPI. +- [`bindings_python_ci.yml`](../../.github/workflows/bindings_python_ci.yml): pull requests. It only checks that generation doesn't fail. The generated files are not published. + +Note that in the repository, `bindings/python/LICENSE` and `NOTICE` are symlinks to the root files, so builds outside the release workflows ship the plain root files. + +## Relevant files + +- `about.hbs`: the `cargo-about` output template. Keep it a plain template. +- `about.toml`: configures `cargo-about`. + - `accepted`: the licenses we bundle, matching `deny.toml`. Add new ones at the end, as the order is the preference when a crate offers a choice. + - `targets`: the wheel targets. Keep in sync with the build matrices in `release_python.yml` and `release_python_nightly.yml`. + - Clarifications, for crates whose license files `cargo-about` can't read (such as `libm`). They pin the file by checksum (`shasum -a 256 <file>`). See the [`cargo-about` docs](https://embarkstudios.github.io/cargo-about/cli/clarify.html). +- `third-party-licenses.txt`: otherwise, add the license text here. For example, `brotli-decompressor`'s MIT license, which the crate doesn't ship. It names the crate and version, so re-check it when the crate is upgraded. +- `third-party-notices.txt`: the `NOTICE` entries of dependencies that ship one (Arrow, Avro, moka). Find them from `bindings/python` (the repository's own crates appear too): + + ```shell + cargo about generate -c about.toml --locked --fail --manifest-path Cargo.toml --format json \ + | jq -r '.crates[].package.manifest_path' | xargs -n1 dirname \ + | xargs -I{} find {} -maxdepth 1 -iname 'NOTICE*' + ``` + +## Keeping entries up to date Review Comment: maybe we can automate this, but for now i am happy with the current solution in order to unblock 0.11 release ########## .github/workflows/bindings_python_ci.yml: ########## @@ -93,6 +93,7 @@ jobs: with: key: bindings-python save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} + - uses: $/.github/actions/generate-python-license Review Comment: on PR CI, check if `cargo about` can fail. this gives us a early and clear signal on any dependency changes ########## bindings/python/LICENSING.md: ########## Review Comment: trying to describe most of the changes. I can also leave out this file and just move the content to the PR description -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
