GitHub user CaroFernando added a comment to the discussion: Checks on artifact licensing
I agree with Ian's two proposals (a specific license for every added image, and much tighter license-check allow/ignore paths). I'd add a third guard, because the miss in #8799 wasn't really a header problem. **Why header checks can't catch this class of issue** Header checks such as RAT answer "does this source file carry an ASF header?". They don't answer "is everything shipped in this artifact accounted for in LICENSE/NOTICE?". #8799 was a bundled dataset missing from the docker-compose LICENSE, so it would have passed a header scan. Images are the same: RAT just treats them as binary files (see the comments in [Tomcat's rat-excludes](https://apache.googlesource.com/tomcat80/+/refs/tags/TOMCAT_8_0_51/res/rat/rat-excludes.txt)), so it can't tell us whether we have the right to ship them. The ASF requirement is that [LICENSE and NOTICE exactly represent the contents of each distribution](https://apache.org/dev/licensing-howto.html), including binary and convenience bundles, so the check needs to run against the packaged artifacts too. **How other ASF projects approach it** - **Flink:** [FLINK-19810](https://issues.apache.org/jira/browse/FLINK-19810) was opened because validating NOTICE files by hand for every release was the most time-consuming part of releasing. They added a CI tool that fails when a shaded dependency is not mentioned in the module's NOTICE, and it runs from their CI compile script. They explicitly kept manual review as well, but the tool catches the most common and severe problems early. Their [Licensing wiki](https://cwiki.apache.org/confluence/display/FLINK/Licensing) also requires a `licenses/` directory with the license text of each non-Apache bundled dependency. - **Airflow:** runs RAT in CI through a [pre-commit check](https://apache.googlesource.com/airflow/+/refs/heads/py-client-sync/scripts/ci/prek/check_license.py) with a pinned RAT image and a single `.rat-excludes` file. Its packaging config at the 2.0.0 tag also had an auto-generated list of third-party license files (`licenses/LICENSE-*.txt`), which is close to the provenance record @sshiv012 suggested. - **SkyWalking Eyes** ([apache/skywalking-eyes](https://github.com/apache/skywalking-eyes)): an ASF-maintained GitHub Action that checks headers and also resolves and checks dependency licenses, driven by a `.licenserc.yaml` with explicit `paths-ignore`. Doris uses it in its license workflow. - **Tomcat:** its RAT excludes file documents why each category of file is excluded. That is a good model for Ian's second proposal: every ignore/allow entry carries a reason a reviewer can challenge. - **A reminder of the stakes:** the Incubator's [August 2019 report](https://cwiki.apache.org/confluence/spaces/INCUBATOR/pages/119549978/August2019) notes that a podling release (MXNet) shipped stock photos of unknown origin and licensing. Also, in the ASF's new [release tooling](https://releases.apache.org/docs/license-checks), a custom `.rat-excludes` replaces the default exclusions, and the lightweight check only looks at an allowlist of file extensions, so exclusion lists and file-type lists need regular review. **Concrete suggestion** 1. Adopt Ian's two rules. Each added image gets a named license, and each ignore/allow path gets a comment explaining why it is there. 2. Add an artifact-level check to release CI, not only on the source tree. For the source tarball, the docker-compose bundle and the Docker images, list the non-code third-party content (images, datasets, fonts, vendored code) and fail if any of it is missing from a manifest, such as the shared provenance record @sshiv012 described. The same manifest could generate the LICENSE/NOTICE sections, so the list and the files can't drift apart. This is essentially the Flink model. 3. When a missed item like #8799 is found, require the fix PR to add the check or test that would have caught it. That gives Ian's "why did we miss this" question a concrete answer each time. I haven't looked closely at how the current license workflows treat non-code files, so apologies if some of this is already covered. Happy to help prototype step 2 if people think it's useful. GitHub link: https://github.com/apache/texera/discussions/8808#discussioncomment-19466175 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
