GitHub user CaroFernando added a comment to the discussion: Checks on artifact 
licensing

I agree with Ian's two proposals (a specific license for every added image, and 
much tighter license-check allow/ignore paths). I'd add a third guard, because 
the miss in #8799 wasn't really a header problem.

**Why header checks can't catch this class of issue**

Header checks such as RAT answer "does this source file carry an ASF header?". 
They don't answer "is everything shipped in this artifact accounted for in 
LICENSE/NOTICE?". #8799 was a bundled dataset missing from the docker-compose 
LICENSE, so it would have passed a header scan. Images are the same: RAT just 
treats them as binary files (see the comments in [Tomcat's 
rat-excludes](https://apache.googlesource.com/tomcat80/+/refs/tags/TOMCAT_8_0_51/res/rat/rat-excludes.txt)),
 so it can't tell us whether we have the right to ship them. The ASF 
requirement is that [LICENSE and NOTICE exactly represent the contents of each 
distribution](https://apache.org/dev/licensing-howto.html), including binary 
and convenience bundles, so the check needs to run against the packaged 
artifacts too.

**How other ASF projects approach it**

- **Flink:** [FLINK-19810](https://issues.apache.org/jira/browse/FLINK-19810) 
was opened because validating NOTICE files by hand for every release was the 
most time-consuming part of releasing. They added a CI tool that fails when a 
shaded dependency is not mentioned in the module's NOTICE, and it runs from 
their CI compile script. They explicitly kept manual review as well, but the 
tool catches the most common and severe problems early. Their [Licensing 
wiki](https://cwiki.apache.org/confluence/display/FLINK/Licensing) also 
requires a `licenses/` directory with the license text of each non-Apache 
bundled dependency.
- **Airflow:** runs RAT in CI through a [pre-commit 
check](https://apache.googlesource.com/airflow/+/refs/heads/py-client-sync/scripts/ci/prek/check_license.py)
 with a pinned RAT image and a single `.rat-excludes` file. Its packaging 
config at the 2.0.0 tag also had an auto-generated list of third-party license 
files (`licenses/LICENSE-*.txt`), which is close to the provenance record 
@sshiv012 suggested.
- **SkyWalking Eyes** 
([apache/skywalking-eyes](https://github.com/apache/skywalking-eyes)): an 
ASF-maintained GitHub Action that checks headers and also resolves and checks 
dependency licenses, driven by a `.licenserc.yaml` with explicit 
`paths-ignore`. Doris uses it in its license workflow.
- **Tomcat:** its RAT excludes file documents why each category of file is 
excluded. That is a good model for Ian's second proposal: every ignore/allow 
entry carries a reason a reviewer can challenge.
- **A reminder of the stakes:** the Incubator's [August 2019 
report](https://cwiki.apache.org/confluence/spaces/INCUBATOR/pages/119549978/August2019)
 notes that a podling release (MXNet) shipped stock photos of unknown origin 
and licensing. Also, in the ASF's new [release 
tooling](https://releases.apache.org/docs/license-checks), a custom 
`.rat-excludes` replaces the default exclusions, and the lightweight check only 
looks at an allowlist of file extensions, so exclusion lists and file-type 
lists need regular review.

**Concrete suggestion**

1. Adopt Ian's two rules. Each added image gets a named license, and each 
ignore/allow path gets a comment explaining why it is there.
2. Add an artifact-level check to release CI, not only on the source tree. For 
the source tarball, the docker-compose bundle and the Docker images, list the 
non-code third-party content (images, datasets, fonts, vendored code) and fail 
if any of it is missing from a manifest, such as the shared provenance record 
@sshiv012 described. The same manifest could generate the LICENSE/NOTICE 
sections, so the list and the files can't drift apart. This is essentially the 
Flink model.
3. When a missed item like #8799 is found, require the fix PR to add the check 
or test that would have caught it. That gives Ian's "why did we miss this" 
question a concrete answer each time.

I haven't looked closely at how the current license workflows treat non-code 
files, so apologies if some of this is already covered. Happy to help prototype 
step 2 if people think it's useful.

GitHub link: 
https://github.com/apache/texera/discussions/8808#discussioncomment-19466175

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to