amaechler opened a new pull request, #20774: URL: https://github.com/apache/druid/pull/20774
Part of #20732. This PR is stacked on #20764, which is still open. Its first three commits are #20764's, rebased onto current master, and drop out once #20764 merges; only the last commit is new here. ### Description Every bump of a bundled dependency fails PR CI until someone edits its version in `licenses.yaml` by hand. As proposed on [dev@](https://lists.apache.org/thread/7y1bkvbb6og394mph4lx2x35dktv38nk), PR CI now only warns about these versions, and the release build stays strict. Pinot handles Dependabot the same way and refreshes its LICENSE-binary at release time (apache/pinot#17818). The release-guide step that refreshes the versions before each release branch follows in a separate PR, also part of #20732. #### Warning mode in `check-licenses.py` With `--stale-versions=warn` (the default is `fail`), a reported dependency becomes a warning instead of an error when: - its groupId and artifactId are registered under other versions, all with the reported license, and - one of those registered versions is no longer in any report, i.e. the dependency was bumped. The warning lists the registered versions and the commands to refresh `licenses.yaml`. Everything else still fails: - a dependency that is not registered under any version; - a dependency whose license differs from that of a registered version, for example when a bump changes the license; - a module that starts bundling another version of a library while the registered version still ships elsewhere, since that needs a new entry rather than a version edit; - unknown licenses, `license_file_path` problems and unknown module tags, which are unchanged. If the dependency ships in the release, only registrations outside the excluded modules (`extensions-contrib/`, `extensions/druid-testcontainers`) count, so a contrib entry cannot cover a jar that ships in the release. Some warnings need more than a version edit at release time, for example when only some libraries of one entry moved. `refresh-license-versions.py` lists those for a manual edit, and the warning says so. Matching the warning exactly to what the refresh script can fix would mean moving `refresh_versions()` into a shared module; I kept that out of this PR. #### PR CI warns, release builds stay strict `distribution/pom.xml` passes `--stale-versions=${druid.distribution.licenses.staleVersions}` to the `check-licenses` execution of the `apache-release` profile, with a default of `fail`. Only `.github/scripts/packaging-check.sh` sets it to `warn`, so `mvn clean install -Papache-release,dist,rat` still fails on a stale version. It is a two-valued option rather than a flag because exec-maven-plugin does not accept a blank argument (see `druid.distribution.pulldeps.opts`). `license_checks_script.sh` no longer runs (`static-checks-maven.sh` comments it out), but it passes `--stale-versions=warn` directly so it stays consistent. `static-checks.yml` also runs `packaging-check` for release branches, so CI on a release branch only warns too; the release manager's build is still strict. `packaging-check.sh` could use `fail` when the target branch is not master. I left that out because I could not verify that `GITHUB_BASE_REF` reaches the script through `worker.yml`. Happy to add it if reviewers prefer. #### Contributor docs `dev/license.md` now says what PR CI does when a contributor leaves a version as it is. #### How it was checked - Unit tests in `test_check_licenses.py`, which the `apache-release` build runs: - a bumped version warns, including when another registered version still ships; - each of these fails: an unregistered dependency, a changed license, an unknown reported license, registered versions with mixed licenses, a registered version that still ships, and an old version registered only under an excluded module while the new one ships in the release; - one end-to-end `check_licenses()` run per mode. - Against real dependency reports from a local build, with `licenses.yaml` made stale by hand (OSHI 7.7.1 to 7.7.0, snappy-java 1.1.10.12 to 1.1.10.11): - `fail` exits 1 with 3 missing licenses, and `warn` exits 0 with 3 warnings; - with the OSHI license also changed, `warn` exits 1 on OSHI and still warns on snappy-java; - with snappy-java not registered at all, `warn` exits 1; - `refresh-license-versions.py` then updates exactly those 2 entries, and `fail` passes again. - `./mvnw -pl distribution -Papache-release exec:exec@check-licenses` on a stale version fails by default and passes with `-Ddruid.distribution.licenses.staleVersions=warn`. - Replay of #20765, the first grouped Dependabot PR, without its `licenses.yaml` edits: - `fail` reports caffeine 3.3.1, hibernate-validator 9.1.5.Final and xz 1.12. - `warn` warns on the first two and still fails on xz, whose license changed from Public Domain to 0BSD in 1.12. - #20765 keeps `license_name: Public Domain` for xz 1.12. Master passes it because its check does not know the name `0BSD`; #20764 adds that name. <hr> This PR has: - [x] been self-reviewed. - [x] added documentation for new or modified features or behaviors. - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
