amaechler opened a new pull request, #20528: URL: https://github.com/apache/druid/pull/20528
### Description Since #20148 (Apache parent 25 → 39), `check-licenses.py` only reads the direct compile dependencies from each module's `dependencies.html`. Transitive dependencies are skipped entirely, so the packaging-check job compares 68 reported dependencies instead of ~420 against `licenses.yaml`. A version bump without a matching `licenses.yaml` update now passes CI. I noticed that on #20520 which bumps `at.yawk.lz4:lz4-java` to 1.11.4 without touching `licenses.yaml`, and the check stays green. Transitive dependencies need to be covered: the binary tarball bundles them (`lib/` and `pull-deps` both resolve them), and ASF policy treats bundled transitive dependencies [the same as first-order ones](https://infra.apache.org/licensing-howto.html) for `LICENSE` and `NOTICE` ([release policy](https://www.apache.org/legal/release-policy.html#licensing-documentation)). <details> <summary>More details from Claude analysis</summary> #### Fixed the dependency report parser The newer `maven-project-info-reports-plugin` output marks sections with `<a id="...">` anchors. The parsing path added in #20148 had two problems: - It only matched `Project_Dependencies_compile`, not `Project_Transitive_Dependencies_compile`. - It only looked for the anchor in state `none`. After the first table, the `<h2>test</h2>` heading moved the state machine into the legacy `h2_end` state, where it waited for an `<h3>` that never comes. The anchor check now runs in any state, matches both compile anchors, and jumps straight to waiting for the table. This also removes the two intermediate `modern_compile_*` states. I also added a guard: the parser counts the compile tables it enters and finishes, and raises if any are left unfinished, so a future format change fails loudly instead of silently shrinking coverage. Three license names that the reports use but the alias table did not know (`Go License`, `MIT license`, `The BSD 2-Clause License`) are now mapped, so the license-name comparison runs for `re2j`, `animal-sniffer-annotations` and `stax2-api`. #### Fixed the `licenses.yaml` drift the check now finds With the parser fixed, master has 15 dependencies that are reported but not registered: - Maven resolver upgrade from #20386, only partly recorded: Apache Maven 3.6.0 → 3.9.16, new entries for `maven-model`, `maven-model-builder`, `maven-resolver-provider`, `maven-resolver-named-locks` and `maven-resolver-transport-file` (notice text taken from each jar's `META-INF/NOTICE`), and `plexus-interpolation` 1.25 → 1.29. - Confluent in `druid-protobuf-extensions`: 8.3.1 → 8.3.2 and `kafka-clients` 8.3.1-ccs → 8.3.2-ccs. - `joda-time` 2.14.4, `jackson-jq` 1.6.5, `slf4j-api` 2.0.20, `auto-value-annotations` 1.11.1. #### Verification On reports generated locally from current master with `generate-license-dependency-reports.py`: | | Before | After | | --- | --- | --- | | Reported dependencies | 68 | 423 | | Unchecked (warning only) | 588 | 237 | | Clean `licenses.yaml` | passes | passes | | `lz4-java` registered at the wrong version | passes | fails, naming `lz4-java` | The parsed rows match every compile-table row in all 31 reports, including the ones with a Classifier column. The binary LICENSE and NOTICE generators still run cleanly on the updated `licenses.yaml`. </details> #### Not in this PR - Runtime-scope dependencies (`Project_*_runtime` tables) ship in the distribution but have never been checked, also before #20148. Adding them reports 28 more missing or outdated entries (for example `log4j-slf4j2-impl`, `hibernate-validator`, `janino`, `hadoop-client-runtime`). That needs per-artifact license and notice review, so I will send it as a separate PR. - About 40 `licenses.yaml` entries are registered at an older version than the reports show (for example Guava 32.1.3-jre, the Confluent 6.2.15 entries, `kafka-clients` 6.2.12-ccs and 5.5.1-ccs under `druid-avro-extensions`). They only produce warnings. Some may ship from `hadoop-dependencies/`, so removing them needs a check against the built tarball. <hr> This PR has: - [x] been self-reviewed. - [x] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
