the-other-tim-brown opened a new pull request, #895: URL: https://github.com/apache/incubator-xtable/pull/895
## What Three commits, each fixing one thing, in the order they were found. | | | |---|---| | `[865]` | `META-INF/licenses/` now matches the bundled non-Apache-2.0 dependencies, both ways, and a check enforces it | | `[772]` | the `xtable-hive-metastore` shade whitelist regenerated for Hive 3.1.3 | | `[701]` | one canonical `LICENSE`/`NOTICE` per jar, one `licenses/` directory, and a check enforcing that too | Closes #865. Closes #701. ## The License Check has never run `validate_shaded_license_coverage.sh` discovered modules with `rg`, which is not installed on `ubuntu-latest`. Every run since #822 landed in June logged ``` release/scripts/validate_shaded_license_coverage.sh: line 92: rg: command not found No modules with maven-shade-plugin were found. ``` and then `exit 0`. See the [run for `ef88e02`](https://github.com/apache/incubator-xtable/actions/workflows/mvn-license-check.yml). Discovery now uses `git ls-files`, and an empty module list fails rather than passing. **This is the reason for the other two commits.** Turning the check on is what surfaced everything below, and it means any release-time conclusion drawn from a green License Check since June is unsupported. `validate_bundled_license_texts.py` from #857 *was* running throughout, which is why the texts stayed broadly right even while the dependency set did not. ## `[865]` Orphaned and missing license texts @zabetak's [follow-up on #865](https://github.com/apache/incubator-xtable/issues/865#issuecomment-5326622278) was against the released `0.4.0` jar. #857 had already removed the specific orphans he named from `main`, but left gaps in the other direction, and nothing prevented either from recurring. `validate_shaded_license_coverage.sh` gains a check that maps each shade whitelist entry to its family in `LICENSE-bundled` and requires `LICENSE-<artifactId>` for everything outside Apache 2.0 — failing both on a dependency with no text and on a text with no dependency behind it. Nine texts added to satisfy it, each taken from the artifact itself. `javax.transaction:jta` and `transaction-api` were filed under Apache 2.0 while #857 shipped correct CDDL texts for them; the `GROUP_OVERRIDES` entry was what was wrong, so that is corrected rather than the texts deleted. Their texts also carried cp1252 mojibake from the Sun originals, now normalised. ## `[772]` The hive-metastore bundle was three years of Hive behind Its `<artifactSet><includes>` still described the Hive 2.3.9 tree from before #772 raised `hive.version` to 3.1.3. **97 runtime dependencies were never shaded, and 25 listed artifacts no longer resolve** — shade skips those silently, which is why the jar shipped license texts with no classes behind them. Same symptom as #865, different cause. `<includes>` regenerated from the runtime tree, 182 → 254 entries, `${scala.binary.version}` preserved, then `LICENSE-bundled` and `NOTICE-bundled` regenerated from it. Two families had to be taught to the generator: `org.apache.hadoop`, because `hadoop-project`'s pom contains `<Xlint:-unchecked/>` and is not namespace-well-formed, and `EPL 1.0`, because junit resolves to 4.12 under EPL, not the 4.11 under CPL the stale metadata claimed. 16 texts added, 5 removed, and the three asm texts rebased from `asm:asm:3.1` onto `org.ow2.asm:9.6`. ## `[701]` Duplicate and scattered license files Every jar carried its license twice: `LICENSE-bundled` and `NOTICE-bundled` are inputs to the shade transformers, read from `target/classes`, but were packaged as resources as well. Since the transformers never read the jar, `maven-jar-plugin` can simply exclude them, and both artifacts lose the duplicate. The hive-metastore bundle also had *three* directories of license text: ours, the `META-INF/license` netty ships, and a top-level `license/` from xml-apis that neither #701 nor #885 spotted. Netty's ten texts are checked in under `META-INF/licenses` and xml-apis' W3C and SAX notices are carried in `LICENSE.xml-apis-dom-sax.txt`, so the originals are filtered out with nothing lost and one directory left. `validate_bundled_jar_license_layout.py` enforces this against the built jars. It is deliberately separate from `validate_bundled_license_texts.py`: that one checks *which* texts exist, this one checks *where* they sit. ### Running both found three wrong license families Neither check could have caught these alone — the coverage check trusts `LICENSE-bundled`, which is generated from these same overrides, so a wrong override is self-consistent and invisible to it. | artifact | was | is, per its own pom | |---|---|---| | `javax.servlet.jsp-api` | Apache 2.0 | **CDDL + GPLv2 with classpath exception** — was shipping with no text at all | | `stax-api` | CDDL + GPLv2 | Apache 2.0 | | `xml-apis` | Apache SL 1.1 | Apache 2.0 (pom silent, jar ships the Apache 2.0 text) | The last one answers the question #885 asked for a second opinion on and never got. No Apache Software License 1.1 dependency remains in any bundle. ## Credit The license texts and `validate_bundled_license_texts.py` are @rangareddy's and @vinishjail97's work from #857 and #885. Every point of #885's analysis that I initially doubted turned out to be correct — the 22-of-29 bundled measurement, junit being 4.12/EPL, and the `xml-apis` heading. The stale Hive metadata is what made all three look wrong. ## Testing ``` apache-rat:check Unapproved: 0 across all modules validate_shaded_license_coverage.sh exit 0 validate_bundled_license_texts.py exit 0 — 5 / 42 / 3 non-AL2 deps, all with texts validate_bundled_jar_license_layout.py exit 0 — all 9 jars ``` Each new check was negative-tested: removing a required text, adding an unbacked one, injecting a duplicate `LICENSE-bundled` and a second license directory each produce the expected failure and exit 1. Note that `-Dmaven.build.cache.enabled=false` is needed to see packaging changes locally — the build cache will serve a stale jar across a root-pom edit. CI's shaded-bundle step already passes it. ## Not fixed here - **#701 also asks that thin jars stop carrying `META-INF/licenses/`.** They cannot: shade builds the bundled jar from the thin jar, so excluding the directory there removes it from the bundle too. Fixing it properly needs the texts moved out of `src/main/resources` and re-added via ~50 per-file shade transformers, which is worse than the problem. - **`junit` and `hamcrest-core` are bundled at all**, arriving transitively at compile scope. Shipping test libraries in a production bundle is arguably the real defect; excluding them beats licensing them, but that is a behavioural change. #885 raised this and it was never answered. - **`branch-0.4` has all of the above**, plus `xtable-spark-runtime`, which bundles `checker-qual` under MIT with no `META-INF/licenses/` at all, and whose `LICENSE-bundled` is hand-written rather than generator output. It is an allowlist rather than a mirror of the runtime tree, so the now-live includes-vs-tree check will likely fail there on cherry-pick. - **`generate_shaded_license_metadata.py` crashes on `xtable-utilities`**, which has no `dependency-tree-runtime.txt` and no `LICENSE-bundled`, though its shade transformers reference one. Pre-existing; the validator skips the module so it is never surfaced. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MisuWSVMDBniJz95BEAHAe -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
