Jose Luis López created HADOOP-20078:
----------------------------------------
Summary: Generate the binary distribution's license files from the
built archive and verify them as part of the release build
Key: HADOOP-20078
URL: https://issues.apache.org/jira/browse/HADOOP-20078
Project: Hadoop Common
Issue Type: Wish
Components: build
Reporter: Jose Luis López
h2. Vision
Add one automated step to the binary release process. It generates the
release's license files (LICENSE-binary and the licenses-binary/ texts)
directly from the archive being voted on, and verifies that the files
match what actually ships — so a release cannot go out with unstated,
misattributed, or outdated third-party license information.
This automates the most labor-intensive part of the current process;
NOTICE-binary stays hand-written and the rest of the procedure is
unchanged.
h2. Motivation
The binary release's license files — LICENSE-binary, NOTICE-binary and the
licenses-binary/ directory of license texts — are put together by hand,
with care. But the distribution keeps growing (628 jars in 3.5.0), and
keeping the files in line with what the tarball actually ships gets
harder every release. A mechanical check of branch-3.5 — listing
everything the tarball ships and comparing it with the files — shows
where they have drifted:
* *52 shipped jars have no license statement*: 46 are not listed at all
(jetty (12 modules), hk2/jersey (7), eclipse persistence, gson, jsr305,
commons-lang 2.6, commons-logging, and more) and 6 are listed under the
wrong Maven group.
* *10 version claims match no shipped jar*, including one component
(commons-beanutils) that ships an older version than the file claims.
* *107 of the listed components ship nothing* — about 40% of the file
(full cloud-SDK trees, kerby x8, hbase x5, junit, dom4j, ...).
* *One listed section ships nothing at all.*
* *The source-release LICENSE.txt references 9 paths that no longer exist*
(removed webapp files, a moved lz4 source path, a renamed cJSON path)
while lz4 sources, d3 and uriparser2 files that DO ship are unlisted.
* *61 components need attribution in NOTICE-binary; 32 ship without a
recorded justification.*
None of this shows up unless someone compares the files with the archive.
The check above did that in minutes. The proposal: make that comparison
part of the release process, so problems like these are found and fixed
before a release ships, not after.
h2. Proposal
Generate the binary license files from the built distribution and check
them automatically during the release build. This automates the most
labor-intensive step of the current process; everything else stays the
same. The system already exists and has been validated end-to-end against
the official 3.5.0 release (see Evidence). It works in five steps:
1. *List what ships.* A script walks the tarball and lists every jar —
including jars inside WARs — and works out each jar's Maven name from
its pom.properties, manifest, or filename (a small mapping file covers
shaded jars). Jars it cannot name are reported, never skipped.
2. *Collect the dependency list.* An opt-in Maven profile collects the
license-maven-plugin report (THIRD-PARTY.txt): license names and
versions for the full dependency set.
3. *Generate LICENSE-binary* from the two lists, applying the licensing
policy in code (which license to name for dual-licensed components,
how multi-license families are grouped). Non-jar content — web UI
files, npm packages, native sources — comes from a small,
version-controlled list.
4. *Generate the licenses-binary/ text directory* from that file and a
versioned text store, so the shipped texts always match the license
file.
5. *Verify during the release build.* The build fails unless: every
shipped jar is covered by the license file; everything listed actually
ships (or is inside a shaded jar, or has a recorded reason); nothing
excluded ships; the shipped texts match the license file; and the
committed file equals the freshly generated one — so stale entries and
hand edits are caught automatically. A failure message names the exact
entry to fix.
NOTICE-binary stays hand-written (it is prose), but it is checked too: a
component that needs attribution and has neither coverage nor a recorded
reason fails the build.
h2. Scope
* *Release-time only.* One step in create-release / HowToRelease, plus an
optional CI job. Nothing changes for developers: no new local steps,
and the checks run offline against the already-built tarball and
dependency report.
* *Source-release files unchanged for now* (LICENSE.txt, NOTICE.txt,
licenses/); extending the checks to them is a natural follow-up.
* *Later, each in its own JIRA:* the same checks on every push and pull
request, a precommit-test integration, and any build-infrastructure
work (image caching, download retries).
h2. Evidence: validated against the official 3.5.0 release
The system was adapted to branch-3.5 and run against the official 3.5.0
release artifacts. Both license files — generated and manual — describe
the same 215 third-party components (628 vs 630 jars is just the branch's
normal dependency updates since 3.5.0). Comparing both against what
ships:
| What ships vs what the license file states | Manual |
Generated |
| Shipped jars not listed | 46 | 0 |
| Shipped jars listed under the wrong Maven group | 6 | 0 |
| Version claims matching no shipped jar | 10 | 0 |
| Listed components that ship nothing | 107 | 0* |
| Generated file vs the archive it ships with | — |
exact |
*3 components are listed although they ship only inside shaded jars —
by design.
The last row is the key property: the generated file comes from the
archive it ships with, so it cannot disagree with it — and the release
build enforces that on every run. A reproducible comparison package
(sources separated by origin, the deviation analyses, an executive
summary) is available for review:
https://github.com/joseluisll/hadoop/tree/feat/LICENSES-3.5/analysis-3.5
(executive summary: analysis-3.5/README.md)
h2. Operational cost
Adapting the system to branch-3.5 took one recorded pass: 8 license texts
removed (components that no longer ship) and 8 exception entries
reconciled — each backed by archive evidence. The checks also flag
configuration that no longer matches anything, so stale configuration
gets removed instead of piling up. After that, the only ongoing work is
justifying new entries when dependencies change; the license files
themselves are generated.
h2. Implementation sketch
* The scripts under dev-support/licenses/ (archive listing, file
generation, the checks, plus a test suite) — already implemented and
validated on the feature branch.
* An opt-in Maven profile for the dependency list; no impact on the
default build.
* A create-release step that runs the listing, generation and checks on
the tarballs being voted on, and publishes the dependency report with
the release.
h2. Compatibility
* No runtime or API change. The only visible difference is in the shipped
license files: they match the archive — every shipped jar stated under
its real name and version, nothing listed that does not ship.
* Release managers run one additional automated step; when it finds
something, the message names the exact entry to fix.
* Hand edits to the generated files get caught and reverted; all changes
go through the small, reviewed configuration files.
h2. Alternatives considered
* *Audit-only*: audits find problems after the fact but leave the fixing
to the same manual work; adding generation fixes them automatically.
* *The license-maven-plugin report alone*: it describes the Maven
dependency graph, not the archive, so it does not fully show what the
tarball ships. It stays a key input — the proposal combines it with the
archive listing.
* *Third-party tooling*: no off-the-shelf tool generates the license file
from the archive's actual contents and checks it during the build.
h2. References
* Comparison package: analysis-3.5/ on
https://github.com/joseluisll/hadoop/tree/feat/LICENSES-3.5
* Tooling branch: https://github.com/joseluisll/hadoop/tree/feat/LICENSES
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
