vinishjail97 opened a new pull request, #876: URL: https://github.com/apache/incubator-xtable/pull/876
## What Every jar we publish carried its licensing information twice, and the shaded jars additionally carried each bundled dependency's own license file wherever that dependency happened to put it. This makes the layout canonical: exactly one `META-INF/LICENSE`, one `META-INF/NOTICE`, and third party texts only under `META-INF/licenses/`. Closes #701. `branch-0.4` counterpart is #875, needed for 0.4.0-incubating RC2. ## Why Raised twice in the [0.4.0-incubating RC1 vote](https://lists.apache.org/thread/pjl0xgfns56bx219494oc51qnj2mll8b), by both binding voters: - Stamatis, `-1`, cited #701 as reference `[4]`, noting these issues "persist across releases". - JB, `+1` binding: *"META-INF contains both LICENSE and LICENSE-bundles (and NOTICE/NOTICE-bundled) but it seems to be redundant. Maybe worth to clean this up."* ## Root cause Two independent mechanisms: 1. `META-INF/LICENSE-bundled` and `META-INF/NOTICE-bundled` are **inputs** for the shade plugin, which writes their content out as `META-INF/LICENSE` and `META-INF/NOTICE` via `IncludeResourceTransformer`. Because they also live in `src/main/resources` they were packaged verbatim as well. In the non-shaded jars that second copy was not merely redundant, it described dependencies that are not in the jar at all. 2. `DontIncludeResourceTransformer` is configured with the exact names `LICENSE`, `NOTICE`, `NOTICE.txt` and matches on suffix, so a dependency's `META-INF/LICENSE` was dropped but `LICENSE.txt`, `LICENSE-junit.txt`, `META-INF/license/*`, `about_files/*`, `META-INF/ASM_LICENSE.txt` and `META-INF/CLI-LICENSE.txt` were not. ## Changes - `maven-jar-plugin` excludes the two `-bundled` files from the jar. They stay in `target/classes`, which is where `IncludeResourceTransformer` reads them from, so shaded jars still get the bundled listing as `META-INF/LICENSE` while non-shaded jars keep the plain Apache-2.0 text. - Shade filters drop the non-canonical dependency license files. The patterns are deliberately broad rather than an enumeration, because which ones appear depends on the resolved dependency tree, see the note below. - New `release/scripts/validate_jar_license_layout.sh`, wired into the License Check workflow, asserts the invariant against the **built jars**. It builds the shaded modules itself when their jars are absent, and finds those modules by scanning for `maven-shade-plugin` so it needs no per-branch module list. Files under `src/main/resources/META-INF` are deliberately **not** moved, so `validate_shaded_license_coverage.sh` keeps working against them unchanged and this does not overlap with #857. ## Verification Rebuilt every shaded module from clean on both this branch and `branch-0.4`, and inspected all jars: - non-shaded jars: `META-INF/LICENSE` is the plain Apache-2.0 text, 202 lines, zero dependency coordinates - shaded jars: `META-INF/LICENSE` carries the bundled listing, 56 coordinates for aws, 183 for hive-metastore, 13 for hudi-support-extensions - no `-bundled` file in any jar, and `META-INF/NOTICE` intact everywhere - `xtable-hive-metastore-bundled` went from 19 non-canonical license files to zero Also checked `xtable-api`, `xtable-core` and `xtable-hudi-support-utils` for collateral damage, since the `maven-jar-plugin` config sits in `pluginManagement` and therefore applies to every module. Run against the real 0.4.0-incubating-rc1 artifacts the new script reports all 21 offending entries, so it would have caught this before the vote. `apache-rat:check`, `spotless:check` and `validate_shaded_license_coverage.sh` pass. **Worth knowing:** the first version of the excludes passed on `branch-0.4` and failed on `main`, because the two branches resolve different junit versions and only one ships `LICENSE-junit.txt`. That is why the second commit replaces the suffix enumeration with broader patterns. Verifying on a single branch would have shipped a fix that silently did not hold on the other. ## Known gaps, deliberately left 1. Non-shaded jars still carry `META-INF/licenses/`, redundant for a jar that bundles nothing. Closing it needs one `IncludeResourceTransformer` per license text, because shade reads the project's **built jar** as input, so excluding a resource from the jar also removes it from the bundle. I verified both halves experimentally. At 22 texts today and 37 once #857 lands that is a lot of generated XML and it would conflict with #857, so it is better done afterwards. 2. On this branch `xtable-hive-metastore-bundled` also picks up five license texts from a dependency's own `META-INF/licenses/`: `antlr2-license.txt`, `asm-license.txt`, `jline2-license.txt`, `jsr223-license.txt`, `normalize-stylesheet-license.txt`. This predates the PR, they sit in the canonical location so they do not violate #701, and separating curated from inherited texts belongs with #700 / #865. `normalize-stylesheet-license.txt` in particular looks orphaned and is worth a look there. 3. The `xtable-utilities` shaded jar has no `META-INF/LICENSE` at all. It is skipped here for the same reason `validate_shaded_license_coverage.sh` skips it, its deploy is skipped and `create_source_release.sh` excludes it, so it is not published. Worth a separate issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
