vinishjail97 opened a new pull request, #875: URL: https://github.com/apache/incubator-xtable/pull/875
## What Every jar we publish carried its licensing information twice, and the shaded jars additionally carried each bundled dependency's own license file in whatever location that dependency happened to use. This makes the layout canonical: exactly one `META-INF/LICENSE`, one `META-INF/NOTICE`, and third party texts only under `META-INF/licenses/`. Refs #701. This is the `branch-0.4` copy so that 0.4.0-incubating RC2 ships the fixed layout; `main` counterpart is coming as a separate PR. ## Why Raised twice in the [0.4.0-incubating RC1 vote](https://lists.apache.org/thread/pjl0xgfns56bx219494oc51qnj2mll8b), by both binding voters: - Stamatis, `-1`, cited #701 as reference `[4]`, noting these issues "persist across releases". - JB, `+1` binding: *"META-INF contains both LICENSE and LICENSE-bundles (and NOTICE/NOTICE-bundled) but it seems to be redundant. Maybe worth to clean this up."* ## Root cause Two independent mechanisms: 1. `META-INF/LICENSE-bundled` and `META-INF/NOTICE-bundled` are **inputs** for the shade plugin, which writes their content out as `META-INF/LICENSE` and `META-INF/NOTICE` via `IncludeResourceTransformer`. But they also live in `src/main/resources`, so they were packaged verbatim as well. In the non-shaded jars that second copy was not merely redundant, it described dependencies that are not in the jar at all. 2. `DontIncludeResourceTransformer` is configured with the exact resource names `LICENSE`, `NOTICE` and `NOTICE.txt`, and it matches on suffix. So `META-INF/LICENSE` from a dependency was dropped, but `LICENSE.txt`, `META-INF/license/*`, `about_files/*`, `META-INF/ASM_LICENSE.txt` and `META-INF/CLI-LICENSE.txt` were not. ## Changes - `maven-jar-plugin` excludes the two `-bundled` files from the jar. They remain in `target/classes`, which is where `IncludeResourceTransformer` reads them from, so the shaded jars still get the bundled dependency listing as `META-INF/LICENSE` while the non-shaded jars keep the plain Apache-2.0 text. - Shade filters drop the non-canonical dependency license files. - New `release/scripts/validate_jar_license_layout.sh`, wired into the License Check workflow, asserts the invariant against the **built jars**. It builds the shaded modules itself when their jars are absent, and discovers those modules by scanning for `maven-shade-plugin`, so it needs no per-branch module list. The files under `src/main/resources/META-INF` are deliberately **not** moved, so `validate_shaded_license_coverage.sh` keeps working against them unchanged and this does not overlap with #857. ## Verification Built all shaded modules and inspected every jar. Before, both the plain and bundled `xtable-aws` jars listed `META-INF/LICENSE`, `LICENSE-bundled`, `NOTICE`, `NOTICE-bundled` and six `licenses/` entries. After: - non-shaded jars: `META-INF/LICENSE` is the plain Apache-2.0 text, 202 lines, zero dependency coordinates - shaded jars: `META-INF/LICENSE` carries the bundled listing, for example 56 coordinates for aws, 183 for hive-metastore, 13 for hudi-support-extensions, 8 for spark-runtime - no `-bundled` file in any jar - `xtable-hive-metastore-bundled` went from 19 non-canonical license files to zero Checked the non-shaded modules too (`xtable-api`, `xtable-core`, `xtable-hudi-support-utils`) to confirm no collateral damage. `xtable-spark-runtime` is covered as well, where the shaded jar is the main artifact rather than a classified one. Run against the actual 0.4.0-incubating-rc1 artifacts, the new script reports all 21 offending entries, so it would have caught this before the vote. `apache-rat:check`, `spotless:check` and `validate_shaded_license_coverage.sh` all pass. ## Known gap, deliberately left The non-shaded jars still carry `META-INF/licenses/`, which is redundant for a jar that bundles nothing. Closing that needs one `IncludeResourceTransformer` per license text, because the shade plugin reads the project's **built jar** as its input, so excluding a resource from the jar also removes it from the bundle. I verified both halves of that experimentally. At 22 texts today and 37 once #857 lands, that is a lot of generated XML and it would conflict directly with #857, so it is better done after #857 merges. Also worth a separate issue: the `xtable-utilities` shaded jar has no `META-INF/LICENSE` at all. It is skipped here for the same reason `validate_shaded_license_coverage.sh` skips it, namely that its deploy is skipped and `create_source_release.sh` excludes it, so it is not part of the published set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
