rangareddy opened a new issue, #880:
URL: https://github.com/apache/incubator-xtable/issues/880
## Summary
`xtable-hive-metastore`'s `maven-shade-plugin` `<artifactSet><includes>`
list no longer matches the module's runtime dependency tree.
`release/scripts/validate_shaded_license_coverage.sh` already asserts that they
must match exactly, and that assertion fails.
This has not been caught in CI because of a separate bug in the same script,
fixed in #857 — see "Why this was never caught" below.
## Evidence
Against the current runtime tree (`./mvnw -pl xtable-hive-metastore -am
-DskipTests dependency:tree -Dscope=runtime -DoutputType=text
-DoutputFile=target/dependency-tree-runtime.txt`):
```
FAIL xtable-hive-metastore: shade <includes> must exactly match runtime
dependencies from
xtable-hive-metastore/target/dependency-tree-runtime.txt.
```
**25 includes name dependencies that are no longer in the runtime tree:**
```
ant:ant javax.servlet:jsp-api
asm:asm javax.servlet:servlet-api
asm:asm-commons net.hydromatic:eigenbase-properties
asm:asm-tree
org.apache.geronimo.specs:geronimo-annotation_1.0_spec
com.google.protobuf:protobuf-java
org.apache.geronimo.specs:geronimo-jaspic_1.0_spec
commons-el:commons-el
org.apache.geronimo.specs:geronimo-jta_1.1_spec
commons-httpclient:commons-httpclient org.apache.hbase:hbase-annotations
io.airlift:aircompressor org.apache.slider:slider-core
io.airlift:slice org.apache.velocity:velocity
javax.activation:activation org.eclipse.jetty.aggregate:jetty-all
javax.mail:mail org.eclipse.jetty.orbit:javax.servlet
oro:oro tomcat:jasper-compiler
tomcat:jasper-runtime
```
**97 runtime dependencies are absent from the includes list**, including all
of jetty 9.x, jersey 2.x, hk2, calcite, the `hbase-*` server modules,
`hive-standalone-metastore`, `mssql-jdbc` and `sqlline`.
The shape of both lists points at one cause: the include list was written
for a **Hive 2.x / older HBase** dependency set and was not updated when the
project moved to Hive 3.1.3.
`xtable-aws` and `xtable-hudi-support-extensions` both pass this check; the
drift is isolated to this one module.
## Why this was never caught
`validate_shaded_license_coverage.sh` builds its module list with `ripgrep`:
```bash
rg -l '<artifactId>maven-shade-plugin</artifactId>' --glob 'pom.xml'
```
`ripgrep` is not installed by `.github/workflows/mvn-license-check.yml` and
is **not present on the GitHub `ubuntu-latest` runner**. The call sits inside a
process substitution, where `set -o errexit` cannot see it fail, so the module
list came back empty and the script hit:
```bash
if [[ ${#SHADE_MODULES[@]} -eq 0 ]]; then
echo "No modules with maven-shade-plugin were found."
exit 0
fi
```
Every License Check run on `main` therefore reports success without
validating anything. From the most recent run on `main`:
```
release/scripts/validate_shaded_license_coverage.sh: line 92: rg: command
not found
No modules with maven-shade-plugin were found.
```
The last ten License Check runs on `main` are all green for this reason.
**#857 fixes the fail-open** (`find` + `grep`, and an empty module list is
now a failure), which is what surfaced this drift. Running `main`'s own
unmodified validator with `rg` available reproduces the same failure,
confirming the drift is pre-existing and not introduced by #857.
## Two possible fixes — needs a decision
1. **Grow the includes list to match the tree.** This is what AGENTS.md
currently prescribes ("Includes must reflect the current `dependency:tree
-Dscope=runtime` output"). It would add all of jetty, jersey, hk2, calcite and
the hbase server modules to the shaded jar, substantially changing what the
artifact bundles and its size, and requiring ~15 additional bundled license
texts.
2. **Narrow the dependency scopes instead.** If those dependencies are
expected to be supplied by the host environment (which is plausible for a Hive
Metastore integration), they should be `provided` rather than `runtime`, and
the include list is closer to correct than the tree is.
These have materially different outcomes for the published artifact, so this
is a release/design decision rather than a tooling fix. Option 2 additionally
suggests the validator's "must exactly match" contract deserves a second look.
## Interim state
To keep #857 (bundled license texts) from being blocked on this decision,
#857 adds an explicit, documented `KNOWN_INCLUDES_DRIFT` entry that downgrades
**only** the includes-vs-tree check for **only** `xtable-hive-metastore` to a
loud `WARN` that prints the full diff and references this issue. Every other
check, including both new license-text checks, remains blocking for all modules.
## Acceptance criteria
- [ ] Decide between options 1 and 2 above (worth a `dev@` thread).
- [ ] Apply the fix to `xtable-hive-metastore/pom.xml`.
- [ ] Remove the `KNOWN_INCLUDES_DRIFT` entry from
`release/scripts/validate_shaded_license_coverage.sh` so the check is blocking
again.
- [ ] Regenerate bundled license metadata (`python3
release/scripts/generate_shaded_license_metadata.py`) and add any newly
required license texts.
- [ ] Consider whether `mvn-license-check.yml` should also run on
`branch-0.4`, given the gate has not been effective on any branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
