bbeaudreault opened a new pull request #4304:
URL: https://github.com/apache/hbase/pull/4304
@busbey I took a quick stab at this. This PR solves two issues:
- The one I first created for making hbase-shaded-client-byo-hadoop a
dependency of hbase-shaded-mapreduce
- The followup issue I noted in a comment on HBASE-26909, that actually even
today none of the hadoop deps are coming through as dependencies for downstream
users of these artifacts
This is just a draft PR because I've only done the following testing:
- build locally and verify dependency-reduced-pom.xml has the right deps
that i expect
- spot check the resulting jars with `jar -tf` to verify that a few of the
example classes are in the correct spot:
- hbase-shaded-client should include Cell.class, FileSystem.class, but not
HRegionServer.class
- hbase-shaded-client-byo-hadoop should include Cell.class but not
FileSystem.class or HRegionServer.class
- hbase-shaded-mapreduce should include HRegionServer.class but not
Cell.class or FileSystem.class
- pull into a downstream project and verify with `mvn dependency:tree`
- also verify in that project that I can do typical hbase client things as
well as TableInputFormat base, and verify that the classes came from the right
jars.
- also verify in that project that `mvn dependency:analyze` wants me to
include both hbase-shaded-client-byo-hadoop and hbase-shaded-mapreduce
Here's an example snippet of `mvn dependency:tree` in a downstream project:
```
[INFO] +-
org.apache.hbase:hbase-shaded-mapreduce:jar:2.0-mapreduce-SNAPSHOT:compile
[INFO] | +-
org.apache.hbase:hbase-shaded-client-byo-hadoop:jar:2.0-mapreduce-SNAPSHOT:compile
[INFO] | +-
javax.annotation:javax.annotation-api:jar:1.0-empty-delegate-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.3.1:compile
[INFO] | | +- log4j:log4j:jar:1.2.17:runtime
[INFO] | | +- org.slf4j:slf4j-log4j12:jar:1.7.25:runtime
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.5.5:compile
[INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:9.8.1:compile
[INFO] | | | \-
com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] | | +-
org.apache.zookeeper:zookeeper:jar:3.4.14-fixed-SNAPSHOT:compile
[INFO] | | | \-
com.github.spotbugs:spotbugs-annotations:jar:1-empty-delegate-SNAPSHOT:compile
[INFO] | | +- org.apache.curator:curator-framework:jar:4.2.0:compile
[INFO] | | +- org.apache.kerby:kerb-simplekdc:jar:1.0.1:compile
[INFO] | | | +- org.apache.kerby:kerb-client:jar:1.0.1:compile
[INFO] | | | | +- org.apache.kerby:kerby-config:jar:1.0.1:compile
[INFO] | | | | +- org.apache.kerby:kerb-common:jar:1.0.1:compile
[INFO] | | | | | \- org.apache.kerby:kerb-crypto:jar:1.0.1:compile
[INFO] | | | | +- org.apache.kerby:kerb-util:jar:1.0.1:compile
[INFO] | | | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
[INFO] | | | \- org.apache.kerby:kerb-admin:jar:1.0.1:compile
[INFO] | | | +- org.apache.kerby:kerb-server:jar:1.0.1:compile
[INFO] | | | | \- org.apache.kerby:kerb-identity:jar:1.0.1:compile
[INFO] | | | \- org.apache.kerby:kerby-xdr:jar:1.0.1:compile
[INFO] | | \-
org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.3.1:compile
[INFO] | +-
jakarta.activation:jakarta.activation-api:jar:1.0-empty-delegate-SNAPSHOT:compile
[INFO] | | \- java.se.modules:java.activation:jar:1.2.1-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:3.3.1:compile
[INFO] | | \- com.squareup.okhttp:okhttp:jar:2.7.5:compile
[INFO] | | \- com.squareup.okio:okio:jar:1.6.0:compile
[INFO] | +- org.apache.htrace:htrace-core4:jar:4.2.0-incubating:compile
[INFO] | +-
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0-cdh5.16.2:compile
[INFO] | | +- com.google.protobuf:protobuf-java:jar:3.19.1:compile
[INFO] | | \- org.apache.avro:avro:jar:1.7.6-cdh5.16.2:compile
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.0-cdh5.16.2:compile
[INFO] | | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | | \- commons-cli:commons-cli:jar:1.2:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0-cdh5.16.2:compile
[INFO] | +- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0-cdh5.16.2:compile
[INFO] | | +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] | | +-
javax.servlet:servlet-api:jar:1.0-empty-delegate-SNAPSHOT:compile
[INFO] | | | \- jakarta.servlet:jakarta.servlet-api:jar:4.0.2:compile
[INFO] | | +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.10:compile
[INFO] | | +-
org.codehaus.jackson:jackson-mapper-asl:jar:1.8.10-cloudera.2:compile
[INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.10:compile
[INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.10:compile
[INFO] | +- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:compile
[INFO] | +-
jakarta.annotation:jakarta.annotation-api:jar:1.3.5-hubspot-SNAPSHOT:compile
[INFO] | +- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
[INFO] | +- org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:compile
[INFO] | +- org.javassist:javassist:jar:3.24.1-GA:compile
[INFO] | +- org.apache.hadoop:hadoop-distcp:jar:2.6.0-cdh5.16.2:compile
[INFO] | | \- io.netty:netty-all:jar:4.0.29.Final:compile
[INFO] | +- org.apache.hadoop:hadoop-common:jar:3.3.1:compile
[INFO] | | +-
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile
[INFO] | | +- org.apache.commons:commons-math3:jar:3.5:compile
[INFO] | | +- commons-net:commons-net:jar:3.3:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | | +-
javax.servlet:javax.servlet-api:jar:1.0-empty-delegate-SNAPSHOT:compile
[INFO] | | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
[INFO] | | +- com.sun.jersey:jersey-core:jar:1.19-hubspot-SNAPSHOT:compile
[INFO] | | | +- javax.ws.rs:jsr311-api:jar:1-empty-SNAPSHOT:compile
[INFO] | | | | \- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile
[INFO] | | | \-
javax.xml.bind:jaxb-api:jar:1.0-empty-delegate-SNAPSHOT:compile
[INFO] | | | \-
java.se.modules:java.xml.bind:jar:2.3.2-SNAPSHOT:compile
[INFO] | | | +-
com.sun.istack:istack-commons-runtime:jar:3.0.8:compile
[INFO] | | | +- org.jvnet.staxex:stax-ex:jar:1.8.1:compile
[INFO] | | | \-
com.sun.xml.fastinfoset:FastInfoset:jar:1.2.16:compile
[INFO] | | +-
com.sun.jersey:jersey-servlet:jar:1.19-hubspot-SNAPSHOT:compile
[INFO] | | | +- javax.annotation:jsr250-api:jar:1-empty-SNAPSHOT:compile
[INFO] | | | \- org.ow2.asm:asm:jar:7.2:compile
[INFO] | | +- com.sun.jersey:jersey-json:jar:1.19:compile
[INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | | | \-
com.sun.xml.bind:jaxb-impl:jar:1.0-empty-delegate-SNAPSHOT:compile
[INFO] | | +-
com.sun.jersey:jersey-server:jar:1.19-hubspot-SNAPSHOT:compile
[INFO] | | +- commons-beanutils:commons-beanutils:jar:1.9.4:compile
[INFO] | | +- org.apache.commons:commons-configuration2:jar:2.1.1:compile
[INFO] | | +- org.apache.commons:commons-lang3:jar:3.9:compile
[INFO] | | +- org.apache.commons:commons-text:jar:1.9:compile
[INFO] | | +- com.google.re2j:re2j:jar:1.1:compile
[INFO] | | +- com.google.code.gson:gson:jar:2.8.9:compile
[INFO] | | +- com.jcraft:jsch:jar:0.1.54:compile
[INFO] | | +- org.apache.curator:curator-client:jar:4.2.0:compile
[INFO] | | +- org.apache.curator:curator-recipes:jar:4.2.0:compile
[INFO] | | +- org.apache.kerby:kerb-core:jar:1.0.1:compile
[INFO] | | | \- org.apache.kerby:kerby-pkix:jar:1.0.1:compile
[INFO] | | | +- org.apache.kerby:kerby-asn1:jar:1.0.1:compile
[INFO] | | | \- org.apache.kerby:kerby-util:jar:1.0.1:compile
[INFO] | | +- org.codehaus.woodstox:stax2-api:jar:4.2:compile
[INFO] | | +- com.fasterxml.woodstox:woodstox-core:jar:5.2.0:compile
[INFO] | | \- dnsjava:dnsjava:jar:2.1.8:compile
[INFO] | +- org.apache.hadoop:hadoop-hdfs:jar:2.6.0-cdh5.16.2:compile
[INFO] | | +- commons-daemon:commons-daemon:jar:1.0.13:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- io.netty:netty:jar:3.9.4.Final:compile
[INFO] | | +- xerces:xercesImpl:jar:2.9.1:compile
[INFO] | | | \- xml-apis:xml-apis:jar:1.4.01:compile
[INFO] | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] | \- org.apache.yetus:audience-annotations:jar:0.5.0:compile
```
As noted above, I also verified this with compile checks to ensure that the
imports I'm doing are working as expected, by building a full project which
uses hbase.
I think we need to consider how to automated test this, maybe we can augment
the invariant checks to somehow. But I'm guessing the only true check is with a
sample downstream project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
