[
https://issues.apache.org/jira/browse/HBASE-29016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17903021#comment-17903021
]
Istvan Toth commented on HBASE-29016:
-------------------------------------
Analysis of differences in the assembly with the patch:
{noformat}
Mockito transitive dependencies:
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
byte-buddy-1.12.19.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
byte-buddy-agent-1.12.19.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
objenesis-3.3.jar
Duplicate version. this one comes from guava 27, 3.1.0 comes from caffeine:
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
checker-qual-2.5.2.jar
jaxws-ri:
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
commonj.sdo-2.1.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
gmbal-4.0.0.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
ha-api-3.1.12.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jakarta.jws-api-1.1.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jakarta.mail-api-1.6.3.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jakarta.persistence-api-2.2.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jakarta.xml.soap-api-1.4.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jakarta.xml.ws-api-2.3.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jaxb-jxc-2.3.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jaxb-runtime-2.3.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jaxb-xjc-2.3.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jaxws-eclipselink-plugin-2.3.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jaxws-rt-2.3.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jaxws-tools-2.3.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
management-api-3.2.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
mimepull-1.9.11.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
org.eclipse.persistence.asm-2.7.4.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
org.eclipse.persistence.core-2.7.4.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
org.eclipse.persistence.moxy-2.7.4.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
org.eclipse.persistence.sdo-2.7.4.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
pfl-asm-4.0.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
pfl-basic-4.0.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
pfl-basic-tools-4.0.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
pfl-dynamic-4.0.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
pfl-tf-4.0.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
pfl-tf-tools-4.0.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
policy-2.7.6.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
saaj-impl-1.5.1.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
sdo-eclipselink-plugin-2.3.2.jar
(also in org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.1:test):
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
stax-ex-1.8.1.jar
(also in org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.1:test):
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
txw2-2.3.2.jar
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
streambuffer-1.5.7.jar
hadoop-yarn-server-tests:test-jar:
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
FastInfoset-1.2.16.jar
(1.2.15 is pulled in by jaxws-ri)
hadoop-mapreduce-client-app:test-jar (via hadoop-yarn-server-common and
ehcache):
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
istack-commons-runtime-3.0.8.jar
Duplicate version. This one comes from hadoop-yarn-common, 2.17.2 is dependency
managed by HBase
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jackson-module-jaxb-annotations-2.12.7.jar
Duplicate version. This one came from
org.apache.hbase.thirdparty:hbase-shaded-jackson-jaxrs-json-provider 1.2.1
comes from hadoop-common.
We don't need it at all, activation-api is not a runtime dependency
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jakarta.activation-api-1.2.2.jar
Duplicate version. newer one comes from
org.apache.hbase.thirdparty:hbase-shaded-jackson-jaxrs-json-provider
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jakarta.xml.bind-api-2.3.2.jar
Duplicate version. This one came from came from jaxws-ri 1.3.5 comes from
org.apache.hbase.thirdparty:hbase-shaded-jersey.
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
jakarta.annotation-api-1.3.4.jar
Replaced by jakarta.activation-api-1.2.1. TBH I'm not sure what is happening
here, but that's a direct replacement, so we should be good.
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
javax.activation-api-1.2.0.jar
org.apache.hadoop:hadoop-yarn-server-nodemanager tests, via hbase-testing-util
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
javax.inject-1.jar
hbase test:
duplicate, 2.0.3 comes from hadoop-auth
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
kerb-crypto-1.0.1.jar
duplicate, 2.0.3 comes from hadoop-auth
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
kerb-util-1.0.1.jar
duplicate, 2.0.3 comes from hadoop-auth
Only in
/home/stoty/x/hbase/hbase-assembly/target/hbase-4.0.0-alpha-1-SNAPSHOT/lib:
kerby-config-1.0.1.jar
{noformat}
> Refactor assembly creation to use only DependencySets and move cached
> classpath creation to a new module
> --------------------------------------------------------------------------------------------------------
>
> Key: HBASE-29016
> URL: https://issues.apache.org/jira/browse/HBASE-29016
> Project: HBase
> Issue Type: Improvement
> Components: build
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Major
> Labels: pull-request-available
>
> There are a few problem with the current assembly creation:
> * The assembly and cached classpath creation is done from the same module,
> even though they need to include different JARs.
> * To filter out the test jars, the descriptor uses a combination of
> moduleSets and dependencySets.
> * However, the results are less than ideal, as some test-related JARs are
> still included in the assembly, while some others have multiple versions of
> the same jar added to the assembly.
> Separating the assembly and cached classpath creation lets us build the
> assembly without the extra test dependencies and duplicate versions in a much
> simpler and more robust manner.
> Relying heavily on the maven dependency mechanism solves the problem of
> duplicate versions, and not depending on the test-related modules at all
> solves the problem of lingering test dependencies.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
