[
https://issues.apache.org/jira/browse/HADOOP-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13692:
-----------------------------------
Attachment: HADOOP-13692-branch-2.001.patch
I'm attaching patch 001. We can solve the problem simply by listing the
Jackson 2 dependencies in hadoop-aws/pom.xml. See below for a before and after
comparison of the dependency tree for an application with a dependency on
hadoop-aws. Before, you'll see jackson-databind 2.5.3, coming in transitively
through aws-java-sdk-core. After, you'll see jackson-databind 2.2.3, coming in
transitively as a direct dependency of hadoop-aws.
For testing, I completed a full test run against an S3 bucket in US-west-2. I
also tested some S3A operations through a throw-away client application with a
dependency on hadoop-aws.
We have a similar dependency management challenge for joda-time, but that one
is already listed in hadoop-aws/pom.xml, so we don't need to do anything about
it in this patch.
h3. Before
{code}
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @
test-hadoop-cloud-storage ---
[INFO] cnauroth:test-hadoop-cloud-storage:jar:0.0.1-SNAPSHOT
[INFO] \- org.apache.hadoop:hadoop-aws:jar:2.9.0-SNAPSHOT:compile
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.9.0-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.9.0-SNAPSHOT:compile
[INFO] | | \- jdk.tools:jdk.tools:jar:1.8:system
[INFO] | +- com.google.guava:guava:jar:11.0.2:compile
[INFO] | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] | +- commons-codec:commons-codec:jar:1.4:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- commons-net:commons-net:jar:3.1:compile
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile
[INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
[INFO] | +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] | +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] | | | +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] | | | \- javax.activation:activation:jar:1.1:compile
[INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] | +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] | | \- asm:asm:jar:3.1:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | +- log4j:log4j:jar:1.2.17:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile
[INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.10:compile
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.9.0-SNAPSHOT:compile
[INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:3.9:compile
[INFO] | | | +- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] | | | \- net.minidev:json-smart:jar:1.1.1:compile
[INFO] | | +-
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] | | | +-
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] | | | +-
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] | | \- org.apache.curator:curator-framework:jar:2.7.1:compile
[INFO] | +- com.jcraft:jsch:jar:0.1.51:compile
[INFO] | +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:2.7.1:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO] | +- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] | | \- io.netty:netty:jar:3.7.0.Final:compile
[INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | \- org.tukaani:xz:jar:1.0:compile
[INFO] +- com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile
[INFO] | +- com.amazonaws:aws-java-sdk-kms:jar:1.10.6:compile
[INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile
[INFO] | \- com.fasterxml.jackson.core:jackson-databind:jar:2.5.3:compile
[INFO] | +-
com.fasterxml.jackson.core:jackson-annotations:jar:2.5.0:compile
[INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.5.3:compile
[INFO] \- joda-time:joda-time:jar:2.9.4:compile
{code}
h3. After
{code}
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @
test-hadoop-cloud-storage ---
[INFO] cnauroth:test-hadoop-cloud-storage:jar:0.0.1-SNAPSHOT
[INFO] \- org.apache.hadoop:hadoop-aws:jar:2.9.0-SNAPSHOT:compile
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.9.0-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.9.0-SNAPSHOT:compile
[INFO] | | \- jdk.tools:jdk.tools:jar:1.8:system
[INFO] | +- com.google.guava:guava:jar:11.0.2:compile
[INFO] | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] | +- commons-codec:commons-codec:jar:1.4:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- commons-net:commons-net:jar:3.1:compile
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile
[INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
[INFO] | +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] | +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] | | | +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] | | | \- javax.activation:activation:jar:1.1:compile
[INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] | +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] | | \- asm:asm:jar:3.1:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | +- log4j:log4j:jar:1.2.17:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile
[INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.10:compile
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.9.0-SNAPSHOT:compile
[INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:3.9:compile
[INFO] | | | +- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] | | | \- net.minidev:json-smart:jar:1.1.1:compile
[INFO] | | +-
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] | | | +-
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] | | | +-
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] | | \- org.apache.curator:curator-framework:jar:2.7.1:compile
[INFO] | +- com.jcraft:jsch:jar:0.1.51:compile
[INFO] | +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:2.7.1:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
[INFO] | +- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] | | \- io.netty:netty:jar:3.7.0.Final:compile
[INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | \- org.tukaani:xz:jar:1.0:compile
[INFO] +- com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile
[INFO] | +- com.amazonaws:aws-java-sdk-kms:jar:1.10.6:compile
[INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.2.3:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.2.3:compile
[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile
[INFO] \- joda-time:joda-time:jar:2.9.4:compile
{code}
> hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent
> classpath conflicts.
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-13692
> URL: https://issues.apache.org/jira/browse/HADOOP-13692
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Minor
> Attachments: HADOOP-13692-branch-2.001.patch
>
>
> If an end user's application has a dependency on hadoop-aws and no other
> Hadoop artifacts, then it picks up a transitive dependency on Jackson 2.5.3
> jars through the AWS SDK. This can cause conflicts at deployment time,
> because Hadoop has a dependency on version 2.2.3, and the 2 versions are not
> compatible with one another. We can prevent this problem by changing
> hadoop-aws to declare explicit dependencies on the Jackson artifacts, at the
> version Hadoop wants.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
