[
https://issues.apache.org/jira/browse/HADOOP-13078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281611#comment-15281611
]
Steve Loughran commented on HADOOP-13078:
-----------------------------------------
OK, in 1.10.6+ there is no uber JAR, just a very large collection of cruft
which most code doesn't need.
Which means that unless someone wants hadoop-tools\lib full of many, many aws
JARs, it's probably simplest to pull them in on a case by case basis
I do think we'll need to flag up the JAR repackaging as an incompatible change,
as bits of the SDK are potentially missing. HADOOP-12269 is marked appropriately
notice the versions of amazon core SDK dependencies which got managed. commons
logging doesn't matter, but I suspect that parts of the AWS SDK are already
depending on Jackson versions which are going to break on certain code paths
{code}
[INFO] | +- (commons-logging:commons-logging:jar:1.1.3:compile - version
managed from 1.1.1; omitted for duplicate)
[INFO] | +- (org.apache.httpcomponents:httpclient:jar:4.2.5:compile -
version managed from 4.3.6; omitted for duplicate)
[INFO] | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.2.3:compile
- version managed from 2.5.3; omitted for duplicate)
{code}
For the Amazon STS temporary credentials, another JAR will have to go in. Other
features may, in future, add more of the set. There's nothing we can do there.
Full listing
{code}
[INFO] +- com.amazonaws:aws-java-sdk:jar:1.10.6:compile
[INFO] | +- com.amazonaws:aws-java-sdk-support:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-simpledb:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-simpleworkflow:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-storagegateway:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-route53:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-kms:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-importexport:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-sts:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-sqs:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-rds:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-redshift:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elasticbeanstalk:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-glacier:jar:1.10.6:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-sqs:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | | +- (com.amazonaws:aws-java-sdk-sns:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | | +- (com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile - omitted for
duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-iam:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-datapipeline:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elasticloadbalancing:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-emr:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elasticache:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-elastictranscoder:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ec2:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-dynamodb:jar:1.10.6:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile - omitted for
duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-sns:jar:1.10.6:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-sqs:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudtrail:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudwatch:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-logs:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cognitoidentity:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cognitosync:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-directconnect:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudformation:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudfront:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-kinesis:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-opsworks:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ses:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-autoscaling:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudsearch:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudwatchmetrics:jar:1.10.6:compile
[INFO] | | +- (com.amazonaws:aws-java-sdk-cloudwatch:jar:1.10.6:compile -
omitted for duplicate)
[INFO] | | +- (com.amazonaws:aws-java-sdk-dynamodb:jar:1.10.6:compile -
omitted for duplicate)
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-swf-libraries:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-simpleworkflow:jar:1.10.6:compile -
omitted for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-codedeploy:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-codepipeline:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-kms:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-config:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-lambda:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ecs:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-cloudhsm:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-ssm:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-workspaces:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-machinelearning:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-directory:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-efs:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-codecommit:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | +- com.amazonaws:aws-java-sdk-devicefarm:jar:1.10.6:compile
[INFO] | | \- (com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile - omitted
for duplicate)
[INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile
[INFO] | +- (commons-logging:commons-logging:jar:1.1.3:compile - version
managed from 1.1.1; omitted for duplicate)
[INFO] | +- (org.apache.httpcomponents:httpclient:jar:4.2.5:compile -
version managed from 4.3.6; omitted for duplicate)
[INFO] | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.2.3:compile
- version managed from 2.5.3; omitted for duplicate)
[INFO] | \- joda-time:joda-time:jar:2.8.1:compile
{code}
> switch hadoop-aws back to using the (heavy) amazon-sdk JAR
> ----------------------------------------------------------
>
> Key: HADOOP-13078
> URL: https://issues.apache.org/jira/browse/HADOOP-13078
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
>
> Hadoop 2.6-2.7 uses the full amazon-aws-sdk JAR. Hadoop 2.8+ has switched to
> the amazon-s3-sdk jar because it was lighter weight.
> I want to return to the full JAR before 2.8 switches, for
> # downstream code: if someone is already including/depending-on/upgrading the
> aws SDK, switching to the s3 sdk complicates packaging, distribution. If
> directly depended on via maven dependencies, it breaks the build
> # some of the 2.8+ patches, e.g. HADOOP-12537, have to add another part of
> the S3 SDK to handle temporary credentials. This will make life even more
> complex downstream
> # if the hadoop-aws module ever adds more stuff (e.g. a s3mper style use of
> dynamo db for directory structure storage), then again, more JARs, more
> complexity.
> Let's just change the build to return to the original JAR. Yes it is heavy,
> but it will be a consistent heaviness for all projects downstream.
> This change *must* go in to 2.8 if we don't want to start breaking things
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
