[
https://issues.apache.org/jira/browse/HADOOP-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257768#comment-15257768
]
Steve Loughran commented on HADOOP-13060:
-----------------------------------------
Jenny: you need to have the AWS versions of your libraries in sync with what
Hadoop was built against; amazon changed the signature of a method, moving an
argument from an int to a long, which is something that ends up getting frozen
at compile time.
If you are working with Hadoop 2.7.x, then you will need aws-java-sdk 1.7.4 on
your classpath. Sorry, but we can't do anything about amazon quietly breaking
binary compatibility on their JARs.
Now, if you really want the latest aws SDK against hadoop 2.7.x, you can
actually rebuild hadoop (more specifically, tools/hadoop-aws) against the
latest amazon JAR; it's just you need to do it at compile time, after which the
library choice is frozen
> While trying to perform a Distcp command, we see the error Exception in
> thread "main" java.lang.NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13060
> URL: https://issues.apache.org/jira/browse/HADOOP-13060
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.1
> Reporter: Jenny Dong
>
> While trying to do a distcp from our native HDFS cluster to S3, we get the
> following error/stacktrace :
> We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar
> version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors
> seen in HADOOP-12420 + other authentication errors).
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
> at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
> I dug into both classes com.amazonaws.services.s3.transfer.TransferManager &
> org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem
> created a ThreadPoolExecutor (which implements AbstractExecutorService which
> implements ExecutorService). I also checked on the classpath to make sure the
> version of the jars being picked up is what I expected.
> Help would be much appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
