[
https://issues.apache.org/jira/browse/HADOOP-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13060.
-------------------------------------
Resolution: Works for Me
Fix Version/s: 2.7.3
Just noticed this is still open. Jenny, you are hitting a version mismatch
between JARs: you need to have the hadoop JAR and AWS SDK JAR in sync, which
happens if you download any of the hadoop releases.
Sorry, there's nothing else we can do. Closing as a WORKSFORME as don't have an
entry "incompatible change of external dependency"
> While trying to perform a Distcp command, we see the error Exception in
> thread "main" java.lang.NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13060
> URL: https://issues.apache.org/jira/browse/HADOOP-13060
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.1
> Reporter: Jenny Dong
> Priority: Minor
> Fix For: 2.7.3
>
>
> While trying to do a distcp from our native HDFS cluster to S3, we get the
> following error/stacktrace :
> We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar
> version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors
> seen in HADOOP-12420 + other authentication errors).
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
> at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
> I dug into both classes com.amazonaws.services.s3.transfer.TransferManager &
> org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem
> created a ThreadPoolExecutor (which implements AbstractExecutorService which
> implements ExecutorService). I also checked on the classpath to make sure the
> version of the jars being picked up is what I expected.
> Help would be much appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
