add '-Dmapreduce.user.classpath.first=true' to your command and try again
On Wed, Apr 9, 2014 at 6:27 AM, Kim Chew <[email protected]> wrote: > It seems to me that in Hadoop 2.2.1, using the "libjars" option does not > search the jars located in the the local file system but HDFS. For example, > > hadoop jar target/myJar.jar Foo -libjars /home/kchew/test-libs/testJar.jar > /user/kchew/inputs/raw.vector /user/kchew/outputs hdfs://remoteNN:8020 > remoteJT:8021 > > 14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with > processName=JobTracker, sessionId= > 14/04/08 15:11:02 INFO mapreduce.JobSubmitter: Cleaning up the staging > area > file:/tmp/hadoop-kchew/mapred/staging/kchew202924688/.staging/job_local202924688_0001 > 14/04/08 15:11:02 ERROR security.UserGroupInformation: > PriviledgedActionException as:kchew (auth:SIMPLE) > cause:java.io.FileNotFoundException: File does not exist: > hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar > java.io.FileNotFoundException: File does not exist: > hdfs:/remoteNN:8020/home/kchew/test-libs/testJar.jar > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102) > at > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288) > at > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224) > at > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93) > at > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57) > at > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264) > > So under Hadoop 2.2.1, do I have to explicitly set some configurations so > when using the "libjars" option it will copy the file to hdfs from local fs? > > TIA > > Kim > -- Regards Shengjun
