It seems to me that in Hadoop 2.2.1, using the "libjars" option does not
search the jars located in the the local file system but HDFS. For example,
hadoop jar target/myJar.jar Foo -libjars /home/kchew/test-libs/testJar.jar
/user/kchew/inputs/raw.vector /user/kchew/outputs hdfs://remoteNN:8020
remoteJT:8021
14/04/08 15:11:02 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
14/04/08 15:11:02 INFO mapreduce.JobSubmitter: Cleaning up the staging area
file:/tmp/hadoop-kchew/mapred/staging/kchew202924688/.staging/job_local202924688_0001
14/04/08 15:11:02 ERROR security.UserGroupInformation:
PriviledgedActionException as:kchew (auth:SIMPLE)
cause:java.io.FileNotFoundException: File does not exist:
hdfs://remoteNN:8020/home/kchew/test-libs/testJar.jar
java.io.FileNotFoundException: File does not exist:
hdfs:/remoteNN:8020/home/kchew/test-libs/testJar.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
So under Hadoop 2.2.1, do I have to explicitly set some configurations so
when using the "libjars" option it will copy the file to hdfs from local fs?
TIA
Kim
