And this is the log from TaskTracker when I try to use har file with -archives parameter:
2012-01-02 23:37:25,873 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager (Thread-1824): Creating openCV.har in /mnt/var/lib/hadoop/mapred/taskTracker/hadoop/distcache/-5719433807905296966_537383540_550032302/10.12.115.223/mnt/var/lib/hadoop/tmp/mapred/staging/hadoop/.staging/job_201201021856_0031/libjars/openCV.har-work-2271662679519761974 with rwxr-xr-x 2012-01-02 23:37:30,395 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager (Thread-1824): Extracting /mnt/var/lib/hadoop/mapred/taskTracker/hadoop/distcache/-5719433807905296966_537383540_550032302/10.12.115.223/mnt/var/lib/hadoop/tmp/mapred/staging/hadoop/.staging/job_201201021856_0031/libjars/openCV.har-work-2271662679519761974/openCV.har to /mnt/var/lib/hadoop/mapred/taskTracker/hadoop/distcache/-5719433807905296966_537383540_550032302/10.12.115.223/mnt/var/lib/hadoop/tmp/mapred/staging/hadoop/.staging/job_201201021856_0031/libjars/openCV.har-work-2271662679519761974 2012-01-02 23:37:30,395 WARN org.apache.hadoop.filecache.TrackerDistributedCacheManager (Thread-1824): Cache file /mnt/var/lib/hadoop/mapred/taskTracker/hadoop/distcache/-5719433807905296966_537383540_550032302/10.12.115.223/mnt/var/lib/hadoop/tmp/mapred/staging/hadoop/.staging/job_201201021856_0031/libjars/openCV.har-work-2271662679519761974/openCV.har specified as archive, but not valid extension. 2012-01-02 23:37:30,468 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager (Thread-1824): Cached hdfs://10.12.115.223:9000/mnt/var/lib/hadoop/tmp/mapred/staging/hadoop/.staging/job_201201021856_0031/libjars/openCV.har as /mnt/var/lib/hadoop/mapred/taskTracker/hadoop/distcache/-5719433807905296966_537383540_550032302/10.12.115.223/mnt/var/lib/hadoop/tmp/mapred/staging/hadoop/.staging/job_201201021856_0031/libjars/openCV.har On 3 January 2012 01:11, Samir Eljazovic <samir.eljazo...@gmail.com> wrote: > Small update. Sorry for re-send. > > On 3 January 2012 01:08, Samir Eljazovic <samir.eljazo...@gmail.com>wrote: > >> Hi, >> I need to provide a lot of 3th party libraries (both java and native) to >> my MR job and doing that using generic option parser (-libjars and -files >> arguments) is a little bit messy. I was wandering if it is possible to wrap >> all libraries into single har archive and use that when submitting the job? >> >> Just to mention that I want to avoid putting all libraries into job jar >> for two reasons: >> 1. does not work for native libs >> 2. takes time to upload jar >> >> Thanks, >> Samir >> > >
