Sorry, wrong question. I found it's not caused by packageJobJar. The slowness happens while putting those small files to HDFS.
On Tue, Jul 5, 2011 at 2:18 PM, Guang-Nan Cheng <[email protected]> wrote: > I'm passing the whole ruby home to Hadoop, which contains thousands of > small files. The packaging process takes few minutes, any tips to speed > this up? > > > -files ruby-1.9.2-p180 > -D > mapred.child.env=PATH=ruby-1.9.2-p180/bin:'$PATH',GEM_HOME=ruby-1.9.2-p180,LD_LIBRARY_PATH=ruby-1.9.2-p180/lib,GEM_PATH=ruby-1.9.2-p180,RUBYLIB=ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/site_ruby:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/vendor_ruby:ruby-1.9.2-p180/lib/ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/1.9.1/x86_64-linux > \ > > >
