Ok, Looking/poking at the classes inside streaming.jar I noticed that the partitioner class is loaded from the same class loader that loaded streaming.jar.
A quick and dirty experiment where I put my custom partitioner class in streaming.jar worked. Thing is, I'd hate to rebuild standard jars that come with hadoop. The only way to start a mapreduce job that depends on multiple jar files is using the -libjars parameter, but it is not supported with streaming.... So I ask again - any way to submit a custom partitioner class to a streaming applications? --- On Mon, 3/8/10, Erez Katz <[email protected]> wrote: From: Erez Katz <[email protected]> Subject: Re: custom partitioner with streaming To: [email protected] Date: Monday, March 8, 2010, 4:50 PM Anyone? --- On Fri, 3/5/10, Erez Katz <[email protected]> wrote: > From: Erez Katz <[email protected]> > Subject: custom partitioner with streaming > To: [email protected] > Date: Friday, March 5, 2010, 10:15 AM > Hi, > > I have a python based map reduce application. > > I would like to define my own paritioner, (just like I > would have done with pipes/java). > > How do I specify the jar file that contains my custom > partitioner, assuming it is written in java... can a > paritioner be written in python/ruby? > > Thanks, > > Erez Katz > > p.s. > I am using hadoop 0.19.1 > > > >
