I think the question was about the mapper keys and their assignment to reducers. The Parititioner<K,V> API looks like:
public abstract int getPartition(KEY key, VALUE value, int numPartitions); So I assume it is entirely possible to write a partitioner that distributes the same key to multiple reducers and it does not have to be non-deterministic. It can assign the partition based on the value. Is this correct? On Wed, Jun 9, 2010 at 2:43 PM, Ted Yu <[email protected]> wrote: > I think his question was about the output key from reducer. > > On Wed, Jun 9, 2010 at 2:40 PM, James Seigel <[email protected]> wrote: > > > Oleg, > > > > Are you wanting to have them in different reducers? If so then you can > > write a Comparable object to make that happen. > > > > If you want them to be on the same reducer, then that is what hadoop will > > do. > > > > :) > > > > > > On 2010-06-09, at 3:06 PM, Ted Yu wrote: > > > > > Can you disclose more about how K3 is generated. > > > From your description below, it is possible. > > > > > > On Wed, Jun 9, 2010 at 1:17 AM, Oleg Ruchovets <[email protected]> > > wrote: > > > > > >> Hi , > > >> My hadoop job writes results of map/reduce to HBase. > > >> I have 3 reducers. > > >> > > >> Here is a sequence of input and output parameters for Mapper , > Combiner > > and > > >> Reducer > > >> *input: InputFormat<K1,V1> > > >> mapper: Mapper<K1,V1,K2,V2> > > >> combiner: Reducer<K2,V2,K2,V2> > > >> reducer: Reducer<K2,V2,K3,V3> > > >> output: RecordWriter<K3,V3> > > >> > > >> *My question: > > >> Is it possible that more than one reducer has the same output key K3. > > >> Meaning in case I have 3 reducers is it possible that > > >> reducer1 K3 -* 1* , V3 [1,2,3] > > >> reducer2 K3 - 2 , V3 [5,6,9] > > >> reducer3 K3 - *1* , V3 [10,15,22] > > >> > > >> As you can see reducer1 has K3 - 1 and reducer3 has K3 - 1. > > >> So is that case possible or every and every reducer has unique output > > key? > > >> > > >> Thanks in advance > > >> Oleg. > > >> > > > > >
