Hello I am not talking about custom partioner(custom partitioner is involved but i want to write same pair for more number times) i want it go to two reducer. for example i have partioning attribute two dimensional <x1,x2>
singatue reduce <0,0> 0 <0,1> 1 <1,0> 2 <1,1> 3 for <1,0> it will goto reducer for <1,null> it should goto to reducer 2 and 3 for <0,null> it should goto reducer 0 and 1 On Wed, Mar 13, 2013 at 2:32 PM, Viral Bajaria <[email protected]>wrote: > Do you want the pair to go to both reducers or do you want it to go to > only one but in a random fashion ? > > AFAIK, 1st is not possible. Someone on the list can correct if I am wrong. > 2nd is possible by just implementing your own partitioner which randomizes > where each key goes (not sure what you gain by that). > > > On Wed, Mar 13, 2013 at 1:59 AM, Vikas Jadhav <[email protected]>wrote: > >> >> Hi >> I am specifying requirement again with example. >> >> >> >> I have use case where i need to shufffle same (key,value) pair to >> multiple reducers >> >> >> For Example we have pair (1,"ABC") and two reducers (reducer0 and >> reducer1) are there then >> >> by default this pair will go to reduce1 (cause (key % numOfReducer) = >> (1%2) ) >> >> >> how i should shuffle this pair to both reducer. >> >> Also I willing to change the code of hadoop framework if Necessory. >> >> Thank you >> >> On Wed, Mar 13, 2013 at 12:51 PM, feng lu <[email protected]> wrote: >> >>> Hi >>> >>> you can use Job#setNumReduceTasks(int tasks) method to set the number of >>> reducer to output. >>> >>> >>> On Wed, Mar 13, 2013 at 2:15 PM, Vikas Jadhav >>> <[email protected]>wrote: >>> >>>> Hello, >>>> >>>> As by default Hadoop framework can shuffle (key,value) pair to only one >>>> reducer >>>> >>>> I have use case where i need to shufffle same (key,value) pair to >>>> multiple reducers >>>> >>>> Also I willing to change the code of hadoop framework if Necessory. >>>> >>>> >>>> Thank you >>>> >>>> -- >>>> * >>>> * >>>> * >>>> >>>> Thanx and Regards* >>>> * Vikas Jadhav* >>>> >>> >>> >>> >>> -- >>> Don't Grow Old, Grow Up... :-) >>> >> >> >> >> -- >> * >> * >> * >> >> Thanx and Regards* >> * Vikas Jadhav* >> > > -- * * * Thanx and Regards* * Vikas Jadhav*
