Thanks. I believe this will solve the problem. Lior
On Mon, Mar 28, 2011 at 6:38 PM, Harsh J <[email protected]> wrote: > Are you looking to disable the speculative execution feature of MR? > > You can do so for your job by passing 'false' to > jobConf.setSpeculativeExecution(..), or fine-control map and reduce > speculatives individually using jobConf.setMapSpeculativeExecution(..) > and jobConf.setReduceSpeculativeExecution(..). > > On Mon, Mar 28, 2011 at 8:32 PM, Lior Schachter <[email protected]> > wrote: > > Hi, > > We have a map/reduce task that insert to hbase (in the reduce phase). > > Our problem is that some reduce jobs finish early and then the framework > use > > them in-order to "backup" running reducers (As it should do). > > This causes to multiple redundant writes to hbase (which is costly). > These > > reducers are eventually killed since the original reducers finish their > job. > > > > Can we configure our job to start the "backup" reducer only in case a > > failure actually occurs (we don't want to compromise robustness) ? > > > > Thanks, > > Lior > > > > > > > > -- > Harsh J > http://harshj.com >
