Anil, You can also disable speculative execution on a per-job basis. See http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setMapSpeculativeExecution(boolean) (Which is why it is called a client-sided property - it applies per-job).
If HBase strongly recommends turning it off, HBase should also, by default, turn it off for its own offered jobs? On Sat, Mar 31, 2012 at 4:02 AM, anil gupta <[email protected]> wrote: > Hi Doug, > > Yes, that's why i had set that property as false in my mapred-site.xml. > But, to my surprise i didnt know that setting that property would be > useless for Hadoop jobs unless the mapred-site.xml is in classpath. The > idea of client side property is a little confusing to me at present since > there is no proper nomenclature for client side properties at present. > Thanks for your reply. > > ~Anil > > On Fri, Mar 30, 2012 at 3:26 PM, Doug Meil > <[email protected]>wrote: > >> >> Speculative execution is on by default in Hadoop. One of the Performance >> recommendations in the Hbase RefGuide is to turn it off. >> >> >> >> >> >> On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <[email protected]> wrote: >> >> >Well that's not an HBase configuration, that's Hadoop. I'm not sure if >> >this is listed anywhere, maybe in the book. >> > >> >BTW usually HBase has a "client" somewhere in the same to indicate >> >it's client side. >> > >> >J-D >> > >> >On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[email protected]> wrote: >> >> Thanks for the quick reply, Jean. Is there any link where i can find the >> >> name of all client-side configuration for HBase? >> >> >> >> ~Anil >> >> >> >> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans >> >><[email protected]>wrote: >> >> >> >>> This is a client-side configuration so if your mapred-site.xml is >> >>> _not_ on your classpath when you start the bulk load, it's not going >> >>> to pick it up. So either have that file on your classpath, or put it >> >>> in whatever other configuration file you have. >> >>> >> >>> J-D >> >>> >> >>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[email protected]> >> >>>wrote: >> >>> > Hi All, >> >>> > >> >>> > I am using cdh3u2. I ran HBase bulk loading with property >> >>> > "mapred.reduce.tasks.speculative.execution" set to "false" in >> >>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading job >> >>>and >> >>> > after short analysis i realized that these jobs are killed because >> >>> another >> >>> > worker node completed the task, hence it means that speculative >> >>>execution >> >>> > is still on. Why the HBase Bulk loader is doing speculative execution >> >>> when >> >>> > i have set it to false in mapred-site.xml? Please let me know if i am >> >>> > missing something over here. >> >>> > >> >>> > -- >> >>> > Thanks & Regards, >> >>> > Anil Gupta >> >>> >> >> >> >> >> >> >> >> -- >> >> Thanks & Regards, >> >> Anil Gupta >> > >> >> >> > > > -- > Thanks & Regards, > Anil Gupta -- Harsh J
