Hash I think you have a good point here - It is a good practice that the utilities given from HBase also adapt HBase own recommendations. For example the RowCounter (org.apache.hadoop.hbase.mapreduce.RowCounter.java) utility is neither setting the speculative execution to 'false', nor the scan cache to a value different to zero, which is not according to the hbase book/ref guide ( http://hbase.apache.org/book.html#mapreduce )
Mikael.S On Sat, Mar 31, 2012 at 7:54 AM, Harsh J <[email protected]> wrote: > Anil, > > You can also disable speculative execution on a per-job basis. See > > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setMapSpeculativeExecution(boolean) > (Which is why it is called a client-sided property - it applies > per-job). > > If HBase strongly recommends turning it off, HBase should also, by > default, turn it off for its own offered jobs? > > On Sat, Mar 31, 2012 at 4:02 AM, anil gupta <[email protected]> wrote: > > Hi Doug, > > > > Yes, that's why i had set that property as false in my mapred-site.xml. > > But, to my surprise i didnt know that setting that property would be > > useless for Hadoop jobs unless the mapred-site.xml is in classpath. The > > idea of client side property is a little confusing to me at present since > > there is no proper nomenclature for client side properties at present. > > Thanks for your reply. > > > > ~Anil > > > > On Fri, Mar 30, 2012 at 3:26 PM, Doug Meil < > [email protected]>wrote: > > > >> > >> Speculative execution is on by default in Hadoop. One of the > Performance > >> recommendations in the Hbase RefGuide is to turn it off. > >> > >> > >> > >> > >> > >> On 3/30/12 6:12 PM, "Jean-Daniel Cryans" <[email protected]> wrote: > >> > >> >Well that's not an HBase configuration, that's Hadoop. I'm not sure if > >> >this is listed anywhere, maybe in the book. > >> > > >> >BTW usually HBase has a "client" somewhere in the same to indicate > >> >it's client side. > >> > > >> >J-D > >> > > >> >On Fri, Mar 30, 2012 at 3:08 PM, anil gupta <[email protected]> > wrote: > >> >> Thanks for the quick reply, Jean. Is there any link where i can find > the > >> >> name of all client-side configuration for HBase? > >> >> > >> >> ~Anil > >> >> > >> >> On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans > >> >><[email protected]>wrote: > >> >> > >> >>> This is a client-side configuration so if your mapred-site.xml is > >> >>> _not_ on your classpath when you start the bulk load, it's not going > >> >>> to pick it up. So either have that file on your classpath, or put it > >> >>> in whatever other configuration file you have. > >> >>> > >> >>> J-D > >> >>> > >> >>> On Fri, Mar 30, 2012 at 2:52 PM, anil gupta <[email protected]> > >> >>>wrote: > >> >>> > Hi All, > >> >>> > > >> >>> > I am using cdh3u2. I ran HBase bulk loading with property > >> >>> > "mapred.reduce.tasks.speculative.execution" set to "false" in > >> >>> > mapred-site.xml. Still, i can see 6 killed task in Bulk Loading > job > >> >>>and > >> >>> > after short analysis i realized that these jobs are killed because > >> >>> another > >> >>> > worker node completed the task, hence it means that speculative > >> >>>execution > >> >>> > is still on. Why the HBase Bulk loader is doing speculative > execution > >> >>> when > >> >>> > i have set it to false in mapred-site.xml? Please let me know if > i am > >> >>> > missing something over here. > >> >>> > > >> >>> > -- > >> >>> > Thanks & Regards, > >> >>> > Anil Gupta > >> >>> > >> >> > >> >> > >> >> > >> >> -- > >> >> Thanks & Regards, > >> >> Anil Gupta > >> > > >> > >> > >> > > > > > > -- > > Thanks & Regards, > > Anil Gupta > > > > -- > Harsh J > -- Mikael.S
