We have multiple clusters (in multiple datacenters), and web serving is separated from MR processing. We do have some jobs running on prod but they are usually of lower intensity, run during low traffic hours, and we limited the number of map and reduce slots to 1 each per node.
The data between the clusters can be copied via any tool of your fancy. In 0.20 you have access to Import/Export+distcp and in 0.89 there's a CopyTable job that does the copying from one HBase setup directly into another HBase cluster. Finally in 0.89 there's also the cluster replication feature (that I wrote) that basically works like MySQL master/slave replication. We currently only use it for disaster recovery tho, I have yet written the multi-slave part :) J-D On Thu, Sep 23, 2010 at 10:11 PM, Bishal Acharya <[email protected]> wrote: > I am running a 20 node cluster with hadoop/hbase. Currently what I am doing > is that, I run the MR jobs in the cluster and at the same time I am serving > my web application directly from Hbase in the same cluster. What happens is, > when I am not running any MR jobs the applications are running perfectly > fine, But when I run MR jobs at the same time as I am browsing my > application, I am faced with this increase in latency while browsing. How > could I properly manage my cluster so that I don't have to face the added > latency due to cluster being saturated by MR jobs. I wanted to know > specifically how this is done in companies using Hbase for front serving for > example at StumbleUpon ? How do they manage this issue ? > > > -- > > Sincerely, > > > *Bishal Acharya* > > /Software Engineer | D2HawkeyeServices Pvt. Ltd. | Subsidiary of Verisk > Health,USA/ > ** > > Cell +977-9849378541 | [email protected] | www.d2hawkeyeservices.com > > P Â Request : Unless absolutely necessary, please do not print this e-mail. > Help save environment. Thank you. > > > > > This email is intended for the recipient only. If you are not the intended > recipient please disregard, and do not use the information for any purpose.
