Ted, your question is a good one. I hadn't thought about that much. I suppose if HBase were to be aware of local resources, then we'd have to configure the RS memory in accordance w/ the relative size of the storage. I would think we would allocate larger heaps (up to a point to avoid GC-induced timeouts) to processes but leave the various fractions (e.g. block cache fraction) intact.
Now that you bring the memory mismatch into focus, I am thinking it is perhaps better to add some drives (if we can) to existing nodes to get them to parity w/ newer higher storage density nodes. Thanks for discussion! From: Ted Yu [mailto:[email protected]] Sent: Friday, August 22, 2014 2:40 PM To: Sean Busbey Cc: [email protected]; Development Subject: Re: RS storage capacity mismatches Thanks for the link, Sean. Ted: What about available memory on the new nodes ? How much larger is it compared to old nodes ? Cheers On Fri, Aug 22, 2014 at 2:35 PM, Sean Busbey <[email protected]<mailto:[email protected]>> wrote: Hi Ted! HBase doesn't currently take local node resources into account when balancing across region servers. There's currently some discussion on HBASE-11780 about building this funtionality[1]. I'm sure they'd love more input or details about the scale you'd need to work at. -Sean [1]: https://issues.apache.org/jira/browse/HBASE-11780 On Fri, Aug 22, 2014 at 4:07 PM, Ted Tuttle <[email protected]<mailto:[email protected]>> wrote: Hello Ted- We are considering added new nodes to existing cluster that have higher storage capacity than current nodes. We hope, but do not know, that HBase would assign more regions to RSs that have more storage capacity. We are considering 2x storage on new nodes vs current node. Thanks, Ted From: Ted Yu [mailto:[email protected]<mailto:[email protected]>] Sent: Friday, August 22, 2014 1:58 PM To: [email protected]<mailto:[email protected]> Cc: Development Subject: Re: RS storage capacity mismatches Some clarification: by unequal storage capacity, you are expecting certain region servers to host fewer regions, right ? How big is the difference between memory capacity among the old and new servers ? Cheers On Fri, Aug 22, 2014 at 1:52 PM, Ted Tuttle <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: Hello- How does HBase handle region servers with unequal storage capacity? We made the original nodes of our cluster a bit light on storage and are wondering what would happen is we started adding nodes w/ more storage capacity. Thanks, Ted -- Sean
