Thanks for catching that -- it was a typo -- one wal per region.
On Tue, Apr 15, 2014 at 8:21 AM, Ted Yu <[email protected]> wrote: > bq. In the case of an SSD world, it makes more sense to have one wal per > node > > Was there a typo in the sentence above (one wal per node) ? > > Cheers > > > On Tue, Apr 15, 2014 at 7:11 AM, Jonathan Hsieh <[email protected]> wrote: > > > It makes sense to have as many wals as # of spindles / replication factor > > per machine. This should be decoupled from the number of regions on a > > region server. So for a cluster with 12 spindles we should likely have > at > > least 4 wals (12 spindles / 3 replication factor), and need to do > > experiments to see if going to 8 or some higher number makes sense (new > wal > > uses a disruptor pattern which avoids much contention on individual > > writes). So with your example, your 1000 regions would get sharded into > > the 4 wals which would maximize io throughput, disk utilization, and > reduce > > time for recovery in the face of failure. > > > > In the case of an SSD world, it makes more sense to have one wal per node > > once we have decent HSM support in HDFS. The key win here will be in > > recovery time -- if any RS goes down we only have to replay a regions > edits > > and not have to split or demux different region's edits. > > > > Jon. > > > > > > On Mon, Apr 14, 2014 at 10:37 PM, Vladimir Rodionov > > <[email protected]>wrote: > > > > > Todd, how about 300 regions with 3x replication? Or 1000 regions? This > > is > > > going to be 3000 files. on HDFS. per one RS. When I said that it does > not > > > scale, I meant that exactly that. > > > > > > > > > > > -- > > // Jonathan Hsieh (shay) > > // HBase Tech Lead, Software Engineer, Cloudera > > // [email protected] // @jmhsieh > > > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // [email protected] // @jmhsieh
