Also the hbase version is 0.94.1
On Sun, Sep 7, 2014 at 12:00 AM, kiran <[email protected]> wrote: > Lars, > > We are facing a similar situation on the similar cluster configuration... > We are having high I/O wait percentages on some machines in our cluster... > We have short circuit reads enabled but still we are facing the similar > problem.. the cpu wait goes upto 50% also in some case while issuing scan > commands with multiple threads.. Is there a work around other than applying > the patch for 0.94.4 ?? > > Thanks > Kiran > > > On Thu, Apr 25, 2013 at 12:12 AM, lars hofhansl <[email protected]> wrote: > >> You may have run into https://issues.apache.org/jira/browse/HBASE-7336 >> (which is in 0.94.4) >> (Although I had not observed this effect as much when short circuit reads >> are enabled) >> >> >> >> ----- Original Message ----- >> From: kzurek <[email protected]> >> To: [email protected] >> Cc: >> Sent: Wednesday, April 24, 2013 3:12 AM >> Subject: HBase - Performance issue >> >> The problem is that when I'm putting my data (multithreaded client, >> ~30MB/s >> traffic outgoing) into the cluster the load is equally spread over all >> RegionServer with 3.5% average CPU wait time (average CPU user: 51%). When >> I've added similar, mutlithreaded client that Scans for, let say, 100 last >> samples of randomly generated key from chosen time range, I'm getting high >> CPU wait time (20% and up) on two (or more if there is higher number of >> threads, default 10) random RegionServers. Therefore, machines that held >> those RS are getting very hot - one of the consequences is that number of >> store file is constantly increasing, up to the maximum limit. Rest of the >> RS >> are having 10-12% CPU wait time and everything seems to be OK (number of >> store files varies so they are being compacted and not increasing over >> time). Any ideas? Maybe I could prioritize writes over reads somehow? Is >> it >> possible? If so what would be the best way to that and where it should be >> placed - on the client or cluster side)? >> >> Cluster specification: >> HBase Version 0.94.2-cdh4.2.0 >> Hadoop Version 2.0.0-cdh4.2.0 >> There are 6xDataNodes (5xHDD for storing data), 1xMasterNodes >> Other settings: >> - Bloom filters (ROWCOL) set >> - Short circuit turned on >> - HDFS Block Size: 128MB >> - Java Heap Size of Namenode/Secondary Namenode in Bytes: 8 GiB >> - Java Heap Size of HBase RegionServer in Bytes: 12 GiB >> - Java Heap Size of HBase Master in Bytes: 4 GiB >> - Java Heap Size of DataNode in Bytes: 1 GiB (default) >> Number of regions per RegionServer: 19 (total 114 regions on 6 RS) >> Key design: <UUID><TIMESTAMP> -> UUID: 1-10M, TIMESTAMP: 1-N >> Table design: 1 column family with 20 columns of 8 bytes >> >> Get client: >> Multiple threads >> Each thread have its own tables instance with their Scanner. >> Each thread have its own range of UUIDs and randomly draws beginning of >> time >> range to build rowkey properly (see above). >> Each time Scan requests same amount of rows, but with random rowkey. >> >> >> >> >> >> -- >> View this message in context: >> http://apache-hbase.679495.n3.nabble.com/HBase-Performance-issue-tp4042836.html >> Sent from the HBase User mailing list archive at Nabble.com. >> >> > > > -- > Thank you > Kiran Sarvabhotla > > -----Even a correct decision is wrong when it is taken late > > -- Thank you Kiran Sarvabhotla -----Even a correct decision is wrong when it is taken late
