Re: hbase 0.90.2 - incredibly slow response
It seem likes my case. My test data: Puts：75090 ops/s, average latency：2.7 ms. scan：494 ops/s ,average latency：1356 ms. (HMaster 1 name node, 3 zoo keeper, 7 Region server/Data node) about my test, some schema may be slower in version 0.90.2. How do you design your schema? If there is any hot spot, Please see Issue HBASE-3779 which may be help you. https://issues.apache.org/jira/browse/HBASE-3779 -邮件原件- 发件人: Venkatesh [mailto:vramanatha...@aol.com] 发送时间: 2011年4月22日 12:00 收件人: user@hbase.apache.org 主题: Re: hbase 0.90.2 - incredibly slow response Thanks St.Ack Yes..will try the upgrade in a smaller setup with ..production like load will investigate/compare -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Thu, Apr 21, 2011 11:47 pm Subject: Re: hbase 0.90.2 - incredibly slow response Sorry to hear you rolled back. I think its fair to say that going to 0.90.2 usually has things running faster and more efficiently. As to why your experience, I'm not sure what it could be though of course; it sounds likes something we've not come across before since we passed you all that we could think of. Whats your plan now? Are you going to try the upgrade again? You might research how your current install is running. Do what Jack Levin did this afternoon where he enabled rpc DEBUG for a while to get a sense of the type of requests and how long hbase is taking to process them (In his case he found that upping the handlers cured a slow scan issue). You could study the 0.20.6 response times and then when you upgrade to 0.90.2, check what its showing. That would at least give us a clue as to where to start digging. St.Ack On Thu, Apr 21, 2011 at 8:21 PM, Venkatesh vramanatha...@aol.com wrote: Thanks St. Ack.. Sorry I had to roll back to 0.20.6..as our system is down way too long.. so..i don't have log rt now..i'll try to recreate in a different machine at a later time.. yes..700 mil puts per day cluster is 20 node (20 datanode+ region server) ..besides that 1 machine with HMaster 1 name node, 3 zoo keeper we do , new HTable, put, close in a multi threaded servlet (tomcat based) (HBase configuration object is constructed in init) same logic works great in 0.20.6.. In 0.90.2 all I changed was retrofit HTable constructor .. it crawls we rolled back to 0.20.6 it works great again..obviously some major logic change in 0.90.2 that requires perhaps different coding practice for client api..If you can shed some light that wld be helpfull My hbase config is pretty much default except region size (using 4gig) -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 2:11 pm Subject: Re: hbase 0.90.2 - incredibly slow response Want to paste your configuration up in pastebin? Is that 700million puts a day? Remind us of your cluster size. Paste some of a regionserver log too. That can be informative. St.Ack On Wed, Apr 20, 2011 at 10:41 AM, Venkatesh vramanatha...@aol.com wrote: shell is no problems..ones/twos..i've tried mass puts from shell we cant handle our production load (even 1/3 of it) 700 mill per day is full load..same load we handled with absolutely no issues in 0.20.6.. there is several pause between batch of puts as wel -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 1:30 pm Subject: Re: hbase 0.90.2 - incredibly slow response On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh vramanatha...@aol.com wrote: I was hoping that too.. I don't have scripts to generate # requests from shell..I will try that.. Did you try it? Above you seem to say that a simple put of 100 bytes takes 2 seconds where in 0.20.6 it took 10 milliseconds. A put from shell of 100 bytes is easy enough to do. hbase put 'YOUR_TABLE', 'SOME_ROW', 'SOME_COLUMN', 'SOME_STRING_OF_100_BYTES' The shell will print out rough numbers on how long it takes to do the put (from ruby). I did n't pre-create regions in 0.20.6 it handled fine the same load.. I'll try performance in 0.90.2 by precreating regions.. Would sharing a single HBaseConfiguration object for all threads hurt performance? I'd doubt that this is the issue. It should help usually. St.Ack
Re: hbase 0.90.2 - incredibly slow response
Thanks St. Ack.. Sorry I had to roll back to 0.20.6..as our system is down way too long.. so..i don't have log rt now..i'll try to recreate in a different machine at a later time.. yes..700 mil puts per day cluster is 20 node (20 datanode+ region server) ..besides that 1 machine with HMaster 1 name node, 3 zoo keeper we do , new HTable, put, close in a multi threaded servlet (tomcat based) (HBase configuration object is constructed in init) same logic works great in 0.20.6.. In 0.90.2 all I changed was retrofit HTable constructor .. it crawls we rolled back to 0.20.6 it works great again..obviously some major logic change in 0.90.2 that requires perhaps different coding practice for client api..If you can shed some light that wld be helpfull My hbase config is pretty much default except region size (using 4gig) -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 2:11 pm Subject: Re: hbase 0.90.2 - incredibly slow response Want to paste your configuration up in pastebin? Is that 700million puts a day? Remind us of your cluster size. Paste some of a regionserver log too. That can be informative. St.Ack On Wed, Apr 20, 2011 at 10:41 AM, Venkatesh vramanatha...@aol.com wrote: shell is no problems..ones/twos..i've tried mass puts from shell we cant handle our production load (even 1/3 of it) 700 mill per day is full load..same load we handled with absolutely no issues in 0.20.6.. there is several pause between batch of puts as wel -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 1:30 pm Subject: Re: hbase 0.90.2 - incredibly slow response On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh vramanatha...@aol.com wrote: I was hoping that too.. I don't have scripts to generate # requests from shell..I will try that.. Did you try it? Above you seem to say that a simple put of 100 bytes takes 2 seconds where in 0.20.6 it took 10 milliseconds. A put from shell of 100 bytes is easy enough to do. hbase put 'YOUR_TABLE', 'SOME_ROW', 'SOME_COLUMN', 'SOME_STRING_OF_100_BYTES' The shell will print out rough numbers on how long it takes to do the put (from ruby). I did n't pre-create regions in 0.20.6 it handled fine the same load.. I'll try performance in 0.90.2 by precreating regions.. Would sharing a single HBaseConfiguration object for all threads hurt performance? I'd doubt that this is the issue. It should help usually. St.Ack
Re: hbase 0.90.2 - incredibly slow response
Sorry to hear you rolled back. I think its fair to say that going to 0.90.2 usually has things running faster and more efficiently. As to why your experience, I'm not sure what it could be though of course; it sounds likes something we've not come across before since we passed you all that we could think of. Whats your plan now? Are you going to try the upgrade again? You might research how your current install is running. Do what Jack Levin did this afternoon where he enabled rpc DEBUG for a while to get a sense of the type of requests and how long hbase is taking to process them (In his case he found that upping the handlers cured a slow scan issue). You could study the 0.20.6 response times and then when you upgrade to 0.90.2, check what its showing. That would at least give us a clue as to where to start digging. St.Ack On Thu, Apr 21, 2011 at 8:21 PM, Venkatesh vramanatha...@aol.com wrote: Thanks St. Ack.. Sorry I had to roll back to 0.20.6..as our system is down way too long.. so..i don't have log rt now..i'll try to recreate in a different machine at a later time.. yes..700 mil puts per day cluster is 20 node (20 datanode+ region server) ..besides that 1 machine with HMaster 1 name node, 3 zoo keeper we do , new HTable, put, close in a multi threaded servlet (tomcat based) (HBase configuration object is constructed in init) same logic works great in 0.20.6.. In 0.90.2 all I changed was retrofit HTable constructor .. it crawls we rolled back to 0.20.6 it works great again..obviously some major logic change in 0.90.2 that requires perhaps different coding practice for client api..If you can shed some light that wld be helpfull My hbase config is pretty much default except region size (using 4gig) -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 2:11 pm Subject: Re: hbase 0.90.2 - incredibly slow response Want to paste your configuration up in pastebin? Is that 700million puts a day? Remind us of your cluster size. Paste some of a regionserver log too. That can be informative. St.Ack On Wed, Apr 20, 2011 at 10:41 AM, Venkatesh vramanatha...@aol.com wrote: shell is no problems..ones/twos..i've tried mass puts from shell we cant handle our production load (even 1/3 of it) 700 mill per day is full load..same load we handled with absolutely no issues in 0.20.6.. there is several pause between batch of puts as wel -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 1:30 pm Subject: Re: hbase 0.90.2 - incredibly slow response On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh vramanatha...@aol.com wrote: I was hoping that too.. I don't have scripts to generate # requests from shell..I will try that.. Did you try it? Above you seem to say that a simple put of 100 bytes takes 2 seconds where in 0.20.6 it took 10 milliseconds. A put from shell of 100 bytes is easy enough to do. hbase put 'YOUR_TABLE', 'SOME_ROW', 'SOME_COLUMN', 'SOME_STRING_OF_100_BYTES' The shell will print out rough numbers on how long it takes to do the put (from ruby). I did n't pre-create regions in 0.20.6 it handled fine the same load.. I'll try performance in 0.90.2 by precreating regions.. Would sharing a single HBaseConfiguration object for all threads hurt performance? I'd doubt that this is the issue. It should help usually. St.Ack
Re: hbase 0.90.2 - incredibly slow response
Thanks St.Ack Yes..will try the upgrade in a smaller setup with ..production like load will investigate/compare -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Thu, Apr 21, 2011 11:47 pm Subject: Re: hbase 0.90.2 - incredibly slow response Sorry to hear you rolled back. I think its fair to say that going to 0.90.2 usually has things running faster and more efficiently. As to why your experience, I'm not sure what it could be though of course; it sounds likes something we've not come across before since we passed you all that we could think of. Whats your plan now? Are you going to try the upgrade again? You might research how your current install is running. Do what Jack Levin did this afternoon where he enabled rpc DEBUG for a while to get a sense of the type of requests and how long hbase is taking to process them (In his case he found that upping the handlers cured a slow scan issue). You could study the 0.20.6 response times and then when you upgrade to 0.90.2, check what its showing. That would at least give us a clue as to where to start digging. St.Ack On Thu, Apr 21, 2011 at 8:21 PM, Venkatesh vramanatha...@aol.com wrote: Thanks St. Ack.. Sorry I had to roll back to 0.20.6..as our system is down way too long.. so..i don't have log rt now..i'll try to recreate in a different machine at a later time.. yes..700 mil puts per day cluster is 20 node (20 datanode+ region server) ..besides that 1 machine with HMaster 1 name node, 3 zoo keeper we do , new HTable, put, close in a multi threaded servlet (tomcat based) (HBase configuration object is constructed in init) same logic works great in 0.20.6.. In 0.90.2 all I changed was retrofit HTable constructor .. it crawls we rolled back to 0.20.6 it works great again..obviously some major logic change in 0.90.2 that requires perhaps different coding practice for client api..If you can shed some light that wld be helpfull My hbase config is pretty much default except region size (using 4gig) -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 2:11 pm Subject: Re: hbase 0.90.2 - incredibly slow response Want to paste your configuration up in pastebin? Is that 700million puts a day? Remind us of your cluster size. Paste some of a regionserver log too. That can be informative. St.Ack On Wed, Apr 20, 2011 at 10:41 AM, Venkatesh vramanatha...@aol.com wrote: shell is no problems..ones/twos..i've tried mass puts from shell we cant handle our production load (even 1/3 of it) 700 mill per day is full load..same load we handled with absolutely no issues in 0.20.6.. there is several pause between batch of puts as wel -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 1:30 pm Subject: Re: hbase 0.90.2 - incredibly slow response On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh vramanatha...@aol.com wrote: I was hoping that too.. I don't have scripts to generate # requests from shell..I will try that.. Did you try it? Above you seem to say that a simple put of 100 bytes takes 2 seconds where in 0.20.6 it took 10 milliseconds. A put from shell of 100 bytes is easy enough to do. hbase put 'YOUR_TABLE', 'SOME_ROW', 'SOME_COLUMN', 'SOME_STRING_OF_100_BYTES' The shell will print out rough numbers on how long it takes to do the put (from ruby). I did n't pre-create regions in 0.20.6 it handled fine the same load.. I'll try performance in 0.90.2 by precreating regions.. Would sharing a single HBaseConfiguration object for all threads hurt performance? I'd doubt that this is the issue. It should help usually. St.Ack
Re: hbase 0.90.2 - incredibly slow response
On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh vramanatha...@aol.com wrote: I was hoping that too.. I don't have scripts to generate # requests from shell..I will try that.. Did you try it? Above you seem to say that a simple put of 100 bytes takes 2 seconds where in 0.20.6 it took 10 milliseconds. A put from shell of 100 bytes is easy enough to do. hbase put 'YOUR_TABLE', 'SOME_ROW', 'SOME_COLUMN', 'SOME_STRING_OF_100_BYTES' The shell will print out rough numbers on how long it takes to do the put (from ruby). I did n't pre-create regions in 0.20.6 it handled fine the same load.. I'll try performance in 0.90.2 by precreating regions.. Would sharing a single HBaseConfiguration object for all threads hurt performance? I'd doubt that this is the issue. It should help usually. St.Ack
Re: hbase 0.90.2 - incredibly slow response
shell is no problems..ones/twos..i've tried mass puts from shell we cant handle our production load (even 1/3 of it) 700 mill per day is full load..same load we handled with absolutely no issues in 0.20.6.. there is several pause between batch of puts as wel -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 1:30 pm Subject: Re: hbase 0.90.2 - incredibly slow response On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh vramanatha...@aol.com wrote: I was hoping that too.. I don't have scripts to generate # requests from shell..I will try that.. Did you try it? Above you seem to say that a simple put of 100 bytes takes 2 seconds where in 0.20.6 it took 10 milliseconds. A put from shell of 100 bytes is easy enough to do. hbase put 'YOUR_TABLE', 'SOME_ROW', 'SOME_COLUMN', 'SOME_STRING_OF_100_BYTES' The shell will print out rough numbers on how long it takes to do the put (from ruby). I did n't pre-create regions in 0.20.6 it handled fine the same load.. I'll try performance in 0.90.2 by precreating regions.. Would sharing a single HBaseConfiguration object for all threads hurt performance? I'd doubt that this is the issue. It should help usually. St.Ack
Re: hbase 0.90.2 - incredibly slow response
Want to paste your configuration up in pastebin? Is that 700million puts a day? Remind us of your cluster size. Paste some of a regionserver log too. That can be informative. St.Ack On Wed, Apr 20, 2011 at 10:41 AM, Venkatesh vramanatha...@aol.com wrote: shell is no problems..ones/twos..i've tried mass puts from shell we cant handle our production load (even 1/3 of it) 700 mill per day is full load..same load we handled with absolutely no issues in 0.20.6.. there is several pause between batch of puts as wel -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Wed, Apr 20, 2011 1:30 pm Subject: Re: hbase 0.90.2 - incredibly slow response On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh vramanatha...@aol.com wrote: I was hoping that too.. I don't have scripts to generate # requests from shell..I will try that.. Did you try it? Above you seem to say that a simple put of 100 bytes takes 2 seconds where in 0.20.6 it took 10 milliseconds. A put from shell of 100 bytes is easy enough to do. hbase put 'YOUR_TABLE', 'SOME_ROW', 'SOME_COLUMN', 'SOME_STRING_OF_100_BYTES' The shell will print out rough numbers on how long it takes to do the put (from ruby). I did n't pre-create regions in 0.20.6 it handled fine the same load.. I'll try performance in 0.90.2 by precreating regions.. Would sharing a single HBaseConfiguration object for all threads hurt performance? I'd doubt that this is the issue. It should help usually. St.Ack
hbase 0.90.2 - incredibly slow response
Just upgraded to 0.90.2 from 0.20.6..Doing a simple put to table ( 100 bytes per put).. Only code change was to retrofit the HTable API to work with 0.90.2 Initializing HBaseConfiguration in servlet.init()... reusing that config for HTable constructor doing put Performance is very slow 90% of requests are well over 2 sec..(With 0.20.6, 90% use to be 10 milli sec) I did run set_meta_memstore_size.rb as per the book.. Any help to debug is appreciated..I also see periodic pauses between hbase puts thanks v
Re: hbase 0.90.2 - incredibly slow response
I was hoping that too.. I don't have scripts to generate # requests from shell..I will try that.. I did n't pre-create regions in 0.20.6 it handled fine the same load.. I'll try performance in 0.90.2 by precreating regions.. Would sharing a single HBaseConfiguration object for all threads hurt performance? frustrating..thanks for your help -Original Message- From: Stack st...@duboce.net To: user@hbase.apache.org Sent: Tue, Apr 19, 2011 1:40 pm Subject: Re: hbase 0.90.2 - incredibly slow response 0.90.2 should be faster. Running same query from shell, it gives you same lag? St.Ack On Tue, Apr 19, 2011 at 10:35 AM, Venkatesh vramanatha...@aol.com wrote: Just upgraded to 0.90.2 from 0.20.6..Doing a simple put to table ( 100 bytes per put).. Only code change was to retrofit the HTable API to work with 0.90.2 Initializing HBaseConfiguration in servlet.init()... reusing that config for HTable constructor doing put Performance is very slow 90% of requests are well over 2 sec..(With 0.20.6, 90% use to be 10 milli sec) I did run set_meta_memstore_size.rb as per the book.. Any help to debug is appreciated..I also see periodic pauses between hbase puts thanks v
