and also there so many Puts maintained by background hbase threads that consuming too much resources
On Wed, Jun 18, 2014 at 6:54 PM, Li Li <[email protected]> wrote: > I mean client slow itself down. e.g. > my client code(one of many threads) > while(true){ > // process data and generate data need to insert to hbase > List<Put> puts=...; > table.put(puts); > } > > the client (I mean threads created by hbase api, not by codes) is > slowed down by retries. But if table.put(puts) don't block. my codes > will send requests. And the server is too busy and gc stop the world > and zookeeper don't receive heartbeat and regards it as dead > > On Wed, Jun 18, 2014 at 6:17 PM, Nicolas Liochon <[email protected]> wrote: >> What do you mean by down? Does it crash? >> >> The server does not block on 0.96, it immediately sends back an exception. >> (See HBASE-9467) >> The client is implicitly slowed down by the retries, w/o blocking on the >> server. It's managed by the hbase client itself, and it's transparent for >> the client application. You've got some logs (info level), to tell you that >> something is happening, but that's it. >> >> You can configure the client to be less more or less verbose >> (hbase.client.start.log.errors.counter) or more or less pushy on the server >> (for example hbase.client.max.perserver.tasks). >> >> Cheers, >> >> Nicolas >> >> >> On Wed, Jun 18, 2014 at 11:59 AM, Li Li <[email protected]> wrote: >> >>> hi all, >>> the hbase client send too much requests and the some region server >>> down. >>> 1. region server down because of gc pause >>> I can see it from log: >>> [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM >>> or host machine (eg GC): pause of approximately 3056ms >>> I can adjust zookeeper config >>> 2. as a client, how can I know server is busy? >>> I am using hbase 0.96 with hadoop1. I can see log such as: >>> org.apache.hadoop.hbase.client.AsyncProcess #178, table=vc2.url_db, >>> attempt=13/35 SUCCEEDED on mphbase1,60020,1403084938641, tracking >>> started Wed Jun 18 17:31:10 CST 2014 >>> I googled and found >>> >>> http://apache-hbase.679495.n3.nabble.com/RegionTooBusyException-Above-memstore-limit-td4056339.html >>> it says in hbase 0.94, if the server is busy, it will block >>> the client request. That's what I want, if the client send too much >>> request than server can handle, it will be blocked. But in 0.96, it >>> seems not. >>> I did not find any exception in client. So I can't slow the clien >>> down. >>>
