On Fri, Mar 30, 2012 at 1:48 AM, Stack <[email protected]> wrote: > On Mon, Mar 26, 2012 at 8:18 PM, Juhani Connolly > <[email protected]> wrote: >> Also put in our configs... They had more in them but we stripped them down a >> lot to try to get at the problem source, no luck though(we took them down to >> the bare minimum as well but that didn't change things so we restored some >> of the settings). >> > > The regionserver thread dumps are near unrecognizable as regionserver > thread dumps.... at least it took me a while to figure they were not > from another process altogether. > > These regionservers are making no progress. All are blocked (which in > itself doesn't ring right). > > How did you make these thread dumps? They seem to be missing some > basic info. That might have been part of the issue. What JVM are you > running? Mind giving us the output that is in the .out when you do a > kill -QUIT PID instead of however you did this thread dumping? >
The dumps are taken with "sudo jstack -F <pid>" Without setting the force flag I get a "Unable to open socket file: target process not responding or HotSpot VM not loaded" error. I'm not entirely sure why this is happening but I just found now that when I try to take a jstack it actually makes a dump in the log directory, this one has the thread names so maybe it's better. There are several dumps included in it, the later ones are under load. As to the jvm, w'ere using sun java 6u26 [juhani_connolly@hornet-slave01 java]$ java -version java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) > Mind posting some regionserver logs while its under load? > Attached. This is while running 100 threads making autoFlushed writes. The throughput on this regionserver is about 1000 wps > Thanks Juhani, > St.Ack Thanks, Juhani
