my yourkit version expired :)... but here is the jstack when it happens: http://pastebin.com/5v6mHg3t
On Mon, May 2, 2011 at 1:00 PM, Todd Lipcon <[email protected]> wrote: > On Mon, May 2, 2011 at 12:56 PM, Jack Levin <[email protected]> wrote: > >> Tried removing yourkit and run on javasun, same thing. We have some >> threads blocked, does anyone know what they block on? >> > > Which threads are blocked? Can you get some jstacks without yourkit? > > -Todd > > >> >> -Jack >> >> On Mon, May 2, 2011 at 7:53 AM, Todd Lipcon <[email protected]> wrote: >> > Hi Jack, >> > >> > Does this happen even if you aren't running Yourkit on the DN? >> > >> > Can you try using a Sun JDK instead of OpenJDK? >> > >> > -Todd >> > >> > On Sun, May 1, 2011 at 7:34 PM, Jack Levin <[email protected]> wrote: >> > >> >> Version: 0.20.2+320 hdfs >> >> .89 HBASE >> >> >> >> ulimit is 32k >> >> xcievers is 5k >> >> >> >> Note from the jstack, I am not exceeding xcievers. >> >> >> >> -Jack >> >> >> >> On Sun, May 1, 2011 at 6:19 PM, Michael Segel < >> [email protected]> >> >> wrote: >> >> > >> >> > >> >> > What's your xceivers set to? >> >> > What's the ulimit -n set for hdfs/hadoop user... (You didn't say >> which >> >> release/version you were using.) >> >> > >> >> >> Date: Sun, 1 May 2011 17:47:18 -0700 >> >> >> Subject: one of our datanodes stops working after few hours >> >> >> From: [email protected] >> >> >> To: [email protected] >> >> >> >> >> >> I took a jstack (http://pastebin.com/5v6mHg3t). After few hours, >> its >> >> >> literally staggers to a halt and gets very very slow... Any ideas >> >> >> whats its blocking on? >> >> >> (main issue is that fsreads for RS get really slow when that >> happens). >> >> >> >> >> >> -Jack >> >> > >> >> >> > >> > >> > >> > -- >> > Todd Lipcon >> > Software Engineer, Cloudera >> > >> > > > > -- > Todd Lipcon > Software Engineer, Cloudera >
