Hello Jean, Thanks for the response! Sorry I missed it (and this one too, just missed it). I was exhausted yesterday, and my e-mails were kind of all over the place yesterday, so I figured they just didn't make any sense ;)
And spot on with the xceivers/xcievers. The error is showing up all over the datanodes. I upp'd the open file limit as per the FAQ a month or two ago, so I at least I'm not seeing any "too many open files". I don't know how I got the ulimit but missed the xceiver thing on the faq. Just a dumb mistake I guess. Thanks for the catch. Take care, -stu --- On Wed, 7/14/10, Jean-Daniel Cryans <[email protected]> wrote: > From: Jean-Daniel Cryans <[email protected]> > Subject: Re: Cluster destabilizes recovering from crash > To: [email protected] > Date: Wednesday, July 14, 2010, 8:55 PM > Check xcievers too, giving more mem > to HBase won't fix HDFS. > > J-D > > On Wed, Jul 14, 2010 at 5:53 PM, Stuart Smith <[email protected]> > wrote: > > > > Ok, back up again. > > > > This time, in addition to just watching & > restarting, the error logs led me to this: > > > > http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201004.mbox/%[email protected]%3e > > > > Which led me to up both HBASE_HEAP and HADOOP_HEAP > from 1 -> 4 GB. Kind of scary, since my regionservers > only have 8GB of memory (NN/ZK/MASTER) has 16GB. But it did > seem to start up a little quicker. > > > > I was seeing about bunch of edits being recovered and > then errors similar to these: > > > > INFO org.apache.hadoop.hdfs.DFSClient: Exception in > > createBlockOutputStream java.io.EOFException > > INFO org.apache.hadoop.hdfs.DFSClient: Abandoning > > block blk_ > > > > I already did the file handle thing a while ago. > > > > Let's see if this fixes it! > > > > Take care, > > -stu > > > > --- On Wed, 7/14/10, Stuart Smith <[email protected]> > wrote: > > > >> From: Stuart Smith <[email protected]> > >> Subject: Cluster destabilizes recovering from > crash > >> To: [email protected] > >> Date: Wednesday, July 14, 2010, 7:44 PM > >> Hello All, > >> > >> Ok, I'm seeing similar behavior to what I > mentioned > >> yesterday, but the cause seems to be hbase not > handling the > >> Delete/Put load. > >> > >> I have: > >> > >> - A small cluster with 4 1950s as > regionservers > >> & 1 M600 as zookeeper & master. NTP is > active, and > >> the datetimes are in sync. > >> > >> - 1 java client with about 32 threads all > uploading > >> files to hbase. Pretty slow though, only 1-4 files > a second, > >> this due to non-hbase issues. > >> > >> - A M/R task that is iterating over a table, > >> and gathering data about each row, and deleting > rows with > >> improperly formed keys. There are about 40 million > rows, and > >> probably around 7-10 million improperly formed > keys. > >> > >> I do the validation & delete in the Map task. > My > >> connection to hbase is initialized in the Map > class > >> constructor. I noticed some map tasks are failing > due to > >> being unable to connect (to namenode proxy, or > something). > >> The number of failures is low compared to the > total number > >> of map tasks, so it's not every connection that's > failing > >> (not a network visibility issue). > >> > >> The hbase master eventually went down, and I > restarted > >> everything. Hbase now takes a long time coming > back, and > >> appears to be re-generating a lot of edits trying > to > >> recover. However, as it goes through the recovery > process, > >> region servers just start dying off, and I have > to > >> continuously restart them. > >> > >> Is this known behavior? Is hbase just kind of > flaky on > >> recovery at this point? Or did I do something > wrong? Should > >> I just not push it that hard with inserts & > deletes? > >> > >> I went through this yesterday, and eventually it > should > >> stabilize and come back, but it takes several > hours of > >> sitting around and babysitting the cluster. > >> > >> I'm wondering if it would be bad to just write a > cron job > >> that calls my "log on and try to start all > regionservers" > >> script. Is there anything bad about trying to > start > >> regionservers via ./hbase start regionserver every > minute or > >> so? They die off pretty quick. > >> > >> Take care, > >> -stu > >> > >> > >> --- On Tue, 7/13/10, Stuart Smith <[email protected]> > >> wrote: > >> > >> > From: Stuart Smith <[email protected]> > >> > Subject: Re: Trying to write too much to > stdout > >> destabilizes cluster across reboots > >> > To: [email protected] > >> > Date: Tuesday, July 13, 2010, 9:38 PM > >> > Last follow up: > >> > > >> > It appears just rebooting, babysitting > regionservers, > >> > watching it die.. and repeating the process > >> eventually > >> > recovered the cluster. > >> > > >> > I still haven't gone through all the errors > very > >> carefully. > >> > I did have hundreds of threads of hbase > clients > >> uploading > >> > data while I was running my M/R task that > dumped the > >> output. > >> > Based on the logs, I'm wondering if it was > just > >> getting > >> > swamped trying to recover & rebuild all > the > >> writes? Not > >> > sure. > >> > > >> > Well, it worked eventually, so I'm going to > just > >> upgrade to > >> > 0.20.5 ASAP, and try to be more careful about > how much > >> I > >> > write to stdout. > >> > > >> > Take care, > >> > -stu > >> > > >> > --- On Tue, 7/13/10, Stuart Smith <[email protected]> > >> > wrote: > >> > > >> > > From: Stuart Smith <[email protected]> > >> > > Subject: Re: Trying to write too much to > stdout > >> > destabilizes cluster across reboots > >> > > To: [email protected] > >> > > Date: Tuesday, July 13, 2010, 9:10 PM > >> > > One follow up. > >> > > > >> > > I noticed the master was saying this as > well: > >> > > > >> > > 2010-07-13 17:48:17,595 WARN > >> > > org.apache.hadoop.hbase.master.HMaster: > >> Processing > >> > pending > >> > > operations: ProcessRegionClose of > >> > > > >> > > >> > filestore,5c4ee356835d82de844600895a7df1595c8b53bd378436f6e8d2a1a99acb550c,1279043794914, > >> > > false, reassign: true > >> > > java.lang.RuntimeException: > >> > java.lang.NullPointerException > >> > > at > >> > > > >> > > >> > org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:96) > >> > > at > >> > > > >> > > >> > org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) > >> > > at > >> > > > >> > > >> > org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:494) > >> > > at > >> > > > >> > > >> > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:429) > >> > > Caused by: > java.lang.NullPointerException > >> > > > >> > > Which lead me to this: > >> > > > >> > > https://issues.apache.org/jira/browse/HBASE-2428 > >> > > > >> > > I guess I shouldn't have put off > upgrading to > >> 0.20.5 > >> > :( > >> > > I'm curious if by this comment: > >> > > > >> > > "Since delayqueue is processed first, > all we did > >> was > >> > try to > >> > > process the close even though in the > other more > >> live > >> > todo > >> > > queue, we might have the process of the > >> regionserver > >> > > shutdown waiting, the process of the > regionserver > >> that > >> > was > >> > > carrying .meta. We'd never recover. " > >> > > > >> > > They mean never as in _never_ or just as > in until > >> the > >> > next > >> > > reboot. Or is my cluster (and my data) > dead and > >> gone > >> > now? > >> > > > >> > > Take care, > >> > > -stu > >> > > > >> > > --- On Tue, 7/13/10, Stuart Smith <[email protected]> > >> > > wrote: > >> > > > >> > > > From: Stuart Smith <[email protected]> > >> > > > Subject: Trying to write too much > to stdout > >> > > destabilizes cluster across reboots > >> > > > To: [email protected] > >> > > > Date: Tuesday, July 13, 2010, 8:58 > PM > >> > > > Hello, > >> > > > > >> > > > I have a M/R task running > over hbase > >> > > > tables. I added > >> > > > a one line of output in my reduce > tasks > >> > running > >> > > over a > >> > > > set > >> > > > of tens of millions of records. > This was > >> a > >> > > little silly, > >> > > > but > >> > > > I was kind of curious what would > happen. > >> > > > > >> > > > Well the reduce ran very slowly, > and > >> > eventually, > >> > > I had to > >> > > > kill the tasks, and I re-started > the > >> whole > >> > > cluster, as > >> > > > other > >> > > > clients were slowing up > considerably, and > >> the > >> > > M/R job > >> > > > started dumping all the stdout > directly to > >> the > >> > > console it > >> > > > was running on. > >> > > > > >> > > > I've re-started it twice, but I > couldn't > >> get > >> > > hbase to > >> > > > come > >> > > > backup (fully). And when I shut > down, the > >> > region > >> > > servers > >> > > > will go down when I tell them, > but the > >> master > >> > > needs to be > >> > > > killed with -s 9 (SIG_KILL? I'm a > little > >> > rusty). > >> > > It > >> > > > eventually seemed to hobble > towards > >> coming > >> > back > >> > > up.. it > >> > > > took > >> > > > a looong.., with messages like: > >> > > > > >> > > > > >> > > > 2010-07-13 16:32:47,722 INFO > >> > > > > org.apache.hadoop.hdfs.DFSClient: > >> Abandoning > >> > > block > >> > > > blk_-9203944767743886537_123926 > >> > > > 2010-07-13 16:32:48,207 DEBUG > >> > > > > >> org.apache.hadoop.hbase.regionserver.HLog: > >> > > Waiting for > >> > > > hlog > >> > > > writers to terminate, iteration > #93 > >> > > > 2010-07-13 16:32:52,205 INFO > >> > > > > org.apache.hadoop.hdfs.DFSClient: > >> Exception in > >> > > > createBlockOutputStream > >> java.io.EOFException > >> > > > 2010-07-13 16:32:52,205 INFO > >> > > > > org.apache.hadoop.hdfs.DFSClient: > >> Abandoning > >> > > block > >> > > > blk_-7742458689391305082_123926 > >> > > > 2010-07-13 16:32:53,072 INFO > >> > > > > org.apache.hadoop.hdfs.DFSClient: > >> Exception in > >> > > > createBlockOutputStream > >> java.io.EOFException > >> > > > 2010-07-13 16:32:53,072 INFO > >> > > > > org.apache.hadoop.hdfs.DFSClient: > >> Abandoning > >> > > block > >> > > > blk_-281711885322381263_123926 > >> > > > 2010-07-13 16:32:53,207 DEBUG > >> > > > > >> org.apache.hadoop.hbase.regionserver.HLog: > >> > > Waiting for > >> > > > hlog > >> > > > writers to terminate, iteration > #94 > >> > > > 2010-07-13 16:32:53,726 INFO > >> > > > > org.apache.hadoop.hdfs.DFSClient: > >> Exception in > >> > > > createBlockOutputStream > >> java.io.IOException: > >> > Bad > >> > > connect > >> > > > ack > >> > > > with firstBadLink > 192.168.193.155:50010 > >> > > > 2010-07-13 16:32:53,726 INFO > >> > > > > org.apache.hadoop.hdfs.DFSClient: > >> Abandoning > >> > > block > >> > > > blk_-2730636850915812524_123926 > >> > > > > >> > > > This just seems odd.. dumping > that much > >> > standard > >> > > output > >> > > > was > >> > > > kind silly, but I wouldn't think > it would > >> > wedge > >> > > the > >> > > > cluster > >> > > > like this... or did something > else > >> happen? > >> > > Everything has > >> > > > been fine for a while until I did > this. > >> > > > > >> > > > I let the cluster sit around for > a while, > >> and > >> > it > >> > > did come > >> > > > back for a little while, after > printing > >> out > >> > this > >> > > over and > >> > > > over (on > >> > > > master): > >> > > > > >> > > > da7738a3ad9,1279039241464 from > >> > > > > ubuntu-hadoop-2,60020,1279066792598; 1 of > >> 1 > >> > > > 2010-07-13 17:28:30,897 INFO > >> > > > > >> org.apache.hadoop.hbase.master.ServerManager: > >> > > Processing > >> > > > MSG_REPORT_PROCESS_OPEN: > >> > > > > >> > > > > >> > > > >> > > >> > filemetadata,unknown_23bf6f189f81218a877a41d1ccd82751527abf19beb61281b8a0f6475e682764,1278533813300 > >> > > > from > ubuntu-hadoop-4,60020,1279066792615; > >> 1 of > >> > > 3 > >> > > > 2010-07-13 17:28:30,897 INFO > >> > > > > >> org.apache.hadoop.hbase.master.ServerManager: > >> > > Processing > >> > > > MSG_REPORT_PROCESS_OPEN: > >> > > > > >> > > > > >> > > > >> > > >> > filestore,7f45c6a2f5c7d09ae8618bf6d0db5b04f346c583a1dbf24d9db135f2c8da9b57,1279050466715 > >> > > > from > ubuntu-hadoop-4,60020,1279066792615; > >> 2 of > >> > > 3 > >> > > > 2010-07-13 17:28:30,897 INFO > >> > > > > >> org.apache.hadoop.hbase.master.ServerManager: > >> > > Processing > >> > > > MSG_REPORT_PROCESS_OPEN: > .META.,,1 from > >> > > > > ubuntu-hadoop-4,60020,1279066792615; 3 of > >> 3 > >> > > > > >> > > > But regionservers kept coming > down, and > >> > bringing > >> > > down the > >> > > > .META region.. eventually the > master > >> seemed to > >> > > recover, > >> > > > and > >> > > > I would bring the regionservers > back up > >> > > individually.. > >> > > > one > >> > > > of them would soon crash again, > repeating > >> the > >> > > cycle. And > >> > > > the > >> > > > master would would die with this > on the > >> web > >> > > page: > >> > > > > >> > > > Trying to contact region server > >> > > 192.168.193.155:60020 for > >> > > > region .META.,,1, row '', but > failed after > >> 3 > >> > > attempts. > >> > > > Exceptions: > >> > > > java.net.ConnectException: > Connection > >> refused > >> > > > java.net.ConnectException: > Connection > >> refused > >> > > > java.net.ConnectException: > Connection > >> refused > >> > > > > >> > > > RequestURI=/master.jsp > >> > > > > >> > > > The regionservers appear to start > dying > >> with > >> > > this > >> > > > statement: > >> > > > > >> > > > > >> > > > 010-07-13 02:12:59,257 DEBUG > >> > > > > >> > > > >> > > >> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: > >> > > > Compaction requested for region > >> > > > > >> > > > > >> > > > >> > > >> > filestore,5a1ec52137fb8bb38affb5ca034ec3817c6611adc2f5a2b399a4ab92effcdedd,1278970890717/1436533481 > >> > > > because: > >> > > > regionserver/192.168.193.145:60020.cacheFlusher > >> > > > 2010-07-13 02:13:19,336 WARN > >> > > > org.apache.hadoop.hdfs.DFSClient: > Failed > >> to > >> > > connect to > >> > > > /192.168.193.145:50010 for file > >> > > > > >> > > > /hbase/.META./1028785192/info/3836661676936096933 > >> for > >> > > > block > >> > > > > -649923918996291451:java.io.EOFException > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > java.io.DataInputStream.readShort(DataInputStream.java:298) > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1384) > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:1853) > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1922) > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46) > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:101) > >> > > > > >> > > > > >> > > > Note that it is failing to > contact > >> > > 192.168.193.145 ... > >> > > > this > >> > > > is the ip of the node itself (the > dying > >> one). > >> > > > > >> > > > There's a long sequence of > exceptions, and > >> the > >> > > > regionservers death ends with > this: > >> > > > > >> > > > 010-07-13 17:36:57,068 INFO > >> > > > org.apache.zookeeper.ClientCnxn: > >> EventThread > >> > > shut down > >> > > > 2010-07-13 17:36:57,069 ERROR > >> > > > > org.apache.hadoop.hdfs.DFSClient: > >> Exception > >> > > closing file > >> > > > > >> > > > > >> > > > >> > > >> > /hbase/filestore/compaction.dir/1408434857/4172852130952668434 > >> > > > : java.io.EOFException > >> > > > java.io.EOFException > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > java.io.DataInputStream.readByte(DataInputStream.java:250) > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298) > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319) > >> > > > at > >> > > > > >> > > > >> > org.apache.hadoop.io.Text.readString(Text.java:400) > >> > > > at > >> > > > > >> > > > > >> > > > >> > > >> > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream > >> > > > > >> > > > It just seems like a lot of > trouble from > >> one > >> > > little > >> > > > System.out.println() statement.. > >> > > > > >> > > > Restarting hbase just repeats the > cycle > >> "Bad > >> > > connect ack > >> > > > with firstBadLink", eventual > recovery, > >> > regionservers > >> > > start > >> > > > crashing, cluster becomes > unusable.. > >> > > > > >> > > > In summary, it would seem > printing out > >> too > >> > much > >> > > data to > >> > > > stdout can render the whole > cluster > >> unstable - > >> > > across > >> > > > reboots - and I haven't figured > out how > >> to > >> > > stabilize it > >> > > > again. > >> > > > > >> > > > Take care, > >> > > > -stu > >> > > > > >> > > > PS. The address for the mailing > list on > >> the > >> > > hbase page is > >> > > > wrong. It shows: > >> > > > > >> > > > [email protected] > >> > > > But it should be > >> > > > [email protected]. > >> > > > I hope! (If this mail goes through, > I got > >> it > >> > right). > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > > >> > >> > >> > >> > > > > > > > > >
