Ok, back up again. This time, in addition to just watching & restarting, the error logs led me to this:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201004.mbox/%[email protected]%3e Which led me to up both HBASE_HEAP and HADOOP_HEAP from 1 -> 4 GB. Kind of scary, since my regionservers only have 8GB of memory (NN/ZK/MASTER) has 16GB. But it did seem to start up a little quicker. I was seeing about bunch of edits being recovered and then errors similar to these: INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.EOFException INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_ I already did the file handle thing a while ago. Let's see if this fixes it! Take care, -stu --- On Wed, 7/14/10, Stuart Smith <[email protected]> wrote: > From: Stuart Smith <[email protected]> > Subject: Cluster destabilizes recovering from crash > To: [email protected] > Date: Wednesday, July 14, 2010, 7:44 PM > Hello All, > > Ok, I'm seeing similar behavior to what I mentioned > yesterday, but the cause seems to be hbase not handling the > Delete/Put load. > > I have: > > - A small cluster with 4 1950s as regionservers > & 1 M600 as zookeeper & master. NTP is active, and > the datetimes are in sync. > > - 1 java client with about 32 threads all uploading > files to hbase. Pretty slow though, only 1-4 files a second, > this due to non-hbase issues. > > - A M/R task that is iterating over a table, > and gathering data about each row, and deleting rows with > improperly formed keys. There are about 40 million rows, and > probably around 7-10 million improperly formed keys. > > I do the validation & delete in the Map task. My > connection to hbase is initialized in the Map class > constructor. I noticed some map tasks are failing due to > being unable to connect (to namenode proxy, or something). > The number of failures is low compared to the total number > of map tasks, so it's not every connection that's failing > (not a network visibility issue). > > The hbase master eventually went down, and I restarted > everything. Hbase now takes a long time coming back, and > appears to be re-generating a lot of edits trying to > recover. However, as it goes through the recovery process, > region servers just start dying off, and I have to > continuously restart them. > > Is this known behavior? Is hbase just kind of flaky on > recovery at this point? Or did I do something wrong? Should > I just not push it that hard with inserts & deletes? > > I went through this yesterday, and eventually it should > stabilize and come back, but it takes several hours of > sitting around and babysitting the cluster. > > I'm wondering if it would be bad to just write a cron job > that calls my "log on and try to start all regionservers" > script. Is there anything bad about trying to start > regionservers via ./hbase start regionserver every minute or > so? They die off pretty quick. > > Take care, > -stu > > > --- On Tue, 7/13/10, Stuart Smith <[email protected]> > wrote: > > > From: Stuart Smith <[email protected]> > > Subject: Re: Trying to write too much to stdout > destabilizes cluster across reboots > > To: [email protected] > > Date: Tuesday, July 13, 2010, 9:38 PM > > Last follow up: > > > > It appears just rebooting, babysitting regionservers, > > watching it die.. and repeating the process > eventually > > recovered the cluster. > > > > I still haven't gone through all the errors very > carefully. > > I did have hundreds of threads of hbase clients > uploading > > data while I was running my M/R task that dumped the > output. > > Based on the logs, I'm wondering if it was just > getting > > swamped trying to recover & rebuild all the > writes? Not > > sure. > > > > Well, it worked eventually, so I'm going to just > upgrade to > > 0.20.5 ASAP, and try to be more careful about how much > I > > write to stdout. > > > > Take care, > > -stu > > > > --- On Tue, 7/13/10, Stuart Smith <[email protected]> > > wrote: > > > > > From: Stuart Smith <[email protected]> > > > Subject: Re: Trying to write too much to stdout > > destabilizes cluster across reboots > > > To: [email protected] > > > Date: Tuesday, July 13, 2010, 9:10 PM > > > One follow up. > > > > > > I noticed the master was saying this as well: > > > > > > 2010-07-13 17:48:17,595 WARN > > > org.apache.hadoop.hbase.master.HMaster: > Processing > > pending > > > operations: ProcessRegionClose of > > > > > > filestore,5c4ee356835d82de844600895a7df1595c8b53bd378436f6e8d2a1a99acb550c,1279043794914, > > > false, reassign: true > > > java.lang.RuntimeException: > > java.lang.NullPointerException > > > at > > > > > > org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:96) > > > at > > > > > > org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) > > > at > > > > > > org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:494) > > > at > > > > > > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:429) > > > Caused by: java.lang.NullPointerException > > > > > > Which lead me to this: > > > > > > https://issues.apache.org/jira/browse/HBASE-2428 > > > > > > I guess I shouldn't have put off upgrading to > 0.20.5 > > :( > > > I'm curious if by this comment: > > > > > > "Since delayqueue is processed first, all we did > was > > try to > > > process the close even though in the other more > live > > todo > > > queue, we might have the process of the > regionserver > > > shutdown waiting, the process of the regionserver > that > > was > > > carrying .meta. We'd never recover. " > > > > > > They mean never as in _never_ or just as in until > the > > next > > > reboot. Or is my cluster (and my data) dead and > gone > > now? > > > > > > Take care, > > > -stu > > > > > > --- On Tue, 7/13/10, Stuart Smith <[email protected]> > > > wrote: > > > > > > > From: Stuart Smith <[email protected]> > > > > Subject: Trying to write too much to stdout > > > destabilizes cluster across reboots > > > > To: [email protected] > > > > Date: Tuesday, July 13, 2010, 8:58 PM > > > > Hello, > > > > > > > > I have a M/R task running over hbase > > > > tables. I added > > > > a one line of output in my reduce tasks > > running > > > over a > > > > set > > > > of tens of millions of records. This was > a > > > little silly, > > > > but > > > > I was kind of curious what would happen. > > > > > > > > Well the reduce ran very slowly, and > > eventually, > > > I had to > > > > kill the tasks, and I re-started the > whole > > > cluster, as > > > > other > > > > clients were slowing up considerably, and > the > > > M/R job > > > > started dumping all the stdout directly to > the > > > console it > > > > was running on. > > > > > > > > I've re-started it twice, but I couldn't > get > > > hbase to > > > > come > > > > backup (fully). And when I shut down, the > > region > > > servers > > > > will go down when I tell them, but the > master > > > needs to be > > > > killed with -s 9 (SIG_KILL? I'm a little > > rusty). > > > It > > > > eventually seemed to hobble towards > coming > > back > > > up.. it > > > > took > > > > a looong.., with messages like: > > > > > > > > > > > > 2010-07-13 16:32:47,722 INFO > > > > org.apache.hadoop.hdfs.DFSClient: > Abandoning > > > block > > > > blk_-9203944767743886537_123926 > > > > 2010-07-13 16:32:48,207 DEBUG > > > > > org.apache.hadoop.hbase.regionserver.HLog: > > > Waiting for > > > > hlog > > > > writers to terminate, iteration #93 > > > > 2010-07-13 16:32:52,205 INFO > > > > org.apache.hadoop.hdfs.DFSClient: > Exception in > > > > createBlockOutputStream > java.io.EOFException > > > > 2010-07-13 16:32:52,205 INFO > > > > org.apache.hadoop.hdfs.DFSClient: > Abandoning > > > block > > > > blk_-7742458689391305082_123926 > > > > 2010-07-13 16:32:53,072 INFO > > > > org.apache.hadoop.hdfs.DFSClient: > Exception in > > > > createBlockOutputStream > java.io.EOFException > > > > 2010-07-13 16:32:53,072 INFO > > > > org.apache.hadoop.hdfs.DFSClient: > Abandoning > > > block > > > > blk_-281711885322381263_123926 > > > > 2010-07-13 16:32:53,207 DEBUG > > > > > org.apache.hadoop.hbase.regionserver.HLog: > > > Waiting for > > > > hlog > > > > writers to terminate, iteration #94 > > > > 2010-07-13 16:32:53,726 INFO > > > > org.apache.hadoop.hdfs.DFSClient: > Exception in > > > > createBlockOutputStream > java.io.IOException: > > Bad > > > connect > > > > ack > > > > with firstBadLink 192.168.193.155:50010 > > > > 2010-07-13 16:32:53,726 INFO > > > > org.apache.hadoop.hdfs.DFSClient: > Abandoning > > > block > > > > blk_-2730636850915812524_123926 > > > > > > > > This just seems odd.. dumping that much > > standard > > > output > > > > was > > > > kind silly, but I wouldn't think it would > > wedge > > > the > > > > cluster > > > > like this... or did something else > happen? > > > Everything has > > > > been fine for a while until I did this. > > > > > > > > I let the cluster sit around for a while, > and > > it > > > did come > > > > back for a little while, after printing > out > > this > > > over and > > > > over (on > > > > master): > > > > > > > > da7738a3ad9,1279039241464 from > > > > ubuntu-hadoop-2,60020,1279066792598; 1 of > 1 > > > > 2010-07-13 17:28:30,897 INFO > > > > > org.apache.hadoop.hbase.master.ServerManager: > > > Processing > > > > MSG_REPORT_PROCESS_OPEN: > > > > > > > > > > > > > > filemetadata,unknown_23bf6f189f81218a877a41d1ccd82751527abf19beb61281b8a0f6475e682764,1278533813300 > > > > from ubuntu-hadoop-4,60020,1279066792615; > 1 of > > > 3 > > > > 2010-07-13 17:28:30,897 INFO > > > > > org.apache.hadoop.hbase.master.ServerManager: > > > Processing > > > > MSG_REPORT_PROCESS_OPEN: > > > > > > > > > > > > > > filestore,7f45c6a2f5c7d09ae8618bf6d0db5b04f346c583a1dbf24d9db135f2c8da9b57,1279050466715 > > > > from ubuntu-hadoop-4,60020,1279066792615; > 2 of > > > 3 > > > > 2010-07-13 17:28:30,897 INFO > > > > > org.apache.hadoop.hbase.master.ServerManager: > > > Processing > > > > MSG_REPORT_PROCESS_OPEN: .META.,,1 from > > > > ubuntu-hadoop-4,60020,1279066792615; 3 of > 3 > > > > > > > > But regionservers kept coming down, and > > bringing > > > down the > > > > .META region.. eventually the master > seemed to > > > recover, > > > > and > > > > I would bring the regionservers back up > > > individually.. > > > > one > > > > of them would soon crash again, repeating > the > > > cycle. And > > > > the > > > > master would would die with this on the > web > > > page: > > > > > > > > Trying to contact region server > > > 192.168.193.155:60020 for > > > > region .META.,,1, row '', but failed after > 3 > > > attempts. > > > > Exceptions: > > > > java.net.ConnectException: Connection > refused > > > > java.net.ConnectException: Connection > refused > > > > java.net.ConnectException: Connection > refused > > > > > > > > RequestURI=/master.jsp > > > > > > > > The regionservers appear to start dying > with > > > this > > > > statement: > > > > > > > > > > > > 010-07-13 02:12:59,257 DEBUG > > > > > > > > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: > > > > Compaction requested for region > > > > > > > > > > > > > > filestore,5a1ec52137fb8bb38affb5ca034ec3817c6611adc2f5a2b399a4ab92effcdedd,1278970890717/1436533481 > > > > because: > > > regionserver/192.168.193.145:60020.cacheFlusher > > > > 2010-07-13 02:13:19,336 WARN > > > > org.apache.hadoop.hdfs.DFSClient: Failed > to > > > connect to > > > > /192.168.193.145:50010 for file > > > > > > > /hbase/.META./1028785192/info/3836661676936096933 > for > > > > block > > > > -649923918996291451:java.io.EOFException > > > > at > > > > > > > > > > > > > > java.io.DataInputStream.readShort(DataInputStream.java:298) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1384) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:1853) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1922) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:101) > > > > > > > > > > > > Note that it is failing to contact > > > 192.168.193.145 ... > > > > this > > > > is the ip of the node itself (the dying > one). > > > > > > > > There's a long sequence of exceptions, and > the > > > > regionservers death ends with this: > > > > > > > > 010-07-13 17:36:57,068 INFO > > > > org.apache.zookeeper.ClientCnxn: > EventThread > > > shut down > > > > 2010-07-13 17:36:57,069 ERROR > > > > org.apache.hadoop.hdfs.DFSClient: > Exception > > > closing file > > > > > > > > > > > > > > /hbase/filestore/compaction.dir/1408434857/4172852130952668434 > > > > : java.io.EOFException > > > > java.io.EOFException > > > > at > > > > > > > > > > > > > > java.io.DataInputStream.readByte(DataInputStream.java:250) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319) > > > > at > > > > > > > > org.apache.hadoop.io.Text.readString(Text.java:400) > > > > at > > > > > > > > > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream > > > > > > > > It just seems like a lot of trouble from > one > > > little > > > > System.out.println() statement.. > > > > > > > > Restarting hbase just repeats the cycle > "Bad > > > connect ack > > > > with firstBadLink", eventual recovery, > > regionservers > > > start > > > > crashing, cluster becomes unusable.. > > > > > > > > In summary, it would seem printing out > too > > much > > > data to > > > > stdout can render the whole cluster > unstable - > > > across > > > > reboots - and I haven't figured out how > to > > > stabilize it > > > > again. > > > > > > > > Take care, > > > > -stu > > > > > > > > PS. The address for the mailing list on > the > > > hbase page is > > > > wrong. It shows: > > > > > > > > [email protected] > > > > But it should be > > > > [email protected]. > > > > I hope! (If this mail goes through, I got > it > > right). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
