Check xcievers too, giving more mem to HBase won't fix HDFS. J-D
On Wed, Jul 14, 2010 at 5:53 PM, Stuart Smith <[email protected]> wrote: > > Ok, back up again. > > This time, in addition to just watching & restarting, the error logs led me > to this: > > http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201004.mbox/%[email protected]%3e > > Which led me to up both HBASE_HEAP and HADOOP_HEAP from 1 -> 4 GB. Kind of > scary, since my regionservers only have 8GB of memory (NN/ZK/MASTER) has > 16GB. But it did seem to start up a little quicker. > > I was seeing about bunch of edits being recovered and then errors similar to > these: > > INFO org.apache.hadoop.hdfs.DFSClient: Exception in > createBlockOutputStream java.io.EOFException > INFO org.apache.hadoop.hdfs.DFSClient: Abandoning > block blk_ > > I already did the file handle thing a while ago. > > Let's see if this fixes it! > > Take care, > -stu > > --- On Wed, 7/14/10, Stuart Smith <[email protected]> wrote: > >> From: Stuart Smith <[email protected]> >> Subject: Cluster destabilizes recovering from crash >> To: [email protected] >> Date: Wednesday, July 14, 2010, 7:44 PM >> Hello All, >> >> Ok, I'm seeing similar behavior to what I mentioned >> yesterday, but the cause seems to be hbase not handling the >> Delete/Put load. >> >> I have: >> >> - A small cluster with 4 1950s as regionservers >> & 1 M600 as zookeeper & master. NTP is active, and >> the datetimes are in sync. >> >> - 1 java client with about 32 threads all uploading >> files to hbase. Pretty slow though, only 1-4 files a second, >> this due to non-hbase issues. >> >> - A M/R task that is iterating over a table, >> and gathering data about each row, and deleting rows with >> improperly formed keys. There are about 40 million rows, and >> probably around 7-10 million improperly formed keys. >> >> I do the validation & delete in the Map task. My >> connection to hbase is initialized in the Map class >> constructor. I noticed some map tasks are failing due to >> being unable to connect (to namenode proxy, or something). >> The number of failures is low compared to the total number >> of map tasks, so it's not every connection that's failing >> (not a network visibility issue). >> >> The hbase master eventually went down, and I restarted >> everything. Hbase now takes a long time coming back, and >> appears to be re-generating a lot of edits trying to >> recover. However, as it goes through the recovery process, >> region servers just start dying off, and I have to >> continuously restart them. >> >> Is this known behavior? Is hbase just kind of flaky on >> recovery at this point? Or did I do something wrong? Should >> I just not push it that hard with inserts & deletes? >> >> I went through this yesterday, and eventually it should >> stabilize and come back, but it takes several hours of >> sitting around and babysitting the cluster. >> >> I'm wondering if it would be bad to just write a cron job >> that calls my "log on and try to start all regionservers" >> script. Is there anything bad about trying to start >> regionservers via ./hbase start regionserver every minute or >> so? They die off pretty quick. >> >> Take care, >> -stu >> >> >> --- On Tue, 7/13/10, Stuart Smith <[email protected]> >> wrote: >> >> > From: Stuart Smith <[email protected]> >> > Subject: Re: Trying to write too much to stdout >> destabilizes cluster across reboots >> > To: [email protected] >> > Date: Tuesday, July 13, 2010, 9:38 PM >> > Last follow up: >> > >> > It appears just rebooting, babysitting regionservers, >> > watching it die.. and repeating the process >> eventually >> > recovered the cluster. >> > >> > I still haven't gone through all the errors very >> carefully. >> > I did have hundreds of threads of hbase clients >> uploading >> > data while I was running my M/R task that dumped the >> output. >> > Based on the logs, I'm wondering if it was just >> getting >> > swamped trying to recover & rebuild all the >> writes? Not >> > sure. >> > >> > Well, it worked eventually, so I'm going to just >> upgrade to >> > 0.20.5 ASAP, and try to be more careful about how much >> I >> > write to stdout. >> > >> > Take care, >> > -stu >> > >> > --- On Tue, 7/13/10, Stuart Smith <[email protected]> >> > wrote: >> > >> > > From: Stuart Smith <[email protected]> >> > > Subject: Re: Trying to write too much to stdout >> > destabilizes cluster across reboots >> > > To: [email protected] >> > > Date: Tuesday, July 13, 2010, 9:10 PM >> > > One follow up. >> > > >> > > I noticed the master was saying this as well: >> > > >> > > 2010-07-13 17:48:17,595 WARN >> > > org.apache.hadoop.hbase.master.HMaster: >> Processing >> > pending >> > > operations: ProcessRegionClose of >> > > >> > >> filestore,5c4ee356835d82de844600895a7df1595c8b53bd378436f6e8d2a1a99acb550c,1279043794914, >> > > false, reassign: true >> > > java.lang.RuntimeException: >> > java.lang.NullPointerException >> > > at >> > > >> > >> org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:96) >> > > at >> > > >> > >> org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63) >> > > at >> > > >> > >> org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:494) >> > > at >> > > >> > >> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:429) >> > > Caused by: java.lang.NullPointerException >> > > >> > > Which lead me to this: >> > > >> > > https://issues.apache.org/jira/browse/HBASE-2428 >> > > >> > > I guess I shouldn't have put off upgrading to >> 0.20.5 >> > :( >> > > I'm curious if by this comment: >> > > >> > > "Since delayqueue is processed first, all we did >> was >> > try to >> > > process the close even though in the other more >> live >> > todo >> > > queue, we might have the process of the >> regionserver >> > > shutdown waiting, the process of the regionserver >> that >> > was >> > > carrying .meta. We'd never recover. " >> > > >> > > They mean never as in _never_ or just as in until >> the >> > next >> > > reboot. Or is my cluster (and my data) dead and >> gone >> > now? >> > > >> > > Take care, >> > > -stu >> > > >> > > --- On Tue, 7/13/10, Stuart Smith <[email protected]> >> > > wrote: >> > > >> > > > From: Stuart Smith <[email protected]> >> > > > Subject: Trying to write too much to stdout >> > > destabilizes cluster across reboots >> > > > To: [email protected] >> > > > Date: Tuesday, July 13, 2010, 8:58 PM >> > > > Hello, >> > > > >> > > > I have a M/R task running over hbase >> > > > tables. I added >> > > > a one line of output in my reduce tasks >> > running >> > > over a >> > > > set >> > > > of tens of millions of records. This was >> a >> > > little silly, >> > > > but >> > > > I was kind of curious what would happen. >> > > > >> > > > Well the reduce ran very slowly, and >> > eventually, >> > > I had to >> > > > kill the tasks, and I re-started the >> whole >> > > cluster, as >> > > > other >> > > > clients were slowing up considerably, and >> the >> > > M/R job >> > > > started dumping all the stdout directly to >> the >> > > console it >> > > > was running on. >> > > > >> > > > I've re-started it twice, but I couldn't >> get >> > > hbase to >> > > > come >> > > > backup (fully). And when I shut down, the >> > region >> > > servers >> > > > will go down when I tell them, but the >> master >> > > needs to be >> > > > killed with -s 9 (SIG_KILL? I'm a little >> > rusty). >> > > It >> > > > eventually seemed to hobble towards >> coming >> > back >> > > up.. it >> > > > took >> > > > a looong.., with messages like: >> > > > >> > > > >> > > > 2010-07-13 16:32:47,722 INFO >> > > > org.apache.hadoop.hdfs.DFSClient: >> Abandoning >> > > block >> > > > blk_-9203944767743886537_123926 >> > > > 2010-07-13 16:32:48,207 DEBUG >> > > > >> org.apache.hadoop.hbase.regionserver.HLog: >> > > Waiting for >> > > > hlog >> > > > writers to terminate, iteration #93 >> > > > 2010-07-13 16:32:52,205 INFO >> > > > org.apache.hadoop.hdfs.DFSClient: >> Exception in >> > > > createBlockOutputStream >> java.io.EOFException >> > > > 2010-07-13 16:32:52,205 INFO >> > > > org.apache.hadoop.hdfs.DFSClient: >> Abandoning >> > > block >> > > > blk_-7742458689391305082_123926 >> > > > 2010-07-13 16:32:53,072 INFO >> > > > org.apache.hadoop.hdfs.DFSClient: >> Exception in >> > > > createBlockOutputStream >> java.io.EOFException >> > > > 2010-07-13 16:32:53,072 INFO >> > > > org.apache.hadoop.hdfs.DFSClient: >> Abandoning >> > > block >> > > > blk_-281711885322381263_123926 >> > > > 2010-07-13 16:32:53,207 DEBUG >> > > > >> org.apache.hadoop.hbase.regionserver.HLog: >> > > Waiting for >> > > > hlog >> > > > writers to terminate, iteration #94 >> > > > 2010-07-13 16:32:53,726 INFO >> > > > org.apache.hadoop.hdfs.DFSClient: >> Exception in >> > > > createBlockOutputStream >> java.io.IOException: >> > Bad >> > > connect >> > > > ack >> > > > with firstBadLink 192.168.193.155:50010 >> > > > 2010-07-13 16:32:53,726 INFO >> > > > org.apache.hadoop.hdfs.DFSClient: >> Abandoning >> > > block >> > > > blk_-2730636850915812524_123926 >> > > > >> > > > This just seems odd.. dumping that much >> > standard >> > > output >> > > > was >> > > > kind silly, but I wouldn't think it would >> > wedge >> > > the >> > > > cluster >> > > > like this... or did something else >> happen? >> > > Everything has >> > > > been fine for a while until I did this. >> > > > >> > > > I let the cluster sit around for a while, >> and >> > it >> > > did come >> > > > back for a little while, after printing >> out >> > this >> > > over and >> > > > over (on >> > > > master): >> > > > >> > > > da7738a3ad9,1279039241464 from >> > > > ubuntu-hadoop-2,60020,1279066792598; 1 of >> 1 >> > > > 2010-07-13 17:28:30,897 INFO >> > > > >> org.apache.hadoop.hbase.master.ServerManager: >> > > Processing >> > > > MSG_REPORT_PROCESS_OPEN: >> > > > >> > > > >> > > >> > >> filemetadata,unknown_23bf6f189f81218a877a41d1ccd82751527abf19beb61281b8a0f6475e682764,1278533813300 >> > > > from ubuntu-hadoop-4,60020,1279066792615; >> 1 of >> > > 3 >> > > > 2010-07-13 17:28:30,897 INFO >> > > > >> org.apache.hadoop.hbase.master.ServerManager: >> > > Processing >> > > > MSG_REPORT_PROCESS_OPEN: >> > > > >> > > > >> > > >> > >> filestore,7f45c6a2f5c7d09ae8618bf6d0db5b04f346c583a1dbf24d9db135f2c8da9b57,1279050466715 >> > > > from ubuntu-hadoop-4,60020,1279066792615; >> 2 of >> > > 3 >> > > > 2010-07-13 17:28:30,897 INFO >> > > > >> org.apache.hadoop.hbase.master.ServerManager: >> > > Processing >> > > > MSG_REPORT_PROCESS_OPEN: .META.,,1 from >> > > > ubuntu-hadoop-4,60020,1279066792615; 3 of >> 3 >> > > > >> > > > But regionservers kept coming down, and >> > bringing >> > > down the >> > > > .META region.. eventually the master >> seemed to >> > > recover, >> > > > and >> > > > I would bring the regionservers back up >> > > individually.. >> > > > one >> > > > of them would soon crash again, repeating >> the >> > > cycle. And >> > > > the >> > > > master would would die with this on the >> web >> > > page: >> > > > >> > > > Trying to contact region server >> > > 192.168.193.155:60020 for >> > > > region .META.,,1, row '', but failed after >> 3 >> > > attempts. >> > > > Exceptions: >> > > > java.net.ConnectException: Connection >> refused >> > > > java.net.ConnectException: Connection >> refused >> > > > java.net.ConnectException: Connection >> refused >> > > > >> > > > RequestURI=/master.jsp >> > > > >> > > > The regionservers appear to start dying >> with >> > > this >> > > > statement: >> > > > >> > > > >> > > > 010-07-13 02:12:59,257 DEBUG >> > > > >> > > >> > >> org.apache.hadoop.hbase.regionserver.CompactSplitThread: >> > > > Compaction requested for region >> > > > >> > > > >> > > >> > >> filestore,5a1ec52137fb8bb38affb5ca034ec3817c6611adc2f5a2b399a4ab92effcdedd,1278970890717/1436533481 >> > > > because: >> > > regionserver/192.168.193.145:60020.cacheFlusher >> > > > 2010-07-13 02:13:19,336 WARN >> > > > org.apache.hadoop.hdfs.DFSClient: Failed >> to >> > > connect to >> > > > /192.168.193.145:50010 for file >> > > > >> > > /hbase/.META./1028785192/info/3836661676936096933 >> for >> > > > block >> > > > -649923918996291451:java.io.EOFException >> > > > at >> > > > >> > > > >> > > >> > >> java.io.DataInputStream.readShort(DataInputStream.java:298) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1384) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:1853) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1922) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:101) >> > > > >> > > > >> > > > Note that it is failing to contact >> > > 192.168.193.145 ... >> > > > this >> > > > is the ip of the node itself (the dying >> one). >> > > > >> > > > There's a long sequence of exceptions, and >> the >> > > > regionservers death ends with this: >> > > > >> > > > 010-07-13 17:36:57,068 INFO >> > > > org.apache.zookeeper.ClientCnxn: >> EventThread >> > > shut down >> > > > 2010-07-13 17:36:57,069 ERROR >> > > > org.apache.hadoop.hdfs.DFSClient: >> Exception >> > > closing file >> > > > >> > > > >> > > >> > >> /hbase/filestore/compaction.dir/1408434857/4172852130952668434 >> > > > : java.io.EOFException >> > > > java.io.EOFException >> > > > at >> > > > >> > > > >> > > >> > >> java.io.DataInputStream.readByte(DataInputStream.java:250) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319) >> > > > at >> > > > >> > > >> org.apache.hadoop.io.Text.readString(Text.java:400) >> > > > at >> > > > >> > > > >> > > >> > >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream >> > > > >> > > > It just seems like a lot of trouble from >> one >> > > little >> > > > System.out.println() statement.. >> > > > >> > > > Restarting hbase just repeats the cycle >> "Bad >> > > connect ack >> > > > with firstBadLink", eventual recovery, >> > regionservers >> > > start >> > > > crashing, cluster becomes unusable.. >> > > > >> > > > In summary, it would seem printing out >> too >> > much >> > > data to >> > > > stdout can render the whole cluster >> unstable - >> > > across >> > > > reboots - and I haven't figured out how >> to >> > > stabilize it >> > > > again. >> > > > >> > > > Take care, >> > > > -stu >> > > > >> > > > PS. The address for the mailing list on >> the >> > > hbase page is >> > > > wrong. It shows: >> > > > >> > > > [email protected] >> > > > But it should be >> > > > [email protected]. >> > > > I hope! (If this mail goes through, I got >> it >> > right). >> > > > >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > > >> > >> > >> > >> > >> >> >> >> > > > >
