Hello,
I have a M/R task running over hbase tables. I added
a one line of output in my reduce tasks running over a set
of tens of millions of records. This was a little silly, but
I was kind of curious what would happen.
Well the reduce ran very slowly, and eventually, I had to
kill the tasks, and I re-started the whole cluster, as other
clients were slowing up considerably, and the M/R job
started dumping all the stdout directly to the console it
was running on.
I've re-started it twice, but I couldn't get hbase to come
backup (fully). And when I shut down, the region servers
will go down when I tell them, but the master needs to be
killed with -s 9 (SIG_KILL? I'm a little rusty). It
eventually seemed to hobble towards coming back up.. it took
a looong.., with messages like:
2010-07-13 16:32:47,722 INFO
org.apache.hadoop.hdfs.DFSClient: Abandoning block
blk_-9203944767743886537_123926
2010-07-13 16:32:48,207 DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Waiting for hlog
writers to terminate, iteration #93
2010-07-13 16:32:52,205 INFO
org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.io.EOFException
2010-07-13 16:32:52,205 INFO
org.apache.hadoop.hdfs.DFSClient: Abandoning block
blk_-7742458689391305082_123926
2010-07-13 16:32:53,072 INFO
org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.io.EOFException
2010-07-13 16:32:53,072 INFO
org.apache.hadoop.hdfs.DFSClient: Abandoning block
blk_-281711885322381263_123926
2010-07-13 16:32:53,207 DEBUG
org.apache.hadoop.hbase.regionserver.HLog: Waiting for hlog
writers to terminate, iteration #94
2010-07-13 16:32:53,726 INFO
org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.io.IOException: Bad connect ack
with firstBadLink 192.168.193.155:50010
2010-07-13 16:32:53,726 INFO
org.apache.hadoop.hdfs.DFSClient: Abandoning block
blk_-2730636850915812524_123926
This just seems odd.. dumping that much standard output was
kind silly, but I wouldn't think it would wedge the cluster
like this... or did something else happen? Everything has
been fine for a while until I did this.
I let the cluster sit around for a while, and it did come
back for a little while, after printing out this over and over (on
master):
da7738a3ad9,1279039241464 from
ubuntu-hadoop-2,60020,1279066792598; 1 of 1
2010-07-13 17:28:30,897 INFO
org.apache.hadoop.hbase.master.ServerManager: Processing
MSG_REPORT_PROCESS_OPEN:
filemetadata,unknown_23bf6f189f81218a877a41d1ccd82751527abf19beb61281b8a0f6475e682764,1278533813300
from ubuntu-hadoop-4,60020,1279066792615; 1 of 3
2010-07-13 17:28:30,897 INFO
org.apache.hadoop.hbase.master.ServerManager: Processing
MSG_REPORT_PROCESS_OPEN:
filestore,7f45c6a2f5c7d09ae8618bf6d0db5b04f346c583a1dbf24d9db135f2c8da9b57,1279050466715
from ubuntu-hadoop-4,60020,1279066792615; 2 of 3
2010-07-13 17:28:30,897 INFO
org.apache.hadoop.hbase.master.ServerManager: Processing
MSG_REPORT_PROCESS_OPEN: .META.,,1 from
ubuntu-hadoop-4,60020,1279066792615; 3 of 3
But regionservers kept coming down, and bringing down the
.META region.. eventually the master seemed to recover, and
I would bring the regionservers back up individually.. one
of them would soon crash again, repeating the cycle. And the
master would would die with this on the web page:
Trying to contact region server 192.168.193.155:60020 for
region .META.,,1, row '', but failed after 3 attempts.
Exceptions:
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
RequestURI=/master.jsp
The regionservers appear to start dying with this
statement:
010-07-13 02:12:59,257 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Compaction requested for region
filestore,5a1ec52137fb8bb38affb5ca034ec3817c6611adc2f5a2b399a4ab92effcdedd,1278970890717/1436533481
because: regionserver/192.168.193.145:60020.cacheFlusher
2010-07-13 02:13:19,336 WARN
org.apache.hadoop.hdfs.DFSClient: Failed to connect to
/192.168.193.145:50010 for file
/hbase/.META./1028785192/info/3836661676936096933 for block
-649923918996291451:java.io.EOFException
at
java.io.DataInputStream.readShort(DataInputStream.java:298)
at
org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1384)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:1853)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1922)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46)
at
org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:101)
Note that it is failing to contact 192.168.193.145 ... this
is the ip of the node itself (the dying one).
There's a long sequence of exceptions, and the
regionservers death ends with this:
010-07-13 17:36:57,068 INFO
org.apache.zookeeper.ClientCnxn: EventThread shut down
2010-07-13 17:36:57,069 ERROR
org.apache.hadoop.hdfs.DFSClient: Exception closing file
/hbase/filestore/compaction.dir/1408434857/4172852130952668434
: java.io.EOFException
java.io.EOFException
at
java.io.DataInputStream.readByte(DataInputStream.java:250)
at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at
org.apache.hadoop.io.Text.readString(Text.java:400)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream
It just seems like a lot of trouble from one little
System.out.println() statement..
Restarting hbase just repeats the cycle "Bad connect ack with firstBadLink",
eventual recovery, regionservers start crashing, cluster becomes unusable..
In summary, it would seem printing out too much data to
stdout can render the whole cluster unstable - across
reboots - and I haven't figured out how to stabilize it
again.
Take care,
-stu
PS. The address for the mailing list on the hbase page is wrong. It shows:
[email protected]
But it should be
[email protected]. I hope! (If this mail goes through, I got it right).