On Thu, Jan 27, 2011 at 11:56 PM, Vidhyashankar Venkataraman <[email protected]> wrote: > 2011-01-28 07:35:49,866 INFO org.apache.hadoop.hbase.master.MasterFileSystem: > Log folder > hdfs://b3110120.yst.yahoo.net:4600/hbase/.logs/b3110270.yst.yahoo.net,60020,1296199618314 > belongs to an existing region server > 2011-01-28 07:35:49,866 INFO org.apache.hadoop.hbase.master.MasterFileSystem: > Log folder > hdfs://b3110120.yst.yahoo.net:4600/hbase/.logs/b3110271.yst.yahoo.net,60020,1296183218615 > doesn't belong to a known region server, splitting
Vidhya: This looks like a master that is starting up. Is that right? Is this new cluster start or master joining an already running cluster (looks like former). > 2011-01-28 07:35:49,867 INFO > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting 1 hlog(s) in > hdfs://b3110120.yst.yahoo.net:4600/hbase/.logs/b3110271.yst.yahoo.net,60020,1296183218615 > 2011-01-28 07:35:49,867 DEBUG > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread > Thread[WriterThread-0,5,main]: starting > 2011-01-28 07:35:49,867 DEBUG > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread > Thread[WriterThread-1,5,main]: starting > 2011-01-28 07:35:49,867 DEBUG > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 1 of 1: > hdfs://b3110120.yst.yahoo.net:4600/hbase/.logs/b3110271.yst.yahoo.net,60020,1296183218615/b3110271.yst.yahoo.net%3A60020.1296183219266, > length=0 > 2011-01-28 07:35:49,867 DEBUG > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread > Thread[WriterThread-2,5,main]: starting > 2011-01-28 07:35:49,867 WARN org.apache.hadoop.hbase.util.FSUtils: Running on > HDFS without append enabled may result in data loss Can you fix this? Can you run w/ a working append? > 2011-01-28 07:35:49,867 WARN > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File > hdfs://b3110120.yst.yahoo.net:4600/hbase/.logs/b3110271.yst.yahoo.net,60020,1296183218615/b3110271.yst.yahoo.net%3A60020.1296183219266 > might be still open, length is 0 > 2011-01-28 07:35:49,869 WARN > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Could not open > hdfs://b3110120.yst.yahoo.net:4600/hbase/.logs/b3110271.yst.yahoo.net,60020,1296183218615/b3110271.yst.yahoo.net%3A60020.1296183219266 > for reading. File is emptyjava.io.EOFException > 2011-01-28 07:35:49,875 INFO > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log > hdfs://b3110120.yst.yahoo.net:4600/hbase/.logs/b3110271.yst.yahoo.net,60020,1296183218615/b3110271.yst.yahoo.net%3A60020.1296183219266 > to > hdfs://b3110120.yst.yahoo.net:4600/hbase/.oldlogs/b3110271.yst.yahoo.net%3A60020.1296183219266 > 2011-01-28 07:35:49,877 INFO > org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split > writer threads to finish > Can you do 'kill -QUIT PID' and see if anything shows in the .out file for hbase? It looks like a master is hung for sure. My guess would be that we're making some presumption based on presence of append. Lets figure it and fix. Good on you V, St.Ack
