I have a periodic process that bulk incremental loads a set of files each time
into my db. The last few runs have been resulting in bulk load failures
complaining of RetriesExhausted. (I am running the last release of 0.89)
Exception in thread "main"
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server b5120229.yst.yahoo.net:60020 for region
vidhyash_test,r:com#mop#lady!/beauty/hair/2010-07-11/136898.shtml!http,1292936192308.7f7e7521764636e108de079799ad9e44.,
row 'r:com#mop#lady!/star/2010-06-10/131380.shtml!http', but failed after 10
attempts.
I looked into the logs of the particular regionserver and I noticed that one of
IPC handlers complains of an output error and throws an exception and after
that, all it does is just validate hfiles whenever there is an attempt at a
bulk incremental load. And, it isnt accessible even through the web interface..
But the region server is still alive according to the master/zk. Can you let me
know what the problem is? (Below is the log where the problem arose).
2010-12-22 21:31:57,679 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117
for inclusion in store metadata region
vidhyash_test,r:jp#co#yahoo#auctions#page17!/jp/show/reviews?aID=v54204554!http,1292936187861.9ad1eccc9cf7f82282757e2b82c45559.2010-12-22
21:31:57,680 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block
blk_8607417804107886121_8839017 from any node: java.io.IOException: No live
nodes contain current block
2010-12-22 21:31:59,610 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
LRU Stats: total=19.67 MB, free=2.32 GB, max=2.34 GB, blocks=0,
accesses=1654363, hits=0, hitRatio=0.00%%, evictions=0, evicted=0,
evictedPerRun=NaN2010-12-22 21:32:00,684 INFO org.apache.hadoop.hdfs.DFSClient:
Could not obtain block blk_8607417804107886121_8839017 from any node:
java.io.IOException: No
live nodes contain current block2010-12-22 21:32:03,687 INFO
org.apache.hadoop.hdfs.DFSClient: Could not obtain block
blk_8607417804107886121_8839017 from any node: java.io.IOException: No live
nodes contain current block
2010-12-22 21:32:06,691 DEBUG org.apache.hadoop.hbase.regionserver.Store: HFile
bounds: first=r:jp#co#yahoo#auctions#page17#www!/jp/auction/v12791536!http
last=r:jp#co#yahoo#auctions#page19!/jp/show/discussion?aID=x120219484&u=chikyuud!http2010-12-22
21:32:06,691 DEBUG org.apache.hadoop.hbase.regionserver.Store: Region bounds:
first=r:jp#co#yahoo#auctions#page17!/jp/show/reviews?aID=v54204554!h
ttp
last=r:jp#co#yahoo#auctions#page19!/jp/show/reviews?aID=x144625371!http2010-12-22
21:32:06,691 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming bulk
load file
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117
to
hdfs://b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata/643287123673932876
2010-12-22 21:32:06,695 INFO org.apache.hadoop.hbase.regionserver.Store: Moved
hfile /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632
567128036272117 into store directory
hdfs://b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata
- updating store file list.2010-12-22 21:32:06,695 INFO
org.apache.hadoop.hbase.regionserver.Store: Successfully loaded store file
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117
into store metadata (new location:
hdfs://b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata/643287123673932876)2010-12-22
21:32:06,695 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder,
call
bulkLoadHFile(/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117,
[...@2b6105a8, [...@6eba6ed7) from 74.6.71.45:52379: output error2010-12-22
21:32:06,696 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 27 on
60020 caught: java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1224)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:708)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:773)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1035)
2010-12-22 21:35:33,312 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2820295752884341300
for inclusion in store content region
vidhyash_test,r:com#careerbuilder#engineering!/en.ic/Texas_Senior-Engineer.htm!http,1292936194810.0658e436cc625b2c786ef80a5dbe4203.
2010-12-22 21:35:41,324 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2820295752884341300
for inclusion in store content region
vidhyash_test,r:com#careerbuilder#engineering!/en.ic/Texas_Senior-Engineer.htm!http,1292936194810.0658e436cc625b2c786ef80a5dbe4203.
2010-12-22 21:35:54,128 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/291138288447336298
for inclusion in store content region
vidhyash_test,r:la#net#kpl#www!/english/news/edn13.htm!http,1292936187014.695817b0e3a8c894240668db0448f8bf.
2010-12-22 21:35:54,529 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/2731042733857819920
for inclusion in store metadata region
vidhyash_test,r:com#homebargear#www!/irish-gift-set.html!http,1292936193313.99fa66ab17756ce4ce5ba3a0d8ee8799.
2010-12-22 21:35:54,823 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/1898525064358474151
for inclusion in store metadata region
vidhyash_test,r:fr#dazibaoueb#www!/tag.php?tag=DESINFORMATION!http,1292936188821.85b144d3f968029a903506bdb4e60cf7.
2010-12-22 21:35:55,198 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2942947977847604920
for inclusion in store content region
vidhyash_test,r:com#pld#mosc95#www!/projects02/ww2/germantanks.html!http,1292936191711.8e3d5df4c05c60f674a7b78474f83eea.
2010-12-22 21:35:57,299 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/2776824162022751481
for inclusion in store metadata region
vidhyash_test,r:cn#com#sina#news!/c/2006-07-12/192710404866.shtml!http,1292936195947.b3b27d1cc94a6378ab4da90acad4efbf.
2010-12-22 21:35:57,547 INFO org.apache.hadoop.hbase.regionserver.Store:
Validating hfile at
/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/1753280038544504583
for inclusion in store metadata region
vidhyash_test,r:com#yoka#space!/blog/34726!http,1292936189782.714fc4e266abca11f578fd90a3561337.
