RegionServer stuck: HLog: Could not append. Requesting close of log
java.io.IOException: Could not get block locations. Aborting...
-----------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-930
URL: https://issues.apache.org/jira/browse/HBASE-930
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
HDFS went wonky. Manifest itself in regionserver with below: i.e. first can't
replicate and then the exceptions every time we append and then every time we
try to rotate the log file. Exception is always: "IOException: Could not get
block locations. Aborting..." Meantime the HDFS is fixed but for whatever
reason the HRS just keeps on with the below failings; as though the error is
stuck in the DFSClient.
{code}
...
2008-10-15 02:54:59,867 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 1399814750/alternate_url store size is 2.9m
2008-10-15 02:54:59,868 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
enwiki_old,7qE8TUI5v-NIZu_9oZhaJF==,1221856888271 in 38sec
2008-10-15 02:59:08,507 INFO org.apache.hadoop.dfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/hbase/aa0-000-8.u.powerset.com/log_208.76.45.180_1223616861290_60020/hlog.dat.1224038569764
could only be re
plicated to 0 nodes, instead of 1
at
org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1117)
at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.ipc.Client.call(Client.java:715)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2440)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2323)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)
...
2008-10-15 02:59:11,357 WARN org.apache.hadoop.dfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/aa0-000-8.u.powerset.com/log_208.76.45.180_1223616861290_60020/hlog.dat.1224038569764
retries left 1
2008-10-15 02:59:14,566 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/hbase/aa0-000-8.u.powerset.com/log_208.76.45.180_1223616861290_60020/hlog.dat.122403
8569764 could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1117)
at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.ipc.Client.call(Client.java:715)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2440)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2323)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1912)
2008-10-15 02:59:14,566 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery
for block null bad datanode[0]
2008-10-15 02:59:14,566 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could
not append. Requesting close of log
java.io.IOException: Could not get block locations. Aborting...
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
2008-10-15 02:59:14,567 INFO org.apache.hadoop.hbase.regionserver.LogRoller:
Rolling hlog. Number of entries: 81
2008-10-15 02:59:14,567 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8
on 60020, call batchUpdate([EMAIL PROTECTED], row => Dnsw4RukHJ5-0FQddY35mF==,
{column => misc:upload_time, value => '...', column => page:mime, value =>
'...', c
olumn => page:content, value => '...', column => page:url, value => '...'}, -1)
from 208.76.44.183:59031: error: java.io.IOException: Could not get block
locations. Aborting...
java.io.IOException: Could not get block locations. Aborting...
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
2008-10-15 02:59:14,588 ERROR org.apache.hadoop.hbase.regionserver.LogRoller:
Log rolling failed
java.io.IOException: Could not get block locations. Aborting...
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
...
{code}
For now adding abort of regionserver if we can't close the log roller file. If
we can't close, then we're losing edits. If this behavior causes us close to
much, then need to dig in more.
Happened on pset cluster running 0.18.1 hadoop and 0.18.0 hbase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
