I'm getting the following exception when using Drill. Asked on Drill mailing
list a few days back; no one had any ideas.
As near as I can tell, the block is there and retrievable. Any thoughts on what
could be causing this exception.
This is the stack trace:
Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain
block: BP-1605794487-10.0.1.3-1435700184285:blk_1073829560_90288
file=/warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
at
org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:880)
~[hadoop-hdfs-2.4.1.jar:na]
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:560)
~[hadoop-hdfs-2.4.1.jar:na]
at
org.apache.hadoop.hdfs.DFSInputStream.seekToNewSource(DFSInputStream.java:1443)
~[hadoop-hdfs-2.4.1.jar:na]
at
org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:766)
~[hadoop-hdfs-2.4.1.jar:na]
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:796)
~[hadoop-hdfs-2.4.1.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:837)
~[hadoop-hdfs-2.4.1.jar:na]
at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.7.0_79]
at
org.apache.drill.exec.store.dfs.DrillFSDataInputStream$WrappedInputStream.read(DrillFSDataInputStream.java:211)
~[drill-java-exec-1.1.0.jar:1.1.0]
at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.7.0_79]
at
parquet.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
~[parquet-format-2.1.1-drill-r1.jar:na]
... 40 common frames omitted
This is a two node cluster, both nodes can retrieve the file.
root@twig03:/opt/drill/log# hdfs dfs -get
/warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
/tmp/.
root@twig03:/opt/drill/log# ls /tmp/*.parquet
/tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
root@twig04:~# hdfs dfs -get
/warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
/tmp/.
root@twig04:~# ls /tmp/*.parquet
/tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
hdfs reports no missing blocks:
root@twig04:~# hdfs dfsadmin -report
Configured Capacity: 7856899358720 (7.15 TB)
Present Capacity: 7856899358720 (7.15 TB)
DFS Remaining: 4310190084096 (3.92 TB)
DFS Used: 3546709274624 (3.23 TB)
DFS Used%: 45.14%
Under replicated blocks: 23854
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Live datanodes (2):
Name: 10.0.1.4:50010 (twig04.twigs)
Hostname: twig04.twigs
Decommission Status : Normal
Configured Capacity: 3928449679360 (3.57 TB)
DFS Used: 1773355933696 (1.61 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 2155093745664 (1.96 TB)
DFS Used%: 45.14%
DFS Remaining%: 54.86%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 3
Last contact: Tue Sep 15 11:18:00 EDT 2015
Name: 10.0.1.3:50010 (twig03.twigs)
Hostname: twig03.twigs
Decommission Status : Normal
Configured Capacity: 3928449679360 (3.57 TB)
DFS Used: 1773353340928 (1.61 TB)
Non DFS Used: 0 (0 B)
DFS Remaining: 2155096338432 (1.96 TB)
DFS Used%: 45.14%
DFS Remaining%: 54.86%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Sep 15 11:18:02 EDT 2015
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
[email protected]<mailto:[email protected]>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you
print.
This email may contain confidential and privileged material for the sole use of
the intended recipient. Any review, use, distribution or disclosure by others
is strictly prohibited. If you are not the intended recipient (or authorized to
receive for the recipient), please contact the sender by reply email and delete
all copies of this message.
Please click
here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
Company Registration Information.
