I'm still getting this, but it seems to go away after a while. It happens with multiple blocks.
I'm seeing the following lines in logs. I suspect it's related: 2015-09-09 12:38:12,556 [WorkManager-1147] WARN o.a.d.exec.rpc.control.WorkEventBus - Fragment 2a0f9fbf-f85b-39a3-dce2-891d7f62b385:1:40 not found in the work bus. Any help or pointers would be greatly appreciated. [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com<http://www.cisco.com/> [email protected]<mailto:[email protected]> Mobile: 865 724 4910 [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. From: "Grant Overby (groverby)" <[email protected]<mailto:[email protected]>> Date: Tuesday, September 8, 2015 at 11:17 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: BlockMissingException Drill is throwing a block missing exception; however, hdfs seems healthy. Thoughts? >From Drill's web ui after executing a query: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: BlockMissingException: Could not obtain block: BP-1605794487-10.0.1.3-1435700184285:blk_1073828756_89484 file=/warehouse2/completed/events/connection_events/1441707300/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet Fragment 1:19 [Error Id: 9a0b442a-e5e1-44f9-9200-fff9f59b990a on twig03.twigs:31010] Retrieving the file from hdfs: root@twig03:~# hdfs dfs -get /warehouse2/completed/events/connection_events/1441707300/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet /tmp/. root@twig03:~# ls /tmp/*.parquet /tmp/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet hdfs report: root@twig03:~# hdfs dfsadmin -report Configured Capacity: 7856899358720 (7.15 TB) Present Capacity: 7856899358720 (7.15 TB) DFS Remaining: 4567228003960 (4.15 TB) DFS Used: 3289671354760 (2.99 TB) DFS Used%: 41.87% Under replicated blocks: 22108 Blocks with corrupt replicas: 0 Missing blocks: 0 ------------------------------------------------- Live datanodes (2): Name: 10.0.1.4:50010 (twig04.twigs) Hostname: twig04.twigs Decommission Status : Normal Configured Capacity: 3928449679360 (3.57 TB) DFS Used: 1644836539588 (1.50 TB) Non DFS Used: 0 (0 B) DFS Remaining: 2283613139772 (2.08 TB) DFS Used%: 41.87% DFS Remaining%: 58.13% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 3 Last contact: Tue Sep 08 11:15:47 EDT 2015 Name: 10.0.1.3:50010 (twig03.twigs) Hostname: twig03.twigs Decommission Status : Normal Configured Capacity: 3928449679360 (3.57 TB) DFS Used: 1644834815172 (1.50 TB) Non DFS Used: 0 (0 B) DFS Remaining: 2283614864188 (2.08 TB) DFS Used%: 41.87% DFS Remaining%: 58.13% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Sep 08 11:15:47 EDT 2015 [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com<http://www.cisco.com/> [email protected]<mailto:[email protected]> Mobile: 865 724 4910 [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
