Hi Grant, Do you see any other errors in the logs ?
I don't think the WorkEventBus warning has anything to do with the issue. It's a warning you can expect to see for failed/cancelled queries. Thanks On Wed, Sep 9, 2015 at 10:32 AM, Grant Overby (groverby) <[email protected] > wrote: > I'm still getting this, but it seems to go away after a while. It happens > with multiple blocks. > > I'm seeing the following lines in logs. I suspect it's related: > > > 2015-09-09 12:38:12,556 [WorkManager-1147] WARN > o.a.d.exec.rpc.control.WorkEventBus - Fragment > 2a0f9fbf-f85b-39a3-dce2-891d7f62b385:1:40 not found in the work bus. > > Any help or pointers would be greatly appreciated. > > [ > http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726 > ] > > Grant Overby > Software Engineer > Cisco.com<http://www.cisco.com/> > [email protected]<mailto:[email protected]> > Mobile: 865 724 4910 > > > > > > > [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think > before you print. > > This email may contain confidential and privileged material for the sole > use of the intended recipient. Any review, use, distribution or disclosure > by others is strictly prohibited. If you are not the intended recipient (or > authorized to receive for the recipient), please contact the sender by > reply email and delete all copies of this message. > > Please click here< > http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for > Company Registration Information. > > > > > > From: "Grant Overby (groverby)" <[email protected]<mailto: > [email protected]>> > Date: Tuesday, September 8, 2015 at 11:17 AM > To: "[email protected]<mailto:[email protected]>" < > [email protected]<mailto:[email protected]>> > Subject: BlockMissingException > > Drill is throwing a block missing exception; however, hdfs seems healthy. > Thoughts? > > From Drill's web ui after executing a query: > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > BlockMissingException: Could not obtain block: > BP-1605794487-10.0.1.3-1435700184285:blk_1073828756_89484 > file=/warehouse2/completed/events/connection_events/1441707300/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet > Fragment 1:19 [Error Id: 9a0b442a-e5e1-44f9-9200-fff9f59b990a on > twig03.twigs:31010] > > Retrieving the file from hdfs: > > root@twig03:~# hdfs dfs -get > /warehouse2/completed/events/connection_events/1441707300/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet > /tmp/. > > root@twig03:~# ls /tmp/*.parquet > > /tmp/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet > > hdfs report: > > root@twig03:~# hdfs dfsadmin -report > > Configured Capacity: 7856899358720 (7.15 TB) > > Present Capacity: 7856899358720 (7.15 TB) > > DFS Remaining: 4567228003960 (4.15 TB) > > DFS Used: 3289671354760 (2.99 TB) > > DFS Used%: 41.87% > > Under replicated blocks: 22108 > > Blocks with corrupt replicas: 0 > > Missing blocks: 0 > > > ------------------------------------------------- > > Live datanodes (2): > > > Name: 10.0.1.4:50010 (twig04.twigs) > > Hostname: twig04.twigs > > Decommission Status : Normal > > Configured Capacity: 3928449679360 (3.57 TB) > > DFS Used: 1644836539588 (1.50 TB) > > Non DFS Used: 0 (0 B) > > DFS Remaining: 2283613139772 (2.08 TB) > > DFS Used%: 41.87% > > DFS Remaining%: 58.13% > > Configured Cache Capacity: 0 (0 B) > > Cache Used: 0 (0 B) > > Cache Remaining: 0 (0 B) > > Cache Used%: 100.00% > > Cache Remaining%: 0.00% > > Xceivers: 3 > > Last contact: Tue Sep 08 11:15:47 EDT 2015 > > > > Name: 10.0.1.3:50010 (twig03.twigs) > > Hostname: twig03.twigs > > Decommission Status : Normal > > Configured Capacity: 3928449679360 (3.57 TB) > > DFS Used: 1644834815172 (1.50 TB) > > Non DFS Used: 0 (0 B) > > DFS Remaining: 2283614864188 (2.08 TB) > > DFS Used%: 41.87% > > DFS Remaining%: 58.13% > > Configured Cache Capacity: 0 (0 B) > > Cache Used: 0 (0 B) > > Cache Remaining: 0 (0 B) > > Cache Used%: 100.00% > > Cache Remaining%: 0.00% > > Xceivers: 2 > > Last contact: Tue Sep 08 11:15:47 EDT 2015 > > > > [ > http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726 > ] > > Grant Overby > Software Engineer > Cisco.com<http://www.cisco.com/> > [email protected]<mailto:[email protected]> > Mobile: 865 724 4910 > > > > > > > [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think > before you print. > > This email may contain confidential and privileged material for the sole > use of the intended recipient. Any review, use, distribution or disclosure > by others is strictly prohibited. If you are not the intended recipient (or > authorized to receive for the recipient), please contact the sender by > reply email and delete all copies of this message. > > Please click here< > http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for > Company Registration Information. > > > > > -- Abdelhakim Deneche Software Engineer <http://www.mapr.com/> Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
