This failure prints more to the log than the first failure from my original email. The log from this failure is here: http://pastebin.com/GQMxdckJ
0: jdbc:drill:> SELECT * . . . . . . . > FROM dfs.events.`connection_events` AS rna_flow_stats . . . . . . . > where dir0 = 1441747800 . . . . . . . > LIMIT 25; Error: SYSTEM ERROR: BlockMissingException: Could not obtain block: BP-1605794487-10.0.1.3-1435700184285:blk_1073829563_90291 file=/warehouse2/completed/events/connection_events/1441747800/1441747805646-9-e4e1c8bd-a53a-46c1-8969-ea0ddaa74c16.parquet Fragment 1:7 [Error Id: f26a3f0b-b52b-45a0-bc1c-8172606be740 on twig03.twigs:31010] (state=,code=0) 0: jdbc:drill:> [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com<http://www.cisco.com/> [email protected]<mailto:[email protected]> Mobile: 865 724 4910 [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. From: "Grant Overby (groverby)" <[email protected]<mailto:[email protected]>> Date: Wednesday, September 9, 2015 at 3:44 PM To: Abdel Hakim Deneche <[email protected]<mailto:[email protected]>>, user <[email protected]<mailto:[email protected]>> Subject: Re: BlockMissingException I didn’t see the previous error in log before, but that could be my fault. The exception in the log is correct. /warehouse2/completed/events/connection_events/1441290600 is indeed a directory, not a regular file; however, I don’t understand why Drill would expect it to be a regular file. >From CLI: 0: jdbc:drill:> SELECT * . . . . . . . > FROM dfs.events.`connection_events` AS rna_flow_stats . . . . . . . > where dir0 = 1441747800 . . . . . . . > ORDER BY rna_flow_stats.initiator_ip DESC, rna_flow_stats.first_packet_second DESC, rna_flow_stats.last_packet_second DESC . . . . . . . > LIMIT 25; java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: BlockMissingException: Could not obtain block: BP-1605794487-10.0.1.3-1435700184285:blk_1073829562_90290 file=/warehouse2/completed/events/connection_events/1441747800/1441747805713-13-eff51ede-e264-4a77-b1dd-4813f3f4d518.parquet Fragment 3:29 [Error Id: 79c00987-6de8-41a9-a666-4af771f32c7e on twig03.twigs:31010] at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73) at sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87) at sqlline.TableOutputFormat.print(TableOutputFormat.java:118) at sqlline.SqlLine.print(SqlLine.java:1583) at sqlline.Commands.execute(Commands.java:852) at sqlline.Commands.sql(Commands.java:751) at sqlline.SqlLine.dispatch(SqlLine.java:738) at sqlline.SqlLine.begin(SqlLine.java:612) at sqlline.SqlLine.start(SqlLine.java:366) at sqlline.SqlLine.main(SqlLine.java:259) 0: jdbc:drill:> >From log: 2015-09-09 15:35:00,579 [2a0f761a-eac6-5399-a93e-f382cfd00cd1:foreman] INFO o.a.d.e.store.mock.MockStorageEngine - Failure while attempting to check for Parquet metadata file. java.io.FileNotFoundException: Path is not a file: /warehouse2/completed/events/connection_events/1441290600 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:58) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1903) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1844) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1824) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1796) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:554) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:364) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_79] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_79] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_79] at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_79] at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) ~[hadoop-common-2.4.1.jar:na] at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) ~[hadoop-common-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1144) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1132) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1122) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1295) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.4.1.jar:na] at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) ~[hadoop-common-2.4.1.jar:na] at org.apache.drill.exec.store.dfs.DrillFileSystem.open(DrillFileSystem.java:128) ~[drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.store.dfs.BasicFormatMatcher$MagicStringMatcher.matches(BasicFormatMatcher.java:139) ~[drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.store.dfs.BasicFormatMatcher.isReadable(BasicFormatMatcher.java:108) ~[drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.store.parquet.ParquetFormatPlugin$ParquetFormatMatcher.isDirReadable(ParquetFormatPlugin.java:226) [drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.store.parquet.ParquetFormatPlugin$ParquetFormatMatcher.isReadable(ParquetFormatPlugin.java:206) [drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:291) [drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:118) [drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96) [drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90) [drill-java-exec-1.1.0.jar:1.1.0] at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:241) [drill-java-exec-1.1.0.jar:1.1.0] at org.apache.calcite.jdbc.SimpleCalciteSchema.getTable(SimpleCalciteSchema.java:117) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom(CalciteCatalogReader.java:117) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:100) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:1) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:75) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:124) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:86) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:874) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:863) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2745) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2730) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:2953) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:86) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:874) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:863) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:210) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:837) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:552) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:174) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] at org.apache.calcite.prepare.PlannerImpl.validateAndGetType(PlannerImpl.java:185) [calcite-core-1.1.0-drill-r14.jar:1.1.0-drill-r14] I tried the following similar queries: This was successful: 0: jdbc:drill:> SELECT count(*) . . . . . . . > FROM dfs.events.`connection_events` AS rna_flow_stats . . . . . . . > where dir0 = 1441747800; This failed: 0: jdbc:drill:> SELECT * . . . . . . . > FROM dfs.events.`connection_events` AS rna_flow_stats . . . . . . . > where dir0 = 1441747800 . . . . . . . > LIMIT 25; Error: SYSTEM ERROR: BlockMissingException: Could not obtain block: BP-1605794487-10.0.1.3-1435700184285:blk_1073829567_90295 file=/warehouse2/completed/events/connection_events/1441747800/1441747810272-4-609d62fe-b81b-4a35-acc9-42959a84a78d.parquet Fragment 1:22 [Error Id: fbfb18e4-67d8-44c9-8a60-9f48d3798f4e on twig04.twigs:31010] (state=,code=0) [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com<http://www.cisco.com/> [email protected]<mailto:[email protected]> Mobile: 865 724 4910 [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. From: Abdel Hakim Deneche <[email protected]<mailto:[email protected]>> Date: Wednesday, September 9, 2015 at 1:36 PM To: user <[email protected]<mailto:[email protected]>> Cc: "Grant Overby (groverby)" <[email protected]<mailto:[email protected]>> Subject: Re: BlockMissingException Hi Grant, Do you see any other errors in the logs ? I don't think the WorkEventBus warning has anything to do with the issue. It's a warning you can expect to see for failed/cancelled queries. Thanks On Wed, Sep 9, 2015 at 10:32 AM, Grant Overby (groverby) <[email protected]<mailto:[email protected]>> wrote: I'm still getting this, but it seems to go away after a while. It happens with multiple blocks. I'm seeing the following lines in logs. I suspect it's related: 2015-09-09 12:38:12,556 [WorkManager-1147] WARN o.a.d.exec.rpc.control.WorkEventBus - Fragment 2a0f9fbf-f85b-39a3-dce2-891d7f62b385:1:40 not found in the work bus. Any help or pointers would be greatly appreciated. [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com<http://www.cisco.com/> [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> Mobile: 865 724 4910<tel:865%20724%204910> [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. From: "Grant Overby (groverby)" <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Date: Tuesday, September 8, 2015 at 11:17 AM To: "[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>" <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> Subject: BlockMissingException Drill is throwing a block missing exception; however, hdfs seems healthy. Thoughts? >From Drill's web ui after executing a query: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: BlockMissingException: Could not obtain block: BP-1605794487-10.0.1.3-1435700184285:blk_1073828756_89484 file=/warehouse2/completed/events/connection_events/1441707300/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet Fragment 1:19 [Error Id: 9a0b442a-e5e1-44f9-9200-fff9f59b990a on twig03.twigs:31010] Retrieving the file from hdfs: root@twig03:~# hdfs dfs -get /warehouse2/completed/events/connection_events/1441707300/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet /tmp/. root@twig03:~# ls /tmp/*.parquet /tmp/1441707312267-9-f98c71d9-dff0-4dad-8220-01b7a8a5e1d5.parquet hdfs report: root@twig03:~# hdfs dfsadmin -report Configured Capacity: 7856899358720 (7.15 TB) Present Capacity: 7856899358720 (7.15 TB) DFS Remaining: 4567228003960 (4.15 TB) DFS Used: 3289671354760 (2.99 TB) DFS Used%: 41.87% Under replicated blocks: 22108 Blocks with corrupt replicas: 0 Missing blocks: 0 ------------------------------------------------- Live datanodes (2): Name: 10.0.1.4:50010<http://10.0.1.4:50010> (twig04.twigs) Hostname: twig04.twigs Decommission Status : Normal Configured Capacity: 3928449679360 (3.57 TB) DFS Used: 1644836539588 (1.50 TB) Non DFS Used: 0 (0 B) DFS Remaining: 2283613139772 (2.08 TB) DFS Used%: 41.87% DFS Remaining%: 58.13% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 3 Last contact: Tue Sep 08 11:15:47 EDT 2015 Name: 10.0.1.3:50010<http://10.0.1.3:50010> (twig03.twigs) Hostname: twig03.twigs Decommission Status : Normal Configured Capacity: 3928449679360 (3.57 TB) DFS Used: 1644834815172 (1.50 TB) Non DFS Used: 0 (0 B) DFS Remaining: 2283614864188 (2.08 TB) DFS Used%: 41.87% DFS Remaining%: 58.13% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Sep 08 11:15:47 EDT 2015 [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726] Grant Overby Software Engineer Cisco.com<http://www.cisco.com/> [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> Mobile: 865 724 4910<tel:865%20724%204910> [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. -- Abdelhakim Deneche Software Engineer [http://www.mapr.com/sites/default/files/logos/mapr-logo-signature.png] <http://www.mapr.com/> Now Available - Free Hadoop On-Demand Training<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
