Could you look at the zookeeper logs and see if there is any information there? Zookeeper logs should be at zk install location/ logs. There should be two files. A .log and .out. Please check both.
Regards Ramana On Monday, October 27, 2014, Chris Drawater <[email protected]> wrote: > We have 3 * Ubuntu 14.04 VM nodes each running a single 0.6 Drillbit , with > 1 of the VMs also running a Zookeeper 3.4.6 instance. > > Each VM has an identical data directory structure on local persistent > filesystem and hosts JSON files. > > Zookeeper is aware of the 3 * Drillbits and 'select * from sys.drillbits' > shows 3 Drillbits. > > UDP Multicast for the VM subnet is supposedly enabled. > > > > Using Squirrl/JDBC driver on a Windows we can connect to Drill (vai the > zookeeper instance) and run SQL queries against JSON files. > > However the queries are only returning rows from the 'foreman' drillbit. > > > > We have never managed to run a distributed query ! > > > > Indeed looking at the Drillbit logs we see : > > > > Not a hint of any awareneess of other Drillbits > > No 'fragments' mentioned in any plan apart from 'fragment_id : 0' > > > > but we do see this (Client connected,via Zookeeper on Node 1, to Drillbit > on Node 3) : > > > > 2014-10-27 10:42:48,914 [dbc13c29-bbd8-4890-93fa-a8a98f4cc8fd:frag:0:0] > ERROR o.a.drill.exec.ops.FragmentContext - Fragment Context received > failure. > > java.lang.RuntimeException: Failure while accessing Zookeeper > > at org.apache.drill.exec.store.sys.zk.ZkPStore.put > (ZkPStore.java:111) ~[drill-java-exec-0.6.0-incubating-rebuffed.jar:0.6.0- > incubating] > > at org.apache.drill.exec.work.foreman.QueryStatus.updateCache > (QueryStatus.java:125) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.foreman.QueryStatus.update > (QueryStatus.java:119) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.foreman.QueryManager.updateStatus > (QueryManager.java:173) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.foreman.QueryManager.finished > (QueryManager.java:189) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.foreman.QueryManager.statusUpdate > (QueryManager.java:162) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at > > org.apache.drill.exec.work.foreman.QueryManager$RootStatusHandler.statusChan > ge(QueryManager.java:284) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at > org.apache.drill.exec.work.fragment.AbstractStatusReporter.finished > (AbstractStatusReporter.java:101) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at > org.apache.drill.exec.work.fragment.AbstractStatusReporter.stateChanged > (AbstractStatusReporter.java:73) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.fragment.FragmentExecutor.updateState > (FragmentExecutor.java:172) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.fragment.FragmentExecutor.run > (FragmentExecutor.java:110) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run > (WorkManager.java:250) > [drill-java-exec-0.6.0-incubating-rebuffed.jar:0.6.0- > incubating] > > at java.util.concurrent.ThreadPoolExecutor.runWorker > (ThreadPoolExecutor.java:1145) [na:1.7.0_65] > > at java.util.concurrent.ThreadPoolExecutor$Worker.run > (ThreadPoolExecutor.java:615) [na:1.7.0_65] > > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > > Caused by: java.lang.InterruptedException: null > > at java.lang.Object.wait(Native Method) ~[na:1.7.0_65] > > at java.lang.Object.wait(Object.java:503) ~[na:1.7.0_65] > > at org.apache.zookeeper.ClientCnxn.submitRequest > (ClientCnxn.java:1309) ~[zookeeper-3.4.5.jar:3.4.5-1392090] > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) ~ > [zookeeper-3.4.5.jar:3.4.5-1392090] > > at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call > (ExistsBuilderImpl.java:172) ~[curator-framework-2.5.0.jar:na] > > at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call > (ExistsBuilderImpl.java:161) ~[curator-framework-2.5.0.jar:na] > > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~ > [curator-client-2.5.0.jar:na] > > at > org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground > (ExistsBuilderImpl.java:157) ~[curator-framework-2.5.0.jar:na] > > at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath > (ExistsBuilderImpl.java:148) ~[curator-framework-2.5.0.jar:na] > > at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath > (ExistsBuilderImpl.java:36) ~[curator-framework-2.5.0.jar:na] > > at org.apache.drill.exec.store.sys.zk.ZkPStore.put > (ZkPStore.java:104) ~[drill-java-exec-0.6.0-incubating-rebuffed.jar:0.6.0- > incubating] > > ... 14 common frames omitted > > > > 2014-10-27 10:42:48,926 [dbc13c29-bbd8-4890-93fa-a8a98f4cc8fd:frag:0:0] > ERROR o.a.d.e.w.f.AbstractStatusReporter - Error 6f41051f-af65-4be8-9cce- > fe1895643d70: Failure while running fragment. > > java.lang.InterruptedException: null > > at java.lang.Object.wait(Native Method) ~[na:1.7.0_65] > > at java.lang.Object.wait(Object.java:503) ~[na:1.7.0_65] > > at org.apache.zookeeper.ClientCnxn.submitRequest > (ClientCnxn.java:1309) ~[zookeeper-3.4.5.jar:3.4.5-1392090] > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036) ~ > [zookeeper-3.4.5.jar:3.4.5-1392090] > > at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call > (ExistsBuilderImpl.java:172) ~[curator-framework-2.5.0.jar:na] > > at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call > (ExistsBuilderImpl.java:161) ~[curator-framework-2.5.0.jar:na] > > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~ > [curator-client-2.5.0.jar:na] > > at > org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground > (ExistsBuilderImpl.java:157) ~[curator-framework-2.5.0.jar:na] > > at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath > (ExistsBuilderImpl.java:148) ~[curator-framework-2.5.0.jar:na] > > at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath > (ExistsBuilderImpl.java:36) ~[curator-framework-2.5.0.jar:na] > > at org.apache.drill.exec.store.sys.zk.ZkPStore.put > (ZkPStore.java:104) ~[drill-java-exec-0.6.0-incubating-rebuffed.jar:0.6.0- > incubating] > > at org.apache.drill.exec.work.foreman.QueryStatus.updateCache > (QueryStatus.java:125) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.foreman.QueryStatus.update > (QueryStatus.java:119) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.foreman.QueryManager.updateStatus > (QueryManager.java:173) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.foreman.QueryManager.finished > (QueryManager.java:189) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.foreman.QueryManager.statusUpdate > (QueryManager.java:162) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at > > org.apache.drill.exec.work.foreman.QueryManager$RootStatusHandler.statusChan > ge(QueryManager.java:284) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at > org.apache.drill.exec.work.fragment.AbstractStatusReporter.finished > (AbstractStatusReporter.java:101) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at > org.apache.drill.exec.work.fragment.AbstractStatusReporter.stateChanged > (AbstractStatusReporter.java:73) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.fragment.FragmentExecutor.updateState > (FragmentExecutor.java:172) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.fragment.FragmentExecutor.run > (FragmentExecutor.java:110) ~[drill-java-exec-0.6.0-incubating- > rebuffed.jar:0.6.0-incubating] > > at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run > (WorkManager.java:250) > [drill-java-exec-0.6.0-incubating-rebuffed.jar:0.6.0- > incubating] > > at java.util.concurrent.ThreadPoolExecutor.runWorker > (ThreadPoolExecutor.java:1145) [na:1.7.0_65] > > at java.util.concurrent.ThreadPoolExecutor$Worker.run > (ThreadPoolExecutor.java:615) [na:1.7.0_65] > > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > > > > Does anyone have any ideas or pointers regarding this ? > > > > Also, I have a few questions... > > > > 1. When do the Drillbits become 'aware' of each other ? > > 2. Is there any Drill tracing that can switched on to reveal the (lack > of) communication between the Drillbits ? > > > > Any help once again gratefully received. > > > > Thanks, > > Chris > > > > > >
