The client (solr in this case) is passing a null path to the ZooKeeper.getChildren(path, ... ) call.
java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) I'm afraid you'll have to work with the solr team to determine the cause of this. Patrick On Thu, Aug 26, 2010 at 12:15 AM, Yatir Ben Shlomo <yat...@outbrain.com>wrote: > I am running a zookeeper ensemble of 3 zookeeper instances > and established a solrCloud to work with it (2 masters , 2 slaves) > on one of the masters I keep noticing ZooKeeper related exceptions which I > can't understand: > And the other is java.lang.IllegalArgumentException: Path cannot be null > (PathUtils.java:45) > > Here are my logs (I set the log level to FINE on zookeeper package) > > Anyone can identify the issue? > (I could not yet get any help from the solrCloud community) > > > FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null > serverPath:null finished:false header:: -8,101 replyHeader:: -8,-1,0 > request:: > 30064776552,v{'/collections},v{},v{'/collections/ENPwl/shards/ENPWL1,'/collections/ENPwl/shards/ENPWL4,'/collections/ENPwl/shards/ENPWL2,'/collections,'/collections/ENPwl/shards/ENPWL3,'/collections/ENPwlMaster/shards/ENPWLMaster_3,'/collections/ENPwlMaster/shards/ENPWLMaster_4,'/live_nodes,'/collections/ENPwlMaster/shards/ENPWLMaster_1,'/collections/ENPwlMaster/shards/ENPWLMaster_2} > response:: null > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug > FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null > serverPath:null finished:false header:: 540,8 replyHeader:: 540,-1,0 > request:: '/collections,F response:: v{'ENPwl,'ENPwlMaster} > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error > SEVERE: Error while calling watcher > java.lang.IllegalArgumentException: Path cannot be null > at > org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) > at > org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) > at > org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process > INFO: Updating live > nodes:org.apache.solr.common.cloud.solrzkcli...@55308275 > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Updating live nodes from ZooKeeper... > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug > FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null > serverPath:null finished:false header:: 541,8 replyHeader:: 541,-1,0 > request:: '/live_nodes,F response:: v{'ob1078.nydc1.outbrain.com:8983 > _solr2,'ob1078.nydc1.outbrain.com:8983 > _solr1,'ob1061.nydc1.outbrain.com:8983 > _solr2,'ob1062.nydc1.outbrain.com:8983 > _solr1,'ob1062.nydc1.outbrain.com:8983 > _solr2,'ob1061.nydc1.outbrain.com:8983 > _solr1,'ob1077.nydc1.outbrain.com:8983 > _solr2,'ob1077.nydc1.outbrain.com:8983_solr1} > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error > SEVERE: Error while calling watcher > java.lang.IllegalArgumentException: Path cannot be null > at > org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) > at > org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) > at > org.apache.solr.cloud.ZkController$2.process(ZkController.java:321) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager > process > INFO: Watcher > org.apache.solr.common.cloud.connectionmana...@339bb448name:ZooKeeperConnection > Watcher:zook1:2181,zook2:2181,zook3:2181 got event > WatchedEvent: Server state change. New state: Disconnected path:null > type:None > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_1 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_3 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_2 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_4 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$5 > process > INFO: Detected a new or removed collection > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug > FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null > serverPath:null finished:false header:: 542,8 replyHeader:: 542,-1,0 > request:: '/collections,F response:: v{'ENPwl,'ENPwlMaster} > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error > SEVERE: Error while calling watcher > java.lang.IllegalArgumentException: Path cannot be null > at > org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) > at > org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) > at > org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process > INFO: Updating live > nodes:org.apache.solr.common.cloud.solrzkcli...@55308275 > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Updating live nodes from ZooKeeper... > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug > FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null > serverPath:null finished:false header:: 543,8 replyHeader:: 543,-1,0 > request:: '/live_nodes,F response:: v{'ob1078.nydc1.outbrain.com:8983 > _solr2,'ob1078.nydc1.outbrain.com:8983 > _solr1,'ob1061.nydc1.outbrain.com:8983 > _solr2,'ob1062.nydc1.outbrain.com:8983 > _solr1,'ob1062.nydc1.outbrain.com:8983 > _solr2,'ob1061.nydc1.outbrain.com:8983 > _solr1,'ob1077.nydc1.outbrain.com:8983 > _solr2,'ob1077.nydc1.outbrain.com:8983_solr1} > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error > SEVERE: Error while calling watcher > java.lang.IllegalArgumentException: Path cannot be null > at > org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) > at > org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) > at > org.apache.solr.cloud.ZkController$2.process(ZkController.java:321) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager > process > INFO: Watcher > org.apache.solr.common.cloud.connectionmana...@339bb448name:ZooKeeperConnection > Watcher:zook1:2181,zook2:2181,zook3:2181 got event > WatchedEvent: Server state change. New state: SyncConnected path:null > type:None > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_1 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_3 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info > INFO: Attempting connection to server zook2/192.168.252.62:2181 > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info > INFO: Priming connection to java.nio.channels.SocketChannel[connected > local=/192.168.252.77:54892 remote=zook2/192.168.252.62:2181] > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info > INFO: Server connection successful > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error > SEVERE: Error while calling watcher > java.lang.IllegalArgumentException: Path cannot be null > at > org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) > at > org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) > at > org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_2 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process > INFO: Updating live > nodes:org.apache.solr.common.cloud.solrzkcli...@29d03e78 > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Updating live nodes from ZooKeeper... > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error > SEVERE: Error while calling watcher > java.lang.IllegalArgumentException: Path cannot be null > at > org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) > at > org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) > at > org.apache.solr.cloud.ZkController$2.process(ZkController.java:321) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_4 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_1 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager > process > INFO: Watcher > org.apache.solr.common.cloud.connectionmana...@40bb2bc3name:ZooKeeperConnection > Watcher:zook1:2181,zook2:2181,zook3:2181 got event > WatchedEvent: Server state change. New state: SyncConnected path:null > type:None > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_3 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$5 > process > INFO: Detected a new or removed collection > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error > SEVERE: Error while calling watcher > java.lang.IllegalArgumentException: Path cannot be null > at > org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) > at > org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) > at > org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_2 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process > INFO: Updating live > nodes:org.apache.solr.common.cloud.solrzkcli...@29d03e78 > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Updating live nodes from ZooKeeper... > Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error > SEVERE: Error while calling watcher > java.lang.IllegalArgumentException: Path cannot be null > at > org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) > at > org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) > at > org.apache.solr.cloud.ZkController$2.process(ZkController.java:321) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_4 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 > process > INFO: Detected a shard change under ShardId:ENPWLMaster_1 in > collection:ENPwlMaster > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$6 > process > INFO: Notified of CloudState change > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Cloud state update for ZooKeeper already scheduled > Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process > INFO: Updating live > nodes:org.apache.solr.common.cloud.solrzkcli...@29d03e78 > Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader > updateCloudState > INFO: Updating live nodes from ZooKeeper... > Aug 25, 2010 5:18:22 AM org.apache.log4j.Category debug > FINE: Got ping response for sessionid:0x12a97312613010b after 0ms >