Ben, NiFi provides an embedded ZooKeeper server for convenience, mostly for 'testing and evaluation' types of purposes. For any sort of production or very high-volume flows, I would strongly encourage you to move ZooKeeper to its own servers. You will certainly see a lot of problems when trying to interact with ZooKeeper if the box that ZooKeeper is running on is under heavy load - either CPU-wise or I/O-wise.
Thanks -Mark > On Jan 26, 2017, at 7:26 AM, bmichaud <[email protected]> wrote: > > On Monday, I stood up a cluster with the same configuration as was done > successfully in 1.0.0 a three-server NiFi cluster. Before I started the > cluster, I cleaned out all zookeeper state and data from the old cluster, > but kept the same flow intact, connected to Kafka to pull data from a topic. > This was a performance environment, and there was heavy load on that kafka > topic, so it was immediately busy. > > My strong belief is that, due to the volume of data that the flow needed to > process during the election process, the election of a coordinator never > occurred, and, to this day, each node remains disconnected from the others, > although they are running independently. > > Could this be a defect in NiFi or Zookeeper? What would you suggest that I > do to resolve this issue? > All servers in the cluster are configured in the following manner: > > nifi.properties: > nifi.state.management.embedded.zookeeper.start=true > nifi.cluster.is.node=true > nifi.cluster.node.address=server1 > nifi.zookeeper.connect.string=server1:2181,server2:2181,server3:2181 > > zookeeper.properties: > server.1=server1:2888:3888 > server.2=server2:2888:3888 > server.3=server3:2888:3888 > > state-management.xml: > <cluster-provider> > <id>zk-provider</id> > > <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> > <property name="Connect > String">server1:2181,server2:2181,server3:2181</property> > <property name="Root Node">/nifi</property> > <property name="Session Timeout">10 seconds</property> > <property name="Access Control">Open</property> > </cluster-provider> > > Let me know if you need additional information, please. > > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Upgrade-from-1-0-0-to-1-1-1-cluster-config-under-heavy-load-nodes-do-not-connect-tp14523.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
