This is an automated email from the ASF dual-hosted git repository.
markap14 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 282c271 NIFI-6578 - Upgraded zookeeper framework version. Some code
changes required. This change also required a change to the embedded zookeeper.
- Updating tests to account for the new node /zookeeper/config Upgraded
Zookeeper and Curator versions for zookeeper migrator in toolkit. Updated tests
to allow for new znode /zookeeper/config - Added documentation changes to
reflect changes to the zookeeper.properties file going foward.
282c271 is described below
commit 282c271c7c29933ca6429acff04806aceb6a5029
Author: thenatog <[email protected]>
AuthorDate: Thu Sep 5 12:33:39 2019 -0400
NIFI-6578 - Upgraded zookeeper framework version. Some code changes
required. This change also required a change to the embedded zookeeper.
- Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit.
Updated tests to allow for new znode /zookeeper/config
- Added documentation changes to reflect changes to the
zookeeper.properties file going foward.
This closes #3715.
Signed-off-by: Mark Payne <[email protected]>
---
.../src/main/asciidoc/administration-guide.adoc | 15 +++++-----
nifi-docs/src/main/asciidoc/toolkit-guide.adoc | 6 ++--
.../state/server/ZooKeeperStateServer.java | 10 +++----
.../src/main/resources/conf/zookeeper.properties | 14 +++++----
nifi-nar-bundles/nifi-framework-bundle/pom.xml | 4 +--
.../nifi-toolkit-zookeeper-migrator/pom.xml | 8 ++---
.../zkmigrator/ZooKeeperMigratorTest.groovy | 35 +++++++++++++++-------
7 files changed, 55 insertions(+), 37 deletions(-)
diff --git a/nifi-docs/src/main/asciidoc/administration-guide.adoc
b/nifi-docs/src/main/asciidoc/administration-guide.adoc
index 24d9720..acd5f91 100644
--- a/nifi-docs/src/main/asciidoc/administration-guide.adoc
+++ b/nifi-docs/src/main/asciidoc/administration-guide.adoc
@@ -97,11 +97,11 @@ The following table lists the default ports used by an
<<embedded_zookeeper>> an
[options="header,footer"]
|==================================================================================================================================================
| Function | Property | Default
Value
-|Zookeeper Client Port | `clientPort` | `2181`
+|Zookeeper Client Port (Deprecated: client port is no longer specified on a
separate line as of NiFi 1.10.x) | `clientPort` |
`2181`
|Zookeeper Server Quorum and Leader Election Ports | `server.1` | _none_
|==================================================================================================================================================
-NOTE: Commented examples for the Zookeeper server ports are included in the
_zookeeper.properties_ file in the form
`server.N=nifi-nodeN-hostname:2888:3888`.
+NOTE: Commented examples for the Zookeeper server ports are included in the
_zookeeper.properties_ file in the form
`server.N=nifi-nodeN-hostname:2888:3888;2181`.
== Configuration Best Practices
If you are running on Linux, consider these best practices. Typical Linux
defaults are not necessarily well-tuned for the needs of an IO intensive
application like NiFi. For all of these areas, your distribution's
requirements may vary. Use these sections as advice, but
@@ -1853,12 +1853,11 @@ However, it is up to the administrator to determine the
number of nodes most app
If the `nifi.state.management.embedded.zookeeper.start` property is set to
`true`, the `nifi.state.management.embedded.zookeeper.properties` property
in _nifi.properties_ also becomes relevant. This specifies the ZooKeeper
properties file to use. At a minimum, this properties file needs to be populated
-with the list of ZooKeeper servers. The servers are specified as properties in
the form of `server.1`, `server.2`, to `server.n`. Each of these servers is
-configured as <hostname>:<quorum port>[:<leader election port>]. For example,
`myhost:2888:3888`. This list of nodes should be the same nodes in the NiFi
-cluster that have the `nifi.state.management.embedded.zookeeper.start`
property set to `true`. Also note that because ZooKeeper will be listening on
these
-ports, the firewall may need to be configured to open these ports for incoming
traffic, at least between nodes in the cluster. Additionally, the port to
-listen on for client connections must be opened in the firewall. The default
value for this is `2181` but can be configured via the _clientPort_ property
-in the _zookeeper.properties_ file.
+with the list of ZooKeeper servers. The servers are specified as properties in
the form of `server.1`, `server.2`, to `server.n`. As of NiFi 1.10.x, Zookeeper
+has been upgraded to 3.5.5 and servers are now defined with the client port
appended at the end as per the
link:https://zookeeper.apache.org/doc/r3.5.2-alpha/zookeeperReconfig.html#sc_reconfig_clientport[Zookeeper
Documentation].
+As such, each of these servers is configured as <hostname>:<quorum
port>[:<leader election port>][:role];[<client port address>:]<client port>.
+As a simple example this would be `server.1 = myhost:2888:3888;2181`. This
list of nodes should be the same nodes in the NiFi cluster that have the
`nifi.state.management.embedded.zookeeper.start` property set to `true`.
+Also note that because ZooKeeper will be listening on these ports, the
firewall may need to be configured to open these ports for incoming traffic, at
least between nodes in the cluster.
When using an embedded ZooKeeper, the ./__conf/zookeeper.properties__ file has
a property named `dataDir`. By default, this value is set to
`./state/zookeeper`.
If more than one NiFi node is running an embedded ZooKeeper, it is important
to tell the server which one it is. This is accomplished by creating a file
named
diff --git a/nifi-docs/src/main/asciidoc/toolkit-guide.adoc
b/nifi-docs/src/main/asciidoc/toolkit-guide.adoc
index 9572d62..5443e42 100644
--- a/nifi-docs/src/main/asciidoc/toolkit-guide.adoc
+++ b/nifi-docs/src/main/asciidoc/toolkit-guide.adoc
@@ -1240,8 +1240,8 @@ Before you begin, confirm that:
|*Required Information*|*Description*
|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the
hosts running in the ZooKeeper ensemble, which can be found in _<NiFi
installation dir>/conf/zookeeper.properties_. Any of the hostnames declared in
the `server.N` properties can be used.
|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must be
one of the hosts running in the ZooKeeper ensemble, which can be found in
_<NiFi installation dir>/conf/zookeeper.properties_. Any of the hostnames
declared in the `server.N` properties can be used.
-|Source ZooKeeper port (*sourceClientPort*)|This can be found in _<NiFi
installation dir>/conf/zookeeper.properties_. The port is specified in the
`clientPort` property.
-|Destination ZooKeeper port (*destinationClientPort*)|This can be found in
_<NiFi installation dir>/conf/zookeeper.properties_. The port is specified in
the `clientPort` property.
+|Source ZooKeeper port (*sourceClientPort*)|This can be found in _<NiFi
installation dir>/conf/zookeeper.properties_. The port is specified in the
`clientPort` property or at the end of the server string.
+|Destination ZooKeeper port (*destinationClientPort*)|This can be found in
_<NiFi installation dir>/conf/zookeeper.properties_. The port is specified in
the `clientPort` property or at the end of the server string.
|Export data path|Determine the path that will store a json file containing
the export of data from ZooKeeper. It must be readable and writable by the
user running the zk-migrator tool.
|Source ZooKeeper Authentication Information|This information is in _<NiFi
installation dir>/conf/state-management.xml_. For NiFi 0.x, if Creator Only is
specified in _state-management.xml_, you need to supply authentication
information using the `-a,--auth` argument with the values from the Username
and Password properties in _state-management.xml_. For NiFi 1.x, supply
authentication information using the `-k,--krb-conf` argument.
@@ -1252,6 +1252,8 @@ If the _state-management.xml_ specifies Open, no
authentication is required.
|Root path to which NiFi writes data in Source ZooKeeper
(*sourceRootPath*)|This information can be found in `<NiFi installation
dir>/conf/state-management.xml` under the Root Node property in the
cluster-provider element. (default: `/nifi`)
|Root path to which NiFi writes data in Destination ZooKeeper
(*destinationRootPath*)|This information can be found in _<NiFi installation
dir>/conf/state-management.xml_ under the Root Node property in the
cluster-provider element.
|====
+NOTE: As of NiFi 1.10.x, because of an upgrade to ZooKeeper 3.5.x, the
migrator may have a permission error `"NoAuthException: KeeperErrorCode =
NoAuth for /zookeeper/config"` when attempting to use `/` as the destination
root path. This is because the `/zookeeper/config` znode has read-only
permissions. Instead use a destination path of `/nifi/components` or similar.
+
2. Stop all processors in the NiFi flow. If you are migrating between two
NiFi installations, the flows on both must be stopped.
3. Export the NiFi component data from the source ZooKeeper. The following
command reads from the specified ZooKeeper running on the given hostname:port,
using the provided path to the data, and authenticates with ZooKeeper using the
given username and password. The data read from ZooKeeper is written to the
file provided.
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/server/ZooKeeperStateServer.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/server/ZooKeeperStateServer.java
index 9db146d..81e7194 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/server/ZooKeeperStateServer.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/state/server/ZooKeeperStateServer.java
@@ -82,7 +82,7 @@ public class ZooKeeperStateServer extends ZooKeeperServerMain
{
final ServerConfig config = new ServerConfig();
config.readFrom(quorumPeerConfig);
try {
- transactionLog = new FileTxnSnapLog(new
File(config.getDataLogDir()), new File(config.getDataDir()));
+ transactionLog = new FileTxnSnapLog(config.getDataLogDir(),
config.getDataDir());
embeddedZkServer = new ZooKeeperServer();
embeddedZkServer.setTxnLogFactory(transactionLog);
@@ -107,15 +107,13 @@ public class ZooKeeperStateServer extends
ZooKeeperServerMain {
logger.info("Starting Embedded ZooKeeper Peer");
try {
- transactionLog = new FileTxnSnapLog(new
File(quorumPeerConfig.getDataLogDir()), new
File(quorumPeerConfig.getDataDir()));
+ transactionLog = new
FileTxnSnapLog(quorumPeerConfig.getDataLogDir(), quorumPeerConfig.getDataDir());
connectionFactory = ServerCnxnFactory.createFactory();
connectionFactory.configure(quorumPeerConfig.getClientPortAddress(),
quorumPeerConfig.getMaxClientCnxns());
quorumPeer = new QuorumPeer();
-
quorumPeer.setClientPortAddress(quorumPeerConfig.getClientPortAddress());
- quorumPeer.setTxnFactory(new FileTxnSnapLog(new
File(quorumPeerConfig.getDataLogDir()), new
File(quorumPeerConfig.getDataDir())));
- quorumPeer.setQuorumPeers(quorumPeerConfig.getServers());
+ quorumPeer.setTxnFactory(new
FileTxnSnapLog(quorumPeerConfig.getDataLogDir(),
quorumPeerConfig.getDataDir()));
quorumPeer.setElectionType(quorumPeerConfig.getElectionAlg());
quorumPeer.setMyid(quorumPeerConfig.getServerId());
quorumPeer.setTickTime(quorumPeerConfig.getTickTime());
@@ -123,7 +121,7 @@ public class ZooKeeperStateServer extends
ZooKeeperServerMain {
quorumPeer.setMaxSessionTimeout(quorumPeerConfig.getMaxSessionTimeout());
quorumPeer.setInitLimit(quorumPeerConfig.getInitLimit());
quorumPeer.setSyncLimit(quorumPeerConfig.getSyncLimit());
- quorumPeer.setQuorumVerifier(quorumPeerConfig.getQuorumVerifier());
+ quorumPeer.setQuorumVerifier(quorumPeerConfig.getQuorumVerifier(),
false);
quorumPeer.setCnxnFactory(connectionFactory);
quorumPeer.setZKDatabase(new
ZKDatabase(quorumPeer.getTxnFactory()));
quorumPeer.setLearnerType(quorumPeerConfig.getPeerType());
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/zookeeper.properties
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/zookeeper.properties
index fa1d885..8866bb8 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/zookeeper.properties
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/zookeeper.properties
@@ -20,7 +20,6 @@
#
#
-clientPort=2181
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
@@ -28,14 +27,19 @@ tickTime=2000
dataDir=./state/zookeeper
autopurge.snapRetainCount=30
+# The server string has changed as of 3.5.5 and the client port is now
specified at the end of the server string:
+#
https://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html#sc_reconfig_clientport
#
# Specifies the servers that are part of this zookeeper ensemble. For
# every NiFi instance running an embedded zookeeper, there needs to be
-# a server entry below. For instance:
+# a server entry below. Client port is now specified at the end of the string
+# after a semi-colon.
#
-# server.1=nifi-node1-hostname:2888:3888
-# server.2=nifi-node2-hostname:2888:3888
-# server.3=nifi-node3-hostname:2888:3888
+# For instance:
+#
+# server.1=nifi-node1-hostname:2888:3888;2181
+# server.2=nifi-node2-hostname:2888:3888;2181
+# server.3=nifi-node3-hostname:2888:3888;2181
#
# The index of the server corresponds to the myid file that gets created
# in the dataDir of each node running an embedded zookeeper. See the
diff --git a/nifi-nar-bundles/nifi-framework-bundle/pom.xml
b/nifi-nar-bundles/nifi-framework-bundle/pom.xml
index 85819dc..6d0999b 100644
--- a/nifi-nar-bundles/nifi-framework-bundle/pom.xml
+++ b/nifi-nar-bundles/nifi-framework-bundle/pom.xml
@@ -575,7 +575,7 @@
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-test</artifactId>
- <version>2.9.1</version>
+ <version>4.2.0</version>
<scope>test</scope>
</dependency>
<dependency>
@@ -592,7 +592,7 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
- <version>3.4.6</version>
+ <version>3.5.5</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
diff --git a/nifi-toolkit/nifi-toolkit-zookeeper-migrator/pom.xml
b/nifi-toolkit/nifi-toolkit-zookeeper-migrator/pom.xml
index 70c197e..c28ba95 100644
--- a/nifi-toolkit/nifi-toolkit-zookeeper-migrator/pom.xml
+++ b/nifi-toolkit/nifi-toolkit-zookeeper-migrator/pom.xml
@@ -42,7 +42,7 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
- <version>3.4.6</version>
+ <version>3.5.5</version>
<exclusions>
<exclusion>
<groupId>log4j</groupId>
@@ -78,13 +78,13 @@
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-test</artifactId>
- <version>2.9.1</version>
+ <version>4.2.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
- <version>2.11.0</version>
+ <version>4.2.0</version>
<scope>test</scope>
</dependency>
</dependencies>
@@ -135,4 +135,4 @@
</plugins>
</build>
-</project>
\ No newline at end of file
+</project>
diff --git
a/nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/groovy/org/apache/nifi/toolkit/zkmigrator/ZooKeeperMigratorTest.groovy
b/nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/groovy/org/apache/nifi/toolkit/zkmigrator/ZooKeeperMigratorTest.groovy
index 2b9bc00..dd59bc4 100644
---
a/nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/groovy/org/apache/nifi/toolkit/zkmigrator/ZooKeeperMigratorTest.groovy
+++
b/nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/groovy/org/apache/nifi/toolkit/zkmigrator/ZooKeeperMigratorTest.groovy
@@ -93,7 +93,7 @@ class ZooKeeperMigratorTest extends Specification {
then:
noExceptionThrown()
def nodes = ZKPaths.getSortedChildren(client, '/').collect {
ZKUtil.listSubTreeBFS(client, "/$it") }.flatten()
- nodes.size() == 6
+ nodes.size() == 7
}
def "Send to open ZooKeeper without ACL migration with new multi-node
parent"() {
@@ -109,7 +109,7 @@ class ZooKeeperMigratorTest extends Specification {
then:
noExceptionThrown()
def nodes = ZKPaths.getSortedChildren(client, '/').collect {
ZKUtil.listSubTreeBFS(client, "/$it") }.flatten()
- nodes.size() == 7
+ nodes.size() == 8
}
def "Receive all nodes from ZooKeeper root"() {
@@ -128,7 +128,7 @@ class ZooKeeperMigratorTest extends Specification {
then:
noExceptionThrown()
def persistedData = new Gson().fromJson(new JsonReader(new
FileReader(outputFilePath)), List) as List
- persistedData.size() == 5
+ persistedData.size() == 6
}
def "Receive Zookeeper node created with username and password"() {
@@ -168,7 +168,7 @@ class ZooKeeperMigratorTest extends Specification {
then:
noExceptionThrown()
def nodes = ZKPaths.getSortedChildren(client, '/').collect {
ZKUtil.listSubTreeBFS(client, "/$it") }.flatten()
- nodes.size() == 3
+ nodes.size() == 4
}
def "Send to open Zookeeper with ACL migration"() {
@@ -184,7 +184,7 @@ class ZooKeeperMigratorTest extends Specification {
then:
noExceptionThrown()
def nodes = ZKPaths.getSortedChildren(client, '/').collect {
ZKUtil.listSubTreeBFS(client, "/$it") }.flatten()
- nodes.size() == 3
+ nodes.size() == 4
}
def "Send to open Zookeeper using existing ACL"() {
@@ -245,26 +245,41 @@ class ZooKeeperMigratorTest extends Specification {
'127.0.0.1,127.0.0.2:2182,127.0.0.3:2183/path/node' | '/path/node' |
'127.0.0.1,127.0.0.2:2182,127.0.0.3:2183' || _
}
- def "Test ignore source"() {
+ def "Test ignore source" () {
given:
def server = new TestingServer()
+ def migrationPathRoot = '/nifi/components'
def connectString = "$server.connectString"
def dataPath = 'target/test-data-ignore-source.json'
+ def client = new ZooKeeper(server.connectString, 3000, { WatchedEvent
watchedEvent ->
+ })
+
+ // Create some znodes under /nifi/components
+ ZKPaths.mkdirs(client, migrationPathRoot)
+ client.setData(migrationPathRoot, 'some data'.bytes, 0)
+ def componentName = '1'
+ client.create("$migrationPathRoot/$componentName", 'some data'.bytes,
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
+ componentName = '1/a'
+ client.create("$migrationPathRoot/$componentName", 'some data'.bytes,
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL)
+ componentName = '2'
+ client.create("$migrationPathRoot/$componentName", 'some data'.bytes,
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
+ componentName = '3'
+ client.create("$migrationPathRoot/$componentName", 'some data'.bytes,
ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)
when: "data is read from the source zookeeper"
- ZooKeeperMigratorMain.main(['-r', '-z', connectString, '-f', dataPath]
as String[])
+ ZooKeeperMigratorMain.main(['-r', '-z',
"$connectString$migrationPathRoot", '-f', dataPath] as String[])
then: "verify the data has been written to the output file"
new File(dataPath).exists()
when: "data is sent to the same zookeeper as the the source zookeeper
without ignore source"
- ZooKeeperMigratorMain.main(['-s', '-z', connectString, '-f', dataPath]
as String[])
+ ZooKeeperMigratorMain.main(['-s', '-z',
"$connectString$migrationPathRoot", '-f', dataPath] as String[])
then: "verify that an illegal argument exception is thrown"
thrown(IllegalArgumentException)
- when: "data is sent to the same zookeeper as the source zookeeper with
ignore source option is set"
- ZooKeeperMigratorMain.main(['-s', '-z', connectString, '-f', dataPath,
'--ignore-source'] as String[])
+ when: "data is sent to the same zookeeper as the source zookeeper with
ignore source option set"
+ ZooKeeperMigratorMain.main(['-s', '-z',
"$connectString$migrationPathRoot", '-f', dataPath, '--ignore-source',
'--use-existing-acl'] as String[])
then: "no exceptions are thrown"
noExceptionThrown()
