[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665181#comment-15665181
 ] 

ASF subversion and git services commented on NIFI-2957:
---

Commit 7e237341810763ed4631fec8f568c04d8f928788 in nifi's branch 
refs/heads/master from [~jtstorck]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7e23734 ]

NIFI-2957 Corrections of typos, documentation, and strings
Updates made regarding Intellij code inspections

This closes #1218

Signed-off-by: Bryan Rosander 


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665183#comment-15665183
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1218


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665176#comment-15665176
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user brosander commented on the issue:

https://github.com/apache/nifi/pull/1218
  
+1 will merge


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665102#comment-15665102
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user brosander commented on the issue:

https://github.com/apache/nifi/pull/1218
  
Reviewing


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664568#comment-15664568
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

GitHub user jtstorck opened a pull request:

https://github.com/apache/nifi/pull/1218

NIFI-2957 Corrections of typos, documentation, and strings

Updates made regarding Intellij code inspections

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jtstorck/nifi NIFI-2957

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1218.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1218


commit eb97e44cfee5fcc2860b4e651f76319f1971364b
Author: Jeff Storck 
Date:   2016-11-14T18:02:16Z

NIFI-2957 Corrections of typos, documentation, and strings
Updates made regarding Intellij code inspections




> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657907#comment-15657907
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1193


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657902#comment-15657902
 ] 

ASF subversion and git services commented on NIFI-2957:
---

Commit ed6e03399f63d53aafdc2faadfa0878ff0b42188 in nifi's branch 
refs/heads/master from [~jtstorck]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ed6e033 ]

NIFI-2957 ZooKeeper Migration Toolkit

Reads from and writes to a Zookeeper that is open or secured via user/password 
digest or SASL
Supports persisting of data obtained from Zookeeper to a file or standard out 
as JSON
Supports sending of Zookeeper data to Zookeeper from a file or standard out as 
JSON
Does not allow data obtained from a source Zookeeper to be written back to the 
same Zookeeper connect string and path

This closes #1193

Signed-off-by: Bryan Rosander 


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657785#comment-15657785
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1193
  
Also removed tests that specify a port for Zookeeper in favor of tests that 
check the connect string and path parsing from the command line.


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656081#comment-15656081
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user brosander commented on the issue:

https://github.com/apache/nifi/pull/1193
  
With the new build I'm seeing errors running though the steps 
[here](https://github.com/jtstorck/docker-zk-krb#run-the-zk-migrator-to-read-the-protected-nodes-for-client-from-zookeeper).

```
2016-11-11 04:13:03,655 WARN [main-SendThread(zk-kerberos:2181)] 
org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, 
closing socket connection and attempting reconnect
java.lang.NoClassDefFoundError: org/apache/log4j/Logger
at org.apache.zookeeper.Login.(Login.java:44) 
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at 
org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:226)
 ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at 
org.apache.zookeeper.client.ZooKeeperSaslClient.(ZooKeeperSaslClient.java:131)
 ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:949) 
~[zookeeper-3.4.6.jar:3.4.6-1569965]
at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 
~[zookeeper-3.4.6.jar:3.4.6-1569965]
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Logger
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
~[na:1.8.0_111]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
~[na:1.8.0_111]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 
~[na:1.8.0_111]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
~[na:1.8.0_111]
... 5 common frames omitted
2016-11-11 04:13:03,713 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator 
Persisting data from source ZooKeeper: 
ZooKeeperEndpointConfig{connectString=zk-kerberos:2181, path=/}
Exception in thread "main" java.lang.RuntimeException: unable to perform 
operation: KeeperErrorCode = ConnectionLoss for /
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:145)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /
at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.getNode(ZooKeeperMigrator.java:188)
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.readZooKeeper(ZooKeeperMigrator.java:95)
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:137)
```

Unfortunately it looks like Zookeeper has [hard log4j 
dependencies](https://github.com/apache/zookeeper/blob/release-3.4.6/src/java/main/org/apache/zookeeper/Login.java#L44).


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655694#comment-15655694
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1193
  
@brosander Updated the logging config to lessen the logging framework's 
output at startup and added a config to the assembly's classpath dir.


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655515#comment-15655515
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1193#discussion_r87508185
  
--- Diff: 
nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/resources/logback.groovy 
---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import ch.qos.logback.classic.encoder.PatternLayoutEncoder
--- End diff --

Yes, I'll look into that.


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655358#comment-15655358
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user brosander commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1193#discussion_r87499434
  
--- Diff: 
nifi-toolkit/nifi-toolkit-zookeeper-migrator/src/test/resources/logback.groovy 
---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import ch.qos.logback.classic.encoder.PatternLayoutEncoder
--- End diff --

I like the dsl config option here but can we put it in the classpath/ 
folder of the toolkit assembly?  That should get picked up still be editable 
without recompile.


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655346#comment-15655346
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user brosander commented on the issue:

https://github.com/apache/nifi/pull/1193
  
@jtstorck having both logback and log4j on the classpath is causing slf4j 
to complain:

```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type 
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]
```


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654616#comment-15654616
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1193
  
@brosander Code has been updated to allow sending data to the root of a 
Zookeeper server.  Also, you shouldn't have to specify the port when sending 
data if Zookeeper is running on the default port.


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654447#comment-15654447
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user brosander commented on the issue:

https://github.com/apache/nifi/pull/1193
  
@jtstorck I'm having some problems trying to save a path (/node) and then 
restore at the same path in a new zookeeper instance.

Here's what I'm getting (with a teardown and reinstantiation of the 
zookeeper container between receive and send commands) :
```
zk-migrator.sh -z zoo1/node -r -f /out/out.json
zk-migrator.sh -z zoo1:2181/ -s -f /out/out.json
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type 
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2016-11-10 16:14:26,590 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator 
Source data was obtained from ZooKeeper: 
ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
2016-11-10 16:14:26,621 INFO [ForkJoinPool.commonPool-worker-1] 
o.a.n.t.zkmigrator.ZooKeeperMigrator transformed original node 
DataStatAclNode{path=/node, acls=[31,s{'world,'anyone}
], ephemeralOwner=0} to DataStatAclNode{path=//node, 
acls=[31,s{'world,'anyone}
], ephemeralOwner=0}
Exception in thread "main" java.lang.RuntimeException: unable to perform 
operation: java.lang.IllegalArgumentException: Invalid path string "//node" 
caused by empty node name specified @1
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:145)
Caused by: java.util.concurrent.ExecutionException: 
java.lang.IllegalArgumentException: Invalid path string "//node" caused by 
empty node name specified @1
at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.writeZooKeeper(ZooKeeperMigrator.java:171)
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigratorMain.main(ZooKeeperMigratorMain.java:139)
Caused by: java.lang.IllegalArgumentException: Invalid path string "//node" 
caused by empty node name specified @1
at 
org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:99)
at 
org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.ensureNodeExists(ZooKeeperMigrator.java:214)
at 
org.apache.nifi.toolkit.zkmigrator.ZooKeeperMigrator.lambda$null$5(ZooKeeperMigrator.java:159)
at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
at 
java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at 
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
```


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a 

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654390#comment-15654390
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user brosander commented on the issue:

https://github.com/apache/nifi/pull/1193
  
@jtstorck I'm getting logging messages in stdout when I don't specify a 
file:

```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/opt/nifi-toolkit/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/opt/nifi-toolkit/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type 
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2016-11-10 15:53:36,446 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator 
Persisting data from source ZooKeeper: 
ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
[
  {
"connectString": "zoo1",
"path": "/node"
  },
  {
"path": "/node",
"data": [
  99,
  111,
  110,
  116,
  101,
  110,
  116
],
"stat": {
  "czxid": 9,
  "mzxid": 9,
  "ctime": 1478792367151,
  "mtime": 1478792367151,
  "version": 0,
  "cversion": 0,
  "aversion": 0,
  "ephemeralOwner": 0,
  "dataLength": 7,
  "numChildren": 0,
  "pzxid": 9
},
"acls": [
  {
"perms": 31,
"id": {
  "scheme": "world",
  "id": "anyone"
}
  }
],
"ephemeralOwner": 0
  }
]2016-11-10 15:53:36,496 INFO [main] o.a.n.t.zkmigrator.ZooKeeperMigrator 1 
nodes read from ZooKeeperEndpointConfig{connectString=zoo1, path=/node}
```


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651843#comment-15651843
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

GitHub user jtstorck opened a pull request:

https://github.com/apache/nifi/pull/1193

NIFI-2957 ZooKeeper Migration Toolkit

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.

Reads from and writes to a Zookeeper that is open or secured via 
user/password digest or SASL
Supports persisting of data obtained from Zookeeper to a file or standard 
out as JSON
Supports sending of Zookeeper data to Zookeeper from a file or standard out 
as JSON
Does not allow data obtained from a source Zookeeper to be written back to 
the same Zookeeper connect string and path

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jtstorck/nifi NIFI-2957

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1193.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1193


commit 7b7bec120164bb2eb8fb7b0018ad5b5fa4fed6c7
Author: Jeff Storck 
Date:   2016-10-27T16:21:20Z

NIFI-2957 ZooKeeper Migration Toolkit
Reads from and writes to a Zookeeper that is open or secured via 
user/password digest or SASL
Supports persisting of data obtained from Zookeeper to a file or standard 
out as JSON
Supports sending of Zookeeper data to Zookeeper from a file or standard out 
as JSON
Does not allow data obtained from a source Zookeeper to be written back to 
the same Zookeeper connect string and path




> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)