[jira] [Updated] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-2525:

Status: Patch Available  (was: Open)

> HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException 
> when sending through proxy that requires authentication
> -
>
> Key: NIFI-2525
> URL: https://issues.apache.org/jira/browse/NIFI-2525
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
> Fix For: 1.0.0
>
>
> Pulling data using Remote Process Group from output port works.
> However, pushing data using Remote Process Group to input port fails with 
> AsynchronousCloseException.
> A RPG sends data via POST, then a proxy server returns 407: proxy auth 
> required. After this, the RPG should resend the request with credential, but 
> the data channel is already closed.
> Currently, it uses chunked encoding so that it can stream data to send. 
> Sending actual data twice won't be efficient. We need to do the 
> authentication before start reading flow-file stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2525:
--
Fix Version/s: 1.0.0

> HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException 
> when sending through proxy that requires authentication
> -
>
> Key: NIFI-2525
> URL: https://issues.apache.org/jira/browse/NIFI-2525
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
> Fix For: 1.0.0
>
>
> Pulling data using Remote Process Group from output port works.
> However, pushing data using Remote Process Group to input port fails with 
> AsynchronousCloseException.
> A RPG sends data via POST, then a proxy server returns 407: proxy auth 
> required. After this, the RPG should resend the request with credential, but 
> the data channel is already closed.
> Currently, it uses chunked encoding so that it can stream data to send. 
> Sending actual data twice won't be efficient. We need to do the 
> authentication before start reading flow-file stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #915: NIFI-2525: Fix Proxy auth issue with async send.

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/915

NIFI-2525: Fix Proxy auth issue with async send.

Without this fix, NiFi fails to send data via HTTP Site-to-Site through
Proxy which requires authentication due to AsynchronousCloseException.

It happens when async client replays producing contents in order to re-send 
the
request with auth credential for the proxy server, however the
connection is already closed.

This fix makes NiFi to send actual data only at the second round of 
requests, so that flow-file
contents can be sent without reading it twice.

Unit test cases using LittleProxy are also added to confirm HTTP 
Site-to-Site works with Proxy with/without authentication.

I've tested with Apache mod_proxy and Squid proxy servers with basic/digest 
auth.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-2525

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #915


commit 12cf7ef186440ef5de07b04ecaa33d93846ece6f
Author: Koji Kawamura 
Date:   2016-08-23T01:57:57Z

NIFI-2525: Fix Proxy auth issue with async send.

Without this fix, NiFi fails to send data via HTTP Site-to-Site through
Proxy which requires authentication due to AsynchronousCloseException.
It happens when async client replays producing contents in order to re-send 
the
request with auth credential for the proxy server, however the
connection is already closed.
This fix makes NiFi to send actual data only at the second round of 
requests, so that flow-file
contents can be sent without reading it twice.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432127#comment-15432127
 ] 

ASF GitHub Bot commented on NIFI-2525:
--

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/915

NIFI-2525: Fix Proxy auth issue with async send.

Without this fix, NiFi fails to send data via HTTP Site-to-Site through
Proxy which requires authentication due to AsynchronousCloseException.

It happens when async client replays producing contents in order to re-send 
the
request with auth credential for the proxy server, however the
connection is already closed.

This fix makes NiFi to send actual data only at the second round of 
requests, so that flow-file
contents can be sent without reading it twice.

Unit test cases using LittleProxy are also added to confirm HTTP 
Site-to-Site works with Proxy with/without authentication.

I've tested with Apache mod_proxy and Squid proxy servers with basic/digest 
auth.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-2525

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #915


commit 12cf7ef186440ef5de07b04ecaa33d93846ece6f
Author: Koji Kawamura 
Date:   2016-08-23T01:57:57Z

NIFI-2525: Fix Proxy auth issue with async send.

Without this fix, NiFi fails to send data via HTTP Site-to-Site through
Proxy which requires authentication due to AsynchronousCloseException.
It happens when async client replays producing contents in order to re-send 
the
request with auth credential for the proxy server, however the
connection is already closed.
This fix makes NiFi to send actual data only at the second round of 
requests, so that flow-file
contents can be sent without reading it twice.




> HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException 
> when sending through proxy that requires authentication
> -
>
> Key: NIFI-2525
> URL: https://issues.apache.org/jira/browse/NIFI-2525
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>
> Pulling data using Remote Process Group from output port works.
> However, pushing data using Remote Process Group to input port fails with 
> AsynchronousCloseException.
> A RPG sends data via POST, then a proxy server returns 407: proxy auth 
> required. After this, the RPG should resend the request with credential, but 
> the data channel is already closed.
> Currently, it uses chunked encoding so that it can stream data to send. 
> Sending actual data twice won't be efficient. We need to do the 
> authentication before start reading flow-file stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #655: DataDog support added

[irabinovitch]

Github user irabinovitch commented on the issue:

https://github.com/apache/nifi/pull/655
  
Once you're ready we'd be happy to work with you all to create a tile / 
listing in the Datadog app, as well as a default dashboard so that users can 
hit the ground running with this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2620) Add ability to write Row Identifier as binary in hbase using the PutHbaseCell

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432069#comment-15432069
 ] 

ASF GitHub Bot commented on NIFI-2620:
--

GitHub user apsaltis opened a pull request:

https://github.com/apache/nifi/pull/914

Adding support for Binary Row Keys 

Adding support for Binary Row Keys for both PutHBaseCell and PutHBaseJSON. 
This also involved making changes to PutFlowFile and PutColumn to carry around 
byte[] and not all strings. These changes are per JIRA NIFI-2620

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apsaltis/nifi nifi-2620

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #914


commit 34659ab69a4bfa6ca0e21f10d158c5d07f44cf26
Author: Andrew Psaltis 
Date:   2016-08-23T02:34:45Z

Adding support for Binary Row Keys for both PutHBaseCell and PutHBaseJSON. 
This also involved making changes to PutFlowFile and PutColumn to carry around 
byte[] and not all strings. These changes are per JIRA NIFI-2620




> Add ability to write Row Identifier as binary in hbase using the PutHbaseCell
> -
>
> Key: NIFI-2620
> URL: https://issues.apache.org/jira/browse/NIFI-2620
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0, 0.7.0, 0.6.1
>Reporter: Andrew Psaltis
>Assignee: Andrew Psaltis
>
> Today the PutHBaseCell processor makes the assumption that all row keys are 
> text. However, this does not work when the row key in the HBase table is 
> binary. 
> If the row key is specified in the binary string format, such as:
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x01\x01\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x01\x00\x00\x01\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x01\x00\x00\x01\x00\x00\x00\x00\x01 
> \x01\x01\x00\x01\x00\x01\x01\x01\x00\x00\x 
> 00\x00\x00\x00\x01\x01\x01\x01\x00\x00\x00 
> \x00\x00\x00\x01\x01\x00\x01\x00\x01\x00\x 
> 00\x01\x01\x01\x01\x00\x00\x01\x01\x01\x00 
> \x01\x00\x00
> Which is the textual representation that the HBase CLI would return, NiFi 
> calls getBytes on that string. Appropriately HBase will encode the '\' with 
> the hex code: x5C, resulting in an output string that looks like:
> x5Cx00\x5Cx00\ ...
> To address this the proposed solution would be to:
> *  Add "toBytesBinary" method to HBaseClientService  similar to the ones 
> already added [1]. 
> * Update the PutFlowFile and PutColumn to pass around mostly byte[] and not 
> strings that they do today.
> For this JIRA only support for Text and Binary will be added for the RowKey
> [1] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java#L427



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #914: Adding support for Binary Row Keys

[apsaltis]

GitHub user apsaltis opened a pull request:

https://github.com/apache/nifi/pull/914

Adding support for Binary Row Keys 

Adding support for Binary Row Keys for both PutHBaseCell and PutHBaseJSON. 
This also involved making changes to PutFlowFile and PutColumn to carry around 
byte[] and not all strings. These changes are per JIRA NIFI-2620

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apsaltis/nifi nifi-2620

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #914


commit 34659ab69a4bfa6ca0e21f10d158c5d07f44cf26
Author: Andrew Psaltis 
Date:   2016-08-23T02:34:45Z

Adding support for Binary Row Keys for both PutHBaseCell and PutHBaseJSON. 
This also involved making changes to PutFlowFile and PutColumn to carry around 
byte[] and not all strings. These changes are per JIRA NIFI-2620




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2630) Allow PublishJMS processor to create TextMessages

[James Anderson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Anderson updated NIFI-2630:
-
Labels: patch  (was: )
Status: Patch Available  (was: Open)

> Allow PublishJMS processor to create TextMessages
> -
>
> Key: NIFI-2630
> URL: https://issues.apache.org/jira/browse/NIFI-2630
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.0
>Reporter: James Anderson
>Priority: Minor
>  Labels: patch
> Fix For: 0.6.0
>
> Attachments: 
> 0001-NIFI-2630-Allow-PublishJMS-processor-to-create-TextM.patch
>
>
> Create a new configuration option for PublishJMS that allows the processor to 
> be configured to emit instances of TextMessages as well as BytesMessage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2630) Allow PublishJMS processor to create TextMessages

[James Anderson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Anderson updated NIFI-2630:
-
Attachment: 0001-NIFI-2630-Allow-PublishJMS-processor-to-create-TextM.patch

Proposed patch to allow PublishJMS processor to create TextMessages.

> Allow PublishJMS processor to create TextMessages
> -
>
> Key: NIFI-2630
> URL: https://issues.apache.org/jira/browse/NIFI-2630
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.0
>Reporter: James Anderson
>Priority: Minor
> Fix For: 0.6.0
>
> Attachments: 
> 0001-NIFI-2630-Allow-PublishJMS-processor-to-create-TextM.patch
>
>
> Create a new configuration option for PublishJMS that allows the processor to 
> be configured to emit instances of TextMessages as well as BytesMessage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2521) License and Notice - TestSNMPAgent classes appear to be copied from external sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432011#comment-15432011
 ] 

ASF GitHub Bot commented on NIFI-2521:
--

Github user joewitt closed the pull request at:

https://github.com/apache/nifi/pull/816


> License and Notice - TestSNMPAgent classes appear to be copied from external 
> sources
> 
>
> Key: NIFI-2521
> URL: https://issues.apache.org/jira/browse/NIFI-2521
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.0.0, 0.8.0
>
>
> TestSNMPAgentV1 and TestSNMPAgentV2c appear to be sourced at least in large 
> part from https://code.google.com/archive/p/springside-sub/
> TestSNMPAgentV3 appears to be sourced from  XYZ as found here 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-snmp-bundle/nifi-snmp-processors/src/test/java/org/apache/nifi/snmp/processors/TestSnmpAgentV3.java#L68.
> These classes should be completely removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #816: NIFI-2521 removed offending test classes and referen...

[joewitt]

Github user joewitt closed the pull request at:

https://github.com/apache/nifi/pull/816


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2630) Allow PublishJMS processor to create TextMessages

[James Anderson (JIRA)]

James Anderson created NIFI-2630:


 Summary: Allow PublishJMS processor to create TextMessages
 Key: NIFI-2630
 URL: https://issues.apache.org/jira/browse/NIFI-2630
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 0.6.0
Reporter: James Anderson
Priority: Minor
 Fix For: 0.6.0


Create a new configuration option for PublishJMS that allows the processor to 
be configured to emit instances of TextMessages as well as BytesMessage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2269) DataDog support added

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431963#comment-15431963
 ] 

ASF subversion and git services commented on NIFI-2269:
---

Commit 6e82ec738cbd6c8461ea90183cbf418f1f0e1f1f in nifi's branch 
refs/heads/master from [~JPercivall]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=6e82ec7 ]

NIFI-2269 Fixing minor issues with style check, licensing, etc.


> DataDog support added
> -
>
> Key: NIFI-2269
> URL: https://issues.apache.org/jira/browse/NIFI-2269
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Ramiz Abadulla
>  Labels: datadog, metrics, nifi, reporting
> Fix For: 1.0.0
>
>
> Added reporting task, that reports NiFi flow and JVM metrics to DataDog 
> service. It also reports data to DataDog from every processor. All metrics 
> are represented as Gauges. The name of each metric for processors has 
> following format: “nifi..”. In order to use this module, you need to have 
> DataDog agent installed on your machine. You can configure time period to 
> send metrics to DataDog by changing “DataDog reporting period” property when 
> adding reporting task in NiFi UI. 
> Attached link to corresponding pull request on GitHub.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2269) DataDog support added

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431962#comment-15431962
 ] 

ASF subversion and git services commented on NIFI-2269:
---

Commit 376d3c4ef4a837fd18213fa3d77eda7d12beed25 in nifi's branch 
refs/heads/master from [~abadu...@live.ru]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=376d3c4 ]

NIFI-2269 Added datadog support

This closes #655

Signed-off-by: jpercivall 


> DataDog support added
> -
>
> Key: NIFI-2269
> URL: https://issues.apache.org/jira/browse/NIFI-2269
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Ramiz Abadulla
>  Labels: datadog, metrics, nifi, reporting
> Fix For: 1.0.0
>
>
> Added reporting task, that reports NiFi flow and JVM metrics to DataDog 
> service. It also reports data to DataDog from every processor. All metrics 
> are represented as Gauges. The name of each metric for processors has 
> following format: “nifi..”. In order to use this module, you need to have 
> DataDog agent installed on your machine. You can configure time period to 
> send metrics to DataDog by changing “DataDog reporting period” property when 
> adding reporting task in NiFi UI. 
> Attached link to corresponding pull request on GitHub.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2269) DataDog support added

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-2269:
---
   Resolution: Fixed
Fix Version/s: 1.0.0
   Status: Resolved  (was: Patch Available)

> DataDog support added
> -
>
> Key: NIFI-2269
> URL: https://issues.apache.org/jira/browse/NIFI-2269
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Ramiz Abadulla
>  Labels: datadog, metrics, nifi, reporting
> Fix For: 1.0.0
>
>
> Added reporting task, that reports NiFi flow and JVM metrics to DataDog 
> service. It also reports data to DataDog from every processor. All metrics 
> are represented as Gauges. The name of each metric for processors has 
> following format: “nifi..”. In order to use this module, you need to have 
> DataDog agent installed on your machine. You can configure time period to 
> send metrics to DataDog by changing “DataDog reporting period” property when 
> adding reporting task in NiFi UI. 
> Attached link to corresponding pull request on GitHub.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #655: DataDog support added

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/655


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #655: DataDog support added

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/655
  
I will be adding a patch on top of this to fix a couple issues with styling 
(see travis-ci build) and licensing but:

+1 

Visually verified the code and did a contrib check build. In a standalone 
instance and secure cluster I did basic functional testing in both Agent and 
HTTP mode (both require adding the dashboard in Datadog with the appropriate 
metrics). Thanks @Ramizjon for the contribution, I will merge it in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #900: NIFI-2605: Fixing a regression bug where nodes would...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/900#discussion_r75789301
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/integration/Node.java
 ---
@@ -311,6 +320,11 @@ public boolean hasRole(final String roleName) {
 return leaderAddress.equals(getClusterAddress());
 }
 
+public void addProcessor(final Processor example) throws 
ProcessorInstantiationException {
--- End diff --

You're right - it shouldn't be there. I started to add some additional 
methods to improve the power of the testing by allowing it to change the flow 
so that we could easily test how the cluster behaves when the flows differ 
between nodes. But then I removed the methods because they didn't belong on 
this PR and it looks like I forgot to remove that method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431933#comment-15431933
 ] 

ASF GitHub Bot commented on NIFI-2605:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/900#discussion_r75789301
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/integration/Node.java
 ---
@@ -311,6 +320,11 @@ public boolean hasRole(final String roleName) {
 return leaderAddress.equals(getClusterAddress());
 }
 
+public void addProcessor(final Processor example) throws 
ProcessorInstantiationException {
--- End diff --

You're right - it shouldn't be there. I started to add some additional 
methods to improve the power of the testing by allowing it to change the flow 
so that we could easily test how the cluster behaves when the flows differ 
between nodes. But then I removed the methods because they didn't belong on 
this PR and it looks like I forgot to remove that method.


> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}
> Edit: As per the discussion below, this manifested due to 3 different 
> problems. The one being solved in this ticket is the one Mark addressed in 
> the linked PR, that a node will always register to be Cluster Coordinator 
> when it joins a cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431926#comment-15431926
 ] 

ASF GitHub Bot commented on NIFI-2605:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/900
  
@JPercivall rebased against master and also removed that addProcessor() 
method from the Node class


> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}
> Edit: As per the discussion below, this manifested due to 3 different 
> problems. The one being solved in this ticket is the one Mark addressed in 
> the linked PR, that a node will always register to be Cluster Coordinator 
> when it joins a cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #900: NIFI-2605: Fixing a regression bug where nodes would potent...

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/900
  
@JPercivall rebased against master and also removed that addProcessor() 
method from the Node class


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #900: NIFI-2605: Fixing a regression bug where nodes would...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/900#discussion_r75788681
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/leader/election/LeaderElectionManager.java
 ---
@@ -40,7 +35,8 @@
 void register(String roleName, LeaderElectionStateChangeListener 
listener);
 
 /**
- * Adds a new role for which a leader is required, providing the given 
value for this node as the Participant ID
+ * Adds a new role for which a leader is required, providing the given 
value for this node as the Participant ID. If the Participant ID
+ * is null, this node will never be elected leader but 
will passively observe changes to the leadership.
--- End diff --

@JPercivall that means the node will never been elected leader... unless it 
later re-registers, providing a Participant ID. I.e., the node can register as 
an observer and later re-register as a participant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431920#comment-15431920
 ] 

ASF GitHub Bot commented on NIFI-2605:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/900#discussion_r75788681
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/leader/election/LeaderElectionManager.java
 ---
@@ -40,7 +35,8 @@
 void register(String roleName, LeaderElectionStateChangeListener 
listener);
 
 /**
- * Adds a new role for which a leader is required, providing the given 
value for this node as the Participant ID
+ * Adds a new role for which a leader is required, providing the given 
value for this node as the Participant ID. If the Participant ID
+ * is null, this node will never be elected leader but 
will passively observe changes to the leadership.
--- End diff --

@JPercivall that means the node will never been elected leader... unless it 
later re-registers, providing a Participant ID. I.e., the node can register as 
an observer and later re-register as a participant.


> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}
> Edit: As per the discussion below, this manifested due to 3 different 
> problems. The one being solved in this ticket is the one Mark addressed in 
> the linked PR, that a node will always register to be Cluster Coordinator 
> when it joins a cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2629) Hide "+ Add Property" in Processors that don't allow dynamic properties

[Kirk Tarou (JIRA)]

Kirk Tarou created NIFI-2629:


 Summary: Hide "+ Add Property" in Processors that don't allow 
dynamic properties
 Key: NIFI-2629
 URL: https://issues.apache.org/jira/browse/NIFI-2629
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework, Core UI
Affects Versions: 0.7.0
Reporter: Kirk Tarou
Priority: Minor


A few Processors expect or allow extra properties to be added on the 
'Properties' tab in the Processor Detail view. The rest of the Processors will 
show a "not a supported property" error.

To avoid confusion, the "+ New property" widget should be hidden if dynamic 
properties are not allowed.

The existence of a "DynamicProperty" annotation in a processor could be used as 
an indicator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431813#comment-15431813
 ] 

ASF GitHub Bot commented on NIFI-2605:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/900#discussion_r75781210
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/leader/election/LeaderElectionManager.java
 ---
@@ -40,7 +35,8 @@
 void register(String roleName, LeaderElectionStateChangeListener 
listener);
 
 /**
- * Adds a new role for which a leader is required, providing the given 
value for this node as the Participant ID
+ * Adds a new role for which a leader is required, providing the given 
value for this node as the Participant ID. If the Participant ID
+ * is null, this node will never be elected leader but 
will passively observe changes to the leadership.
--- End diff --

Does this mean that in the future this node could never become the leader?


> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}
> Edit: As per the discussion below, this manifested due to 3 different 
> problems. The one being solved in this ticket is the one Mark addressed in 
> the linked PR, that a node will always register to be Cluster Coordinator 
> when it joins a cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #900: NIFI-2605: Fixing a regression bug where nodes would...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/900#discussion_r75781210
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/leader/election/LeaderElectionManager.java
 ---
@@ -40,7 +35,8 @@
 void register(String roleName, LeaderElectionStateChangeListener 
listener);
 
 /**
- * Adds a new role for which a leader is required, providing the given 
value for this node as the Participant ID
+ * Adds a new role for which a leader is required, providing the given 
value for this node as the Participant ID. If the Participant ID
+ * is null, this node will never be elected leader but 
will passively observe changes to the leadership.
--- End diff --

Does this mean that in the future this node could never become the leader?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2628) NiFi node does not save flow returned by cluster coodinator to disk immediately

[Arpit Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Gupta updated NIFI-2628:
--
Summary: NiFi node does not save flow returned by cluster coodinator to 
disk immediately  (was: NiFi node does not save flow returned by CC to disk 
immediately)

> NiFi node does not save flow returned by cluster coodinator to disk 
> immediately
> ---
>
> Key: NIFI-2628
> URL: https://issues.apache.org/jira/browse/NIFI-2628
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
> Fix For: 1.1.0
>
>
> While investigating NIFI-2605 it was determined that NiFi node can return the 
> flow to the end user from memory and not immediately save the flow to disk. 
> This in some rare cases can lead to issues. 
> See for more detail 
> https://issues.apache.org/jira/browse/NIFI-2605?focusedCommentId=15431737=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15431737



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2628) NiFi node does not save flow returned by cluster coodinator to disk immediately

[Arpit Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Gupta updated NIFI-2628:
--
Description: 
While investigating NIFI-2605 it was determined that NiFi node does not save 
the flow returned by cluster coordinator to disk immediately. This in some rare 
cases can lead to issues. 

See for more detail 

https://issues.apache.org/jira/browse/NIFI-2605?focusedCommentId=15431737=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15431737

  was:
While investigating NIFI-2605 it was determined that NiFi node can return the 
flow to the end user from memory and not immediately save the flow to disk. 
This in some rare cases can lead to issues. 

See for more detail 

https://issues.apache.org/jira/browse/NIFI-2605?focusedCommentId=15431737=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15431737


> NiFi node does not save flow returned by cluster coodinator to disk 
> immediately
> ---
>
> Key: NIFI-2628
> URL: https://issues.apache.org/jira/browse/NIFI-2628
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
> Fix For: 1.1.0
>
>
> While investigating NIFI-2605 it was determined that NiFi node does not save 
> the flow returned by cluster coordinator to disk immediately. This in some 
> rare cases can lead to issues. 
> See for more detail 
> https://issues.apache.org/jira/browse/NIFI-2605?focusedCommentId=15431737=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15431737



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2440) Add last modified time & timestamp attributes to flow files generated by ListSFTP processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431800#comment-15431800
 ] 

ASF GitHub Bot commented on NIFI-2440:
--

GitHub user kirkalicious opened a pull request:

https://github.com/apache/nifi/pull/913

NIFI-2440 - Add 'file.lastModifiedTime' attribute to ListSFTP processor

Added 'file.lastModifiedTime' attribute to ListFileTransfer, which is
the abstract class extended by ListSFTP.
String literal attribute names were replaced with static references to
attribute name constants in ListFile.
ListFileTransfer stores the 'file.lastModifiedTime' attribute in the
format specified in ListFile.FILE_MODIFY_DATE_ATTR_FORMAT

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kirkalicious/nifi NIFI-2440

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #913


commit 9018a3794cdfc5570115a877b3dee32bbe3c8ae5
Author: Kirk Tarou 
Date:   2016-08-22T23:34:46Z

NIFI-2440 - Add 'file.lastModifiedTime' attribute to ListSFTP processor

Added 'file.lastModifiedTime' attribute to ListFileTransfer, which is
the abstract class extended by ListSFTP.
String literal attribute names were replaced with static references to
attribute name constants in ListFile.
ListFileTransfer stores the 'file.lastModifiedTime' attribute in the
format specified in ListFile.FILE_MODIFY_DATE_ATTR_FORMAT




> Add last modified time & timestamp attributes to flow files generated by 
> ListSFTP processor
> ---
>
> Key: NIFI-2440
> URL: https://issues.apache.org/jira/browse/NIFI-2440
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Kirk Tarou
>Priority: Trivial
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The timestamp & last modified time attributes are not exposed in ListSFTP so 
> there's no way to preserve the timestamp of the remotely collected files when 
> writing them out to a file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2628) NiFi node does not save flow returned by CC to disk immediately

[Arpit Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Gupta updated NIFI-2628:
--
Summary: NiFi node does not save flow returned by CC to disk immediately  
(was: NiFi node does not save flow to disk immediately and returns flow from 
memory)

> NiFi node does not save flow returned by CC to disk immediately
> ---
>
> Key: NIFI-2628
> URL: https://issues.apache.org/jira/browse/NIFI-2628
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
> Fix For: 1.1.0
>
>
> While investigating NIFI-2605 it was determined that NiFi node can return the 
> flow to the end user from memory and not immediately save the flow to disk. 
> This in some rare cases can lead to issues. 
> See for more detail 
> https://issues.apache.org/jira/browse/NIFI-2605?focusedCommentId=15431737=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15431737



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[Arpit Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431792#comment-15431792
 ] 

Arpit Gupta commented on NIFI-2605:
---

Logged NIFI-2628 for #2.

> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}
> Edit: As per the discussion below, this manifested due to 3 different 
> problems. The one being solved in this ticket is the one Mark addressed in 
> the linked PR, that a node will always register to be Cluster Coordinator 
> when it joins a cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2628) NiFi node does not save flow to disk immediately and returns flow from memory

[Arpit Gupta (JIRA)]

Arpit Gupta created NIFI-2628:
-

 Summary: NiFi node does not save flow to disk immediately and 
returns flow from memory
 Key: NIFI-2628
 URL: https://issues.apache.org/jira/browse/NIFI-2628
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.0.0
Reporter: Arpit Gupta
 Fix For: 1.1.0


While investigating NIFI-2605 it was determined that NiFi node can return the 
flow to the end user from memory and not immediately save the flow to disk. 
This in some rare cases can lead to issues. 

See for more detail 

https://issues.apache.org/jira/browse/NIFI-2605?focusedCommentId=15431737=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15431737



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431778#comment-15431778
 ] 

ASF GitHub Bot commented on NIFI-2605:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/900
  
@markap14 it appears this now has conflicts with Master. When you get a 
chance, could you address them?


> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}
> Edit: As per the discussion below, this manifested due to 3 different 
> problems. The one being solved in this ticket is the one Mark addressed in 
> the linked PR, that a node will always register to be Cluster Coordinator 
> when it joins a cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #900: NIFI-2605: Fixing a regression bug where nodes would potent...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/900
  
@markap14 it appears this now has conflicts with Master. When you get a 
chance, could you address them?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-2605:
---
Description: 
Follow stack trace was present in the node that did not connect

{code}
2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
bytes) from host in 61 millis
2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
Failed to load flow from cluster due to: 
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
to cluster because local flow is different than cluster flow.
at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
 ~[nifi-framework-core-1.0.0.jar:1.0.0]
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
 ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
[nifi-jetty-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.(NiFi.java:137) 
[nifi-runtime-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.main(NiFi.java:227) 
[nifi-runtime-1.0.0.jar:1.0.0]
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
configuration is not inheritable by the flow controller because of flow 
differences: Found difference in Flows:
Local Fingerprint:   
7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
{code}

Edit: As per the discussion below, this manifested due to 3 different problems. 
The one being solved in this ticket is the one Mark addressed in the linked PR, 
that a node will always register to be Cluster Coordinator when it joins a 
cluster.

  was:
Follow stack trace was present in the node that did not connect

{code}
2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
bytes) from host in 61 millis
2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
Failed to load flow from cluster due to: 
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
to cluster because local flow is different than cluster flow.
org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
to cluster because local flow is different than cluster flow.
at 
org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
 ~[nifi-framework-core-1.0.0.jar:1.0.0]
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
 ~[nifi-framework-core-1.0.0.jar:1.0.0]
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
[nifi-jetty-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.(NiFi.java:137) 
[nifi-runtime-1.0.0.jar:1.0.0]
at org.apache.nifi.NiFi.main(NiFi.java:227) 
[nifi-runtime-1.0.0.jar:1.0.0]
Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
configuration is not inheritable by the flow controller because of flow 
differences: Found difference in Flows:
Local Fingerprint:   
7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
{code}


> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> 

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431772#comment-15431772
 ] 

Joseph Percivall commented on NIFI-2605:


That all makes sense and I agree with your path forward. I'll modify the ticket 
body a bit to reflect it's intended problem and continue reviewing Mark's PR.

Could you create a new Jira for #2 and assign it for 1.1.0? It's minor issue 
that, while inconvenient, doesn't represent any threat (data loss, etc.) and 
easily could lead to unforeseen/unintended problems.

> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-2605:
---
Comment: was deleted

(was: That all makes sense and I agree with your path forward. I'll modify the 
ticket body a bit to reflect it's intended problem and continue reviewing 
Mark's PR.

Could you create a new Jira for #2 and assign it for 1.1.0? It's minor issue 
that, while inconvenient, doesn't represent any threat (data loss, etc.) and 
easily could lead to unforeseen/unintended problems.)

> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431771#comment-15431771
 ] 

Joseph Percivall commented on NIFI-2605:


That all makes sense and I agree with your path forward. I'll modify the ticket 
body a bit to reflect it's intended problem and continue reviewing Mark's PR.

Could you create a new Jira for #2 and assign it for 1.1.0? It's minor issue 
that, while inconvenient, doesn't represent any threat (data loss, etc.) and 
easily could lead to unforeseen/unintended problems.

> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #709: NIFI-2369 - Added EL support to CSV properties

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/709#discussion_r75773261
  
--- Diff: 
nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
 ---
@@ -191,57 +192,40 @@ public void testEmptyContent() throws IOException {
 }
 
 @Test
-public void testCSVProperties() throws IOException {
+public void testBasicConversionNoErrors() throws IOException {
 TestRunner runner = 
TestRunners.newTestRunner(ConvertCSVToAvro.class);
-ConvertCSVToAvro processor = new ConvertCSVToAvro();
-ProcessContext context = runner.getProcessContext();
-
-// check defaults
-processor.createCSVProperties(context);
-Assert.assertEquals("Charset should match",
-"utf8", processor.props.charset);
-Assert.assertEquals("Delimiter should match",
-",", processor.props.delimiter);
-Assert.assertEquals("Quote should match",
-"\"", processor.props.quote);
-Assert.assertEquals("Escape should match",
-"\\", processor.props.escape);
-Assert.assertEquals("Header flag should match",
-false, processor.props.useHeader);
-Assert.assertEquals("Lines to skip should match",
-0, processor.props.linesToSkip);
+runner.assertNotValid();
+runner.setProperty(ConvertCSVToAvro.SCHEMA, SCHEMA.toString());
+runner.assertValid();
 
-runner.setProperty(ConvertCSVToAvro.CHARSET, "utf16");
-runner.setProperty(ConvertCSVToAvro.DELIMITER, "|");
-runner.setProperty(ConvertCSVToAvro.QUOTE, "'");
-runner.setProperty(ConvertCSVToAvro.ESCAPE, "\u2603");
-runner.setProperty(ConvertCSVToAvro.HAS_HEADER, "true");
-runner.setProperty(ConvertCSVToAvro.LINES_TO_SKIP, "2");
-
-// check updates
-processor.createCSVProperties(context);
-Assert.assertEquals("Charset should match",
-"utf16", processor.props.charset);
-Assert.assertEquals("Delimiter should match",
-"|", processor.props.delimiter);
-Assert.assertEquals("Quote should match",
-"'", processor.props.quote);
-Assert.assertEquals("Escape should match",
-"\u2603", processor.props.escape);
-Assert.assertEquals("Header flag should match",
-true, processor.props.useHeader);
-Assert.assertEquals("Lines to skip should match",
-2, processor.props.linesToSkip);
+runner.enqueue(streamFor("1,green\n2,blue,\n3,grey,12.95"));
+runner.run();
+
+long converted = runner.getCounterValue("Converted records");
+long errors = runner.getCounterValue("Conversion errors");
+Assert.assertEquals("Should convert 3 rows", 3, converted);
+Assert.assertEquals("Should reject 0 row", 0, errors);
+
+runner.assertTransferCount("success", 1);
+runner.assertTransferCount("failure", 0);
+runner.assertTransferCount("incompatible", 0);
 }
 
 @Test
-public void testBasicConversionNoErrors() throws IOException {
+public void testExpresionLanguageBasedCSVProperties() throws 
IOException {
 TestRunner runner = 
TestRunners.newTestRunner(ConvertCSVToAvro.class);
 runner.assertNotValid();
 runner.setProperty(ConvertCSVToAvro.SCHEMA, SCHEMA.toString());
 runner.assertValid();
 
-runner.enqueue(streamFor("1,green\n2,blue,\n3,grey,12.95"));
+runner.setProperty(ConvertCSVToAvro.DELIMITER, "${csv.delimiter}");
+runner.setProperty(ConvertCSVToAvro.QUOTE, "${csv.quote}");
+
+HashMap flowFileAttributes = new 
HashMap();
+flowFileAttributes.put("csv.delimiter", "|");
+flowFileAttributes.put("csv.quote", "~");
+
+
runner.enqueue(streamFor("1|green\n2|~blue|field~|\n3|grey|12.95"), 
flowFileAttributes);
--- End diff --

Might be helpful to check that the fields are converted as expected (by 
reading the resulting Avro file(s)) instead of just the total number of records 
produced.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2369) ConvertCSVtoAvro should allow EL for CSV Properties

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431743#comment-15431743
 ] 

ASF GitHub Bot commented on NIFI-2369:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/709#discussion_r75773261
  
--- Diff: 
nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
 ---
@@ -191,57 +192,40 @@ public void testEmptyContent() throws IOException {
 }
 
 @Test
-public void testCSVProperties() throws IOException {
+public void testBasicConversionNoErrors() throws IOException {
 TestRunner runner = 
TestRunners.newTestRunner(ConvertCSVToAvro.class);
-ConvertCSVToAvro processor = new ConvertCSVToAvro();
-ProcessContext context = runner.getProcessContext();
-
-// check defaults
-processor.createCSVProperties(context);
-Assert.assertEquals("Charset should match",
-"utf8", processor.props.charset);
-Assert.assertEquals("Delimiter should match",
-",", processor.props.delimiter);
-Assert.assertEquals("Quote should match",
-"\"", processor.props.quote);
-Assert.assertEquals("Escape should match",
-"\\", processor.props.escape);
-Assert.assertEquals("Header flag should match",
-false, processor.props.useHeader);
-Assert.assertEquals("Lines to skip should match",
-0, processor.props.linesToSkip);
+runner.assertNotValid();
+runner.setProperty(ConvertCSVToAvro.SCHEMA, SCHEMA.toString());
+runner.assertValid();
 
-runner.setProperty(ConvertCSVToAvro.CHARSET, "utf16");
-runner.setProperty(ConvertCSVToAvro.DELIMITER, "|");
-runner.setProperty(ConvertCSVToAvro.QUOTE, "'");
-runner.setProperty(ConvertCSVToAvro.ESCAPE, "\u2603");
-runner.setProperty(ConvertCSVToAvro.HAS_HEADER, "true");
-runner.setProperty(ConvertCSVToAvro.LINES_TO_SKIP, "2");
-
-// check updates
-processor.createCSVProperties(context);
-Assert.assertEquals("Charset should match",
-"utf16", processor.props.charset);
-Assert.assertEquals("Delimiter should match",
-"|", processor.props.delimiter);
-Assert.assertEquals("Quote should match",
-"'", processor.props.quote);
-Assert.assertEquals("Escape should match",
-"\u2603", processor.props.escape);
-Assert.assertEquals("Header flag should match",
-true, processor.props.useHeader);
-Assert.assertEquals("Lines to skip should match",
-2, processor.props.linesToSkip);
+runner.enqueue(streamFor("1,green\n2,blue,\n3,grey,12.95"));
+runner.run();
+
+long converted = runner.getCounterValue("Converted records");
+long errors = runner.getCounterValue("Conversion errors");
+Assert.assertEquals("Should convert 3 rows", 3, converted);
+Assert.assertEquals("Should reject 0 row", 0, errors);
+
+runner.assertTransferCount("success", 1);
+runner.assertTransferCount("failure", 0);
+runner.assertTransferCount("incompatible", 0);
 }
 
 @Test
-public void testBasicConversionNoErrors() throws IOException {
+public void testExpresionLanguageBasedCSVProperties() throws 
IOException {
 TestRunner runner = 
TestRunners.newTestRunner(ConvertCSVToAvro.class);
 runner.assertNotValid();
 runner.setProperty(ConvertCSVToAvro.SCHEMA, SCHEMA.toString());
 runner.assertValid();
 
-runner.enqueue(streamFor("1,green\n2,blue,\n3,grey,12.95"));
+runner.setProperty(ConvertCSVToAvro.DELIMITER, "${csv.delimiter}");
+runner.setProperty(ConvertCSVToAvro.QUOTE, "${csv.quote}");
+
+HashMap flowFileAttributes = new 
HashMap();
+flowFileAttributes.put("csv.delimiter", "|");
+flowFileAttributes.put("csv.quote", "~");
+
+
runner.enqueue(streamFor("1|green\n2|~blue|field~|\n3|grey|12.95"), 
flowFileAttributes);
--- End diff --

Might be helpful to check that the fields are converted as expected (by 
reading the resulting Avro file(s)) instead of just the total number of records 
produced.


> ConvertCSVtoAvro should allow EL for CSV Properties
> ---
>
> Key: NIFI-2369
> URL: 

[jira] [Commented] (NIFI-2369) ConvertCSVtoAvro should allow EL for CSV Properties

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431742#comment-15431742
 ] 

ASF GitHub Bot commented on NIFI-2369:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/709#discussion_r75773057
  
--- Diff: 
nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
 ---
@@ -191,57 +192,40 @@ public void testEmptyContent() throws IOException {
 }
 
 @Test
-public void testCSVProperties() throws IOException {
+public void testBasicConversionNoErrors() throws IOException {
 TestRunner runner = 
TestRunners.newTestRunner(ConvertCSVToAvro.class);
-ConvertCSVToAvro processor = new ConvertCSVToAvro();
-ProcessContext context = runner.getProcessContext();
-
-// check defaults
-processor.createCSVProperties(context);
-Assert.assertEquals("Charset should match",
-"utf8", processor.props.charset);
-Assert.assertEquals("Delimiter should match",
-",", processor.props.delimiter);
-Assert.assertEquals("Quote should match",
-"\"", processor.props.quote);
-Assert.assertEquals("Escape should match",
-"\\", processor.props.escape);
-Assert.assertEquals("Header flag should match",
-false, processor.props.useHeader);
-Assert.assertEquals("Lines to skip should match",
-0, processor.props.linesToSkip);
+runner.assertNotValid();
+runner.setProperty(ConvertCSVToAvro.SCHEMA, SCHEMA.toString());
+runner.assertValid();
 
-runner.setProperty(ConvertCSVToAvro.CHARSET, "utf16");
-runner.setProperty(ConvertCSVToAvro.DELIMITER, "|");
-runner.setProperty(ConvertCSVToAvro.QUOTE, "'");
-runner.setProperty(ConvertCSVToAvro.ESCAPE, "\u2603");
-runner.setProperty(ConvertCSVToAvro.HAS_HEADER, "true");
-runner.setProperty(ConvertCSVToAvro.LINES_TO_SKIP, "2");
-
-// check updates
-processor.createCSVProperties(context);
-Assert.assertEquals("Charset should match",
-"utf16", processor.props.charset);
-Assert.assertEquals("Delimiter should match",
-"|", processor.props.delimiter);
-Assert.assertEquals("Quote should match",
-"'", processor.props.quote);
-Assert.assertEquals("Escape should match",
-"\u2603", processor.props.escape);
-Assert.assertEquals("Header flag should match",
-true, processor.props.useHeader);
-Assert.assertEquals("Lines to skip should match",
-2, processor.props.linesToSkip);
+runner.enqueue(streamFor("1,green\n2,blue,\n3,grey,12.95"));
+runner.run();
+
+long converted = runner.getCounterValue("Converted records");
+long errors = runner.getCounterValue("Conversion errors");
+Assert.assertEquals("Should convert 3 rows", 3, converted);
+Assert.assertEquals("Should reject 0 row", 0, errors);
+
+runner.assertTransferCount("success", 1);
+runner.assertTransferCount("failure", 0);
+runner.assertTransferCount("incompatible", 0);
 }
 
 @Test
-public void testBasicConversionNoErrors() throws IOException {
+public void testExpresionLanguageBasedCSVProperties() throws 
IOException {
--- End diff --

Minor typo in test name here


> ConvertCSVtoAvro should allow EL for CSV Properties
> ---
>
> Key: NIFI-2369
> URL: https://issues.apache.org/jira/browse/NIFI-2369
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.1
> Environment: N/A
>Reporter: Simon Elliston Ball
>Assignee: Joseph Witt
>Priority: Minor
>  Labels: starter
> Attachments: ConvertCSVAvro_-_Delimiter_Example.xml
>
>
> When creating a common processing flow, it would be useful to be able to 
> parameterise the CSV properties, delimiter, quote and escape characters and 
> other processing settings. It should be possible to set these based on 
> attributes of the incoming CSV file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #709: NIFI-2369 - Added EL support to CSV properties

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/709#discussion_r75773057
  
--- Diff: 
nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/test/java/org/apache/nifi/processors/kite/TestCSVToAvroProcessor.java
 ---
@@ -191,57 +192,40 @@ public void testEmptyContent() throws IOException {
 }
 
 @Test
-public void testCSVProperties() throws IOException {
+public void testBasicConversionNoErrors() throws IOException {
 TestRunner runner = 
TestRunners.newTestRunner(ConvertCSVToAvro.class);
-ConvertCSVToAvro processor = new ConvertCSVToAvro();
-ProcessContext context = runner.getProcessContext();
-
-// check defaults
-processor.createCSVProperties(context);
-Assert.assertEquals("Charset should match",
-"utf8", processor.props.charset);
-Assert.assertEquals("Delimiter should match",
-",", processor.props.delimiter);
-Assert.assertEquals("Quote should match",
-"\"", processor.props.quote);
-Assert.assertEquals("Escape should match",
-"\\", processor.props.escape);
-Assert.assertEquals("Header flag should match",
-false, processor.props.useHeader);
-Assert.assertEquals("Lines to skip should match",
-0, processor.props.linesToSkip);
+runner.assertNotValid();
+runner.setProperty(ConvertCSVToAvro.SCHEMA, SCHEMA.toString());
+runner.assertValid();
 
-runner.setProperty(ConvertCSVToAvro.CHARSET, "utf16");
-runner.setProperty(ConvertCSVToAvro.DELIMITER, "|");
-runner.setProperty(ConvertCSVToAvro.QUOTE, "'");
-runner.setProperty(ConvertCSVToAvro.ESCAPE, "\u2603");
-runner.setProperty(ConvertCSVToAvro.HAS_HEADER, "true");
-runner.setProperty(ConvertCSVToAvro.LINES_TO_SKIP, "2");
-
-// check updates
-processor.createCSVProperties(context);
-Assert.assertEquals("Charset should match",
-"utf16", processor.props.charset);
-Assert.assertEquals("Delimiter should match",
-"|", processor.props.delimiter);
-Assert.assertEquals("Quote should match",
-"'", processor.props.quote);
-Assert.assertEquals("Escape should match",
-"\u2603", processor.props.escape);
-Assert.assertEquals("Header flag should match",
-true, processor.props.useHeader);
-Assert.assertEquals("Lines to skip should match",
-2, processor.props.linesToSkip);
+runner.enqueue(streamFor("1,green\n2,blue,\n3,grey,12.95"));
+runner.run();
+
+long converted = runner.getCounterValue("Converted records");
+long errors = runner.getCounterValue("Conversion errors");
+Assert.assertEquals("Should convert 3 rows", 3, converted);
+Assert.assertEquals("Should reject 0 row", 0, errors);
+
+runner.assertTransferCount("success", 1);
+runner.assertTransferCount("failure", 0);
+runner.assertTransferCount("incompatible", 0);
 }
 
 @Test
-public void testBasicConversionNoErrors() throws IOException {
+public void testExpresionLanguageBasedCSVProperties() throws 
IOException {
--- End diff --

Minor typo in test name here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (NIFI-2611) UnpackContent cannot unpack any type of flowfile

[Joseph Gresock (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Gresock closed NIFI-2611.


> UnpackContent cannot unpack any type of flowfile
> 
>
> Key: NIFI-2611
> URL: https://issues.apache.org/jira/browse/NIFI-2611
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.7.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
> Fix For: 1.0.0, 0.8.0
>
>
> Two possibly separate problems:
> *Flowfile-stream-v2 and v3*
> This may be a problem with either MergeContent's production of 
> flowfile-stream v2 and v3, or with UnpackContent's inability to unpack them, 
> not sure which.  Here is a screen shot with how to reproduce it: 
> https://ibin.co/2sCwqbFbAs3a.png
> Essentially, when you pack a flow file as flowfile-stream v2 or v3, a 
> subsequent UnpackContent set to the respective type fails with the error 
> "Cannot unpack {} because it does not appear to have any entries".
> *Flowfile-tar-v1*
> When selecting flowfile-tar-v1 from UnpackContent, you immediately get 
> @OnScheduled error failure as soon as you start the processor, which prevents 
> it from processing any incoming flow files.  Here is a screenshot: 
> https://ibin.co/2sCxI4iDm88t.png



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[Arpit Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431737#comment-15431737
 ] 

Arpit Gupta edited comment on NIFI-2605 at 8/22/16 10:37 PM:
-

Here is some more detail about the issue.

This issue was seen when a node was disconnected and flow.xml was removed and 
node was started back up. After node came back up it had inherited the flow and 
then a restart of the whole cluster was issued.

So i think we might have multiple issues here

1. node will always think there is no CC when connecting and will register to 
be the CC.

2. node will then ask the current CC for the latest flow and holds it in memory 
(but does not save it to disk to immediately). In the above case the node that 
was disconnected became the CC and when asked for the latest flow returns the 
flow from memory.

3. when starting back up the node that never saved the flow to disk is elected 
CC, causing all other nodes to have un-inheritable flows

I think this jira/pr is solving #1. #2 is a new issue and #3 is being resolved 
as part of NIFI-1966



was (Author: arpitgupta):
Here is some more detail about the issue.

This issue was seen when a node was disconnected and flow.xml was removed and 
node was started back up. After node came back up it had inherited the flow and 
then a restart of the whole cluster was issued.

So i think we might have multiple issues here

1. node will always think there is no CC when connecting and will register to 
be the CC.

2. node will then ask the current CC for the latest flow and holds it in memory 
(but does not save it to disk to immediately). In the above case the node that 
was disconnected became the CC and when asked for the latest flow returns the 
flow from memory.

3. when starting back up the node that never saved the flow to disk is elected 
CC, causing all other nodes to have un-inheritable flows

I think the jira is solving #1. #2 is a new issue and #3 is being resolved as 
part of NIFI-1966


> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[Arpit Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431737#comment-15431737
 ] 

Arpit Gupta commented on NIFI-2605:
---

Here is some more detail about the issue.

This issue was seen when a node was disconnected and flow.xml was removed and 
node was started back up. After node came back up it had inherited the flow and 
then a restart of the whole cluster was issued.

So i think we might have multiple issues here

1. node will always think there is no CC when connecting and will register to 
be the CC.

2. node will then ask the current CC for the latest flow and holds it in memory 
(but does not save it to disk to immediately). In the above case the node that 
was disconnected became the CC and when asked for the latest flow returns the 
flow from memory.

3. when starting back up the node that never saved the flow to disk is elected 
CC, causing all other nodes to have un-inheritable flows

I think the jira is solving #1. #2 is a new issue and #3 is being resolved as 
part of NIFI-1966


> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431608#comment-15431608
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
Thanks for all the comments @bbende . I have incorporated them and will be 
pushing a new commit as soon as the build & checkstyle complete. 


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
> 020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:16 $ more password.enc
> VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:55 $
> {code}
> In 

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
Thanks for all the comments @bbende . I have incorporated them and will be 
pushing a new commit as soon as the build & checkstyle complete. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #911: NIFI-2610 added initial support for categorizing tes...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/911#discussion_r75756188
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/testutil/IntegrationTest.java ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.testutil;
+
+import org.junit.experimental.categories.Category;
--- End diff --

Just javadoc


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #911: NIFI-2610 added initial support for categorizing tes...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/911#discussion_r75756079
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestProcessorLifecycle.java
 ---
@@ -191,6 +193,7 @@ public void 
validateStopCallsAreMeaninglessIfProcessorNotStarted() throws Except
  * operations can only be @OnScheduled, @OnUnscheduled, @OnStopped.
  */
 @Test
+@Category(IntegrationTest.class)
--- End diff --

Forgot to mention. The two tests that you see here (as an example) were 
failing sporadically, due to unexpected delays related to the load of the full 
build.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2610) TestProcessorLifecycle and TestStandardProcessScheduler classes causes brittle builds and appears to be an integration test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431600#comment-15431600
 ] 

ASF GitHub Bot commented on NIFI-2610:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/911#discussion_r75755889
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestProcessorLifecycle.java
 ---
@@ -191,6 +193,7 @@ public void 
validateStopCallsAreMeaninglessIfProcessorNotStarted() throws Except
  * operations can only be @OnScheduled, @OnUnscheduled, @OnStopped.
  */
 @Test
+@Category(IntegrationTest.class)
--- End diff --

These are the tests that deal with concurrency which is hard to tune when 
are running under load of full build. So these annotation will exclude them 
from full build yet they are still valid JUnit tests and will run under JUnit 
as before essentially sparing you from commenting/uncommenting @Ignore.


> TestProcessorLifecycle and TestStandardProcessScheduler classes causes 
> brittle builds and appears to be an integration test
> ---
>
> Key: NIFI-2610
> URL: https://issues.apache.org/jira/browse/NIFI-2610
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> The tests in TestProcessorLifecycle appear to be attempting to replicate 
> various threading scenarios.  Such tests are notoriously difficult to get 
> right and indeed the build is brittle as a result.  These tests are likely 
> valuable and should be improved but they also should be considered 
> integration tests it appears.
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.708 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateSuccessfullAndOrderlyShutdown(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 6.313 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateSuccessfullAndOrderlyShutdown(TestProcessorLifecycle.java:224)
> This test also causes build problems and seems to be of a similar style
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.447 sec 
> <<< FAILURE! - in 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler
> validateEnabledDisableMultiThread(org.apache.nifi.controller.scheduling.TestStandardProcessScheduler)
>  Time elapsed: 5.667 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler.validateEnabledDisableMultiThread(TestStandardProcessScheduler.java:373)
> Brittle tests like this risk the build process which harms the review cycle 
> and complicates release voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #911: NIFI-2610 added initial support for categorizing tes...

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/911#discussion_r75755889
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestProcessorLifecycle.java
 ---
@@ -191,6 +193,7 @@ public void 
validateStopCallsAreMeaninglessIfProcessorNotStarted() throws Except
  * operations can only be @OnScheduled, @OnUnscheduled, @OnStopped.
  */
 @Test
+@Category(IntegrationTest.class)
--- End diff --

These are the tests that deal with concurrency which is hard to tune when 
are running under load of full build. So these annotation will exclude them 
from full build yet they are still valid JUnit tests and will run under JUnit 
as before essentially sparing you from commenting/uncommenting @Ignore.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2610) TestProcessorLifecycle and TestStandardProcessScheduler classes causes brittle builds and appears to be an integration test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431595#comment-15431595
 ] 

ASF GitHub Bot commented on NIFI-2610:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/911#discussion_r75755578
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/testutil/IntegrationTest.java ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.testutil;
+
+import org.junit.experimental.categories.Category;
--- End diff --

I haven't compiled yet but I can't see where (except for Javadoc) the 
Category class is used. Does this cause checkstyle errors, or does Javadoc need 
it as an import or something?


> TestProcessorLifecycle and TestStandardProcessScheduler classes causes 
> brittle builds and appears to be an integration test
> ---
>
> Key: NIFI-2610
> URL: https://issues.apache.org/jira/browse/NIFI-2610
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> The tests in TestProcessorLifecycle appear to be attempting to replicate 
> various threading scenarios.  Such tests are notoriously difficult to get 
> right and indeed the build is brittle as a result.  These tests are likely 
> valuable and should be improved but they also should be considered 
> integration tests it appears.
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.708 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateSuccessfullAndOrderlyShutdown(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 6.313 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateSuccessfullAndOrderlyShutdown(TestProcessorLifecycle.java:224)
> This test also causes build problems and seems to be of a similar style
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.447 sec 
> <<< FAILURE! - in 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler
> validateEnabledDisableMultiThread(org.apache.nifi.controller.scheduling.TestStandardProcessScheduler)
>  Time elapsed: 5.667 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler.validateEnabledDisableMultiThread(TestStandardProcessScheduler.java:373)
> Brittle tests like this risk the build process which harms the review cycle 
> and complicates release voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #911: NIFI-2610 added initial support for categorizing tes...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/911#discussion_r75755578
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/testutil/IntegrationTest.java ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.testutil;
+
+import org.junit.experimental.categories.Category;
--- End diff --

I haven't compiled yet but I can't see where (except for Javadoc) the 
Category class is used. Does this cause checkstyle errors, or does Javadoc need 
it as an import or something?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2610) TestProcessorLifecycle and TestStandardProcessScheduler classes causes brittle builds and appears to be an integration test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431592#comment-15431592
 ] 

ASF GitHub Bot commented on NIFI-2610:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/911#discussion_r75755360
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/scheduling/TestProcessorLifecycle.java
 ---
@@ -191,6 +193,7 @@ public void 
validateStopCallsAreMeaninglessIfProcessorNotStarted() throws Except
  * operations can only be @OnScheduled, @OnUnscheduled, @OnStopped.
  */
 @Test
+@Category(IntegrationTest.class)
--- End diff --

This test (and the one following) were not marked as @Ignore before and 
aren't in a named integration test file (starting or ending in IT). Are these 
actually integration tests that have been running as unit tests, or do they 
just take too long to run, or are they issuing failures? In the last case, I 
recommend against labelling failing tests with the Category and keep the 
@Ignore so we can easily find them in source without having to run all 
(integration) tests to see which pass and which fail.


> TestProcessorLifecycle and TestStandardProcessScheduler classes causes 
> brittle builds and appears to be an integration test
> ---
>
> Key: NIFI-2610
> URL: https://issues.apache.org/jira/browse/NIFI-2610
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> The tests in TestProcessorLifecycle appear to be attempting to replicate 
> various threading scenarios.  Such tests are notoriously difficult to get 
> right and indeed the build is brittle as a result.  These tests are likely 
> valuable and should be improved but they also should be considered 
> integration tests it appears.
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.708 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateSuccessfullAndOrderlyShutdown(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 6.313 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateSuccessfullAndOrderlyShutdown(TestProcessorLifecycle.java:224)
> This test also causes build problems and seems to be of a similar style
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.447 sec 
> <<< FAILURE! - in 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler
> validateEnabledDisableMultiThread(org.apache.nifi.controller.scheduling.TestStandardProcessScheduler)
>  Time elapsed: 5.667 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler.validateEnabledDisableMultiThread(TestStandardProcessScheduler.java:373)
> Brittle tests like this risk the build process which harms the review cycle 
> and complicates release voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1613) ConvertJSONToSQL Drops Type Information

[Charles Bryan Clifford (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431579#comment-15431579
 ] 

Charles Bryan Clifford commented on NIFI-1613:
--

As first pointed out by Aaron, ConvertJsonToSQL is the likely source of the 
in-coming flow file attribute entry value truncation. 


> ConvertJSONToSQL Drops Type Information
> ---
>
> Key: NIFI-1613
> URL: https://issues.apache.org/jira/browse/NIFI-1613
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.4.1, 0.5.1
> Environment: Ubuntu 14.04 LTS
>Reporter: Aaron Stephens
>Assignee: Toivo Adams
>  Labels: ConvertJSONToSQL, Phoenix, SQL
>
> It appears that the ConvertJSONToSQL processor is turning Boolean (and 
> possibly Integer and Float) values into Strings.  This is okay for some 
> drivers (like PostgreSQL) which can coerce a String back into a Boolean, but 
> it causes issues for others (specifically Phoenix in my case).
> {noformat}
> org.apache.phoenix.schema.ConstraintViolationException: 
> org.apache.phoenix.schema.TypeMismatchException: ERROR 203 (22005): Type 
> mismatch. VARCHAR cannot be coerced to BOOLEAN
> at 
> org.apache.phoenix.schema.types.PDataType.throwConstraintViolationException(PDataType.java:282)
>  ~[na:na]
> at 
> org.apache.phoenix.schema.types.PBoolean.toObject(PBoolean.java:136) ~[na:na]
> at 
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.setObject(PhoenixPreparedStatement.java:442)
>  ~[na:na]
> at 
> org.apache.commons.dbcp.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:166)
>  ~[na:na]
> at 
> org.apache.commons.dbcp.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:166)
>  ~[na:na]
> at 
> org.apache.nifi.processors.standard.PutSQL.setParameter(PutSQL.java:728) 
> ~[na:na]
> at 
> org.apache.nifi.processors.standard.PutSQL.setParameters(PutSQL.java:606) 
> ~[na:na]
> at 
> org.apache.nifi.processors.standard.PutSQL.onTrigger(PutSQL.java:223) ~[na:na]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>  ~[nifi-api-0.4.1.jar:0.4.1]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1146)
>  ~[nifi-framework-core-0.4.1.jar:0.4.1]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:139)
>  [nifi-framework-core-0.4.1.jar:0.4.1]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
>  [nifi-framework-core-0.4.1.jar:0.4.1]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
>  [nifi-framework-core-0.4.1.jar:0.4.1]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
> [na:1.7.0_79]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) 
> [na:1.7.0_79]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>  [na:1.7.0_79]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  [na:1.7.0_79]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  [na:1.7.0_79]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  [na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> Caused by: org.apache.phoenix.schema.TypeMismatchException: ERROR 203 
> (22005): Type mismatch. VARCHAR cannot be coerced to BOOLEAN
> at 
> org.apache.phoenix.exception.SQLExceptionCode$1.newException(SQLExceptionCode.java:71)
>  ~[na:na]
> at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
>  ~[na:na]
> ... 20 common frames omitted
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #774: Nifi 2440 - Add Last Modified Time attribute to List...

[kirkalicious]

Github user kirkalicious closed the pull request at:

https://github.com/apache/nifi/pull/774


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
Thanks Bryan. The tests were brittle and expected a 256-bit key in some 
cases. I have made adjustments to the test to be more robust in the absence of 
the JCE unlimited strength cryptography jurisdiction policy files. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431558#comment-15431558
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75752491
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/AESSensitivePropertyProvider.java
 ---
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties;
+
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.crypto.BadPaddingException;
+import javax.crypto.Cipher;
+import javax.crypto.IllegalBlockSizeException;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.IvParameterSpec;
+import javax.crypto.spec.SecretKeySpec;
+import org.apache.commons.lang3.StringUtils;
+import org.bouncycastle.util.encoders.Base64;
+import org.bouncycastle.util.encoders.DecoderException;
+import org.bouncycastle.util.encoders.EncoderException;
+import org.bouncycastle.util.encoders.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AESSensitivePropertyProvider implements 
SensitivePropertyProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(AESSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AES Sensitive 
Property Provider";
+private static final String IMPLEMENTATION_KEY = "aes/gcm/";
+private static final String ALGORITHM = "AES/GCM/NoPadding";
+private static final String PROVIDER = "BC";
+private static final String DELIMITER = "||"; // "|" is not a valid 
Base64 character, so ensured not to be present in cipher text
+private static final int IV_LENGTH = 12;
+private static final int MIN_CIPHER_TEXT_LENGTH = IV_LENGTH * 4 / 3 + 
DELIMITER.length() + 1;
+
+private Cipher cipher;
+private SecretKey key;
+
+public AESSensitivePropertyProvider() {
--- End diff --

There was a use case for this originally, but now the only access is from 
the constructor, so I will make this fix. 


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation 

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75752491
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/AESSensitivePropertyProvider.java
 ---
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties;
+
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.crypto.BadPaddingException;
+import javax.crypto.Cipher;
+import javax.crypto.IllegalBlockSizeException;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.IvParameterSpec;
+import javax.crypto.spec.SecretKeySpec;
+import org.apache.commons.lang3.StringUtils;
+import org.bouncycastle.util.encoders.Base64;
+import org.bouncycastle.util.encoders.DecoderException;
+import org.bouncycastle.util.encoders.EncoderException;
+import org.bouncycastle.util.encoders.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AESSensitivePropertyProvider implements 
SensitivePropertyProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(AESSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AES Sensitive 
Property Provider";
+private static final String IMPLEMENTATION_KEY = "aes/gcm/";
+private static final String ALGORITHM = "AES/GCM/NoPadding";
+private static final String PROVIDER = "BC";
+private static final String DELIMITER = "||"; // "|" is not a valid 
Base64 character, so ensured not to be present in cipher text
+private static final int IV_LENGTH = 12;
+private static final int MIN_CIPHER_TEXT_LENGTH = IV_LENGTH * 4 / 3 + 
DELIMITER.length() + 1;
+
+private Cipher cipher;
+private SecretKey key;
+
+public AESSensitivePropertyProvider() {
--- End diff --

There was a use case for this originally, but now the only access is from 
the constructor, so I will make this fix. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431549#comment-15431549
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75751891
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/AESSensitivePropertyProvider.java
 ---
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties;
+
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.crypto.BadPaddingException;
+import javax.crypto.Cipher;
+import javax.crypto.IllegalBlockSizeException;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.IvParameterSpec;
+import javax.crypto.spec.SecretKeySpec;
+import org.apache.commons.lang3.StringUtils;
+import org.bouncycastle.util.encoders.Base64;
+import org.bouncycastle.util.encoders.DecoderException;
+import org.bouncycastle.util.encoders.EncoderException;
+import org.bouncycastle.util.encoders.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AESSensitivePropertyProvider implements 
SensitivePropertyProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(AESSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AES Sensitive 
Property Provider";
+private static final String IMPLEMENTATION_KEY = "aes/gcm/";
+private static final String ALGORITHM = "AES/GCM/NoPadding";
+private static final String PROVIDER = "BC";
+private static final String DELIMITER = "||"; // "|" is not a valid 
Base64 character, so ensured not to be present in cipher text
+private static final int IV_LENGTH = 12;
+private static final int MIN_CIPHER_TEXT_LENGTH = IV_LENGTH * 4 / 3 + 
DELIMITER.length() + 1;
+
+private Cipher cipher;
+private SecretKey key;
+
+public AESSensitivePropertyProvider() {
+
+}
+
+public AESSensitivePropertyProvider(String key) throws 
NoSuchPaddingException, NoSuchAlgorithmException, NoSuchProviderException {
+setKey(key);
+}
+
+public AESSensitivePropertyProvider(byte[] key) throws 
NoSuchPaddingException, NoSuchAlgorithmException, NoSuchProviderException {
+this(key == null ? "" : Hex.toHexString(key));
+}
+
+public void setKey(String keyHex) throws 
SensitivePropertyProtectionException {
+if (!isKeySet()) {
+if (keyHex == null || keyHex.length() == 0) {
--- End diff --

Will do. 


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, 

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431548#comment-15431548
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75751825
  
--- Diff: nifi-commons/nifi-properties/pom.xml ---
@@ -21,4 +21,10 @@
 1.0.0-SNAPSHOT
 
 nifi-properties
+
+
+org.bouncycastle
+bcprov-jdk15on
--- End diff --

Good catch. Thanks Bryan. 


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
> 020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:16 $ more 

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75751891
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/AESSensitivePropertyProvider.java
 ---
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties;
+
+import java.nio.charset.StandardCharsets;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.crypto.BadPaddingException;
+import javax.crypto.Cipher;
+import javax.crypto.IllegalBlockSizeException;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import javax.crypto.spec.IvParameterSpec;
+import javax.crypto.spec.SecretKeySpec;
+import org.apache.commons.lang3.StringUtils;
+import org.bouncycastle.util.encoders.Base64;
+import org.bouncycastle.util.encoders.DecoderException;
+import org.bouncycastle.util.encoders.EncoderException;
+import org.bouncycastle.util.encoders.Hex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AESSensitivePropertyProvider implements 
SensitivePropertyProvider {
+
+private static final Logger logger = 
LoggerFactory.getLogger(AESSensitivePropertyProvider.class);
+
+private static final String IMPLEMENTATION_NAME = "AES Sensitive 
Property Provider";
+private static final String IMPLEMENTATION_KEY = "aes/gcm/";
+private static final String ALGORITHM = "AES/GCM/NoPadding";
+private static final String PROVIDER = "BC";
+private static final String DELIMITER = "||"; // "|" is not a valid 
Base64 character, so ensured not to be present in cipher text
+private static final int IV_LENGTH = 12;
+private static final int MIN_CIPHER_TEXT_LENGTH = IV_LENGTH * 4 / 3 + 
DELIMITER.length() + 1;
+
+private Cipher cipher;
+private SecretKey key;
+
+public AESSensitivePropertyProvider() {
+
+}
+
+public AESSensitivePropertyProvider(String key) throws 
NoSuchPaddingException, NoSuchAlgorithmException, NoSuchProviderException {
+setKey(key);
+}
+
+public AESSensitivePropertyProvider(byte[] key) throws 
NoSuchPaddingException, NoSuchAlgorithmException, NoSuchProviderException {
+this(key == null ? "" : Hex.toHexString(key));
+}
+
+public void setKey(String keyHex) throws 
SensitivePropertyProtectionException {
+if (!isKeySet()) {
+if (keyHex == null || keyHex.length() == 0) {
--- End diff --

Will do. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75751825
  
--- Diff: nifi-commons/nifi-properties/pom.xml ---
@@ -21,4 +21,10 @@
 1.0.0-SNAPSHOT
 
 nifi-properties
+
+
+org.bouncycastle
+bcprov-jdk15on
--- End diff --

Good catch. Thanks Bryan. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2604) JDBC Connection Pool support for lib directory and expression language

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431546#comment-15431546
 ] 

ASF GitHub Bot commented on NIFI-2604:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/912

NIFI-2604: Added validators & logic for multiple URLs/files/folders for DB 
driver location



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-2604

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/912.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #912


commit 6cbd6bdda4268e316d9816eb633fecae6f7f3a93
Author: Matt Burgess 
Date:   2016-08-22T20:25:57Z

NIFI-2604: Added validators and logic for multiple URLs/files/folders for 
DB driver location




> JDBC Connection Pool support for lib directory and expression language
> --
>
> Key: NIFI-2604
> URL: https://issues.apache.org/jira/browse/NIFI-2604
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Witt
>Assignee: Matt Burgess
>
> It would be ideal if the JDBC Connection Service supported specifying a 
> directory instead of particular driver jars.  It would also be helpful if it 
> accepted expression language statements so that it could refer to a location 
> that is based on variable registry values so it is more portable between 
> environments.
> This stems from a user list thread titled "adding dependencies like jdbc 
> drivers to the build" on Aug 18 2016



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2604) JDBC Connection Pool support for lib directory and expression language

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-2604:
---
Status: Patch Available  (was: In Progress)

> JDBC Connection Pool support for lib directory and expression language
> --
>
> Key: NIFI-2604
> URL: https://issues.apache.org/jira/browse/NIFI-2604
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Joseph Witt
>Assignee: Matt Burgess
>
> It would be ideal if the JDBC Connection Service supported specifying a 
> directory instead of particular driver jars.  It would also be helpful if it 
> accepted expression language statements so that it could refer to a location 
> that is based on variable registry values so it is more portable between 
> environments.
> This stems from a user list thread titled "adding dependencies like jdbc 
> drivers to the build" on Aug 18 2016



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #912: NIFI-2604: Added validators & logic for multiple URL...

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/912

NIFI-2604: Added validators & logic for multiple URLs/files/folders for DB 
driver location



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-2604

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/912.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #912


commit 6cbd6bdda4268e316d9816eb633fecae6f7f3a93
Author: Matt Burgess 
Date:   2016-08-22T20:25:57Z

NIFI-2604: Added validators and logic for multiple URLs/files/folders for 
DB driver location




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2536) Update Documentation for Templates

[James Wing (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431524#comment-15431524
 ] 

James Wing commented on NIFI-2536:
--

Thanks for reviewing and merging, [~mcgilman].

> Update Documentation for Templates
> --
>
> Key: NIFI-2536
> URL: https://issues.apache.org/jira/browse/NIFI-2536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.0.0
>
>
> The documentation for importing a template does not reflect recent UI changes 
> for 1.0.  Required updates include:
> * Template features moved to the Operate Palette (create, upload)
> ** Association between template actions and current process group permissions
> * Template features still in the Templates dialog (list, filter, export, 
> delete)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431517#comment-15431517
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
I ran into some test failures when building this PR, possibly due to the 
fact that I don't have JCE installed?

```
Failed tests:
  ConfigEncryptionToolTest.testShouldDeriveKeyFromPassword:337 assert 
derivedKey == KEY_HEX
   |  |  |
   |  |  
0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
   |  false
   0123456789ABCDEFFEDCBA9876543210
  ConfigEncryptionToolTest.testShouldPerformFullOperation Wrong exit status 
expected:<0> but was:<2>

Tests in error:
  ConfigEncryptionToolTest.testShouldEncryptSensitiveProperties:512 » 
IllegalArgument
  ConfigEncryptionToolTest.testShouldParseKey:279 » Key The key (64 hex 
chars) m...
  
ConfigEncryptionToolTest.testShouldSerializeNiFiPropertiesAndPreserveFormat:751 
» IllegalArgument
  
ConfigEncryptionToolTest.testShouldSerializeNiFiPropertiesAndPreserveFormatWithNewPropertyAtEOF:798
 » IllegalArgument
```


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
I ran into some test failures when building this PR, possibly due to the 
fact that I don't have JCE installed?

```
Failed tests:
  ConfigEncryptionToolTest.testShouldDeriveKeyFromPassword:337 assert 
derivedKey == KEY_HEX
       |          |  |
       
|          |  0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
       |          false
       0123456789ABCDEFFEDCBA9876543210
  ConfigEncryptionToolTest.testShouldPerformFullOperation Wrong exit 
status expected:<0> but was:<2>

Tests in error:
  ConfigEncryptionToolTest.testShouldEncryptSensitiveProperties:512 » 
IllegalArgument
  ConfigEncryptionToolTest.testShouldParseKey:279 » Key The key (64 
hex chars) m...

  ConfigEncryptionToolTest.testShouldSerializeNiFiPropertiesAndPreserveFormat:751
 » IllegalArgument

  ConfigEncryptionToolTest.testShouldSerializeNiFiPropertiesAndPreserveFormatWithNewPropertyAtEOF:798
 » IllegalArgument
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2626) LIstenHTTP leaks threads when address is already in use

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431506#comment-15431506
 ] 

ASF subversion and git services commented on NIFI-2626:
---

Commit 184dc219b7e8a9f1c31062d6bc935abc91a94930 in nifi's branch refs/heads/0.x 
from [~jtstorck]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=184dc21 ]

NIFI-2626 Fixes jetty server thread leak. This closes #910


> LIstenHTTP leaks threads when address is already in use
> ---
>
> Key: NIFI-2626
> URL: https://issues.apache.org/jira/browse/NIFI-2626
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.0.0, 0.8.0
>
>
> While testing the current 1.0.0 master branch for release, I ran into a 
> thread leak issue with ListenHTTP.  Though not the fault of ListenHTTP 
> explicitly, threads are being leaked by Jetty after attempting to bind to an 
> address that is already in use.  Jetty starts a thread when server.start() is 
> called, but does not clean up after itself if the start() method throws an 
> exception.  Over time, these threads pile up and eventually the JVM will run 
> out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2626) LIstenHTTP leaks threads when address is already in use

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431509#comment-15431509
 ] 

ASF GitHub Bot commented on NIFI-2626:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/910
  
Thanks @jtstorck. I've merged this into master and cherry picked back into 
0.x


> LIstenHTTP leaks threads when address is already in use
> ---
>
> Key: NIFI-2626
> URL: https://issues.apache.org/jira/browse/NIFI-2626
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.0.0, 0.8.0
>
>
> While testing the current 1.0.0 master branch for release, I ran into a 
> thread leak issue with ListenHTTP.  Though not the fault of ListenHTTP 
> explicitly, threads are being leaked by Jetty after attempting to bind to an 
> address that is already in use.  Jetty starts a thread when server.start() is 
> called, but does not clean up after itself if the start() method throws an 
> exception.  Over time, these threads pile up and eventually the JVM will run 
> out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #910: NIFI-2626 Fixes jetty server thread leak

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/910
  
Thanks @jtstorck. I've merged this into master and cherry picked back into 
0.x


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-2626) LIstenHTTP leaks threads when address is already in use

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman resolved NIFI-2626.
---
   Resolution: Fixed
Fix Version/s: 0.8.0

> LIstenHTTP leaks threads when address is already in use
> ---
>
> Key: NIFI-2626
> URL: https://issues.apache.org/jira/browse/NIFI-2626
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.0.0, 0.8.0
>
>
> While testing the current 1.0.0 master branch for release, I ran into a 
> thread leak issue with ListenHTTP.  Though not the fault of ListenHTTP 
> explicitly, threads are being leaked by Jetty after attempting to bind to an 
> address that is already in use.  Jetty starts a thread when server.start() is 
> called, but does not clean up after itself if the start() method throws an 
> exception.  Over time, these threads pile up and eventually the JVM will run 
> out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2626) LIstenHTTP leaks threads when address is already in use

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431502#comment-15431502
 ] 

ASF GitHub Bot commented on NIFI-2626:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/910


> LIstenHTTP leaks threads when address is already in use
> ---
>
> Key: NIFI-2626
> URL: https://issues.apache.org/jira/browse/NIFI-2626
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.0.0
>
>
> While testing the current 1.0.0 master branch for release, I ran into a 
> thread leak issue with ListenHTTP.  Though not the fault of ListenHTTP 
> explicitly, threads are being leaked by Jetty after attempting to bind to an 
> address that is already in use.  Jetty starts a thread when server.start() is 
> called, but does not clean up after itself if the start() method throws an 
> exception.  Over time, these threads pile up and eventually the JVM will run 
> out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2610) TestProcessorLifecycle and TestStandardProcessScheduler classes causes brittle builds and appears to be an integration test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431500#comment-15431500
 ] 

ASF GitHub Bot commented on NIFI-2610:
--

GitHub user olegz opened a pull request:

https://github.com/apache/nifi/pull/911

NIFI-2610 added initial support for categorizing tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/olegz/nifi NIFI-2610

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/911.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #911


commit f280931cd3cec130bd774db2070889a48d340bf7
Author: Oleg Zhurakousky 
Date:   2016-08-22T19:48:31Z

NIFI-2610 added initial support for categorizing tests




> TestProcessorLifecycle and TestStandardProcessScheduler classes causes 
> brittle builds and appears to be an integration test
> ---
>
> Key: NIFI-2610
> URL: https://issues.apache.org/jira/browse/NIFI-2610
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> The tests in TestProcessorLifecycle appear to be attempting to replicate 
> various threading scenarios.  Such tests are notoriously difficult to get 
> right and indeed the build is brittle as a result.  These tests are likely 
> valuable and should be improved but they also should be considered 
> integration tests it appears.
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.708 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateSuccessfullAndOrderlyShutdown(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 6.313 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateSuccessfullAndOrderlyShutdown(TestProcessorLifecycle.java:224)
> This test also causes build problems and seems to be of a similar style
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.447 sec 
> <<< FAILURE! - in 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler
> validateEnabledDisableMultiThread(org.apache.nifi.controller.scheduling.TestStandardProcessScheduler)
>  Time elapsed: 5.667 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler.validateEnabledDisableMultiThread(TestStandardProcessScheduler.java:373)
> Brittle tests like this risk the build process which harms the review cycle 
> and complicates release voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2610) TestProcessorLifecycle and TestStandardProcessScheduler classes causes brittle builds and appears to be an integration test

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-2610:
---
Status: Patch Available  (was: Open)

> TestProcessorLifecycle and TestStandardProcessScheduler classes causes 
> brittle builds and appears to be an integration test
> ---
>
> Key: NIFI-2610
> URL: https://issues.apache.org/jira/browse/NIFI-2610
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> The tests in TestProcessorLifecycle appear to be attempting to replicate 
> various threading scenarios.  Such tests are notoriously difficult to get 
> right and indeed the build is brittle as a result.  These tests are likely 
> valuable and should be improved but they also should be considered 
> integration tests it appears.
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.708 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateSuccessfullAndOrderlyShutdown(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 6.313 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateSuccessfullAndOrderlyShutdown(TestProcessorLifecycle.java:224)
> This test also causes build problems and seems to be of a similar style
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.447 sec 
> <<< FAILURE! - in 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler
> validateEnabledDisableMultiThread(org.apache.nifi.controller.scheduling.TestStandardProcessScheduler)
>  Time elapsed: 5.667 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler.validateEnabledDisableMultiThread(TestStandardProcessScheduler.java:373)
> Brittle tests like this risk the build process which harms the review cycle 
> and complicates release voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #910: NIFI-2626 Fixes jetty server thread leak

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/910


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2619) ClassLoaderUtils doesn't support spaces between paths or URLs in the paths

[Yolanda M. Davis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yolanda M. Davis updated NIFI-2619:
---
   Resolution: Resolved
Fix Version/s: 1.0.0
   Status: Resolved  (was: Patch Available)

> ClassLoaderUtils doesn't support spaces between paths or URLs in the paths
> --
>
> Key: NIFI-2619
> URL: https://issues.apache.org/jira/browse/NIFI-2619
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.0.0
>
>
> ClassLoaderUtils.getCustomClassLoader() splits a module path list on a comma, 
> but doesn't trim the individual entries.
> Also in ClassLoaderUtils.getURLsForClasspath(), it is assumed that the paths 
> are Files, but they could also already be URLs. Logic should be added to 
> check if the path is already a valid URL, and add it to the list if so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2619) ClassLoaderUtils doesn't support spaces between paths or URLs in the paths

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431458#comment-15431458
 ] 

ASF GitHub Bot commented on NIFI-2619:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/907


> ClassLoaderUtils doesn't support spaces between paths or URLs in the paths
> --
>
> Key: NIFI-2619
> URL: https://issues.apache.org/jira/browse/NIFI-2619
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.0.0
>
>
> ClassLoaderUtils.getCustomClassLoader() splits a module path list on a comma, 
> but doesn't trim the individual entries.
> Also in ClassLoaderUtils.getURLsForClasspath(), it is assumed that the paths 
> are Files, but they could also already be URLs. Logic should be added to 
> check if the path is already a valid URL, and add it to the list if so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2619) ClassLoaderUtils doesn't support spaces between paths or URLs in the paths

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431457#comment-15431457
 ] 

ASF subversion and git services commented on NIFI-2619:
---

Commit 7123a1a2761d238b2d1e300e33f2f3ade54347e1 in nifi's branch 
refs/heads/master from [~mattyb149]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7123a1a ]

NIFI-2619: Added unit test showing bugs, Added logic to ClassLoaderUtils to 
trim module paths and accept URLs

Signed-off-by: Yolanda M. Davis 

This closes #907


> ClassLoaderUtils doesn't support spaces between paths or URLs in the paths
> --
>
> Key: NIFI-2619
> URL: https://issues.apache.org/jira/browse/NIFI-2619
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> ClassLoaderUtils.getCustomClassLoader() splits a module path list on a comma, 
> but doesn't trim the individual entries.
> Also in ClassLoaderUtils.getURLsForClasspath(), it is assumed that the paths 
> are Files, but they could also already be URLs. Logic should be added to 
> check if the path is already a valid URL, and add it to the list if so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #907: NIFI-2619: Added logic to ClassLoaderUtils to trim m...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/907


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2605) On restart of all nodes in nifi cluster one of the nodes failed to join the cluster with fingerprint mismatch

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431452#comment-15431452
 ] 

Joseph Percivall commented on NIFI-2605:


Hey [~arpitgupta] and [~markap14], I think there has been some 
miscommunication. This ticket seems to be about when a cluster is restarting as 
a whole with one node having and un-inheritable flow and is potentially a 
duplicate of NIFI-1966[1].

Mark's PR addresses the issue where the cluster is in a state of no leader 
elected and a new node joins the cluster. In this event (before Mark's change) 
there is the chance that the new node becomes the Cluster Coordinator and kicks 
out all the other nodes.

If this seems correct to you two, I'd suggest closing this as a duplicate of 
NIFI-1966 and opening a new Jira for the issue Mark is fixing.

[1] https://issues.apache.org/jira/browse/NIFI-1966

> On restart of all nodes in nifi cluster one of the nodes failed to join the 
> cluster with fingerprint mismatch
> -
>
> Key: NIFI-2605
> URL: https://issues.apache.org/jira/browse/NIFI-2605
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Arpit Gupta
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Follow stack trace was present in the node that did not connect
> {code}
> 2016-08-18 12:04:55,628 INFO [Process Cluster Protocol Request-1] 
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request 
> ea80ad62-585c-4460-9ee9-93cc12c8db54 (type=NODE_STATUS_CHANGE, length=1052 
> bytes) from host in 61 millis
> 2016-08-18 12:04:55,806 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> org.apache.nifi.controller.UninheritableFlowException: Failed to connect node 
> to cluster because local flow is different than cluster flow.
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:866)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:492)
>  ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:746) 
> [nifi-jetty-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:137) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227) 
> [nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed 
> configuration is not inheritable by the flow controller because of flow 
> differences: Found difference in Flows:
> Local Fingerprint:   
> 7c84501d-d10c-407c-b9f3-1d80e38fe36a9d7d39c0-0156-1000--c6ce3a7d9d7d3cd1-0156-1000--
> Cluster Fingerprint: 9d89d844-0156-1000-e4bc-8ae5e0566749
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-2611) UnpackContent cannot unpack any type of flowfile

[Michael Moser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Moser resolved NIFI-2611.
-
   Resolution: Fixed
Fix Version/s: 0.8.0
   1.0.0

Thanks [~joewitt]

> UnpackContent cannot unpack any type of flowfile
> 
>
> Key: NIFI-2611
> URL: https://issues.apache.org/jira/browse/NIFI-2611
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.7.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
> Fix For: 1.0.0, 0.8.0
>
>
> Two possibly separate problems:
> *Flowfile-stream-v2 and v3*
> This may be a problem with either MergeContent's production of 
> flowfile-stream v2 and v3, or with UnpackContent's inability to unpack them, 
> not sure which.  Here is a screen shot with how to reproduce it: 
> https://ibin.co/2sCwqbFbAs3a.png
> Essentially, when you pack a flow file as flowfile-stream v2 or v3, a 
> subsequent UnpackContent set to the respective type fails with the error 
> "Cannot unpack {} because it does not appear to have any entries".
> *Flowfile-tar-v1*
> When selecting flowfile-tar-v1 from UnpackContent, you immediately get 
> @OnScheduled error failure as soon as you start the processor, which prevents 
> it from processing any incoming flow files.  Here is a screenshot: 
> https://ibin.co/2sCxI4iDm88t.png



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2611) UnpackContent cannot unpack any type of flowfile

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431443#comment-15431443
 ] 

Joseph Witt commented on NIFI-2611:
---

[~mosermw] [~jgresock] Friendly reminder to close ticket if this is resolved 
and be sure to set fix versions for any branches it has been pushed to.

> UnpackContent cannot unpack any type of flowfile
> 
>
> Key: NIFI-2611
> URL: https://issues.apache.org/jira/browse/NIFI-2611
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.7.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>
> Two possibly separate problems:
> *Flowfile-stream-v2 and v3*
> This may be a problem with either MergeContent's production of 
> flowfile-stream v2 and v3, or with UnpackContent's inability to unpack them, 
> not sure which.  Here is a screen shot with how to reproduce it: 
> https://ibin.co/2sCwqbFbAs3a.png
> Essentially, when you pack a flow file as flowfile-stream v2 or v3, a 
> subsequent UnpackContent set to the respective type fails with the error 
> "Cannot unpack {} because it does not appear to have any entries".
> *Flowfile-tar-v1*
> When selecting flowfile-tar-v1 from UnpackContent, you immediately get 
> @OnScheduled error failure as soon as you start the processor, which prevents 
> it from processing any incoming flow files.  Here is a screenshot: 
> https://ibin.co/2sCxI4iDm88t.png



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2626) LIstenHTTP leaks threads when address is already in use

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431437#comment-15431437
 ] 

ASF GitHub Bot commented on NIFI-2626:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/910
  
Reviewing...


> LIstenHTTP leaks threads when address is already in use
> ---
>
> Key: NIFI-2626
> URL: https://issues.apache.org/jira/browse/NIFI-2626
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.0.0
>
>
> While testing the current 1.0.0 master branch for release, I ran into a 
> thread leak issue with ListenHTTP.  Though not the fault of ListenHTTP 
> explicitly, threads are being leaked by Jetty after attempting to bind to an 
> address that is already in use.  Jetty starts a thread when server.start() is 
> called, but does not clean up after itself if the start() method throws an 
> exception.  Over time, these threads pile up and eventually the JVM will run 
> out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #910: NIFI-2626 Fixes jetty server thread leak

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/910
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2536) Update Documentation for Templates

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2536:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Update Documentation for Templates
> --
>
> Key: NIFI-2536
> URL: https://issues.apache.org/jira/browse/NIFI-2536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.0.0
>
>
> The documentation for importing a template does not reflect recent UI changes 
> for 1.0.  Required updates include:
> * Template features moved to the Operate Palette (create, upload)
> ** Association between template actions and current process group permissions
> * Template features still in the Templates dialog (list, filter, export, 
> delete)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2536) Update Documentation for Templates

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431431#comment-15431431
 ] 

ASF subversion and git services commented on NIFI-2536:
---

Commit 7884d09948381c655571c5868a40ba5b7e9c1c76 in nifi's branch 
refs/heads/master from [~jameswing]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7884d09 ]

NIFI-2536 Update User Guide Template section. This closes #852


> Update Documentation for Templates
> --
>
> Key: NIFI-2536
> URL: https://issues.apache.org/jira/browse/NIFI-2536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.0.0
>
>
> The documentation for importing a template does not reflect recent UI changes 
> for 1.0.  Required updates include:
> * Template features moved to the Operate Palette (create, upload)
> ** Association between template actions and current process group permissions
> * Template features still in the Templates dialog (list, filter, export, 
> delete)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2536) Update Documentation for Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431434#comment-15431434
 ] 

ASF GitHub Bot commented on NIFI-2536:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/852
  
Looks good @jvwing. This has been merged into master. Thanks!


> Update Documentation for Templates
> --
>
> Key: NIFI-2536
> URL: https://issues.apache.org/jira/browse/NIFI-2536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.0.0
>
>
> The documentation for importing a template does not reflect recent UI changes 
> for 1.0.  Required updates include:
> * Template features moved to the Operate Palette (create, upload)
> ** Association between template actions and current process group permissions
> * Template features still in the Templates dialog (list, filter, export, 
> delete)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2536) Update Documentation for Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431433#comment-15431433
 ] 

ASF GitHub Bot commented on NIFI-2536:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/852


> Update Documentation for Templates
> --
>
> Key: NIFI-2536
> URL: https://issues.apache.org/jira/browse/NIFI-2536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.0.0
>
>
> The documentation for importing a template does not reflect recent UI changes 
> for 1.0.  Required updates include:
> * Template features moved to the Operate Palette (create, upload)
> ** Association between template actions and current process group permissions
> * Template features still in the Templates dialog (list, filter, export, 
> delete)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2536) Update Documentation for Templates

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2536:
--
Fix Version/s: 1.0.0

> Update Documentation for Templates
> --
>
> Key: NIFI-2536
> URL: https://issues.apache.org/jira/browse/NIFI-2536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
> Fix For: 1.0.0
>
>
> The documentation for importing a template does not reflect recent UI changes 
> for 1.0.  Required updates include:
> * Template features moved to the Operate Palette (create, upload)
> ** Association between template actions and current process group permissions
> * Template features still in the Templates dialog (list, filter, export, 
> delete)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #852: NIFI-2536 Update User Guide Template section

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/852


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2619) ClassLoaderUtils doesn't support spaces between paths or URLs in the paths

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431424#comment-15431424
 ] 

ASF GitHub Bot commented on NIFI-2619:
--

Github user YolandaMDavis commented on the issue:

https://github.com/apache/nifi/pull/907
  
@mattyb149 thanks for applying that change.  I tested on standalone node 
with Jolt Transform processor using a list of customjars (included extra 
commas, spaces etc) and works as expected with no exceptions thrown.

+1

Will squash and merge shortly


> ClassLoaderUtils doesn't support spaces between paths or URLs in the paths
> --
>
> Key: NIFI-2619
> URL: https://issues.apache.org/jira/browse/NIFI-2619
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> ClassLoaderUtils.getCustomClassLoader() splits a module path list on a comma, 
> but doesn't trim the individual entries.
> Also in ClassLoaderUtils.getURLsForClasspath(), it is assumed that the paths 
> are Files, but they could also already be URLs. Logic should be added to 
> check if the path is already a valid URL, and add it to the list if so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #907: NIFI-2619: Added logic to ClassLoaderUtils to trim module p...

[YolandaMDavis]

Github user YolandaMDavis commented on the issue:

https://github.com/apache/nifi/pull/907
  
@mattyb149 thanks for applying that change.  I tested on standalone node 
with Jolt Transform processor using a list of customjars (included extra 
commas, spaces etc) and works as expected with no exceptions thrown.

+1

Will squash and merge shortly


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2611) UnpackContent cannot unpack any type of flowfile

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431420#comment-15431420
 ] 

ASF GitHub Bot commented on NIFI-2611:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/905


> UnpackContent cannot unpack any type of flowfile
> 
>
> Key: NIFI-2611
> URL: https://issues.apache.org/jira/browse/NIFI-2611
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.7.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>
> Two possibly separate problems:
> *Flowfile-stream-v2 and v3*
> This may be a problem with either MergeContent's production of 
> flowfile-stream v2 and v3, or with UnpackContent's inability to unpack them, 
> not sure which.  Here is a screen shot with how to reproduce it: 
> https://ibin.co/2sCwqbFbAs3a.png
> Essentially, when you pack a flow file as flowfile-stream v2 or v3, a 
> subsequent UnpackContent set to the respective type fails with the error 
> "Cannot unpack {} because it does not appear to have any entries".
> *Flowfile-tar-v1*
> When selecting flowfile-tar-v1 from UnpackContent, you immediately get 
> @OnScheduled error failure as soon as you start the processor, which prevents 
> it from processing any incoming flow files.  Here is a screenshot: 
> https://ibin.co/2sCxI4iDm88t.png



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #905: NIFI-2611: Fixing bugs in UnpackContent with flow fi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/905


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2611) UnpackContent cannot unpack any type of flowfile

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431418#comment-15431418
 ] 

ASF subversion and git services commented on NIFI-2611:
---

Commit 17dec04939efda59345f5c2de49b575fd7d8dfcc in nifi's branch 
refs/heads/master from Joe Gresock
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=17dec04 ]

NIFI-2611: Fixing bugs in UnpackContent with flow file unpackers

Signed-off-by: Mike Moser 

This closes #905


> UnpackContent cannot unpack any type of flowfile
> 
>
> Key: NIFI-2611
> URL: https://issues.apache.org/jira/browse/NIFI-2611
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.7.0
>Reporter: Joseph Gresock
>Assignee: Joseph Gresock
>
> Two possibly separate problems:
> *Flowfile-stream-v2 and v3*
> This may be a problem with either MergeContent's production of 
> flowfile-stream v2 and v3, or with UnpackContent's inability to unpack them, 
> not sure which.  Here is a screen shot with how to reproduce it: 
> https://ibin.co/2sCwqbFbAs3a.png
> Essentially, when you pack a flow file as flowfile-stream v2 or v3, a 
> subsequent UnpackContent set to the respective type fails with the error 
> "Cannot unpack {} because it does not appear to have any entries".
> *Flowfile-tar-v1*
> When selecting flowfile-tar-v1 from UnpackContent, you immediately get 
> @OnScheduled error failure as soon as you start the processor, which prevents 
> it from processing any incoming flow files.  Here is a screenshot: 
> https://ibin.co/2sCxI4iDm88t.png



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2536) Update Documentation for Templates

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431415#comment-15431415
 ] 

ASF GitHub Bot commented on NIFI-2536:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/852
  
Reviewing...


> Update Documentation for Templates
> --
>
> Key: NIFI-2536
> URL: https://issues.apache.org/jira/browse/NIFI-2536
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: James Wing
>Assignee: James Wing
>Priority: Minor
>
> The documentation for importing a template does not reflect recent UI changes 
> for 1.0.  Required updates include:
> * Template features moved to the Operate Palette (create, upload)
> ** Association between template actions and current process group permissions
> * Template features still in the Templates dialog (list, filter, export, 
> delete)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #852: NIFI-2536 Update User Guide Template section

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/852
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2626) LIstenHTTP leaks threads when address is already in use

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431411#comment-15431411
 ] 

ASF GitHub Bot commented on NIFI-2626:
--

GitHub user jtstorck opened a pull request:

https://github.com/apache/nifi/pull/910

NIFI-2626 Fixes jetty server thread leak



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jtstorck/nifi NIFI-2626

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/910.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #910


commit 684bbb687c3040b4a8ef9d6088b03d4a1032fe9e
Author: Jeff Storck 
Date:   2016-08-22T18:48:06Z

NIFI-2626 Fixes jetty server thread leak




> LIstenHTTP leaks threads when address is already in use
> ---
>
> Key: NIFI-2626
> URL: https://issues.apache.org/jira/browse/NIFI-2626
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.0.0
>
>
> While testing the current 1.0.0 master branch for release, I ran into a 
> thread leak issue with ListenHTTP.  Though not the fault of ListenHTTP 
> explicitly, threads are being leaked by Jetty after attempting to bind to an 
> address that is already in use.  Jetty starts a thread when server.start() is 
> called, but does not clean up after itself if the start() method throws an 
> exception.  Over time, these threads pile up and eventually the JVM will run 
> out of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)