[GitHub] nifi pull request #926: NIFI-2451 Added encrypt-config documentation to Admi...

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/926

NIFI-2451 Added encrypt-config documentation to Admin Guide

As with [PR 925](https://github.com/apache/nifi/pull/925) this PR has an 
extra commit 
[`c638191`](https://github.com/apache/nifi/commit/c638191a47ae5767de82b383ecb04496ce18877b)
 which has been merged into Apache's git repository but has not yet synced to 
the GitHub mirror repo. Once that sync occurs, this PR will only contain 
commits 
[`a6c543b`](https://github.com/apache/nifi/commit/a6c543bb3d56b4b6f28f4a429f833d51d29e16ff)
 by @andrewmlim and 
[`5a3f247`](https://github.com/apache/nifi/commit/5a3f24762e2e0a7d8403384fc8bb7217130186cc)
 by me (1 file, ~100 lines changed). 

This should supersede [PR 855](https://github.com/apache/nifi/pull/855). 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-2451

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/926.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #926


commit c638191a47ae5767de82b383ecb04496ce18877b
Author: Andy LoPresto 
Date:   2016-08-16T03:18:47Z

NIFI-1831 Added internal logic and command-line tool to allow AES-encrypted 
sensitive configuration values in nifi.properties.

This closes #834.

commit a6c543bb3d56b4b6f28f4a429f833d51d29e16ff
Author: Andrew Lim 
Date:   2016-08-13T03:54:11Z

NIFI-2451 Update Admin guide for encrypt-config command utility, new 
nifi.sensitive.props.additional.keys property, and removal of Java 7 reference 
for JCE Unlimited Strength Jurisdiction Policy

commit 5a3f24762e2e0a7d8403384fc8bb7217130186cc
Author: Andy LoPresto 
Date:   2016-08-24T04:54:53Z

NIFI-2451 Added new documentation regarding encrypt-config tool after 
changes in NIFI-1831.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2451) Need to update Encryption Configuration content in Admin guide for 1.0 changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434173#comment-15434173
 ] 

ASF GitHub Bot commented on NIFI-2451:
--

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/926

NIFI-2451 Added encrypt-config documentation to Admin Guide

As with [PR 925](https://github.com/apache/nifi/pull/925) this PR has an 
extra commit 
[`c638191`](https://github.com/apache/nifi/commit/c638191a47ae5767de82b383ecb04496ce18877b)
 which has been merged into Apache's git repository but has not yet synced to 
the GitHub mirror repo. Once that sync occurs, this PR will only contain 
commits 
[`a6c543b`](https://github.com/apache/nifi/commit/a6c543bb3d56b4b6f28f4a429f833d51d29e16ff)
 by @andrewmlim and 
[`5a3f247`](https://github.com/apache/nifi/commit/5a3f24762e2e0a7d8403384fc8bb7217130186cc)
 by me (1 file, ~100 lines changed). 

This should supersede [PR 855](https://github.com/apache/nifi/pull/855). 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-2451

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/926.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #926


commit c638191a47ae5767de82b383ecb04496ce18877b
Author: Andy LoPresto 
Date:   2016-08-16T03:18:47Z

NIFI-1831 Added internal logic and command-line tool to allow AES-encrypted 
sensitive configuration values in nifi.properties.

This closes #834.

commit a6c543bb3d56b4b6f28f4a429f833d51d29e16ff
Author: Andrew Lim 
Date:   2016-08-13T03:54:11Z

NIFI-2451 Update Admin guide for encrypt-config command utility, new 
nifi.sensitive.props.additional.keys property, and removal of Java 7 reference 
for JCE Unlimited Strength Jurisdiction Policy

commit 5a3f24762e2e0a7d8403384fc8bb7217130186cc
Author: Andy LoPresto 
Date:   2016-08-24T04:54:53Z

NIFI-2451 Added new documentation regarding encrypt-config tool after 
changes in NIFI-1831.




> Need to update Encryption Configuration content in Admin guide for 1.0 changes
> --
>
> Key: NIFI-2451
> URL: https://issues.apache.org/jira/browse/NIFI-2451
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
> Fix For: 1.0.0
>
>
> Will need to document changes in Admin Guide for:
> NIFI-1831 for Encryption Configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2640) Fix Windows command-line utility for encrypted configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434157#comment-15434157
 ] 

ASF GitHub Bot commented on NIFI-2640:
--

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/925

NIFI-2640 Fixed Windows encrypt-config.bat script

The Apache repo hasn't synced with GitHub yet, so [PR 834] still shows as 
open, and this PR includes the 
[`c638191`](https://github.com/apache/nifi/commit/c638191a47ae5767de82b383ecb04496ce18877b)
 commit which will close it. I'll update this PR after Apache syncs.  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-2640

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/925.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #925


commit c638191a47ae5767de82b383ecb04496ce18877b
Author: Andy LoPresto 
Date:   2016-08-16T03:18:47Z

NIFI-1831 Added internal logic and command-line tool to allow AES-encrypted 
sensitive configuration values in nifi.properties.

This closes #834.

commit 6705aa21430bcc54568bf35f075e6a7cf6fe4e9e
Author: Andy LoPresto 
Date:   2016-08-24T04:13:07Z

NIFI-2640 Fixed Windows encrypt-config.bat script to correctly invoke 
ConfigEncryptionTool.




> Fix Windows command-line utility for encrypted configuration files
> --
>
> Key: NIFI-2640
> URL: https://issues.apache.org/jira/browse/NIFI-2640
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>  Labels: configuration, encryption, security, windows
> Fix For: 1.0.0
>
>
> The *nix script {{encrypt-config.sh}} properly invokes the 
> {{ConfigEncryptionTool}} but the Windows {{encrypt-config.bat}} does not. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #925: NIFI-2640 Fixed Windows encrypt-config.bat script

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/925

NIFI-2640 Fixed Windows encrypt-config.bat script

The Apache repo hasn't synced with GitHub yet, so [PR 834] still shows as 
open, and this PR includes the 
[`c638191`](https://github.com/apache/nifi/commit/c638191a47ae5767de82b383ecb04496ce18877b)
 commit which will close it. I'll update this PR after Apache syncs.  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-2640

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/925.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #925


commit c638191a47ae5767de82b383ecb04496ce18877b
Author: Andy LoPresto 
Date:   2016-08-16T03:18:47Z

NIFI-1831 Added internal logic and command-line tool to allow AES-encrypted 
sensitive configuration values in nifi.properties.

This closes #834.

commit 6705aa21430bcc54568bf35f075e6a7cf6fe4e9e
Author: Andy LoPresto 
Date:   2016-08-24T04:13:07Z

NIFI-2640 Fixed Windows encrypt-config.bat script to correctly invoke 
ConfigEncryptionTool.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2640) Fix Windows command-line utility for encrypted configuration files

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-2640:
---

 Summary: Fix Windows command-line utility for encrypted 
configuration files
 Key: NIFI-2640
 URL: https://issues.apache.org/jira/browse/NIFI-2640
 Project: Apache NiFi
  Issue Type: Bug
  Components: Configuration
Affects Versions: 1.0.0
Reporter: Andy LoPresto
Assignee: Andy LoPresto
 Fix For: 1.0.0


The *nix script {{encrypt-config.sh}} properly invokes the 
{{ConfigEncryptionTool}} but the Windows {{encrypt-config.bat}} does not. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1831) Allow encrypted passwords in configuration files

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-1831:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

The framework for this feature has been committed to master. The internal logic 
allows the application to load {{nifi.properties}} files with sensitive 
properties protected by AES encryption in Galois/Counter Mode (GCM) (which 
provides AEAD -- authenticated encryption to mitigate cipher text manipulation) 
with 128 bit or 256 bit key lengths -- and provides a command-line utility to 
accept a key or password and encrypt the properties from an existing plaintext 
{{nifi.properties}} file and write the encrypted properties and the key out to 
the configuration files. Follow on issues can be raised against this to allow 
for future enhancements. 

> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434130#comment-15434130
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
I added *groovy-all* to `NOTICE` and added some Javadoc throughout the new 
classes. With Bryan's +1 I am going to squash the commits and merge to master. 


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
> 020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:16 $ more password.enc
> VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:55 $
> 

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
I added *groovy-all* to `NOTICE` and added some Javadoc throughout the new 
classes. With Bryan's +1 I am going to squash the commits and merge to master. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2525) HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException when sending through proxy that requires authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433834#comment-15433834
 ] 

ASF GitHub Bot commented on NIFI-2525:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/915#discussion_r75969046
  
--- Diff: nifi-assembly/NOTICE ---
@@ -909,6 +909,11 @@ The following binary components are provided under the 
Apache Software License v
 Expert Group and released to the public domain, as explained at
 http://creativecommons.org/publicdomain/zero/1.0/
 
+  (ASLv2) LittleProxy
--- End diff --

@markap14 Thanks for pointing that out. Removed it.


> HTTP Site-to-Site fails with java.nio.channels.AsynchronousCloseException 
> when sending through proxy that requires authentication
> -
>
> Key: NIFI-2525
> URL: https://issues.apache.org/jira/browse/NIFI-2525
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
> Fix For: 1.0.0
>
>
> Pulling data using Remote Process Group from output port works.
> However, pushing data using Remote Process Group to input port fails with 
> AsynchronousCloseException.
> A RPG sends data via POST, then a proxy server returns 407: proxy auth 
> required. After this, the RPG should resend the request with credential, but 
> the data channel is already closed.
> Currently, it uses chunked encoding so that it can stream data to send. 
> Sending actual data twice won't be efficient. We need to do the 
> authentication before start reading flow-file stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #915: NIFI-2525: Fix Proxy auth issue with async send.

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/915#discussion_r75969046
  
--- Diff: nifi-assembly/NOTICE ---
@@ -909,6 +909,11 @@ The following binary components are provided under the 
Apache Software License v
 Expert Group and released to the public domain, as explained at
 http://creativecommons.org/publicdomain/zero/1.0/
 
+  (ASLv2) LittleProxy
--- End diff --

@markap14 Thanks for pointing that out. Removed it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433813#comment-15433813
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
Performed a couple of test runs and everything is working nicely, thanks 
for making all the updates.

My last comment is that we may need to add something to the toolkit NOTICE 
for Grooby. The main NiFi assembly seems to have this for the scripting 
processors bringing in groovy, only difference being groovy.jar vs. 
groovy-all.jar: 
https://github.com/apache/nifi/blob/master/nifi-assembly/NOTICE#L815

If we add that (or we have a reason we don't need it) then I am a +1 and 
good to merge, nice work!


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
Performed a couple of test runs and everything is working nicely, thanks 
for making all the updates.

My last comment is that we may need to add something to the toolkit NOTICE 
for Grooby. The main NiFi assembly seems to have this for the scripting 
processors bringing in groovy, only difference being groovy.jar vs. 
groovy-all.jar: 
https://github.com/apache/nifi/blob/master/nifi-assembly/NOTICE#L815

If we add that (or we have a reason we don't need it) then I am a +1 and 
good to merge, nice work!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2608) Thread-safety issue with ConsumeKafka

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433772#comment-15433772
 ] 

Joseph Witt commented on NIFI-2608:
---

Status of this work:
* Not ready to submit a PR but the evolving work is here 
https://github.com/joewitt/incubator-nifi/commits/NIFI-2608
* Changes made so far:
** now uses a ConsumerPool to encapsulate the KafkaConsumer and to ensure 
multi-threaded access to the pool ensures only a single KafkaConsumer is used 
by a single thread at a time. The ConsumerPool returns ConsumerLeases and that 
supports all necessary calls for interaction with the true kafka api (poll, 
commitOffsets).
** ConsumeKafka reworked to use the ConsumerPool exclusively and supports easy 
mocking of the ConsumerPool to ensure unit testing focuses on the processor and 
its behavior rather than the kafka client.  Several benefits came from this 
approach:
*** We support multiple topic pulling now via a comma separated list of topics.
*** We ensure data when being merged into flowFiles is merged in a manner that 
maintains its topic/partion/offset ordering and grouping and ensures the first 
offset is stored with the flow file and we don't store the key for groups since 
the key wouldn't be meaningful.
*** The ConsumerPool and usage logic makes reasoning over handling of consumers 
very straightforward and in the event of an exception using a consumer its 
lease can be poisoned thus ensuring it will be actually closed and a new one 
made.  
*** Using an easy to reason over try w/resources approach to handling 
ConsumerLeases from the pool.
*** We ensure the offsets for all topics and partitions are accounted for and 
committed once all items are written to the session and committed to the 
session.
** The key of a kafka message is now being hex encoded and is called 
'kafka.key.hex'.  This means it can be used by PublishKafka or others without 
worrying about it having been improperly interpreted into UTF-8 or something 
else.  Its raw byte value is thus maintained.

Work that remains:
* Fix tests for producer after moving code.
* Update WritesAttributes annotations for ConsumeKafka to account for all items
* Consider providing Consume and Publish Kafka processors for the 0.10 API.  
While the API is the same there is apparently a way to limit how many messages 
come back from Kafka as a result of a single poll call which we definitely want 
to use.  Also, while the 0.9 clients can talk to 0.10 brokers the Kafka docs 
indicate there may be a steep kafka server/broker performance penalty due to 
changes in how the compaction or compression mechanism works.  Could call these 
processors ConsumeKafka010 and PublishKafka010.

> Thread-safety issue with ConsumeKafka
> -
>
> Key: NIFI-2608
> URL: https://issues.apache.org/jira/browse/NIFI-2608
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Oleg Zhurakousky
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> KafkaConsumer went fro thread-safe in 0.8 to not-thread-safe in 0.9 which was 
> overlooked while implementing new ConsumeKafka processor which relied on 0.9 
> Client API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2639) Update graphics in "Apache NiFi In Depth" doc

[Joseph Percivall (JIRA)]

Joseph Percivall created NIFI-2639:
--

 Summary: Update graphics in "Apache NiFi In Depth" doc
 Key: NIFI-2639
 URL: https://issues.apache.org/jira/browse/NIFI-2639
 Project: Apache NiFi
  Issue Type: Improvement
Affects Versions: 1.0.0
Reporter: Joseph Percivall
Priority: Minor
 Fix For: 1.1.0


There are a number of graphics in the "Apache NiFi In Depth" that are screen 
shots of a flow that currently are of the 0.x UI. 

For aesthetics purposes, these should be updated using the new UI .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-2638) Update iconProcessor.png with image of the new GUI.

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall resolved NIFI-2638.

Resolution: Fixed

> Update iconProcessor.png with image of the new GUI.
> ---
>
> Key: NIFI-2638
> URL: https://issues.apache.org/jira/browse/NIFI-2638
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Reporter: Sarah Olson
>Assignee: Sarah Olson
> Fix For: 1.0.0
>
>
> The iconProcessor screenshot still reflects the old GUI. 
> Updating with image of the new icon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2638) Update iconProcessor.png with image of the new GUI.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433734#comment-15433734
 ] 

ASF GitHub Bot commented on NIFI-2638:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/924


> Update iconProcessor.png with image of the new GUI.
> ---
>
> Key: NIFI-2638
> URL: https://issues.apache.org/jira/browse/NIFI-2638
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Reporter: Sarah Olson
>Assignee: Sarah Olson
> Fix For: 1.0.0
>
>
> The iconProcessor screenshot still reflects the old GUI. 
> Updating with image of the new icon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #924: NIFI-2638: Updated inconProcessor.png with image fro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/924


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2638) Update iconProcessor.png with image of the new GUI.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433731#comment-15433731
 ] 

ASF subversion and git services commented on NIFI-2638:
---

Commit 7a4fed189c8275414694f53f33c599240ec672bd in nifi's branch 
refs/heads/master from [~solson75]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7a4fed1 ]

NIFI-2638: Updated iconProcessor.png with image from new UI.

This closes #924

Signed-off-by: jpercivall 


> Update iconProcessor.png with image of the new GUI.
> ---
>
> Key: NIFI-2638
> URL: https://issues.apache.org/jira/browse/NIFI-2638
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Reporter: Sarah Olson
>Assignee: Sarah Olson
> Fix For: 1.0.0
>
>
> The iconProcessor screenshot still reflects the old GUI. 
> Updating with image of the new icon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2638) Update iconProcessor.png with image of the new GUI.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433726#comment-15433726
 ] 

ASF GitHub Bot commented on NIFI-2638:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/924
  
+1

Visually verified image looks good and did a contrib check build. In a 
standealone instance went through and made sure all of the images in the 
"general" docs are updated. Looks good @thesolson I will merge it in.


> Update iconProcessor.png with image of the new GUI.
> ---
>
> Key: NIFI-2638
> URL: https://issues.apache.org/jira/browse/NIFI-2638
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Reporter: Sarah Olson
>Assignee: Sarah Olson
> Fix For: 1.0.0
>
>
> The iconProcessor screenshot still reflects the old GUI. 
> Updating with image of the new icon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #924: NIFI-2638: Updated inconProcessor.png with image from new U...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/924
  
+1

Visually verified image looks good and did a contrib check build. In a 
standealone instance went through and made sure all of the images in the 
"general" docs are updated. Looks good @thesolson I will merge it in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2638) Update iconProcessor.png with image of the new GUI.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433634#comment-15433634
 ] 

ASF GitHub Bot commented on NIFI-2638:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/924
  
Reviewing


> Update iconProcessor.png with image of the new GUI.
> ---
>
> Key: NIFI-2638
> URL: https://issues.apache.org/jira/browse/NIFI-2638
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Reporter: Sarah Olson
>Assignee: Sarah Olson
> Fix For: 1.0.0
>
>
> The iconProcessor screenshot still reflects the old GUI. 
> Updating with image of the new icon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2603) Bringing Some UI Color Back

[Peter Wicks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Wicks updated NIFI-2603:
--
Description: 
In the new 1.0 UI all of the colors associated with status (except the orange 
triangle) are gone; replaced with a dark gray color.

I propose bringing back color.  The screenshots are in the format of before on 
the top and after on the bottom, except were labeled in the picture itself:

 - Top Status Menu: https://goo.gl/photos/se1JnvhRwU7N4Fap7
 - Process Group: https://goo.gl/photos/dqjG4KvC6xqxQfgT7
 - Processes (Running/Stopped/Invalid): https://goo.gl/photos/dSS8vgE2RkrXtc77A
 - Operate Play/Stop buttons (only on mouse hover): 
https://goo.gl/photos/Am5SUEEn7G9RjmMe6
 - Processor/Processor Group Context Menu: 
https://goo.gl/photos/Jq3qFg4ezaN91qms5

This is not a "100% done, I've covered everything" before/after list.  I know I 
need to do the NiFi summary page also at the minimum.



  was:
In the new 1.0 UI all of the colors associated with status (except the orange 
triangle) are gone; replaced with a dark gray color.

I propose bringing back color; see screenshot: 
https://goo.gl/photos/RZzz45RGeTAmWwiC7


> Bringing Some UI Color Back
> ---
>
> Key: NIFI-2603
> URL: https://issues.apache.org/jira/browse/NIFI-2603
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Peter Wicks
>Priority: Minor
>
> In the new 1.0 UI all of the colors associated with status (except the orange 
> triangle) are gone; replaced with a dark gray color.
> I propose bringing back color.  The screenshots are in the format of before 
> on the top and after on the bottom, except were labeled in the picture itself:
>  - Top Status Menu: https://goo.gl/photos/se1JnvhRwU7N4Fap7
>  - Process Group: https://goo.gl/photos/dqjG4KvC6xqxQfgT7
>  - Processes (Running/Stopped/Invalid): 
> https://goo.gl/photos/dSS8vgE2RkrXtc77A
>  - Operate Play/Stop buttons (only on mouse hover): 
> https://goo.gl/photos/Am5SUEEn7G9RjmMe6
>  - Processor/Processor Group Context Menu: 
> https://goo.gl/photos/Jq3qFg4ezaN91qms5
> This is not a "100% done, I've covered everything" before/after list.  I know 
> I need to do the NiFi summary page also at the minimum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2638) Update iconProcessor.png with image of the new GUI.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433578#comment-15433578
 ] 

ASF GitHub Bot commented on NIFI-2638:
--

GitHub user thesolson opened a pull request:

https://github.com/apache/nifi/pull/924

NIFI-2638: Updated inconProcessor.png with image from new UI.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thesolson/nifi NIFI-2638

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/924.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #924


commit 69256a0cac5872b2eb2796b6b796375fd923200a
Author: Sarah Olson 
Date:   2016-08-23T20:47:14Z

NIFI-2638: Updated inconProcessor.png with image from new UI.




> Update iconProcessor.png with image of the new GUI.
> ---
>
> Key: NIFI-2638
> URL: https://issues.apache.org/jira/browse/NIFI-2638
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Reporter: Sarah Olson
>Assignee: Sarah Olson
> Fix For: 1.0.0
>
>
> The iconProcessor screenshot still reflects the old GUI. 
> Updating with image of the new icon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #924: NIFI-2638: Updated inconProcessor.png with image fro...

[thesolson]

GitHub user thesolson opened a pull request:

https://github.com/apache/nifi/pull/924

NIFI-2638: Updated inconProcessor.png with image from new UI.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thesolson/nifi NIFI-2638

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/924.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #924


commit 69256a0cac5872b2eb2796b6b796375fd923200a
Author: Sarah Olson 
Date:   2016-08-23T20:47:14Z

NIFI-2638: Updated inconProcessor.png with image from new UI.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2632) Add fragment attributes for SplitJson and SplitXml

[Yolanda M. Davis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yolanda M. Davis updated NIFI-2632:
---
   Resolution: Resolved
Fix Version/s: 1.0.0
   Status: Resolved  (was: Patch Available)

> Add fragment attributes for SplitJson and SplitXml
> --
>
> Key: NIFI-2632
> URL: https://issues.apache.org/jira/browse/NIFI-2632
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.0.0
>
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitJson and SplitXml processors should 
> write these same attributes for their split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2636) UnpackContent has concurrent thread safety issue, causes flowfiles to fail

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433570#comment-15433570
 ] 

ASF GitHub Bot commented on NIFI-2636:
--

GitHub user mosermw opened a pull request:

https://github.com/apache/nifi/pull/923

NIFI-2636 resolve thread safety problem in UnpackContent



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mosermw/nifi NIFI-2636

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/923.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #923


commit 83bd31552852aad23402449c05f0ebf3a9b6acfc
Author: Mike Moser 
Date:   2016-08-23T20:41:13Z

NIFI-2636 resolve thread safety problem in UnpackContent




> UnpackContent has concurrent thread safety issue, causes flowfiles to fail
> --
>
> Key: NIFI-2636
> URL: https://issues.apache.org/jira/browse/NIFI-2636
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 0.7.0
>Reporter: Michael Moser
>Assignee: Michael Moser
>
> Shortly after merging NIFI-2611 I took a last look at the code and noticed 
> that each onTrigger() call, when the Packaging Format property is set to "use 
> mime.type attribute", that the class instance variable "private Unpacker 
> unpacker" can change.  When UnpackContent is set to > 1 concurrent task, this 
> isn't thread safe.  Thread A can set the unpacker to the TarUnpacker, but 
> before it gets a chance to unpack its tar file, Thread B changes the unpacker 
> to a FlowFileUnpackagerV3 which causes Thread A to fail its unpack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2636) UnpackContent has concurrent thread safety issue, causes flowfiles to fail

[Michael Moser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Moser updated NIFI-2636:

Status: Patch Available  (was: Open)

> UnpackContent has concurrent thread safety issue, causes flowfiles to fail
> --
>
> Key: NIFI-2636
> URL: https://issues.apache.org/jira/browse/NIFI-2636
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 0.7.0
>Reporter: Michael Moser
>Assignee: Michael Moser
>
> Shortly after merging NIFI-2611 I took a last look at the code and noticed 
> that each onTrigger() call, when the Packaging Format property is set to "use 
> mime.type attribute", that the class instance variable "private Unpacker 
> unpacker" can change.  When UnpackContent is set to > 1 concurrent task, this 
> isn't thread safe.  Thread A can set the unpacker to the TarUnpacker, but 
> before it gets a chance to unpack its tar file, Thread B changes the unpacker 
> to a FlowFileUnpackagerV3 which causes Thread A to fail its unpack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #923: NIFI-2636 resolve thread safety problem in UnpackCon...

[mosermw]

GitHub user mosermw opened a pull request:

https://github.com/apache/nifi/pull/923

NIFI-2636 resolve thread safety problem in UnpackContent



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mosermw/nifi NIFI-2636

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/923.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #923


commit 83bd31552852aad23402449c05f0ebf3a9b6acfc
Author: Mike Moser 
Date:   2016-08-23T20:41:13Z

NIFI-2636 resolve thread safety problem in UnpackContent




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2632) Add fragment attributes for SplitJson and SplitXml

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433554#comment-15433554
 ] 

ASF subversion and git services commented on NIFI-2632:
---

Commit ee9bd94082cb27694e94b6bd5907b52a641e86d9 in nifi's branch 
refs/heads/master from [~mattyb149]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ee9bd94 ]

NIFI-2632: Added fragment attributes to SplitJson and SplitXml

Signed-off-by: Yolanda M. Davis 

This closes #919


> Add fragment attributes for SplitJson and SplitXml
> --
>
> Key: NIFI-2632
> URL: https://issues.apache.org/jira/browse/NIFI-2632
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitJson and SplitXml processors should 
> write these same attributes for their split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2632) Add fragment attributes for SplitJson and SplitXml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433555#comment-15433555
 ] 

ASF GitHub Bot commented on NIFI-2632:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/919


> Add fragment attributes for SplitJson and SplitXml
> --
>
> Key: NIFI-2632
> URL: https://issues.apache.org/jira/browse/NIFI-2632
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitJson and SplitXml processors should 
> write these same attributes for their split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #919: NIFI-2632: Added fragment attributes to SplitJson an...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/919


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-2638) Update iconProcessor.png with image of the new GUI.

[Sarah Olson (JIRA)]

Sarah Olson created NIFI-2638:
-

 Summary: Update iconProcessor.png with image of the new GUI.
 Key: NIFI-2638
 URL: https://issues.apache.org/jira/browse/NIFI-2638
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Documentation & Website
Reporter: Sarah Olson
Assignee: Sarah Olson
 Fix For: 1.0.0


The iconProcessor screenshot still reflects the old GUI. 
Updating with image of the new icon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2632) Add fragment attributes for SplitJson and SplitXml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433539#comment-15433539
 ] 

ASF GitHub Bot commented on NIFI-2632:
--

Github user YolandaMDavis commented on the issue:

https://github.com/apache/nifi/pull/919
  
@mattyb149 Understood on the above comment.  Just to note it (for maybe 
future refactor) it may be cool to have some abstract SplitProcessor that has 
attributes, relationships and some of the split/original flow file handling 
that seems to be similar across many of the split processors.

This looks good tested with flows using json and xml. Splits were executed 
as expected.  Extracted attributes in both cases and confirmed new attributes 
for flow files transferred to split queue (original files did not contain 
attributes).  Attributes were also available for expression languages.

+1

I'll merge this in shortly



> Add fragment attributes for SplitJson and SplitXml
> --
>
> Key: NIFI-2632
> URL: https://issues.apache.org/jira/browse/NIFI-2632
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitJson and SplitXml processors should 
> write these same attributes for their split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #919: NIFI-2632: Added fragment attributes to SplitJson and Split...

[YolandaMDavis]

Github user YolandaMDavis commented on the issue:

https://github.com/apache/nifi/pull/919
  
@mattyb149 Understood on the above comment.  Just to note it (for maybe 
future refactor) it may be cool to have some abstract SplitProcessor that has 
attributes, relationships and some of the split/original flow file handling 
that seems to be similar across many of the split processors.

This looks good tested with flows using json and xml. Splits were executed 
as expected.  Extracted attributes in both cases and confirmed new attributes 
for flow files transferred to split queue (original files did not contain 
attributes).  Attributes were also available for expression languages.

+1

I'll merge this in shortly



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2622) SelectHiveQL with Avro output doesn't handle complex types

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-2622:
--
Fix Version/s: (was: 1.1.0)
   1.0.0

> SelectHiveQL with Avro output doesn't handle complex types
> --
>
> Key: NIFI-2622
> URL: https://issues.apache.org/jira/browse/NIFI-2622
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.0.0
>
>
> Hive supports table columns of complex type, such as Arrays, Structs, Maps, 
> etc.  However when using SelectHiveQL with Avro output, the result set is not 
> being converted correctly and an error occurs.
> This is a fundamental issue with the way SelectHiveQL attempts to create an 
> Avro schema without having seen any of the rows, since the sub-types of 
> complex items cannot be determined from the SQL type alone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-2622) SelectHiveQL with Avro output doesn't handle complex types

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende resolved NIFI-2622.
---
Resolution: Fixed

> SelectHiveQL with Avro output doesn't handle complex types
> --
>
> Key: NIFI-2622
> URL: https://issues.apache.org/jira/browse/NIFI-2622
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.0.0
>
>
> Hive supports table columns of complex type, such as Arrays, Structs, Maps, 
> etc.  However when using SelectHiveQL with Avro output, the result set is not 
> being converted correctly and an error occurs.
> This is a fundamental issue with the way SelectHiveQL attempts to create an 
> Avro schema without having seen any of the rows, since the sub-types of 
> complex items cannot be determined from the SQL type alone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2622) SelectHiveQL with Avro output doesn't handle complex types

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433493#comment-15433493
 ] 

ASF subversion and git services commented on NIFI-2622:
---

Commit 5d1a4f343f11606af5f45d1db4d58ca31a59e524 in nifi's branch 
refs/heads/master from [~mattyb149]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5d1a4f3 ]

NIFI-2622: Added support for complex types in SelectHiveQL

This closes #922.

Signed-off-by: Bryan Bende 


> SelectHiveQL with Avro output doesn't handle complex types
> --
>
> Key: NIFI-2622
> URL: https://issues.apache.org/jira/browse/NIFI-2622
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Hive supports table columns of complex type, such as Arrays, Structs, Maps, 
> etc.  However when using SelectHiveQL with Avro output, the result set is not 
> being converted correctly and an error occurs.
> This is a fundamental issue with the way SelectHiveQL attempts to create an 
> Avro schema without having seen any of the rows, since the sub-types of 
> complex items cannot be determined from the SQL type alone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #922: NIFI-2622: Added support for complex types in Select...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/922


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2622) SelectHiveQL with Avro output doesn't handle complex types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433489#comment-15433489
 ] 

ASF GitHub Bot commented on NIFI-2622:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/922
  
Looks good, verified this work with array and map columns, will merge to 
master.


> SelectHiveQL with Avro output doesn't handle complex types
> --
>
> Key: NIFI-2622
> URL: https://issues.apache.org/jira/browse/NIFI-2622
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Hive supports table columns of complex type, such as Arrays, Structs, Maps, 
> etc.  However when using SelectHiveQL with Avro output, the result set is not 
> being converted correctly and an error occurs.
> This is a fundamental issue with the way SelectHiveQL attempts to create an 
> Avro schema without having seen any of the rows, since the sub-types of 
> complex items cannot be determined from the SQL type alone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #922: NIFI-2622: Added support for complex types in SelectHiveQL

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/922
  
Looks good, verified this work with array and map columns, will merge to 
master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #919: NIFI-2632: Added fragment attributes to SplitJson an...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/919#discussion_r75937909
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitJson.java
 ---
@@ -61,6 +63,16 @@
 + "Each generated FlowFile is comprised of an element of the 
specified array and transferred to relationship 'split,' "
 + "with the original file transferred to the 'original' 
relationship. If the specified JsonPath is not found or "
 + "does not evaluate to an array element, the original file is 
routed to 'failure' and no files are generated.")
+@WritesAttributes({
+@WritesAttribute(attribute = "fragment.identifier",
--- End diff --

I don't like the name either, but wanted to keep it consistent with the 
other split processors 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2632) Add fragment attributes for SplitJson and SplitXml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433487#comment-15433487
 ] 

ASF GitHub Bot commented on NIFI-2632:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/919#discussion_r75937909
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitJson.java
 ---
@@ -61,6 +63,16 @@
 + "Each generated FlowFile is comprised of an element of the 
specified array and transferred to relationship 'split,' "
 + "with the original file transferred to the 'original' 
relationship. If the specified JsonPath is not found or "
 + "does not evaluate to an array element, the original file is 
routed to 'failure' and no files are generated.")
+@WritesAttributes({
+@WritesAttribute(attribute = "fragment.identifier",
--- End diff --

I don't like the name either, but wanted to keep it consistent with the 
other split processors 


> Add fragment attributes for SplitJson and SplitXml
> --
>
> Key: NIFI-2632
> URL: https://issues.apache.org/jira/browse/NIFI-2632
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitJson and SplitXml processors should 
> write these same attributes for their split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433476#comment-15433476
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
Ok I got it to a point where it can be run multiple times on an 
already-encrypted file and the only side-effect is that previously encrypted 
properties will look like (after 2 executions):

```

nifi.sensitive.props.key=C7AlglOx5QOmggfM||EG+qM9ubldyr8J73RmXCyWIxRKCzn6JyXv7mN6DkV51oDoRSdnWWXbALfKcKFw
nifi.sensitive.props.key.protected=aes/gcm/256
nifi.sensitive.props.key.protected=aes/gcm/256
```

Fixing that and then will push. 


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
> 020: 4d38 6d32 7952 4776 5739 

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
Ok I got it to a point where it can be run multiple times on an 
already-encrypted file and the only side-effect is that previously encrypted 
properties will look like (after 2 executions):

```

nifi.sensitive.props.key=C7AlglOx5QOmggfM||EG+qM9ubldyr8J73RmXCyWIxRKCzn6JyXv7mN6DkV51oDoRSdnWWXbALfKcKFw
nifi.sensitive.props.key.protected=aes/gcm/256
nifi.sensitive.props.key.protected=aes/gcm/256
```

Fixing that and then will push. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2632) Add fragment attributes for SplitJson and SplitXml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433434#comment-15433434
 ] 

ASF GitHub Bot commented on NIFI-2632:
--

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/919#discussion_r75933924
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitJson.java
 ---
@@ -61,6 +63,16 @@
 + "Each generated FlowFile is comprised of an element of the 
specified array and transferred to relationship 'split,' "
 + "with the original file transferred to the 'original' 
relationship. If the specified JsonPath is not found or "
 + "does not evaluate to an array element, the original file is 
routed to 'failure' and no files are generated.")
+@WritesAttributes({
+@WritesAttribute(attribute = "fragment.identifier",
--- End diff --

Suggest a different name to indicate this is more of a fragment from a the 
same parent (e.g. fragment.parent.identifier or fragment.group.identifier)?  To 
me using fragment alone implies a unique id for that split instance (even 
though I know the attribute description explains more)


> Add fragment attributes for SplitJson and SplitXml
> --
>
> Key: NIFI-2632
> URL: https://issues.apache.org/jira/browse/NIFI-2632
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitJson and SplitXml processors should 
> write these same attributes for their split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #919: NIFI-2632: Added fragment attributes to SplitJson an...

[YolandaMDavis]

Github user YolandaMDavis commented on a diff in the pull request:

https://github.com/apache/nifi/pull/919#discussion_r75933924
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitJson.java
 ---
@@ -61,6 +63,16 @@
 + "Each generated FlowFile is comprised of an element of the 
specified array and transferred to relationship 'split,' "
 + "with the original file transferred to the 'original' 
relationship. If the specified JsonPath is not found or "
 + "does not evaluate to an array element, the original file is 
routed to 'failure' and no files are generated.")
+@WritesAttributes({
+@WritesAttribute(attribute = "fragment.identifier",
--- End diff --

Suggest a different name to indicate this is more of a fragment from a the 
same parent (e.g. fragment.parent.identifier or fragment.group.identifier)?  To 
me using fragment alone implies a unique id for that split instance (even 
though I know the attribute description explains more)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433416#comment-15433416
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
Running the latest commit, nifi.properties ends up with:
```
nifi.sensitive.props.key.protected=aes/gcm/128
nifi.sensitive.props.key.protected=
```
The app then fails to start because I think it finds the second one and 
can't decrypt the sensitive properties key. I can paste the stacktrace if 
helpful, but removing the empty one lets the app start.


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
> 020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords 

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
Running the latest commit, nifi.properties ends up with:
```
nifi.sensitive.props.key.protected=aes/gcm/128
nifi.sensitive.props.key.protected=
```
The app then fails to start because I think it finds the second one and 
can't decrypt the sensitive properties key. I can paste the stacktrace if 
helpful, but removing the empty one lets the app start.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433398#comment-15433398
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
Ok that is not working for me... If I run the tool a second time using the 
same password, even without changing anything I get:
```
./nifi-toolkit-1.0.0-SNAPSHOT/bin/encrypt-config.sh -b 
nifi-1.0.0-SNAPSHOT/conf/bootstrap.conf -n 
nifi-1.0.0-SNAPSHOT/conf/nifi.properties
2016-08-23 14:56:51,772 WARN [main] 
o.a.nifi.properties.ConfigEncryptionTool The source nifi.properties and 
destination nifi.properties are identical 
[nifi-1.0.0-SNAPSHOT/conf/nifi.properties] so the original will be overwritten
Enter the password:
2016-08-23 14:56:56,604 INFO [main] 
o.a.nifi.properties.NiFiPropertiesLoader Loaded 118 properties from 
/Users/bbende/Projects/bbende-nifi/nifi-assembly/target/nifi-1.0.0-SNAPSHOT-bin/nifi-1.0.0-SNAPSHOT/conf/nifi.properties
2016-08-23 14:56:56,609 INFO [main] 
o.a.n.properties.ProtectedNiFiProperties Loaded 118 properties (including 3 
protection schemes) into ProtectedNiFiProperties
Cannot load NiFiProperties from [nifi-1.0.0-SNAPSHOT/conf/nifi.properties]

usage: org.apache.nifi.properties.ConfigEncryptionTool [-b ] [-h] [-k 
] [-n ] [-o ] [-p ] [-r]
```


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> 

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
Ok that is not working for me... If I run the tool a second time using the 
same password, even without changing anything I get:
```
./nifi-toolkit-1.0.0-SNAPSHOT/bin/encrypt-config.sh -b 
nifi-1.0.0-SNAPSHOT/conf/bootstrap.conf -n 
nifi-1.0.0-SNAPSHOT/conf/nifi.properties
2016-08-23 14:56:51,772 WARN [main] 
o.a.nifi.properties.ConfigEncryptionTool The source nifi.properties and 
destination nifi.properties are identical 
[nifi-1.0.0-SNAPSHOT/conf/nifi.properties] so the original will be overwritten
Enter the password:
2016-08-23 14:56:56,604 INFO [main] 
o.a.nifi.properties.NiFiPropertiesLoader Loaded 118 properties from 
/Users/bbende/Projects/bbende-nifi/nifi-assembly/target/nifi-1.0.0-SNAPSHOT-bin/nifi-1.0.0-SNAPSHOT/conf/nifi.properties
2016-08-23 14:56:56,609 INFO [main] 
o.a.n.properties.ProtectedNiFiProperties Loaded 118 properties (including 3 
protection schemes) into ProtectedNiFiProperties
Cannot load NiFiProperties from [nifi-1.0.0-SNAPSHOT/conf/nifi.properties]

usage: org.apache.nifi.properties.ConfigEncryptionTool [-b ] [-h] [-k 
] [-n ] [-o ] [-p ] [-r]
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433381#comment-15433381
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
My intention is that you can run it repeatedly against the 
already-protected file if you add a new value. It should skip over the already 
protected values and only protect any new ones (marked as sensitive). Obviously 
you have to use the same password. Does this not work for you?



> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
> 020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:16 $ more password.enc
> 

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
My intention is that you can run it repeatedly against the 
already-protected file if you add a new value. It should skip over the already 
protected values and only protect any new ones (marked as sensitive). Obviously 
you have to use the same password. Does this not work for you?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2637) Allow CreateHadoopSequenceFile compression codec to be configurable

[Caleb Fenton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Fenton updated NIFI-2637:
---
Description: 
Currently, when creating a Hadoop Sequence File, the compression codec is 
hardcoded as DefaultCodec, which is deflate. It would be better if this was 
configurable or the codec is inferred from the Hadoop configuration.

Here's the relevant line in code:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/SequenceFileWriterImpl.java#L100

Here's the relevant block of code where the writer is created with the 
DefaultCodec:
{noformat}
final SequenceFile.Writer writer = SequenceFile.createWriter(configuration,
SequenceFile.Writer.stream(fsDataOutputStream),
SequenceFile.Writer.keyClass(Text.class),

SequenceFile.Writer.valueClass(InputStreamWritable.class),

SequenceFile.Writer.compression(compressionType, new DefaultCodec(
{noformat}

Basically, I want to use Snappy for certain sequence files, and I can't do that 
by changing Hadoop or processor configuration. It looks like I'll have to 
change the source.

  was:
Currently, when creating a Hadoop Sequence File, the compression codec is 
hardcoded as DefaultCodec, which is deflate. It would be better if this was 
configurable or the codec is inferred from the Hadoop configuration.

Here's the relevant line in code:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/SequenceFileWriterImpl.java#L100

Here's the relevant block of code where the writer is created with the 
DefaultCodec:
{noformat}
final SequenceFile.Writer writer = SequenceFile.createWriter(configuration,
SequenceFile.Writer.stream(fsDataOutputStream),
SequenceFile.Writer.keyClass(Text.class),

SequenceFile.Writer.valueClass(InputStreamWritable.class),

SequenceFile.Writer.compression(compressionType, new DefaultCodec(
{noformat}


> Allow CreateHadoopSequenceFile compression codec to be configurable
> ---
>
> Key: NIFI-2637
> URL: https://issues.apache.org/jira/browse/NIFI-2637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Caleb Fenton
>Priority: Minor
>
> Currently, when creating a Hadoop Sequence File, the compression codec is 
> hardcoded as DefaultCodec, which is deflate. It would be better if this was 
> configurable or the codec is inferred from the Hadoop configuration.
> Here's the relevant line in code:
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/SequenceFileWriterImpl.java#L100
> Here's the relevant block of code where the writer is created with the 
> DefaultCodec:
> {noformat}
> final SequenceFile.Writer writer = SequenceFile.createWriter(configuration,
> 
> SequenceFile.Writer.stream(fsDataOutputStream),
> SequenceFile.Writer.keyClass(Text.class),
> 
> SequenceFile.Writer.valueClass(InputStreamWritable.class),
> 
> SequenceFile.Writer.compression(compressionType, new DefaultCodec(
> {noformat}
> Basically, I want to use Snappy for certain sequence files, and I can't do 
> that by changing Hadoop or processor configuration. It looks like I'll have 
> to change the source.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2637) Allow CreateHadoopSequenceFile compression codec to be configurable

[Caleb Fenton (JIRA)]

Caleb Fenton created NIFI-2637:
--

 Summary: Allow CreateHadoopSequenceFile compression codec to be 
configurable
 Key: NIFI-2637
 URL: https://issues.apache.org/jira/browse/NIFI-2637
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Caleb Fenton
Priority: Minor


Currently, when creating a Hadoop Sequence File, the compression codec is 
hardcoded as DefaultCodec, which is deflate. It would be better if this was 
configurable or the codec is inferred from the Hadoop configuration.

Here's the relevant line in code:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/SequenceFileWriterImpl.java#L100

Here's the relevant block of code where the writer is created with the 
DefaultCodec:
{noformat}
final SequenceFile.Writer writer = SequenceFile.createWriter(configuration,
SequenceFile.Writer.stream(fsDataOutputStream),
SequenceFile.Writer.keyClass(Text.class),

SequenceFile.Writer.valueClass(InputStreamWritable.class),

SequenceFile.Writer.compression(compressionType, new DefaultCodec(
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433360#comment-15433360
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/834
  
I was actually thinking less about accidentally overwriting, and more about 
something like this... 

I setup nifi.properties, encrypt it, and everything is good, then I decide 
I want to add another encrypted property. Right now you can't run the tool a 
second time against an already protected nifi.properties, which makes sense, 
but it means you have to manually undo everything if you wanted to add another 
protected property.

I'm not hung up on this since it is not very hard to reset the file 
manually, just wanted to see what you thought about the reversal/decryption. 
I'm ok leaving everything as is for now.


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 

[jira] [Commented] (NIFI-2624) JDBC-to-Avro processors handle BigDecimals as Strings

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433354#comment-15433354
 ] 

Matt Burgess commented on NIFI-2624:


Looks like the parsing of logical types (like DECIMAL) was not added until Avro 
1.7.8 (and 1.8.0) under AVRO-1497, I believe we will need to upgrade Avro 
before being able to handle BigDecimals properly.

> JDBC-to-Avro processors handle BigDecimals as Strings
> -
>
> Key: NIFI-2624
> URL: https://issues.apache.org/jira/browse/NIFI-2624
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>
> The original SQL processors implemented BigDecimal values as Strings for 
> Avro, as the version of Avro it used (1.7.6) did not support DECIMAL type.
> As of Avro 1.7.7 (AVRO-1402), this type is supported and so the SQL/HiveQL 
> processors should be updated to handle BigDecimals correctly if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2624) JDBC-to-Avro processors handle BigDecimals as Strings

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-2624:
---
Fix Version/s: (was: 1.1.0)

> JDBC-to-Avro processors handle BigDecimals as Strings
> -
>
> Key: NIFI-2624
> URL: https://issues.apache.org/jira/browse/NIFI-2624
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>
> The original SQL processors implemented BigDecimal values as Strings for 
> Avro, as the version of Avro it used (1.7.6) did not support DECIMAL type.
> As of Avro 1.7.7 (AVRO-1402), this type is supported and so the SQL/HiveQL 
> processors should be updated to handle BigDecimals correctly if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433310#comment-15433310
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75919751
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
 ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import groovy.io.GroovyPrintWriter
+import org.apache.commons.cli.CommandLine
+import org.apache.commons.cli.CommandLineParser
+import org.apache.commons.cli.DefaultParser
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.commons.cli.ParseException
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.toolkit.tls.commandLine.ExitCode
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.charset.StandardCharsets
+import java.security.KeyException
+import java.security.Security
+
+class ConfigEncryptionTool {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionTool.class)
+
+public String bootstrapConfPath
+public String niFiPropertiesPath
+public String outputNiFiPropertiesPath
+public String loginIdentityProvidersPath
+
+private String keyHex
+private String password
+private NiFiProperties niFiProperties
+
+private boolean usingPassword = true
+
+private static final String HELP_ARG = "help"
+private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf"
+private static final String NIFI_PROPERTIES_ARG = "niFiProperties"
+private static final String OUTPUT_NIFI_PROPERTIES_ARG = 
"outputNiFiProperties"
+private static final String KEY_ARG = "key"
+private static final String PASSWORD_ARG = "password"
+private static final String USE_KEY_ARG = "useRawKey"
+
+private static final int MIN_PASSWORD_LENGTH = 12
+
+// Strong parameters as of 12 Aug 2016
+private static final int SCRYPT_N = 2**16
+private static final int SCRYPT_R = 8
+private static final int SCRYPT_P = 1
+
+private static
+final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal 
format for encrypted sensitive configuration values"
+private static final String BOOTSTRAP_KEY_PREFIX = 
"nifi.bootstrap.sensitive.key="
+private static final String JAVA_HOME = "JAVA_HOME"
+private static final String NIFI_TOOLKIT_HOME = "NIFI_TOOLKIT_HOME"
+private static final String SEP = System.lineSeparator()
+
+private static final String FOOTER = buildFooter()
+
+private static
+final String DEFAULT_DESCRIPTION = "This tool reads from a 
nifi.properties file with plain sensitive configuration values, prompts the 
user for a master key, and encrypts each value. It will replace the plain value 
with the protected value in the same file (or write to a new nifi.properties 
file if specified)."
+
+private static String buildHeader(String description = 
DEFAULT_DESCRIPTION) {
+"${SEP}${description}${SEP * 2}"
+}
+
+private static String buildFooter() {
+"${SEP}Java home: ${System.getenv(JAVA_HOME)}${SEP}NiFi Toolkit 
home: ${System.getenv(NIFI_TOOLKIT_HOME)}"
+}
+
+private final Options options;
+private final String header;
 

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433287#comment-15433287
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
I have a note to possibly pause and prompt for confirmation on overwrite 
with the option to pass a flag allowing *force yes* behavior. I am hesitant to 
add "decryption" or "reversal" capability to the tool even requiring the 
password/key as input. 

Do you think what I suggested above would be sufficient? I am worried about 
someone trying to fire the tool without interaction and not realizing it will 
wait for confirmation, but I also accept that the tool is unlikely to be used 
in an unsupervised environment. 


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 

[GitHub] nifi issue #834: NIFI-1831 Implemented encrypted configuration capabilities

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/834
  
I have a note to possibly pause and prompt for confirmation on overwrite 
with the option to pass a flag allowing *force yes* behavior. I am hesitant to 
add "decryption" or "reversal" capability to the tool even requiring the 
password/key as input. 

Do you think what I suggested above would be sufficient? I am worried about 
someone trying to fire the tool without interaction and not realizing it will 
wait for confirmation, but I also accept that the tool is unlikely to be used 
in an unsupervised environment. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433283#comment-15433283
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75917246
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
 ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import groovy.io.GroovyPrintWriter
+import org.apache.commons.cli.CommandLine
+import org.apache.commons.cli.CommandLineParser
+import org.apache.commons.cli.DefaultParser
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.commons.cli.ParseException
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.toolkit.tls.commandLine.ExitCode
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.charset.StandardCharsets
+import java.security.KeyException
+import java.security.Security
+
+class ConfigEncryptionTool {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionTool.class)
+
+public String bootstrapConfPath
+public String niFiPropertiesPath
+public String outputNiFiPropertiesPath
+public String loginIdentityProvidersPath
+
+private String keyHex
+private String password
+private NiFiProperties niFiProperties
+
+private boolean usingPassword = true
+
+private static final String HELP_ARG = "help"
+private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf"
+private static final String NIFI_PROPERTIES_ARG = "niFiProperties"
+private static final String OUTPUT_NIFI_PROPERTIES_ARG = 
"outputNiFiProperties"
+private static final String KEY_ARG = "key"
+private static final String PASSWORD_ARG = "password"
+private static final String USE_KEY_ARG = "useRawKey"
+
+private static final int MIN_PASSWORD_LENGTH = 12
+
+// Strong parameters as of 12 Aug 2016
+private static final int SCRYPT_N = 2**16
+private static final int SCRYPT_R = 8
+private static final int SCRYPT_P = 1
+
+private static
+final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal 
format for encrypted sensitive configuration values"
+private static final String BOOTSTRAP_KEY_PREFIX = 
"nifi.bootstrap.sensitive.key="
+private static final String JAVA_HOME = "JAVA_HOME"
+private static final String NIFI_TOOLKIT_HOME = "NIFI_TOOLKIT_HOME"
+private static final String SEP = System.lineSeparator()
+
+private static final String FOOTER = buildFooter()
+
+private static
+final String DEFAULT_DESCRIPTION = "This tool reads from a 
nifi.properties file with plain sensitive configuration values, prompts the 
user for a master key, and encrypts each value. It will replace the plain value 
with the protected value in the same file (or write to a new nifi.properties 
file if specified)."
+
+private static String buildHeader(String description = 
DEFAULT_DESCRIPTION) {
+"${SEP}${description}${SEP * 2}"
+}
+
+private static String buildFooter() {
+"${SEP}Java home: ${System.getenv(JAVA_HOME)}${SEP}NiFi Toolkit 
home: ${System.getenv(NIFI_TOOLKIT_HOME)}"
+}
+
+private final Options options;
+private final String header;

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75917246
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
 ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import groovy.io.GroovyPrintWriter
+import org.apache.commons.cli.CommandLine
+import org.apache.commons.cli.CommandLineParser
+import org.apache.commons.cli.DefaultParser
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.commons.cli.ParseException
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.toolkit.tls.commandLine.ExitCode
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.charset.StandardCharsets
+import java.security.KeyException
+import java.security.Security
+
+class ConfigEncryptionTool {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionTool.class)
+
+public String bootstrapConfPath
+public String niFiPropertiesPath
+public String outputNiFiPropertiesPath
+public String loginIdentityProvidersPath
+
+private String keyHex
+private String password
+private NiFiProperties niFiProperties
+
+private boolean usingPassword = true
+
+private static final String HELP_ARG = "help"
+private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf"
+private static final String NIFI_PROPERTIES_ARG = "niFiProperties"
+private static final String OUTPUT_NIFI_PROPERTIES_ARG = 
"outputNiFiProperties"
+private static final String KEY_ARG = "key"
+private static final String PASSWORD_ARG = "password"
+private static final String USE_KEY_ARG = "useRawKey"
+
+private static final int MIN_PASSWORD_LENGTH = 12
+
+// Strong parameters as of 12 Aug 2016
+private static final int SCRYPT_N = 2**16
+private static final int SCRYPT_R = 8
+private static final int SCRYPT_P = 1
+
+private static
+final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal 
format for encrypted sensitive configuration values"
+private static final String BOOTSTRAP_KEY_PREFIX = 
"nifi.bootstrap.sensitive.key="
+private static final String JAVA_HOME = "JAVA_HOME"
+private static final String NIFI_TOOLKIT_HOME = "NIFI_TOOLKIT_HOME"
+private static final String SEP = System.lineSeparator()
+
+private static final String FOOTER = buildFooter()
+
+private static
+final String DEFAULT_DESCRIPTION = "This tool reads from a 
nifi.properties file with plain sensitive configuration values, prompts the 
user for a master key, and encrypts each value. It will replace the plain value 
with the protected value in the same file (or write to a new nifi.properties 
file if specified)."
+
+private static String buildHeader(String description = 
DEFAULT_DESCRIPTION) {
+"${SEP}${description}${SEP * 2}"
+}
+
+private static String buildFooter() {
+"${SEP}Java home: ${System.getenv(JAVA_HOME)}${SEP}NiFi Toolkit 
home: ${System.getenv(NIFI_TOOLKIT_HOME)}"
+}
+
+private final Options options;
+private final String header;
+
+
+public ConfigEncryptionTool() {
+this(DEFAULT_DESCRIPTION)
+}
+
+public ConfigEncryptionTool(String description) {
+this.header = buildHeader(description)
+

[jira] [Updated] (NIFI-1831) Allow encrypted passwords in configuration files

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-1831:
---
Status: Patch Available  (was: Open)

> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
> 020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:16 $ more password.enc
> VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:55 $
> {code}
> In {{nifi.properties}}: 
> {code}
> nifi.security.keystorePasswd=VCXVaFbPAX6OWCvFickvJcwD8Tkg71LvM8m2yRGvW9A=
> nifi.security.keystorePasswd.encrypted=AES-CBC-256
> {code}
> Example 2: *here the encrypted password has a header tag indicating 

[GitHub] nifi issue #919: NIFI-2632: Added fragment attributes to SplitJson and Split...

[YolandaMDavis]

Github user YolandaMDavis commented on the issue:

https://github.com/apache/nifi/pull/919
  
@mattyb149 I can review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2632) Add fragment attributes for SplitJson and SplitXml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433257#comment-15433257
 ] 

ASF GitHub Bot commented on NIFI-2632:
--

Github user YolandaMDavis commented on the issue:

https://github.com/apache/nifi/pull/919
  
@mattyb149 I can review


> Add fragment attributes for SplitJson and SplitXml
> --
>
> Key: NIFI-2632
> URL: https://issues.apache.org/jira/browse/NIFI-2632
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Some "splitting" processors such as SplitText and SplitContent write 
> attributes to the split flow files indicating their fragment index, the total 
> count, and the filename of the original flow file. This is done to support a 
> form of "micro-batching" and/or a split-join pattern in a data flow (i.e. 
> split the original file, do work on the individual pieces, and possibly merge 
> them together later).
> For consistency and capability, the SplitJson and SplitXml processors should 
> write these same attributes for their split flow files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-2620) Add ability to write Row Identifier as binary in hbase using the PutHbaseCell

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende resolved NIFI-2620.
---
Resolution: Fixed

> Add ability to write Row Identifier as binary in hbase using the PutHbaseCell
> -
>
> Key: NIFI-2620
> URL: https://issues.apache.org/jira/browse/NIFI-2620
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0, 0.7.0, 0.6.1
>Reporter: Andrew Psaltis
>Assignee: Andrew Psaltis
> Fix For: 1.0.0
>
>
> Today the PutHBaseCell processor makes the assumption that all row keys are 
> text. However, this does not work when the row key in the HBase table is 
> binary. 
> If the row key is specified in the binary string format, such as:
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x01\x01\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x01\x00\x00\x01\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x01\x00\x00\x01\x00\x00\x00\x00\x01 
> \x01\x01\x00\x01\x00\x01\x01\x01\x00\x00\x 
> 00\x00\x00\x00\x01\x01\x01\x01\x00\x00\x00 
> \x00\x00\x00\x01\x01\x00\x01\x00\x01\x00\x 
> 00\x01\x01\x01\x01\x00\x00\x01\x01\x01\x00 
> \x01\x00\x00
> Which is the textual representation that the HBase CLI would return, NiFi 
> calls getBytes on that string. Appropriately HBase will encode the '\' with 
> the hex code: x5C, resulting in an output string that looks like:
> x5Cx00\x5Cx00\ ...
> To address this the proposed solution would be to:
> *  Add "toBytesBinary" method to HBaseClientService  similar to the ones 
> already added [1]. 
> * Update the PutFlowFile and PutColumn to pass around mostly byte[] and not 
> strings that they do today.
> For this JIRA only support for Text and Binary will be added for the RowKey
> [1] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java#L427



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2620) Add ability to write Row Identifier as binary in hbase using the PutHbaseCell

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433242#comment-15433242
 ] 

ASF GitHub Bot commented on NIFI-2620:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/914


> Add ability to write Row Identifier as binary in hbase using the PutHbaseCell
> -
>
> Key: NIFI-2620
> URL: https://issues.apache.org/jira/browse/NIFI-2620
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0, 0.7.0, 0.6.1
>Reporter: Andrew Psaltis
>Assignee: Andrew Psaltis
> Fix For: 1.0.0
>
>
> Today the PutHBaseCell processor makes the assumption that all row keys are 
> text. However, this does not work when the row key in the HBase table is 
> binary. 
> If the row key is specified in the binary string format, such as:
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x01\x01\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x01\x00\x00\x01\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x01\x00\x00\x01\x00\x00\x00\x00\x01 
> \x01\x01\x00\x01\x00\x01\x01\x01\x00\x00\x 
> 00\x00\x00\x00\x01\x01\x01\x01\x00\x00\x00 
> \x00\x00\x00\x01\x01\x00\x01\x00\x01\x00\x 
> 00\x01\x01\x01\x01\x00\x00\x01\x01\x01\x00 
> \x01\x00\x00
> Which is the textual representation that the HBase CLI would return, NiFi 
> calls getBytes on that string. Appropriately HBase will encode the '\' with 
> the hex code: x5C, resulting in an output string that looks like:
> x5Cx00\x5Cx00\ ...
> To address this the proposed solution would be to:
> *  Add "toBytesBinary" method to HBaseClientService  similar to the ones 
> already added [1]. 
> * Update the PutFlowFile and PutColumn to pass around mostly byte[] and not 
> strings that they do today.
> For this JIRA only support for Text and Binary will be added for the RowKey
> [1] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java#L427



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433240#comment-15433240
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75912621
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
 ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import groovy.io.GroovyPrintWriter
+import org.apache.commons.cli.CommandLine
+import org.apache.commons.cli.CommandLineParser
+import org.apache.commons.cli.DefaultParser
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.commons.cli.ParseException
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.toolkit.tls.commandLine.ExitCode
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.charset.StandardCharsets
+import java.security.KeyException
+import java.security.Security
+
+class ConfigEncryptionTool {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionTool.class)
+
+public String bootstrapConfPath
+public String niFiPropertiesPath
+public String outputNiFiPropertiesPath
+public String loginIdentityProvidersPath
+
+private String keyHex
+private String password
+private NiFiProperties niFiProperties
+
+private boolean usingPassword = true
+
+private static final String HELP_ARG = "help"
+private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf"
+private static final String NIFI_PROPERTIES_ARG = "niFiProperties"
+private static final String OUTPUT_NIFI_PROPERTIES_ARG = 
"outputNiFiProperties"
+private static final String KEY_ARG = "key"
+private static final String PASSWORD_ARG = "password"
+private static final String USE_KEY_ARG = "useRawKey"
+
+private static final int MIN_PASSWORD_LENGTH = 12
+
+// Strong parameters as of 12 Aug 2016
+private static final int SCRYPT_N = 2**16
+private static final int SCRYPT_R = 8
+private static final int SCRYPT_P = 1
+
+private static
+final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal 
format for encrypted sensitive configuration values"
+private static final String BOOTSTRAP_KEY_PREFIX = 
"nifi.bootstrap.sensitive.key="
+private static final String JAVA_HOME = "JAVA_HOME"
+private static final String NIFI_TOOLKIT_HOME = "NIFI_TOOLKIT_HOME"
+private static final String SEP = System.lineSeparator()
+
+private static final String FOOTER = buildFooter()
+
+private static
+final String DEFAULT_DESCRIPTION = "This tool reads from a 
nifi.properties file with plain sensitive configuration values, prompts the 
user for a master key, and encrypts each value. It will replace the plain value 
with the protected value in the same file (or write to a new nifi.properties 
file if specified)."
+
+private static String buildHeader(String description = 
DEFAULT_DESCRIPTION) {
+"${SEP}${description}${SEP * 2}"
+}
+
+private static String buildFooter() {
+"${SEP}Java home: ${System.getenv(JAVA_HOME)}${SEP}NiFi Toolkit 
home: ${System.getenv(NIFI_TOOLKIT_HOME)}"
+}
+
+private final Options options;
+private final String header;
 

[jira] [Commented] (NIFI-2620) Add ability to write Row Identifier as binary in hbase using the PutHbaseCell

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433241#comment-15433241
 ] 

ASF subversion and git services commented on NIFI-2620:
---

Commit 0303805c014b54c095827a7b191e5efb136a4e2e in nifi's branch 
refs/heads/master from [~apsaltis]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0303805 ]

NIFI-2620 Adding support for Binary Row Keys for both PutHBaseCell and 
PutHBaseJSON. This also involved making changes to PutFlowFile and PutColumn to 
carry around byte[] and not all strings. This closes #914.

Signed-off-by: Bryan Bende 


> Add ability to write Row Identifier as binary in hbase using the PutHbaseCell
> -
>
> Key: NIFI-2620
> URL: https://issues.apache.org/jira/browse/NIFI-2620
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0, 0.7.0, 0.6.1
>Reporter: Andrew Psaltis
>Assignee: Andrew Psaltis
> Fix For: 1.0.0
>
>
> Today the PutHBaseCell processor makes the assumption that all row keys are 
> text. However, this does not work when the row key in the HBase table is 
> binary. 
> If the row key is specified in the binary string format, such as:
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x01\x01\x00\x 
> 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x00\x00\x00\x00\x01\x00\x00\x01\x00 
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x 
> 00\x00\x01\x00\x00\x01\x00\x00\x00\x00\x01 
> \x01\x01\x00\x01\x00\x01\x01\x01\x00\x00\x 
> 00\x00\x00\x00\x01\x01\x01\x01\x00\x00\x00 
> \x00\x00\x00\x01\x01\x00\x01\x00\x01\x00\x 
> 00\x01\x01\x01\x01\x00\x00\x01\x01\x01\x00 
> \x01\x00\x00
> Which is the textual representation that the HBase CLI would return, NiFi 
> calls getBytes on that string. Appropriately HBase will encode the '\' with 
> the hex code: x5C, resulting in an output string that looks like:
> x5Cx00\x5Cx00\ ...
> To address this the proposed solution would be to:
> *  Add "toBytesBinary" method to HBaseClientService  similar to the ones 
> already added [1]. 
> * Update the PutFlowFile and PutColumn to pass around mostly byte[] and not 
> strings that they do today.
> For this JIRA only support for Text and Binary will be added for the RowKey
> [1] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java#L427



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2610) TestProcessorLifecycle and TestStandardProcessScheduler classes causes brittle builds and appears to be an integration test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433243#comment-15433243
 ] 

ASF GitHub Bot commented on NIFI-2610:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/918


> TestProcessorLifecycle and TestStandardProcessScheduler classes causes 
> brittle builds and appears to be an integration test
> ---
>
> Key: NIFI-2610
> URL: https://issues.apache.org/jira/browse/NIFI-2610
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> The tests in TestProcessorLifecycle appear to be attempting to replicate 
> various threading scenarios.  Such tests are notoriously difficult to get 
> right and indeed the build is brittle as a result.  These tests are likely 
> valuable and should be improved but they also should be considered 
> integration tests it appears.
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.708 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateSuccessfullAndOrderlyShutdown(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 6.313 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateSuccessfullAndOrderlyShutdown(TestProcessorLifecycle.java:224)
> This test also causes build problems and seems to be of a similar style
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.447 sec 
> <<< FAILURE! - in 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler
> validateEnabledDisableMultiThread(org.apache.nifi.controller.scheduling.TestStandardProcessScheduler)
>  Time elapsed: 5.667 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler.validateEnabledDisableMultiThread(TestStandardProcessScheduler.java:373)
> Brittle tests like this risk the build process which harms the review cycle 
> and complicates release voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #918: NIFI-2610 annotated unstable tests with @Ignore

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/918


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75912621
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy
 ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import groovy.io.GroovyPrintWriter
+import org.apache.commons.cli.CommandLine
+import org.apache.commons.cli.CommandLineParser
+import org.apache.commons.cli.DefaultParser
+import org.apache.commons.cli.HelpFormatter
+import org.apache.commons.cli.Options
+import org.apache.commons.cli.ParseException
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.toolkit.tls.commandLine.ExitCode
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.charset.StandardCharsets
+import java.security.KeyException
+import java.security.Security
+
+class ConfigEncryptionTool {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionTool.class)
+
+public String bootstrapConfPath
+public String niFiPropertiesPath
+public String outputNiFiPropertiesPath
+public String loginIdentityProvidersPath
+
+private String keyHex
+private String password
+private NiFiProperties niFiProperties
+
+private boolean usingPassword = true
+
+private static final String HELP_ARG = "help"
+private static final String BOOTSTRAP_CONF_ARG = "bootstrapConf"
+private static final String NIFI_PROPERTIES_ARG = "niFiProperties"
+private static final String OUTPUT_NIFI_PROPERTIES_ARG = 
"outputNiFiProperties"
+private static final String KEY_ARG = "key"
+private static final String PASSWORD_ARG = "password"
+private static final String USE_KEY_ARG = "useRawKey"
+
+private static final int MIN_PASSWORD_LENGTH = 12
+
+// Strong parameters as of 12 Aug 2016
+private static final int SCRYPT_N = 2**16
+private static final int SCRYPT_R = 8
+private static final int SCRYPT_P = 1
+
+private static
+final String BOOTSTRAP_KEY_COMMENT = "# Master key in hexadecimal 
format for encrypted sensitive configuration values"
+private static final String BOOTSTRAP_KEY_PREFIX = 
"nifi.bootstrap.sensitive.key="
+private static final String JAVA_HOME = "JAVA_HOME"
+private static final String NIFI_TOOLKIT_HOME = "NIFI_TOOLKIT_HOME"
+private static final String SEP = System.lineSeparator()
+
+private static final String FOOTER = buildFooter()
+
+private static
+final String DEFAULT_DESCRIPTION = "This tool reads from a 
nifi.properties file with plain sensitive configuration values, prompts the 
user for a master key, and encrypts each value. It will replace the plain value 
with the protected value in the same file (or write to a new nifi.properties 
file if specified)."
+
+private static String buildHeader(String description = 
DEFAULT_DESCRIPTION) {
+"${SEP}${description}${SEP * 2}"
+}
+
+private static String buildFooter() {
+"${SEP}Java home: ${System.getenv(JAVA_HOME)}${SEP}NiFi Toolkit 
home: ${System.getenv(NIFI_TOOLKIT_HOME)}"
+}
+
+private final Options options;
+private final String header;
+
+
+public ConfigEncryptionTool() {
+this(DEFAULT_DESCRIPTION)
+}
+
+public ConfigEncryptionTool(String description) {
+this.header = buildHeader(description)
+

[GitHub] nifi pull request #914: Adding support for Binary Row Keys

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/914


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #914: Adding support for Binary Row Keys

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/914
  
Tested and all looks good, will merge to master, thanks Andrew!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2610) TestProcessorLifecycle and TestStandardProcessScheduler classes causes brittle builds and appears to be an integration test

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-2610:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> TestProcessorLifecycle and TestStandardProcessScheduler classes causes 
> brittle builds and appears to be an integration test
> ---
>
> Key: NIFI-2610
> URL: https://issues.apache.org/jira/browse/NIFI-2610
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> The tests in TestProcessorLifecycle appear to be attempting to replicate 
> various threading scenarios.  Such tests are notoriously difficult to get 
> right and indeed the build is brittle as a result.  These tests are likely 
> valuable and should be improved but they also should be considered 
> integration tests it appears.
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.708 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateSuccessfullAndOrderlyShutdown(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 6.313 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateSuccessfullAndOrderlyShutdown(TestProcessorLifecycle.java:224)
> This test also causes build problems and seems to be of a similar style
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.447 sec 
> <<< FAILURE! - in 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler
> validateEnabledDisableMultiThread(org.apache.nifi.controller.scheduling.TestStandardProcessScheduler)
>  Time elapsed: 5.667 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler.validateEnabledDisableMultiThread(TestStandardProcessScheduler.java:373)
> Brittle tests like this risk the build process which harms the review cycle 
> and complicates release voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2610) TestProcessorLifecycle and TestStandardProcessScheduler classes causes brittle builds and appears to be an integration test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433219#comment-15433219
 ] 

ASF GitHub Bot commented on NIFI-2610:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/918
  
+1 LGTM, ran all unit tests without failures (and Travis passed). Merging 
to master


> TestProcessorLifecycle and TestStandardProcessScheduler classes causes 
> brittle builds and appears to be an integration test
> ---
>
> Key: NIFI-2610
> URL: https://issues.apache.org/jira/browse/NIFI-2610
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> The tests in TestProcessorLifecycle appear to be attempting to replicate 
> various threading scenarios.  Such tests are notoriously difficult to get 
> right and indeed the build is brittle as a result.  These tests are likely 
> valuable and should be improved but they also should be considered 
> integration tests it appears.
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.708 sec 
> <<< FAILURE! - in org.apache.nifi.controller.scheduling.TestProcessorLifecycle
> validateSuccessfullAndOrderlyShutdown(org.apache.nifi.controller.scheduling.TestProcessorLifecycle)
>   Time elapsed: 6.313 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<2>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> org.apache.nifi.controller.scheduling.TestProcessorLifecycle.validateSuccessfullAndOrderlyShutdown(TestProcessorLifecycle.java:224)
> This test also causes build problems and seems to be of a similar style
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.447 sec 
> <<< FAILURE! - in 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler
> validateEnabledDisableMultiThread(org.apache.nifi.controller.scheduling.TestStandardProcessScheduler)
>  Time elapsed: 5.667 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.controller.scheduling.TestStandardProcessScheduler.validateEnabledDisableMultiThread(TestStandardProcessScheduler.java:373)
> Brittle tests like this risk the build process which harms the review cycle 
> and complicates release voting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #918: NIFI-2610 annotated unstable tests with @Ignore

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/918
  
+1 LGTM, ran all unit tests without failures (and Travis passed). Merging 
to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433213#comment-15433213
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75909986
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
 ---
@@ -0,0 +1,1225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import ch.qos.logback.classic.spi.LoggingEvent
+import ch.qos.logback.core.AppenderBase
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.junit.After
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Rule
+import org.junit.Test
+import org.junit.contrib.java.lang.system.Assertion
+import org.junit.contrib.java.lang.system.ExpectedSystemExit
+import org.junit.contrib.java.lang.system.SystemOutRule
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.file.Files
+import java.nio.file.attribute.PosixFilePermission
+import java.security.KeyException
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class ConfigEncryptionToolTest extends GroovyTestCase {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionToolTest.class)
+
+@Rule
+public final ExpectedSystemExit exit = ExpectedSystemExit.none()
+
+@Rule
+public final SystemOutRule systemOutRule = new 
SystemOutRule().enableLog()
+
+private static final String KEY_HEX_128 = 
"0123456789ABCDEFFEDCBA9876543210"
+private static final String KEY_HEX_256 = KEY_HEX_128 * 2
+public static final String KEY_HEX = 
isUnlimitedStrengthCryptoAvailable() ? KEY_HEX_256 : KEY_HEX_128
+private static final String PASSWORD = "thisIsABadPassword"
+
+@BeforeClass
+public static void setUpOnce() throws Exception {
+Security.addProvider(new BouncyCastleProvider())
+
+logger.metaClass.methodMissing = { String name, args ->
+logger.info("[${name?.toUpperCase()}] ${(args as List).join(" 
")}")
+}
+}
+
+@Before
+public void setUp() throws Exception {
+
+}
+
+@After
+public void tearDown() throws Exception {
+TestAppender.reset()
+}
+
+private static boolean isUnlimitedStrengthCryptoAvailable() {
+Cipher.getMaxAllowedKeyLength("AES") > 128
+}
+
+@Test
+void testShouldPrintHelpMessage() {
+// Arrange
+def flags = ["-h", "--help"]
+ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+// Act
+flags.each { String arg ->
+def msg = shouldFail(CommandLineParseException) {
+tool.parse([arg] as String[])
+}
+
+// Assert
+assert msg == null
+assert systemOutRule.getLog().contains("usage: 
org.apache.nifi.properties.ConfigEncryptionTool [")
+}
+}
+
+@Test
+void testShouldParseBootstrapConfArgument() {
+// Arrange
+def flags = ["-b", "--bootstrapConf"]
+String bootstrapPath = "src/test/resources/bootstrap.conf"
+ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+// Act
+flags.each 

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433212#comment-15433212
 ] 

ASF GitHub Bot commented on NIFI-1831:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75909949
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties
 ---
@@ -131,8 +131,10 @@ nifi.web.jetty.threads=${nifi.web.jetty.threads}
 
 # security properties #
 nifi.sensitive.props.key=
+nifi.sensitive.props.key.protected=${nifi.sensitive.props.key.protected}
--- End diff --

Thanks for catching this Bryan. It was an issue where the tool had to merge 
two instances of `Properties` together and the logic wasn't checking the right 
key set. Added a test and fixed. Empty protection values no longer stop the 
field from being protected. 


> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s 

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75909986
  
--- Diff: 
nifi-toolkit/nifi-toolkit-encrypt-config/src/test/groovy/org/apache/nifi/properties/ConfigEncryptionToolTest.groovy
 ---
@@ -0,0 +1,1225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties
+
+import ch.qos.logback.classic.spi.LoggingEvent
+import ch.qos.logback.core.AppenderBase
+import org.apache.commons.codec.binary.Hex
+import org.apache.nifi.toolkit.tls.commandLine.CommandLineParseException
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.console.TextDevice
+import org.apache.nifi.util.console.TextDevices
+import org.bouncycastle.crypto.generators.SCrypt
+import org.bouncycastle.jce.provider.BouncyCastleProvider
+import org.junit.After
+import org.junit.Before
+import org.junit.BeforeClass
+import org.junit.Rule
+import org.junit.Test
+import org.junit.contrib.java.lang.system.Assertion
+import org.junit.contrib.java.lang.system.ExpectedSystemExit
+import org.junit.contrib.java.lang.system.SystemOutRule
+import org.junit.runner.RunWith
+import org.junit.runners.JUnit4
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+import javax.crypto.Cipher
+import java.nio.file.Files
+import java.nio.file.attribute.PosixFilePermission
+import java.security.KeyException
+import java.security.Security
+
+@RunWith(JUnit4.class)
+class ConfigEncryptionToolTest extends GroovyTestCase {
+private static final Logger logger = 
LoggerFactory.getLogger(ConfigEncryptionToolTest.class)
+
+@Rule
+public final ExpectedSystemExit exit = ExpectedSystemExit.none()
+
+@Rule
+public final SystemOutRule systemOutRule = new 
SystemOutRule().enableLog()
+
+private static final String KEY_HEX_128 = 
"0123456789ABCDEFFEDCBA9876543210"
+private static final String KEY_HEX_256 = KEY_HEX_128 * 2
+public static final String KEY_HEX = 
isUnlimitedStrengthCryptoAvailable() ? KEY_HEX_256 : KEY_HEX_128
+private static final String PASSWORD = "thisIsABadPassword"
+
+@BeforeClass
+public static void setUpOnce() throws Exception {
+Security.addProvider(new BouncyCastleProvider())
+
+logger.metaClass.methodMissing = { String name, args ->
+logger.info("[${name?.toUpperCase()}] ${(args as List).join(" 
")}")
+}
+}
+
+@Before
+public void setUp() throws Exception {
+
+}
+
+@After
+public void tearDown() throws Exception {
+TestAppender.reset()
+}
+
+private static boolean isUnlimitedStrengthCryptoAvailable() {
+Cipher.getMaxAllowedKeyLength("AES") > 128
+}
+
+@Test
+void testShouldPrintHelpMessage() {
+// Arrange
+def flags = ["-h", "--help"]
+ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+// Act
+flags.each { String arg ->
+def msg = shouldFail(CommandLineParseException) {
+tool.parse([arg] as String[])
+}
+
+// Assert
+assert msg == null
+assert systemOutRule.getLog().contains("usage: 
org.apache.nifi.properties.ConfigEncryptionTool [")
+}
+}
+
+@Test
+void testShouldParseBootstrapConfArgument() {
+// Arrange
+def flags = ["-b", "--bootstrapConf"]
+String bootstrapPath = "src/test/resources/bootstrap.conf"
+ConfigEncryptionTool tool = new ConfigEncryptionTool()
+
+// Act
+flags.each { String arg ->
+tool.parse([arg, bootstrapPath] as String[])
+logger.info("Parsed bootstrap.conf location: 
${tool.bootstrapConfPath}")
+
+// Assert
+assert 

[GitHub] nifi pull request #834: NIFI-1831 Implemented encrypted configuration capabi...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/834#discussion_r75909949
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/nifi.properties
 ---
@@ -131,8 +131,10 @@ nifi.web.jetty.threads=${nifi.web.jetty.threads}
 
 # security properties #
 nifi.sensitive.props.key=
+nifi.sensitive.props.key.protected=${nifi.sensitive.props.key.protected}
--- End diff --

Thanks for catching this Bryan. It was an issue where the tool had to merge 
two instances of `Properties` together and the logic wasn't checking the right 
key set. Added a test and fixed. Empty protection values no longer stop the 
field from being protected. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2636) UnpackContent has concurrent thread safety issue, causes flowfiles to fail

[Michael Moser (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433185#comment-15433185
 ] 

Michael Moser commented on NIFI-2636:
-

I wrote a unit test that fails only about half the time due to this bug.  I'm 
not sure how to create a unit test that fails all the time.  I'll go ahead with 
an initial patch using this unit test, but suggestions are welcome.

> UnpackContent has concurrent thread safety issue, causes flowfiles to fail
> --
>
> Key: NIFI-2636
> URL: https://issues.apache.org/jira/browse/NIFI-2636
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 0.7.0
>Reporter: Michael Moser
>Assignee: Michael Moser
>
> Shortly after merging NIFI-2611 I took a last look at the code and noticed 
> that each onTrigger() call, when the Packaging Format property is set to "use 
> mime.type attribute", that the class instance variable "private Unpacker 
> unpacker" can change.  When UnpackContent is set to > 1 concurrent task, this 
> isn't thread safe.  Thread A can set the unpacker to the TarUnpacker, but 
> before it gets a chance to unpack its tar file, Thread B changes the unpacker 
> to a FlowFileUnpackagerV3 which causes Thread A to fail its unpack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2636) UnpackContent has concurrent thread safety issue, causes flowfiles to fail

[Michael Moser (JIRA)]

Michael Moser created NIFI-2636:
---

 Summary: UnpackContent has concurrent thread safety issue, causes 
flowfiles to fail
 Key: NIFI-2636
 URL: https://issues.apache.org/jira/browse/NIFI-2636
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Affects Versions: 0.7.0
Reporter: Michael Moser
Assignee: Michael Moser


Shortly after merging NIFI-2611 I took a last look at the code and noticed that 
each onTrigger() call, when the Packaging Format property is set to "use 
mime.type attribute", that the class instance variable "private Unpacker 
unpacker" can change.  When UnpackContent is set to > 1 concurrent task, this 
isn't thread safe.  Thread A can set the unpacker to the TarUnpacker, but 
before it gets a chance to unpack its tar file, Thread B changes the unpacker 
to a FlowFileUnpackagerV3 which causes Thread A to fail its unpack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #914: Adding support for Binary Row Keys

[apsaltis]

Github user apsaltis commented on the issue:

https://github.com/apache/nifi/pull/914
  
Bryan,
Thanks for the review. This is certainly new development and thus I agree 
that this should be put into master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #914: Adding support for Binary Row Keys

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/914
  
Thanks for making the updates, looks good, testing it now... I noticed this 
PR is against 0.x, since most new development is going into 1.x, should we put 
this into master?

I think the changes should apply cleanly against master so we don't have to 
close and re-open the PR, but just wanted to confirm where it will get merged 
in. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi-cpp pull request #7: MINIFI-84 Performing explicit checking on v...

[apiri]

GitHub user apiri opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/7

MINIFI-84 Performing explicit checking on values of processor Properties 
YAML

MINIFI-84 Performing explicit checking on values of processor Properties 
YAML. An empty value is treated as a YAML node in lieu of being null and would 
lead to conversion errors.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apiri/nifi-minifi-cpp MINIFI-84

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/7.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #7


commit 9c18eddfc018913a8295e4da9c60a6243b2ec296
Author: Aldrin Piri 
Date:   2016-08-23T16:31:49Z

MINIFI-84 Performing explicit checking on values of processor Properties 
YAML. An empty string is treated as a YAML node and would lead to conversion 
errors.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2608) Thread-safety issue with ConsumeKafka

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-2608:
---
Assignee: Joseph Witt  (was: Oleg Zhurakousky)

> Thread-safety issue with ConsumeKafka
> -
>
> Key: NIFI-2608
> URL: https://issues.apache.org/jira/browse/NIFI-2608
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Oleg Zhurakousky
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> KafkaConsumer went fro thread-safe in 0.8 to not-thread-safe in 0.9 which was 
> overlooked while implementing new ConsumeKafka processor which relied on 0.9 
> Client API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2622) SelectHiveQL with Avro output doesn't handle complex types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433085#comment-15433085
 ] 

ASF GitHub Bot commented on NIFI-2622:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/922

NIFI-2622: Added support for complex types in SelectHiveQL



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-2622

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/922.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #922


commit b954297a7bf52ebd7f60429df172d274b86dd9b3
Author: Matt Burgess 
Date:   2016-08-23T15:54:09Z

NIFI-2622: Added support for complex types in SelectHiveQL




> SelectHiveQL with Avro output doesn't handle complex types
> --
>
> Key: NIFI-2622
> URL: https://issues.apache.org/jira/browse/NIFI-2622
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Hive supports table columns of complex type, such as Arrays, Structs, Maps, 
> etc.  However when using SelectHiveQL with Avro output, the result set is not 
> being converted correctly and an error occurs.
> This is a fundamental issue with the way SelectHiveQL attempts to create an 
> Avro schema without having seen any of the rows, since the sub-types of 
> complex items cannot be determined from the SQL type alone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #922: NIFI-2622: Added support for complex types in Select...

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/922

NIFI-2622: Added support for complex types in SelectHiveQL



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-2622

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/922.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #922


commit b954297a7bf52ebd7f60429df172d274b86dd9b3
Author: Matt Burgess 
Date:   2016-08-23T15:54:09Z

NIFI-2622: Added support for complex types in SelectHiveQL




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2622) SelectHiveQL with Avro output doesn't handle complex types

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433080#comment-15433080
 ] 

Matt Burgess commented on NIFI-2622:


The ARRAY handling was already moved as part of NIFI-2623, so the remainder of 
the effort is to add STRUCT and JAVA_OBJECT cases to the switch for String 
processing.

> SelectHiveQL with Avro output doesn't handle complex types
> --
>
> Key: NIFI-2622
> URL: https://issues.apache.org/jira/browse/NIFI-2622
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Hive supports table columns of complex type, such as Arrays, Structs, Maps, 
> etc.  However when using SelectHiveQL with Avro output, the result set is not 
> being converted correctly and an error occurs.
> This is a fundamental issue with the way SelectHiveQL attempts to create an 
> Avro schema without having seen any of the rows, since the sub-types of 
> complex items cannot be determined from the SQL type alone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2622) SelectHiveQL with Avro output doesn't handle complex types

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433075#comment-15433075
 ] 

Matt Burgess commented on NIFI-2622:


As it turns out, Hive returns complex types as strings but sets the JDBC type 
"appropriately" (STRUCT, ARRAY, JAVA_OBJECT for Map, etc.). So at a minimum to 
support complex types in SelectHiveQL, HiveJdbcCommon could add these types to 
the String processing case in createSchema(). Please note that since Hive is 
returning strings, the Avro fields will contain strings (not records, arrays, 
etc.). In order to support that, we'd need to add a property for the user to 
provide an Avro schema, then parse the string trying to deconstruct it into 
elements according to the schema. That should be a separate improvement Jira if 
desired.

> SelectHiveQL with Avro output doesn't handle complex types
> --
>
> Key: NIFI-2622
> URL: https://issues.apache.org/jira/browse/NIFI-2622
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Hive supports table columns of complex type, such as Arrays, Structs, Maps, 
> etc.  However when using SelectHiveQL with Avro output, the result set is not 
> being converted correctly and an error occurs.
> This is a fundamental issue with the way SelectHiveQL attempts to create an 
> Avro schema without having seen any of the rows, since the sub-types of 
> complex items cannot be determined from the SQL type alone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2622) SelectHiveQL with Avro output doesn't handle complex types

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-2622:
--

Assignee: Matt Burgess

> SelectHiveQL with Avro output doesn't handle complex types
> --
>
> Key: NIFI-2622
> URL: https://issues.apache.org/jira/browse/NIFI-2622
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Hive supports table columns of complex type, such as Arrays, Structs, Maps, 
> etc.  However when using SelectHiveQL with Avro output, the result set is not 
> being converted correctly and an error occurs.
> This is a fundamental issue with the way SelectHiveQL attempts to create an 
> Avro schema without having seen any of the rows, since the sub-types of 
> complex items cannot be determined from the SQL type alone.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-2623) SelectHiveQL with Avro output doesn't handle binary types

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende resolved NIFI-2623.
---
Resolution: Fixed

> SelectHiveQL with Avro output doesn't handle binary types
> -
>
> Key: NIFI-2623
> URL: https://issues.apache.org/jira/browse/NIFI-2623
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.0.0
>
>
> SelectHiveQL, when specifying Avro output, and having columns of binary 
> types, throws an error when run.
> The error is a SQLException with text "Method not supported". This is coming 
> from the Hive driver because HiveJdbcCommon is calling getBytes() on the 
> ResultSet, and that method has not been implemented in the driver (HIVE-7134).
> It may be possible to use getObject() and convert it into a byte array or 
> ByteBuffer (if it is not already) for storage into an Avro record.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2623) SelectHiveQL with Avro output doesn't handle binary types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433059#comment-15433059
 ] 

ASF GitHub Bot commented on NIFI-2623:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/920


> SelectHiveQL with Avro output doesn't handle binary types
> -
>
> Key: NIFI-2623
> URL: https://issues.apache.org/jira/browse/NIFI-2623
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> SelectHiveQL, when specifying Avro output, and having columns of binary 
> types, throws an error when run.
> The error is a SQLException with text "Method not supported". This is coming 
> from the Hive driver because HiveJdbcCommon is calling getBytes() on the 
> ResultSet, and that method has not been implemented in the driver (HIVE-7134).
> It may be possible to use getObject() and convert it into a byte array or 
> ByteBuffer (if it is not already) for storage into an Avro record.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #920: NIFI-2623: Fixed support for binary types in SelectH...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/920


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2623) SelectHiveQL with Avro output doesn't handle binary types

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433058#comment-15433058
 ] 

ASF subversion and git services commented on NIFI-2623:
---

Commit fbec3b9c13dce360a501790b8e215646101d4e77 in nifi's branch 
refs/heads/master from [~mattyb149]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=fbec3b9 ]

NIFI-2623: Fixed support for binary types in SelectHiveQL

This closes #920.

Signed-off-by: Bryan Bende 


> SelectHiveQL with Avro output doesn't handle binary types
> -
>
> Key: NIFI-2623
> URL: https://issues.apache.org/jira/browse/NIFI-2623
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> SelectHiveQL, when specifying Avro output, and having columns of binary 
> types, throws an error when run.
> The error is a SQLException with text "Method not supported". This is coming 
> from the Hive driver because HiveJdbcCommon is calling getBytes() on the 
> ResultSet, and that method has not been implemented in the driver (HIVE-7134).
> It may be possible to use getObject() and convert it into a byte array or 
> ByteBuffer (if it is not already) for storage into an Avro record.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2623) SelectHiveQL with Avro output doesn't handle binary types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433054#comment-15433054
 ] 

ASF GitHub Bot commented on NIFI-2623:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/920
  
Looks good, verified selecting from a table with a binary column no longer 
produces an error, will merge to master


> SelectHiveQL with Avro output doesn't handle binary types
> -
>
> Key: NIFI-2623
> URL: https://issues.apache.org/jira/browse/NIFI-2623
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> SelectHiveQL, when specifying Avro output, and having columns of binary 
> types, throws an error when run.
> The error is a SQLException with text "Method not supported". This is coming 
> from the Hive driver because HiveJdbcCommon is calling getBytes() on the 
> ResultSet, and that method has not been implemented in the driver (HIVE-7134).
> It may be possible to use getObject() and convert it into a byte array or 
> ByteBuffer (if it is not already) for storage into an Avro record.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #920: NIFI-2623: Fixed support for binary types in SelectHiveQL

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/920
  
Looks good, verified selecting from a table with a binary column no longer 
produces an error, will merge to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2634) Cannot import template from 0.x that contains Remote Process Group

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433022#comment-15433022
 ] 

ASF subversion and git services commented on NIFI-2634:
---

Commit 8cc670c8a690efea8eee4d090486d724d0069cbd in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8cc670c ]

NIFI-2634: Ensure that we check whether or not the Site-to-Site protocol is set 
when importing template, instead of assuming that it will be

This closes #921

Signed-off-by: jpercivall 


> Cannot import template from 0.x that contains Remote Process Group
> --
>
> Key: NIFI-2634
> URL: https://issues.apache.org/jira/browse/NIFI-2634
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Percivall
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> As a user, when attempting to add a template from 0.x that contains an RPG to 
> the graph I get a screen saying "unexpected error". After returning to the 
> canvas, all the processors were added to the graph but there are not process 
> groups, connections or remote process groups. In the user logs there is the 
> below exception.
> I believe this is due to the template not having the type of RPG (raw vs 
> http).
> 2016-08-23 10:03:41,157 ERROR [NiFi Web Server-138] 
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: 
> java.lang.NullPointerException: Name is null. Returning Internal Server Error 
> response.
> java.lang.NullPointerException: Name is null
>   at java.lang.Enum.valueOf(Enum.java:236) ~[na:1.8.0_74]
>   at 
> org.apache.nifi.remote.protocol.SiteToSiteTransportProtocol.valueOf(SiteToSiteTransportProtocol.java:19)
>  ~[nifi-site-to-site-client-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
>   at 
> org.apache.nifi.controller.FlowController.instantiateSnippet(FlowController.java:1744)
>  ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
>   at 
> org.apache.nifi.web.dao.impl.StandardTemplateDAO.instantiateTemplate(StandardTemplateDAO.java:134)
>  ~[classes/:na]
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade.createTemplateInstance(StandardNiFiServiceFacade.java:1727)
>  ~[classes/:1.0.0-SNAPSHOT]
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
>  ~[classes/:1.0.0-SNAPSHOT]
>   at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) 
> ~[spring-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.createLock(NiFiServiceFacadeLock.java:41)
>  ~[classes/:1.0.0-SNAPSHOT]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.8.0_74]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> ~[na:1.8.0_74]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.8.0_74]
>   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_74]
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
>  ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>   at 
> 

[GitHub] nifi pull request #921: NIFI-2634: Ensure that we check whether or not the S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/921


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---