date:20180307

[jira] [Updated] (NIFI-4946) nifi-spark-bundle : Adding support for pyfiles, file, jars options

2018-03-07 Thread Mageswaran (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mageswaran updated NIFI-4946:
-
Attachment: nifi-spark-options.png

> nifi-spark-bundle : Adding support for pyfiles, file, jars options
> --
>
> Key: NIFI-4946
> URL: https://issues.apache.org/jira/browse/NIFI-4946
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.6.0
> Environment: Ubuntu 16.04, IntelliJ
>Reporter: Mageswaran
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: nifi-spark-options.png, nifi-spark.png
>
>
> Adding support for submitting PySpark based Sparks jobs (which is normally 
> structured as modules) over Livy on existing "ExecuteSparkInteractive" 
> processor.
> This is done by reading file paths for pyfiles and file and an option from 
> user whether the processor should trigger a batch job or not.
> [https://livy.incubator.apache.org/docs/latest/rest-api.html]
>  
> More details will be posted in the Git link.
>  
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4946) nifi-spark-bundle : Adding support for pyfiles, file, jars options

2018-03-07 Thread Mageswaran (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mageswaran updated NIFI-4946:
-
Attachment: nifi-spark.png

> nifi-spark-bundle : Adding support for pyfiles, file, jars options
> --
>
> Key: NIFI-4946
> URL: https://issues.apache.org/jira/browse/NIFI-4946
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.6.0
> Environment: Ubuntu 16.04, IntelliJ
>Reporter: Mageswaran
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: nifi-spark.png
>
>
> Adding support for submitting PySpark based Sparks jobs (which is normally 
> structured as modules) over Livy on existing "ExecuteSparkInteractive" 
> processor.
> This is done by reading file paths for pyfiles and file and an option from 
> user whether the processor should trigger a batch job or not.
> [https://livy.incubator.apache.org/docs/latest/rest-api.html]
>  
> More details will be posted in the Git link.
>  
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4946) nifi-spark-bundle : Adding support for pyfiles, file, jars options

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390869#comment-16390869
 ] 

ASF GitHub Bot commented on NIFI-4946:
--

GitHub user Mageswaran1989 opened a pull request:

https://github.com/apache/nifi/pull/2521

NIFI-4946 nifi-spark-bundle : Adding support for pyfiles, file, jars options

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ x] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhiraa/nifi 
NIFI-4946_nifi-spark-bundle_Adding_support_for_pyfiles

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2521.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2521


commit a8301cdad0194328d420282f5e3260127776ed84
Author: Mageswaran 
Date:   2018-03-08T07:39:26Z

NIFI-4946 initial commit




> nifi-spark-bundle : Adding support for pyfiles, file, jars options
> --
>
> Key: NIFI-4946
> URL: https://issues.apache.org/jira/browse/NIFI-4946
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.6.0
> Environment: Ubuntu 16.04, IntelliJ
>Reporter: Mageswaran
>Priority: Major
> Fix For: 1.6.0
>
>
> Adding support for submitting PySpark based Sparks jobs (which is normally 
> structured as modules) over Livy on existing "ExecuteSparkInteractive" 
> processor.
> This is done by reading file paths for pyfiles and file and an option from 
> user whether the processor should trigger a batch job or not.
> [https://livy.incubator.apache.org/docs/latest/rest-api.html]
>  
> More details will be posted in the Git link.
>  
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2521: NIFI-4946 nifi-spark-bundle : Adding support for py...

2018-03-07 Thread Mageswaran1989

GitHub user Mageswaran1989 opened a pull request:

https://github.com/apache/nifi/pull/2521

NIFI-4946 nifi-spark-bundle : Adding support for pyfiles, file, jars options

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ x] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhiraa/nifi 
NIFI-4946_nifi-spark-bundle_Adding_support_for_pyfiles

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2521.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2521


commit a8301cdad0194328d420282f5e3260127776ed84
Author: Mageswaran 
Date:   2018-03-08T07:39:26Z

NIFI-4946 initial commit




---

[jira] [Created] (NIFI-4946) nifi-spark-bundle : Adding support for pyfiles, file, jars options

2018-03-07 Thread Mageswaran (JIRA)

Mageswaran created NIFI-4946:


 Summary: nifi-spark-bundle : Adding support for pyfiles, file, 
jars options
 Key: NIFI-4946
 URL: https://issues.apache.org/jira/browse/NIFI-4946
 Project: Apache NiFi
  Issue Type: New Feature
  Components: Extensions
Affects Versions: 1.6.0
 Environment: Ubuntu 16.04, IntelliJ
Reporter: Mageswaran
 Fix For: 1.6.0


Adding support for submitting PySpark based Sparks jobs (which is normally 
structured as modules) over Livy on existing "ExecuteSparkInteractive" 
processor.

This is done by reading file paths for pyfiles and file and an option from user 
whether the processor should trigger a batch job or not.

[https://livy.incubator.apache.org/docs/latest/rest-api.html]

 

More details will be posted in the Git link.

 

Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4833) NIFI-4833 Add ScanHBase processor

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390794#comment-16390794
 ] 

ASF GitHub Bot commented on NIFI-4833:
--

Github user bdesert commented on the issue:

https://github.com/apache/nifi/pull/2478
  
@bbende ,
Bryan, committed the changes. Tested on a cluster, works as expected.
When have a time, please review.


> NIFI-4833 Add ScanHBase processor
> -
>
> Key: NIFI-4833
> URL: https://issues.apache.org/jira/browse/NIFI-4833
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Ed Berezitsky
>Assignee: Ed Berezitsky
>Priority: Major
>
> Add ScanHBase (new) processor to retrieve records from HBase tables.
> Today there are GetHBase and FetchHBaseRow. GetHBase can pull entire table or 
> only new rows after processor started; it also must be scheduled and doesn't 
> support incoming . FetchHBaseRow can pull rows with known rowkeys only.
> This processor could provide functionality similar to what could be reached 
> by using hbase shell, defining following properties:
> -scan based on range of row key IDs 
> -scan based on range of time stamps
> -limit number of records pulled
> -use filters
> -reverse rows



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2478: NIFI-4833 Add scanHBase Processor

2018-03-07 Thread bdesert

Github user bdesert commented on the issue:

https://github.com/apache/nifi/pull/2478
  
@bbende ,
Bryan, committed the changes. Tested on a cluster, works as expected.
When have a time, please review.


---

[jira] [Created] (NIFI-4945) In Nifi 1.5, START_TLS in combination with LDAP will allow any password during auth

2018-03-07 Thread Matthew Elder (JIRA)

Matthew Elder created NIFI-4945:
---

 Summary: In Nifi 1.5, START_TLS in combination with LDAP will 
allow any password during auth
 Key: NIFI-4945
 URL: https://issues.apache.org/jira/browse/NIFI-4945
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.5.0
 Environment: alpine docker, openjdk 8, jumpcloud ldp service
Reporter: Matthew Elder


In Nifi 1.5, START_TLS in combination with LDAP will allow any password during 
auth

 

This has to do with the login portion of the ldap integration and not the 
groups aspect.

 

START_TLS accepts any password (huge security hole!)

LDAPS,SIMPLE will not allow any password

 

strange!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (MINIFICPP-419) Change the structure of the C2 response

2018-03-07 Thread marco polo (JIRA)

marco polo created MINIFICPP-419:


 Summary: Change the structure of the C2 response
 Key: MINIFICPP-419
 URL: https://issues.apache.org/jira/browse/MINIFICPP-419
 Project: NiFi MiNiFi C++
  Issue Type: Bug
Reporter: marco polo


Change the structure of the response to separate Flow information from agent 
information and limit what is being sent in a given response to only changed 
information. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4928) Upgrade to current version of BouncyCastle (1.55 -> 1.59)

2018-03-07 Thread Andy LoPresto (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-4928:

Status: Patch Available  (was: In Progress)

> Upgrade to current version of BouncyCastle (1.55 -> 1.59)
> -
>
> Key: NIFI-4928
> URL: https://issues.apache.org/jira/browse/NIFI-4928
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: dependencies, security
>
> BouncyCastle 1.59 is now available. Currently we are using 1.55. See [Section 
> 2.1.1 - 2.4.5|https://www.bouncycastle.org/releasenotes.html] for details of 
> new features and bug fixes. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4928) Upgrade to current version of BouncyCastle (1.55 -> 1.59)

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390557#comment-16390557
 ] 

ASF GitHub Bot commented on NIFI-4928:
--

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/2520

NIFI-4928 Updated BouncyCastle dependencies to version 1.59.

This PR updates the version of the library to 1.59. To verify the PGP code 
still works, I have provided [a 
template](https://gist.github.com/alopresto/9c8746375673c88f058a09afe16790d2). 



Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-4928

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2520


commit 887ec25ddd23e38f609b8a022476108bce2fd365
Author: Andy LoPresto 
Date:   2018-03-08T01:18:33Z

NIFI-4928 Updated BouncyCastle dependencies to version 1.59.




> Upgrade to current version of BouncyCastle (1.55 -> 1.59)
> -
>
> Key: NIFI-4928
> URL: https://issues.apache.org/jira/browse/NIFI-4928
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: dependencies, security
>
> BouncyCastle 1.59 is now available. Currently we are using 1.55. See [Section 
> 2.1.1 - 2.4.5|https://www.bouncycastle.org/releasenotes.html] for details of 
> new features and bug fixes. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2520: NIFI-4928 Updated BouncyCastle dependencies to vers...

2018-03-07 Thread alopresto

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/2520

NIFI-4928 Updated BouncyCastle dependencies to version 1.59.

This PR updates the version of the library to 1.59. To verify the PGP code 
still works, I have provided [a 
template](https://gist.github.com/alopresto/9c8746375673c88f058a09afe16790d2). 



Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-4928

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2520


commit 887ec25ddd23e38f609b8a022476108bce2fd365
Author: Andy LoPresto 
Date:   2018-03-08T01:18:33Z

NIFI-4928 Updated BouncyCastle dependencies to version 1.59.




---

[jira] [Commented] (NIFI-4809) Implement a SiteToSiteMetricsReportingTask

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390342#comment-16390342
 ] 

ASF GitHub Bot commented on NIFI-4809:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2430
  
@mattyb149 - pushed another commit with unit tests and doc


> Implement a SiteToSiteMetricsReportingTask
> --
>
> Key: NIFI-4809
> URL: https://issues.apache.org/jira/browse/NIFI-4809
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> At the moment there is an AmbariReportingTask to send the NiFi-related 
> metrics of the host to the Ambari Metrics Service. In a multi-cluster 
> configuration, or when working with MiNiFi (Java) agents, it might not be 
> possible for all the NiFi instances (NiFi and/or MiNiFi) to access the AMS 
> REST API.
> To solve this problem, a solution would be to implement a 
> SiteToSiteMetricsReportingTask to send the data via S2S to the "main" NiFi 
> instance/cluster that will be able to publish the metrics into AMS (using 
> InvokeHTTP). This way, it is possible to have the metrics of all the 
> instances exposed in one AMS instance.
> I propose to send the data formatted as we are doing right now in the Ambari 
> reporting task. If needed, it can be easily converted into another schema 
> using the record processors once received via S2S.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2430: NIFI-4809 - Implement a SiteToSiteMetricsReportingTask

2018-03-07 Thread pvillard31

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2430
  
@mattyb149 - pushed another commit with unit tests and doc


---

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390188#comment-16390188
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172973981
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
 ---
@@ -385,7 +427,18 @@ protected PutFlowFile createPut(ProcessContext 
context, Record record, RecordSch
 
 
 if (fieldValueBytes != null) {
-columns.add(new PutColumn(fam, 
clientService.toBytes(name), fieldValueBytes, timestamp));
+PutColumn column;
+
+String visString = (visField != null && visSettings != 
null && visSettings.containsKey(name))
+? (String)visSettings.get(name) : 
defaultVisibility;
+
+if (visString != null && !visString.equals("") ) {
--- End diff --

Same here


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390189#comment-16390189
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172973696
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
 ---
@@ -194,6 +220,12 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final String fieldEncodingStrategy = 
context.getProperty(FIELD_ENCODING_STRATEGY).getValue();
 final String complexFieldStrategy = 
context.getProperty(COMPLEX_FIELD_STRATEGY).getValue();
 final String rowEncodingStrategy = 
context.getProperty(ROW_ID_ENCODING_STRATEGY).getValue();
+final String recordPathText = 
context.getProperty(VISIBILITY_RECORD_PATH).getValue();
+
+RecordPath recordPath = null;
+if (recordPathCache != null && recordPathText != null && 
!recordPathText.equals("")) {
--- End diff --

java
if (recordPathCache != null && !StringUtils.isEmpty(recordPathText)) {



> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390190#comment-16390190
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172975369
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase-client-service-api/src/main/java/org/apache/nifi/hbase/HBaseClientService.java
 ---
@@ -150,6 +194,40 @@
  */
 void scan(String tableName, byte[] startRow, byte[] endRow, 
Collection columns, ResultHandler handler) throws IOException;
 
+/**
+ * Scans the given table for the given rowId and passes the result to 
the handler.
+ *
+ * @param tableName the name of an HBase table to scan
+ * @param startRow the row identifier to start scanning at
+ * @param endRow the row identifier to end scanning at
+ * @param columns optional columns to return, if not specified all 
columns are returned
+ * @param visibilityLabels optional list of visibility labels that the 
user should be able to see when communicating with HBase
+ * @param handler a handler to process rows of the result
+ * @throws IOException thrown when there are communication errors with 
HBase
+ */
+void scan(String tableName, byte[] startRow, byte[] endRow, 
Collection columns, List visibilityLabels, ResultHandler 
handler) throws IOException;
+
+/**
+ * Get all of the labels in HBase.
+ *
+ * @return a List of all of the labels.
+ */
+List getLabels();
+
+/**
+ * Get all of the labels a given user can see.
+ * @param user the user to lookup.
+ * @return a List of all of the labels a user is allowed to see.
+ */
+List getLabelsForUser(String user);
+
+/**
+ * Get all of the labels the current user (NiFi process user or 
Kerberos keytab principle) can see.
+ *
+ * @return a List of all of the labels the current can see.
--- End diff --

current *user*?


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390192#comment-16390192
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172977733
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -103,6 +103,17 @@
 .defaultValue("UTF-8")
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .build();
+static final PropertyDescriptor VISIBLITY_LABEL = new 
PropertyDescriptor.Builder()
+.name("delete-visibility-label")
+.displayName("Visibility Label")
+.description("If visibility labels are enabled, a row cannot 
be deleted without supplying its visibility label(s) in the delete " +
+"request. Note: this visibility label will be applied 
to all cells within the row that is specified. If some cells have " +
+"different visibility labels, they will not be 
deleted. When that happens, the failure to delete will be considered a success 
" +
+"because HBase does not report it as a failure.")
+.required(false)
--- End diff --

Since this is not required and always valid, a user could force the 
property to be set with an empty string. Is that OK? It means visibility would 
be equal to "". Does that make sense in some cases? Wouldn't be better to add a 
non-empty string validator?


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390193#comment-16390193
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172975481
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase-client-service-api/src/main/java/org/apache/nifi/hbase/HBaseClientService.java
 ---
@@ -126,6 +138,25 @@
 
 void delete(String tableName, List rowIds) throws IOException;
 
+/**
+ * Deletes a list of cells from HBase. This is intended to be used 
with granual delete operations.
--- End diff --

granular?


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390194#comment-16390194
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172976209
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -349,18 +399,54 @@ public boolean checkAndPut(final String tableName, 
final byte[] rowId, final byt
 
 @Override
 public void delete(final String tableName, final byte[] rowId) throws 
IOException {
+delete(tableName, rowId, null);
+}
+
+@Override
+public void delete(String tableName, byte[] rowId, String 
visibilityLabel) throws IOException {
 try (final Table table = 
connection.getTable(TableName.valueOf(tableName))) {
 Delete delete = new Delete(rowId);
+if (visibilityLabel != null && 
!visibilityLabel.trim().equals("")) {
+delete.setCellVisibility(new 
CellVisibility(visibilityLabel));
+}
 table.delete(delete);
 }
 }
 
 @Override
 public void delete(String tableName, List rowIds) throws 
IOException {
+delete(tableName, rowIds);
+}
+
+@Override
+public void deleteCells(String tableName, List deletes) 
throws IOException {
+List deleteRequests = new ArrayList<>();
+for (int index = 0; index < deletes.size(); index++) {
+DeleteRequest req = deletes.get(index);
+Delete delete = new Delete(req.getRowId())
+.addColumn(req.getColumnFamily(), 
req.getColumnQualifier());
+if (req.getVisibilityLabel() != null && 
!req.getVisibilityLabel().trim().equals("")) {
--- End diff --

StringUtils.isBlank(str)


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390191#comment-16390191
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172973274
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseJSON.java
 ---
@@ -255,7 +253,16 @@ public void process(final InputStream in) throws 
IOException {
 final byte[] colFamBytes = 
columnFamily.getBytes(StandardCharsets.UTF_8);
 final byte[] colQualBytes = 
fieldName.getBytes(StandardCharsets.UTF_8);
 final byte[] colValBytes = fieldValueHolder.get();
-columns.add(new PutColumn(colFamBytes, colQualBytes, 
colValBytes, timestamp));
+
+final String visibilityStringToUse = 
pickVisibilityString(visibilityString, columnFamily, fieldName, flowFile);
+PutColumn column;
+if (visibilityStringToUse != null && 
!visibilityStringToUse.equals("")) {
--- End diff --

Same here


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390187#comment-16390187
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172973432
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
 ---
@@ -141,6 +149,16 @@
 .allowableValues(NULL_FIELD_EMPTY, NULL_FIELD_SKIP)
 .build();
 
+protected static final PropertyDescriptor VISIBILITY_RECORD_PATH = new 
PropertyDescriptor.Builder()
+.name("put-hb-rec-visibility-record-path")
+.displayName("Visibility String Record Path Root")
+.description("A record path that points to part of the record 
which contains a path to a mapping of visibility strings to record paths")
+.required(false)
+.addValidator(Validator.VALID) //new 
RecordPathPropertyNameValidator())
--- End diff --

Comment to remove?


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390195#comment-16390195
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172977970
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/GetHBase.java
 ---
@@ -16,28 +16,6 @@
  */
 package org.apache.nifi.hbase;
 
-import java.io.File;
--- End diff --

Just a detail: could you revert the import order changes?


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390196#comment-16390196
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172976240
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -349,18 +399,54 @@ public boolean checkAndPut(final String tableName, 
final byte[] rowId, final byt
 
 @Override
 public void delete(final String tableName, final byte[] rowId) throws 
IOException {
+delete(tableName, rowId, null);
+}
+
+@Override
+public void delete(String tableName, byte[] rowId, String 
visibilityLabel) throws IOException {
 try (final Table table = 
connection.getTable(TableName.valueOf(tableName))) {
 Delete delete = new Delete(rowId);
+if (visibilityLabel != null && 
!visibilityLabel.trim().equals("")) {
+delete.setCellVisibility(new 
CellVisibility(visibilityLabel));
+}
 table.delete(delete);
 }
 }
 
 @Override
 public void delete(String tableName, List rowIds) throws 
IOException {
+delete(tableName, rowIds);
+}
+
+@Override
+public void deleteCells(String tableName, List deletes) 
throws IOException {
+List deleteRequests = new ArrayList<>();
+for (int index = 0; index < deletes.size(); index++) {
+DeleteRequest req = deletes.get(index);
+Delete delete = new Delete(req.getRowId())
+.addColumn(req.getColumnFamily(), 
req.getColumnQualifier());
+if (req.getVisibilityLabel() != null && 
!req.getVisibilityLabel().trim().equals("")) {
+delete.setCellVisibility(new 
CellVisibility(req.getVisibilityLabel()));
+}
+deleteRequests.add(delete);
+}
+batchDelete(tableName, deleteRequests);
+}
+
+@Override
+public void delete(String tableName, List rowIds, String 
visibilityLabel) throws IOException {
 List deletes = new ArrayList<>();
 for (int index = 0; index < rowIds.size(); index++) {
-deletes.add(new Delete(rowIds.get(index)));
+Delete delete = new Delete(rowIds.get(index));
+if (visibilityLabel != null && 
!visibilityLabel.trim().equals("")) {
--- End diff --

same here


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390186#comment-16390186
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172972637
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseCell.java
 ---
@@ -96,16 +97,18 @@ protected PutFlowFile createPut(final ProcessSession 
session, final ProcessConte
 
 
 final byte[] buffer = new byte[(int) flowFile.getSize()];
-session.read(flowFile, new InputStreamCallback() {
-@Override
-public void process(final InputStream in) throws IOException {
-StreamUtils.fillBuffer(in, buffer);
-}
-});
+session.read(flowFile, in -> StreamUtils.fillBuffer(in, buffer));
 
+PutColumn column = null;
+if (visibilityStringToUse != null && 
!visibilityStringToUse.equals("")) {
--- End diff --

Cosmetic comment - feel free to ignore.
java
PutColumn column = StringUtils.isEmpty(visibilityStringToUse)
? new PutColumn(columnFamily.getBytes(StandardCharsets.UTF_8),
columnQualifier.getBytes(StandardCharsets.UTF_8), 
buffer, timestamp)
: new PutColumn(columnFamily.getBytes(StandardCharsets.UTF_8),
columnQualifier.getBytes(StandardCharsets.UTF_8), 
buffer, timestamp, visibilityStringToUse);



> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172973432
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
 ---
@@ -141,6 +149,16 @@
 .allowableValues(NULL_FIELD_EMPTY, NULL_FIELD_SKIP)
 .build();
 
+protected static final PropertyDescriptor VISIBILITY_RECORD_PATH = new 
PropertyDescriptor.Builder()
+.name("put-hb-rec-visibility-record-path")
+.displayName("Visibility String Record Path Root")
+.description("A record path that points to part of the record 
which contains a path to a mapping of visibility strings to record paths")
+.required(false)
+.addValidator(Validator.VALID) //new 
RecordPathPropertyNameValidator())
--- End diff --

Comment to remove?


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172973274
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseJSON.java
 ---
@@ -255,7 +253,16 @@ public void process(final InputStream in) throws 
IOException {
 final byte[] colFamBytes = 
columnFamily.getBytes(StandardCharsets.UTF_8);
 final byte[] colQualBytes = 
fieldName.getBytes(StandardCharsets.UTF_8);
 final byte[] colValBytes = fieldValueHolder.get();
-columns.add(new PutColumn(colFamBytes, colQualBytes, 
colValBytes, timestamp));
+
+final String visibilityStringToUse = 
pickVisibilityString(visibilityString, columnFamily, fieldName, flowFile);
+PutColumn column;
+if (visibilityStringToUse != null && 
!visibilityStringToUse.equals("")) {
--- End diff --

Same here


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172976209
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -349,18 +399,54 @@ public boolean checkAndPut(final String tableName, 
final byte[] rowId, final byt
 
 @Override
 public void delete(final String tableName, final byte[] rowId) throws 
IOException {
+delete(tableName, rowId, null);
+}
+
+@Override
+public void delete(String tableName, byte[] rowId, String 
visibilityLabel) throws IOException {
 try (final Table table = 
connection.getTable(TableName.valueOf(tableName))) {
 Delete delete = new Delete(rowId);
+if (visibilityLabel != null && 
!visibilityLabel.trim().equals("")) {
+delete.setCellVisibility(new 
CellVisibility(visibilityLabel));
+}
 table.delete(delete);
 }
 }
 
 @Override
 public void delete(String tableName, List rowIds) throws 
IOException {
+delete(tableName, rowIds);
+}
+
+@Override
+public void deleteCells(String tableName, List deletes) 
throws IOException {
+List deleteRequests = new ArrayList<>();
+for (int index = 0; index < deletes.size(); index++) {
+DeleteRequest req = deletes.get(index);
+Delete delete = new Delete(req.getRowId())
+.addColumn(req.getColumnFamily(), 
req.getColumnQualifier());
+if (req.getVisibilityLabel() != null && 
!req.getVisibilityLabel().trim().equals("")) {
--- End diff --

StringUtils.isBlank(str)


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172975481
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase-client-service-api/src/main/java/org/apache/nifi/hbase/HBaseClientService.java
 ---
@@ -126,6 +138,25 @@
 
 void delete(String tableName, List rowIds) throws IOException;
 
+/**
+ * Deletes a list of cells from HBase. This is intended to be used 
with granual delete operations.
--- End diff --

granular?


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172977970
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/GetHBase.java
 ---
@@ -16,28 +16,6 @@
  */
 package org.apache.nifi.hbase;
 
-import java.io.File;
--- End diff --

Just a detail: could you revert the import order changes?


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172976240
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -349,18 +399,54 @@ public boolean checkAndPut(final String tableName, 
final byte[] rowId, final byt
 
 @Override
 public void delete(final String tableName, final byte[] rowId) throws 
IOException {
+delete(tableName, rowId, null);
+}
+
+@Override
+public void delete(String tableName, byte[] rowId, String 
visibilityLabel) throws IOException {
 try (final Table table = 
connection.getTable(TableName.valueOf(tableName))) {
 Delete delete = new Delete(rowId);
+if (visibilityLabel != null && 
!visibilityLabel.trim().equals("")) {
+delete.setCellVisibility(new 
CellVisibility(visibilityLabel));
+}
 table.delete(delete);
 }
 }
 
 @Override
 public void delete(String tableName, List rowIds) throws 
IOException {
+delete(tableName, rowIds);
+}
+
+@Override
+public void deleteCells(String tableName, List deletes) 
throws IOException {
+List deleteRequests = new ArrayList<>();
+for (int index = 0; index < deletes.size(); index++) {
+DeleteRequest req = deletes.get(index);
+Delete delete = new Delete(req.getRowId())
+.addColumn(req.getColumnFamily(), 
req.getColumnQualifier());
+if (req.getVisibilityLabel() != null && 
!req.getVisibilityLabel().trim().equals("")) {
+delete.setCellVisibility(new 
CellVisibility(req.getVisibilityLabel()));
+}
+deleteRequests.add(delete);
+}
+batchDelete(tableName, deleteRequests);
+}
+
+@Override
+public void delete(String tableName, List rowIds, String 
visibilityLabel) throws IOException {
 List deletes = new ArrayList<>();
 for (int index = 0; index < rowIds.size(); index++) {
-deletes.add(new Delete(rowIds.get(index)));
+Delete delete = new Delete(rowIds.get(index));
+if (visibilityLabel != null && 
!visibilityLabel.trim().equals("")) {
--- End diff --

same here


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172975369
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase-client-service-api/src/main/java/org/apache/nifi/hbase/HBaseClientService.java
 ---
@@ -150,6 +194,40 @@
  */
 void scan(String tableName, byte[] startRow, byte[] endRow, 
Collection columns, ResultHandler handler) throws IOException;
 
+/**
+ * Scans the given table for the given rowId and passes the result to 
the handler.
+ *
+ * @param tableName the name of an HBase table to scan
+ * @param startRow the row identifier to start scanning at
+ * @param endRow the row identifier to end scanning at
+ * @param columns optional columns to return, if not specified all 
columns are returned
+ * @param visibilityLabels optional list of visibility labels that the 
user should be able to see when communicating with HBase
+ * @param handler a handler to process rows of the result
+ * @throws IOException thrown when there are communication errors with 
HBase
+ */
+void scan(String tableName, byte[] startRow, byte[] endRow, 
Collection columns, List visibilityLabels, ResultHandler 
handler) throws IOException;
+
+/**
+ * Get all of the labels in HBase.
+ *
+ * @return a List of all of the labels.
+ */
+List getLabels();
+
+/**
+ * Get all of the labels a given user can see.
+ * @param user the user to lookup.
+ * @return a List of all of the labels a user is allowed to see.
+ */
+List getLabelsForUser(String user);
+
+/**
+ * Get all of the labels the current user (NiFi process user or 
Kerberos keytab principle) can see.
+ *
+ * @return a List of all of the labels the current can see.
--- End diff --

current *user*?


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172973981
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
 ---
@@ -385,7 +427,18 @@ protected PutFlowFile createPut(ProcessContext 
context, Record record, RecordSch
 
 
 if (fieldValueBytes != null) {
-columns.add(new PutColumn(fam, 
clientService.toBytes(name), fieldValueBytes, timestamp));
+PutColumn column;
+
+String visString = (visField != null && visSettings != 
null && visSettings.containsKey(name))
+? (String)visSettings.get(name) : 
defaultVisibility;
+
+if (visString != null && !visString.equals("") ) {
--- End diff --

Same here


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172973696
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseRecord.java
 ---
@@ -194,6 +220,12 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 final String fieldEncodingStrategy = 
context.getProperty(FIELD_ENCODING_STRATEGY).getValue();
 final String complexFieldStrategy = 
context.getProperty(COMPLEX_FIELD_STRATEGY).getValue();
 final String rowEncodingStrategy = 
context.getProperty(ROW_ID_ENCODING_STRATEGY).getValue();
+final String recordPathText = 
context.getProperty(VISIBILITY_RECORD_PATH).getValue();
+
+RecordPath recordPath = null;
+if (recordPathCache != null && recordPathText != null && 
!recordPathText.equals("")) {
--- End diff --

java
if (recordPathCache != null && !StringUtils.isEmpty(recordPathText)) {



---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172972637
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/PutHBaseCell.java
 ---
@@ -96,16 +97,18 @@ protected PutFlowFile createPut(final ProcessSession 
session, final ProcessConte
 
 
 final byte[] buffer = new byte[(int) flowFile.getSize()];
-session.read(flowFile, new InputStreamCallback() {
-@Override
-public void process(final InputStream in) throws IOException {
-StreamUtils.fillBuffer(in, buffer);
-}
-});
+session.read(flowFile, in -> StreamUtils.fillBuffer(in, buffer));
 
+PutColumn column = null;
+if (visibilityStringToUse != null && 
!visibilityStringToUse.equals("")) {
--- End diff --

Cosmetic comment - feel free to ignore.
java
PutColumn column = StringUtils.isEmpty(visibilityStringToUse)
? new PutColumn(columnFamily.getBytes(StandardCharsets.UTF_8),
columnQualifier.getBytes(StandardCharsets.UTF_8), 
buffer, timestamp)
: new PutColumn(columnFamily.getBytes(StandardCharsets.UTF_8),
columnQualifier.getBytes(StandardCharsets.UTF_8), 
buffer, timestamp, visibilityStringToUse);



---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread pvillard31

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2518#discussion_r172977733
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -103,6 +103,17 @@
 .defaultValue("UTF-8")
 .addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
 .build();
+static final PropertyDescriptor VISIBLITY_LABEL = new 
PropertyDescriptor.Builder()
+.name("delete-visibility-label")
+.displayName("Visibility Label")
+.description("If visibility labels are enabled, a row cannot 
be deleted without supplying its visibility label(s) in the delete " +
+"request. Note: this visibility label will be applied 
to all cells within the row that is specified. If some cells have " +
+"different visibility labels, they will not be 
deleted. When that happens, the failure to delete will be considered a success 
" +
+"because HBase does not report it as a failure.")
+.required(false)
--- End diff --

Since this is not required and always valid, a user could force the 
property to be set with an empty string. Is that OK? It means visibility would 
be equal to "". Does that make sense in some cases? Wouldn't be better to add a 
non-empty string validator?


---

[jira] [Commented] (NIFI-4885) More granular restricted component categories

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390168#comment-16390168
 ] 

ASF GitHub Bot commented on NIFI-4885:
--

Github user andrewmlim commented on the issue:

https://github.com/apache/nifi/pull/2515
  
Also, maybe a separate issue, but wanted to note that the MoveHDFS 
processor is tagged as "restricted" in the Add Processor window, but it is not 
treated like other restricted components.


> More granular restricted component categories
> -
>
> Key: NIFI-4885
> URL: https://issues.apache.org/jira/browse/NIFI-4885
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Update the Restricted annotation to support more granular categories. 
> Available categories will map to new access policies. Example categories and 
> their corresponding access policies may be
>  * read-filesystem (/restricted-components/read-filesystem)
>  * write-filesystem (/restricted-components/write-filesystem)
>  * code-execution (/restricted-components/code-execution)
>  * keytab-access (/restricted-components/keytab-access)
> The hierarchical nature of the access policies will support backward 
> compatibility with existing installations where the policy of 
> /restricted-components was used to enforce all subcategories. Any users with 
> /restricted-components permissions will be granted access to all 
> subcategories. In order to leverage the new granular categories, an 
> administrator will need to use NiFi to update their access policies (remove a 
> user from /restricted-components and place them into the desired subcategory)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2515: NIFI-4885: Granular component restrictions

2018-03-07 Thread andrewmlim

Github user andrewmlim commented on the issue:

https://github.com/apache/nifi/pull/2515
  
Also, maybe a separate issue, but wanted to note that the MoveHDFS 
processor is tagged as "restricted" in the Add Processor window, but it is not 
treated like other restricted components.


---

[jira] [Commented] (NIFI-4885) More granular restricted component categories

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390166#comment-16390166
 ] 

ASF GitHub Bot commented on NIFI-4885:
--

Github user andrewmlim commented on the issue:

https://github.com/apache/nifi/pull/2515
  
@mcgilman 
I tested your latest changes and they look good.  A couple minor things to 
address:

- In the message "Only listing restriction specific users. Users with 
permission "regardless of restriction" not shown but are also allowed."  should 
add an "s" to the second instance of "restriction".  So the correct message 
reads:

Only listing restriction specific users. Users with permission "regardless 
of restrictions" not shown but are also allowed.

- The following text is used in a tooltip and doc:

"Allows users to create/modify restricted components assuming otherwise 
sufficient permissions” 

I thought this might be more clear:

“Allows users to create/modify restricted components assuming other 
permissions are sufficient”

-In the Admin and User Guides, change the content to:

Allows users to create/modify restricted components assuming other 
permissions are sufficient. The restricted components may indicate which 
specific permissions are required. Permissions can be granted for specific 
restrictions or be granted regardless of restrictions. If permission is granted 
regardless of restrictions, the user can create/modify all restricted 
components.


> More granular restricted component categories
> -
>
> Key: NIFI-4885
> URL: https://issues.apache.org/jira/browse/NIFI-4885
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Update the Restricted annotation to support more granular categories. 
> Available categories will map to new access policies. Example categories and 
> their corresponding access policies may be
>  * read-filesystem (/restricted-components/read-filesystem)
>  * write-filesystem (/restricted-components/write-filesystem)
>  * code-execution (/restricted-components/code-execution)
>  * keytab-access (/restricted-components/keytab-access)
> The hierarchical nature of the access policies will support backward 
> compatibility with existing installations where the policy of 
> /restricted-components was used to enforce all subcategories. Any users with 
> /restricted-components permissions will be granted access to all 
> subcategories. In order to leverage the new granular categories, an 
> administrator will need to use NiFi to update their access policies (remove a 
> user from /restricted-components and place them into the desired subcategory)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2515: NIFI-4885: Granular component restrictions

2018-03-07 Thread andrewmlim

Github user andrewmlim commented on the issue:

https://github.com/apache/nifi/pull/2515
  
@mcgilman 
I tested your latest changes and they look good.  A couple minor things to 
address:

- In the message "Only listing restriction specific users. Users with 
permission "regardless of restriction" not shown but are also allowed."  should 
add an "s" to the second instance of "restriction".  So the correct message 
reads:

Only listing restriction specific users. Users with permission "regardless 
of restrictions" not shown but are also allowed.

- The following text is used in a tooltip and doc:

"Allows users to create/modify restricted components assuming otherwise 
sufficient permissionsâ 

I thought this might be more clear:

âAllows users to create/modify restricted components assuming other 
permissions are sufficientâ

-In the Admin and User Guides, change the content to:

Allows users to create/modify restricted components assuming other 
permissions are sufficient. The restricted components may indicate which 
specific permissions are required. Permissions can be granted for specific 
restrictions or be granted regardless of restrictions. If permission is granted 
regardless of restrictions, the user can create/modify all restricted 
components.


---

[jira] [Updated] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently

2018-03-07 Thread Matt Burgess (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4944:
---
Description: 
When data coming into PutHiveStreaming is compressed with Snappy, then multiple 
instances of PutHiveStreaming in a flow can cause a failure, the log often 
shows the following:

{{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
 failed to process due to org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}

This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
NAR) where two classloaders can try to define the native loader class, thus the 
second one would fail, giving the error above.

The proposed solution is to guarantee that Snappy is loaded before this 
situation is encountered (i.e. before the InstanceClassLoaders are created).

  was:
{{When data coming into PutHiveStreaming is compressed with Snappy, then 
multiple instances of PutHiveStreaming in a flow can cause a failure, the log 
often shows the following: }}


{{ 
org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
 failed to process due to org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {} }}


 This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
NAR) where two classloaders can try to define the native loader class, thus the 
second one would fail, giving the error above.

The proposed solution is to guarantee that Snappy is loaded before this 
situation is encountered (i.e. before the InstanceClassLoaders are created).


> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When data coming into PutHiveStreaming is compressed with Snappy, then 
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log 
> often shows the following:
> {{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
>  failed to process due to org.xerial.snappy.SnappyError: 
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
> NAR) where two classloaders can try to define the native loader class, thus 
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this 
> situation is encountered (i.e. before the InstanceClassLoaders are created).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently

2018-03-07 Thread Matt Burgess (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4944:
---
Description: 
{{When data coming into PutHiveStreaming is compressed with Snappy, then 
multiple instances of PutHiveStreaming in a flow can cause a failure, the log 
often shows the following: }}


{{ 
org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
 failed to process due to org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {} }}


 This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
NAR) where two classloaders can try to define the native loader class, thus the 
second one would fail, giving the error above.

The proposed solution is to guarantee that Snappy is loaded before this 
situation is encountered (i.e. before the InstanceClassLoaders are created).

  was:
When data coming into PutHiveStreaming is compressed with Snappy, then multiple 
instances of PutHiveStreaming in a flow can cause a failure, the log often 
shows the following: 
org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
 failed to process due to org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {} 
This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
NAR) where two classloaders can try to define the native loader class, thus the 
second one would fail, giving the error above.

The proposed solution is to guarantee that Snappy is loaded before this 
situation is encountered (i.e. before the InstanceClassLoaders are created).


> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> {{When data coming into PutHiveStreaming is compressed with Snappy, then 
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log 
> often shows the following: }}
> {{ 
> org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
>  failed to process due to org.xerial.snappy.SnappyError: 
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {} }}
>  This is due to a race condition in Snappy 1.0.5 (the version used by the 
> Hive NAR) where two classloaders can try to define the native loader class, 
> thus the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this 
> situation is encountered (i.e. before the InstanceClassLoaders are created).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390144#comment-16390144
 ] 

ASF GitHub Bot commented on NIFI-4944:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2519

NIFI-4944: Guard against race condition in Snappy for PutHiveStreaming

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-4944

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2519.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2519


commit 20de68f5e33039c2bb4d3ecd79eddd5e882aa8db
Author: Matthew Burgess 
Date:   2018-03-07T20:19:39Z

NIFI-4944: Guard against race condition in Snappy for PutHiveStreaming




> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When data coming into PutHiveStreaming is compressed with Snappy, then 
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log 
> often shows the following: 
> org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
>  failed to process due to org.xerial.snappy.SnappyError: 
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {} 
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
> NAR) where two classloaders can try to define the native loader class, thus 
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this 
> situation is encountered (i.e. before the InstanceClassLoaders are created).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (MINIFICPP-413) Implement contains() Expression Language function

2018-03-07 Thread Andrew Christianson (JIRA)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Christianson resolved MINIFICPP-413.
---
Resolution: Fixed

> Implement contains() Expression Language function
> -
>
> Key: MINIFICPP-413
> URL: https://issues.apache.org/jira/browse/MINIFICPP-413
> Project: NiFi MiNiFi C++
>  Issue Type: Sub-task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> Implement contains() Expression Language function



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently

2018-03-07 Thread Matt Burgess (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4944:
---
Status: Patch Available  (was: In Progress)

> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When data coming into PutHiveStreaming is compressed with Snappy, then 
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log 
> often shows the following: 
> org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
>  failed to process due to org.xerial.snappy.SnappyError: 
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {} 
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
> NAR) where two classloaders can try to define the native loader class, thus 
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this 
> situation is encountered (i.e. before the InstanceClassLoaders are created).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (MINIFICPP-416) Support crop properties in TFConvertImageToTensor

2018-03-07 Thread Andrew Christianson (JIRA)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Christianson resolved MINIFICPP-416.
---
Resolution: Fixed

> Support crop properties in TFConvertImageToTensor
> -
>
> Key: MINIFICPP-416
> URL: https://issues.apache.org/jira/browse/MINIFICPP-416
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> In order to allow for modifying input aspect ratio (e.g. 4:3) to match output 
> ratio (e.g. 1:1), support cropping of the input image via the following new 
> optional properties:
>  * Crop Offset X
>  * Crop Offset Y
>  * Crop Size X
>  * Crop Size Y



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2519: NIFI-4944: Guard against race condition in Snappy f...

2018-03-07 Thread mattyb149

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2519

NIFI-4944: Guard against race condition in Snappy for PutHiveStreaming

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-4944

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2519.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2519


commit 20de68f5e33039c2bb4d3ecd79eddd5e882aa8db
Author: Matthew Burgess 
Date:   2018-03-07T20:19:39Z

NIFI-4944: Guard against race condition in Snappy for PutHiveStreaming




---

[jira] [Resolved] (MINIFICPP-415) Implement matches() Expression Language function

2018-03-07 Thread Andrew Christianson (JIRA)


 [ 
https://issues.apache.org/jira/browse/MINIFICPP-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Christianson resolved MINIFICPP-415.
---
Resolution: Fixed

> Implement matches() Expression Language function
> 
>
> Key: MINIFICPP-415
> URL: https://issues.apache.org/jira/browse/MINIFICPP-415
> Project: NiFi MiNiFi C++
>  Issue Type: Sub-task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> Implement matches() Expression Language function



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently

2018-03-07 Thread Matt Burgess (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-4944:
--

Assignee: Matt Burgess

> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When data coming into PutHiveStreaming is compressed with Snappy, then 
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log 
> often shows the following: 
> org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
>  failed to process due to org.xerial.snappy.SnappyError: 
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {} 
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
> NAR) where two classloaders can try to define the native loader class, thus 
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this 
> situation is encountered (i.e. before the InstanceClassLoaders are created).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently

2018-03-07 Thread Matt Burgess (JIRA)

Matt Burgess created NIFI-4944:
--

 Summary: PutHiveStreaming multiple instances with Snappy fail 
intermittently
 Key: NIFI-4944
 URL: https://issues.apache.org/jira/browse/NIFI-4944
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Reporter: Matt Burgess


When data coming into PutHiveStreaming is compressed with Snappy, then multiple 
instances of PutHiveStreaming in a flow can cause a failure, the log often 
shows the following: 
org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
 failed to process due to org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {} 
This is due to a race condition in Snappy 1.0.5 (the version used by the Hive 
NAR) where two classloaders can try to define the native loader class, thus the 
second one would fail, giving the error above.

The proposed solution is to guarantee that Snappy is loaded before this 
situation is encountered (i.e. before the InstanceClassLoaders are created).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-4943) Batch Duration capability from ExecuteProcess added to ExecuteStreamCommand

2018-03-07 Thread Oleksandr Lobunets (JIRA)

Oleksandr Lobunets created NIFI-4943:


 Summary: Batch Duration capability from ExecuteProcess added to 
ExecuteStreamCommand
 Key: NIFI-4943
 URL: https://issues.apache.org/jira/browse/NIFI-4943
 Project: Apache NiFi
  Issue Type: Improvement
Affects Versions: 1.5.0
Reporter: Oleksandr Lobunets


It would be great to let the ExecuteStreamCommand processor to send FlowFiles 
per chunk of stdout using a given separator (common case: for each line from 
stdout).

I have a case  of running the 3rd party CLI (linux) with the following 
behaviour:
- Should be executed upon a FlowFile with attributes/content containing 
parameters to CLI
- Accepts params via flags or environment variables 
- Writes output to stdout as a stream of JSON objects
- The output might be huge (millions and millions of objects), which means 
caching stdout is not an option - each line/object should be sent as a separate 
FlowFile
- The errors/log is written to stderr (might be very chatty)

Using ExecuteProcessor is not an option (cannot be trigger by incoming 
FlowFile), but the way it treats stdout is what is desired.
Using ExecuteStreamCommand is not an option as it buffers the output until the 
binary exists with a status code 0.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-4877) ExecuteStreamCommand tasks stuck

2018-03-07 Thread Boris Tyukin (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boris Tyukin resolved NIFI-4877.

Resolution: Duplicate

duplicate of NIFI-528

> ExecuteStreamCommand tasks stuck
> 
>
> Key: NIFI-4877
> URL: https://issues.apache.org/jira/browse/NIFI-4877
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
> Environment: Oracle Linux 6.8, NiFi 1.5.0
>Reporter: Boris Tyukin
>Priority: Critical
>
> there is no way to stop/kill processes, executed by ExecuteStreamCommand. 
> Steps to reproduce:
>  # In ExecuteStreamCommand processer, use sleep as a command, and 100 as 
> argument - this will make it run for a while. Set ignore STDIN to true.
>  # Run the flow and stop ExecuteStreamCommand processor - it will stop but 
> not really. the task will be stuck forever and the only workaround I found is 
> to restart nifi service which is not cool at all.
>  # to make it worse, processor would not accept new flowfiles and you cannot 
> Run it. Only restart fixes the issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3551) ExecuteStreamCommand processor does not know that .sh script has become a zombie or killed

2018-03-07 Thread Matt Burgess (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389937#comment-16389937
 ] 

Matt Burgess commented on NIFI-3551:


Is this a duplicate of NIFI-528?

> ExecuteStreamCommand processor does not know that .sh script has become a 
> zombie or killed
> --
>
> Key: NIFI-3551
> URL: https://issues.apache.org/jira/browse/NIFI-3551
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0
> Environment: Linux RHEL 6.5
>Reporter: Olav Jordens
>Priority: Minor
> Attachments: ExecuteStreamCommandZombie.png
>
>
> I have a workflow which periodically runs a .sh script using 
> ExecuteStreamCommand processor which usually takes a few minutes. This 
> morning, I noticed that the resulting flow had not produced anything for 
> almost a day. I checked on my Linux box and saw that the .sh script had been 
> running for almost a day - Something must have been wrong here, so killed the 
> process from the command line. However, the Nifi processor will not stop. It 
> continues to show active processes. Restarting the nifi process resolves 
> this, but this is a radical solution on a production system.
> In the image I have shown the zombie extracted from the running workflow. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4877) ExecuteStreamCommand tasks stuck

2018-03-07 Thread Matt Burgess (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389936#comment-16389936
 ] 

Matt Burgess commented on NIFI-4877:


[~boristyukin] Is this a duplicate of NIFI-528?

> ExecuteStreamCommand tasks stuck
> 
>
> Key: NIFI-4877
> URL: https://issues.apache.org/jira/browse/NIFI-4877
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
> Environment: Oracle Linux 6.8, NiFi 1.5.0
>Reporter: Boris Tyukin
>Priority: Critical
>
> there is no way to stop/kill processes, executed by ExecuteStreamCommand. 
> Steps to reproduce:
>  # In ExecuteStreamCommand processer, use sleep as a command, and 100 as 
> argument - this will make it run for a while. Set ignore STDIN to true.
>  # Run the flow and stop ExecuteStreamCommand processor - it will stop but 
> not really. the task will be stuck forever and the only workaround I found is 
> to restart nifi service which is not cool at all.
>  # to make it worse, processor would not accept new flowfiles and you cannot 
> Run it. Only restart fixes the issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2479: NIFI-4859 NIFI-4835 Swagger spec fixes

2018-03-07 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2479
  
@kevdoran Sorry, did not notice the mention. Will review...


---

[jira] [Commented] (NIFI-4859) Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389922#comment-16389922
 ] 

ASF GitHub Bot commented on NIFI-4859:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2479
  
@kevdoran Sorry, did not notice the mention. Will review...


> Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state
> 
>
> Key: NIFI-4859
> URL: https://issues.apache.org/jira/browse/NIFI-4859
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Chaffelson
>Assignee: Kevin Doran
>Priority: Minor
>  Labels: swagger
> Fix For: 1.6.0
>
>
> It is possible to get a Versioned Process Group into a SYNC_FAILURE state, 
> but this is not an allowable state in the code generated from the swagger.json
> This prevents versioned objects from being manipulated via the API in some 
> use cases.
> {noformat}
> @state.setter
> def state(self, state):
> """
> Sets the state of this VersionControlInformationDTO.
> The current state of the Process Group, as it relates to the Versioned Flow
> :param state: The state of this VersionControlInformationDTO.
> :type: str
> """
> allowed_values = ["LOCALLY_MODIFIED_DESCENDANT", "LOCALLY_MODIFIED", "STALE", 
> "LOCALLY_MODIFIED_AND_STALE", "UP_TO_DATE"]
> if state not in allowed_values:
> raise ValueError(
> "Invalid value for `state` ({0}), must be one of {1}"
> > .format(state, allowed_values)
> )
> E ValueError: Invalid value for `state` (SYNC_FAILURE), must be one of 
> ['LOCALLY_MODIFIED_DESCENDANT', 'LOCALLY_MODIFIED', 'STALE', 
> 'LOCALLY_MODIFIED_AND_STALE', 'UP_TO_DATE']
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-333) Implement string search functions in Expression Language

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/MINIFICPP-333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389921#comment-16389921
 ] 

ASF GitHub Bot commented on MINIFICPP-333:
--

GitHub user achristianson opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/273

MINIFICPP-333 Added string searching functions to Expression Language

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [x] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the LICENSE file?
- [x] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-333

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #273


commit 2476947a4a10ed10e8ddc659db2c8580f42b5dab
Author: Andrew I. Christianson 
Date:   2018-03-06T17:37:08Z

MINIFICPP-333 Added string searching functions to Expression Language




> Implement string search functions in Expression Language
> 
>
> Key: MINIFICPP-333
> URL: https://issues.apache.org/jira/browse/MINIFICPP-333
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> Implement these:
> startsWith
> endsWith
> contains
> in
> find
> matches
> indexOf
> lastIndexOf



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #273: MINIFICPP-333 Added string searching func...

2018-03-07 Thread achristianson

GitHub user achristianson opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/273

MINIFICPP-333 Added string searching functions to Expression Language

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [x] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the LICENSE file?
- [x] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-333

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #273


commit 2476947a4a10ed10e8ddc659db2c8580f42b5dab
Author: Andrew I. Christianson 
Date:   2018-03-06T17:37:08Z

MINIFICPP-333 Added string searching functions to Expression Language




---

[GitHub] nifi issue #2512: NIFI-4936 pushed down version declarations to lowest appro...

2018-03-07 Thread jtstorck

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/2512
  
+1 regarding the HDFS component POM changes.  After building with this PR, 
I tested Put/List/DeleteHDFS processors with TDE paths successfully.


---

[jira] [Commented] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389838#comment-16389838
 ] 

ASF GitHub Bot commented on NIFI-4936:
--

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/2512
  
+1 regarding the HDFS component POM changes.  After building with this PR, 
I tested Put/List/DeleteHDFS processors with TDE paths successfully.


> NiFi parent pom dependency management forcing versions to align defeating 
> classloader isolation
> ---
>
> Key: NIFI-4936
> URL: https://issues.apache.org/jira/browse/NIFI-4936
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Extensions, Tools and Build
>Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0
>Reporter: Joseph Witt
>Assignee: Joseph Witt
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: NIFI-4936.patch, build-fixing, 
> old-vs-new-dependencies.txt
>
>
> the top level pom in nifi has a massive dependency management section.  this 
> was used initially to help enforce consistent usage of dependencies across 
> nifi but this also can defeat the purpose of the classloader isolation 
> offered by nars.  We need to push down dependency version declarations to the 
> nar levels where appropriate.
> there have been reported issues of defects happening due to us using much 
> newer versions (or sometimes older) of dependencies due to this dependency 
> management model.  By pushing declarations down to the proper scope each 
> nar/etc.. can use the specific versions of components it needs and we'll stop 
> introducing issues by forcing a different version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389815#comment-16389815
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2518
  
This also adds a new processor: DeleteHBaseCells.


> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2518: NIFI-4637 Added support for visibility labels to the HBase...

2018-03-07 Thread MikeThomsen

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2518
  
This also adds a new processor: DeleteHBaseCells.


---

[jira] [Commented] (NIFI-4637) Add support for HBase visibility labels to HBase processors and controller services

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389812#comment-16389812
 ] 

ASF GitHub Bot commented on NIFI-4637:
--

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2518

NIFI-4637 Added support for visibility labels to the HBase processors.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4637

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2518


commit 78945e97191954915c9467d7d4db8d64d4d5f567
Author: Mike Thomsen 
Date:   2018-02-22T21:15:00Z

NIFI-4637 Added support for visibility labels to the HBase processors.




> Add support for HBase visibility labels to HBase processors and controller 
> services
> ---
>
> Key: NIFI-4637
> URL: https://issues.apache.org/jira/browse/NIFI-4637
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> HBase supports visibility labels, but you can't use them from NiFi because 
> there is no way to set them. The existing processors and services should be 
> upgraded to handle this capability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4885) More granular restricted component categories

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389811#comment-16389811
 ] 

ASF GitHub Bot commented on NIFI-4885:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2515
  
Thanks for the feedback @andrewmlim! I've pushed another commit addressing 
the issue.


> More granular restricted component categories
> -
>
> Key: NIFI-4885
> URL: https://issues.apache.org/jira/browse/NIFI-4885
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Update the Restricted annotation to support more granular categories. 
> Available categories will map to new access policies. Example categories and 
> their corresponding access policies may be
>  * read-filesystem (/restricted-components/read-filesystem)
>  * write-filesystem (/restricted-components/write-filesystem)
>  * code-execution (/restricted-components/code-execution)
>  * keytab-access (/restricted-components/keytab-access)
> The hierarchical nature of the access policies will support backward 
> compatibility with existing installations where the policy of 
> /restricted-components was used to enforce all subcategories. Any users with 
> /restricted-components permissions will be granted access to all 
> subcategories. In order to leverage the new granular categories, an 
> administrator will need to use NiFi to update their access policies (remove a 
> user from /restricted-components and place them into the desired subcategory)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2515: NIFI-4885: Granular component restrictions

2018-03-07 Thread mcgilman

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2515
  
Thanks for the feedback @andrewmlim! I've pushed another commit addressing 
the issue.


---

[GitHub] nifi pull request #2518: NIFI-4637 Added support for visibility labels to th...

2018-03-07 Thread MikeThomsen

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2518

NIFI-4637 Added support for visibility labels to the HBase processors.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4637

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2518


commit 78945e97191954915c9467d7d4db8d64d4d5f567
Author: Mike Thomsen 
Date:   2018-02-22T21:15:00Z

NIFI-4637 Added support for visibility labels to the HBase processors.




---

[jira] [Commented] (NIFI-4364) InfluxDB ControllerService, PutInfluxDB Processor, and ReportingTask

2018-03-07 Thread Richard St. John (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389521#comment-16389521
 ] 

Richard St. John commented on NIFI-4364:


We do have a github repo containing these, and other, NiFi extensions.  Here is 
the link: https://github.com/Asymmetrik/nifi-nar-bundles.  Let me know if you 
have any questions.

> InfluxDB ControllerService, PutInfluxDB Processor, and ReportingTask
> 
>
> Key: NIFI-4364
> URL: https://issues.apache.org/jira/browse/NIFI-4364
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Richard St. John
>Priority: Major
>
> My team has been working storing metric data in influxDB.  As such, we 
> created an InfluxDB service, PutInfluxDB processor and a ReportingTask that 
> sends data to influxDB.  Is this something that others could benefit from?  
> If so, we could contribute these additions to the NiFi codebase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4838) Make GetMongo support multiple commits and give some progress indication

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389501#comment-16389501
 ] 

ASF GitHub Bot commented on NIFI-4838:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2448
  
@mattyb149 I put it back to batch size and got rid of the configuration 
option for progressive commits. The new behavior is the default and for 
individual flowfiles it checks to see if the batch size was set and uses that 
for when to do a commit, otherwise it uses the default of 100 flowfiles. I feel 
like that's a good compromise for when batch size is not set.


> Make GetMongo support multiple commits and give some progress indication
> 
>
> Key: NIFI-4838
> URL: https://issues.apache.org/jira/browse/NIFI-4838
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> It shouldn't wait until the end to do a commit() call because the effect is 
> that GetMongo looks like it has hung to a user who is pulling a very large 
> data set.
> It should also have an option for running a count query to get the current 
> approximate count of documents that would match the query and append an 
> attribute that indicates where a flowfile stands in the total result count. 
> Ex:
> query.progress.point.start = 2500
> query.progress.point.end = 5000
> query.count.estimate = 17,568,231



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2448: NIFI-4838 Added configurable progressive commits to GetMon...

2018-03-07 Thread MikeThomsen

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2448
  
@mattyb149 I put it back to batch size and got rid of the configuration 
option for progressive commits. The new behavior is the default and for 
individual flowfiles it checks to see if the batch size was set and uses that 
for when to do a commit, otherwise it uses the default of 100 flowfiles. I feel 
like that's a good compromise for when batch size is not set.


---

[jira] [Commented] (NIFI-4364) InfluxDB ControllerService, PutInfluxDB Processor, and ReportingTask

2018-03-07 Thread John Smith (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389440#comment-16389440
 ] 

John Smith commented on NIFI-4364:
--

I'd also be very interested in this if you have a repo somewhere?

Thanks

> InfluxDB ControllerService, PutInfluxDB Processor, and ReportingTask
> 
>
> Key: NIFI-4364
> URL: https://issues.apache.org/jira/browse/NIFI-4364
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Richard St. John
>Priority: Major
>
> My team has been working storing metric data in influxDB.  As such, we 
> created an InfluxDB service, PutInfluxDB processor and a ReportingTask that 
> sends data to influxDB.  Is this something that others could benefit from?  
> If so, we could contribute these additions to the NiFi codebase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4938) Offline buffering is not supported by paho client version 1.0.2 used in mqtt processors.

2018-03-07 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/NIFI-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389296#comment-16389296
 ] 

ASF GitHub Bot commented on NIFI-4938:
--

Github user Himanshu-it commented on the issue:

https://github.com/apache/nifi/pull/2514
  
continuous integration with travis falis with concurrent modification error 
in 
"TestFileSystemRepository.testMinimalArchiveCleanupIntervalHonoredAndLogged:151",
 but locally I have build with
mvn clean install and mvn -Pcontrib-check clean install, both are working 
fine. Any pointers on this. 


> Offline buffering is not supported by paho client version 1.0.2 used in mqtt 
> processors.
> 
>
> Key: NIFI-4938
> URL: https://issues.apache.org/jira/browse/NIFI-4938
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Himanshu Mishra
>Priority: Major
>
> Off line buffering is not supported by paho client version 1.0.2 used in mqtt 
> processors which causes problem of "message loss" with QoS 1 and QoS 2 for 
> mqtt messages when the client connectivity is lost.
> The version 1.2.0 of paho client supports this off line buffering. Thus 
> addressing this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2514: NIFI-4938 Upgraded org.eclipse.paho.client.mqttv3 dependen...

2018-03-07 Thread Himanshu-it

Github user Himanshu-it commented on the issue:

https://github.com/apache/nifi/pull/2514
  
continuous integration with travis falis with concurrent modification error 
in 
"TestFileSystemRepository.testMinimalArchiveCleanupIntervalHonoredAndLogged:151",
 but locally I have build with
mvn clean install and mvn -Pcontrib-check clean install, both are working 
fine. Any pointers on this. 


---

[jira] [Resolved] (NIFI-4937) Cannot GET /nifi-api/flow/bulletin-board behind reverse proxy

2018-03-07 Thread Damian Czaja (JIRA)


 [ 
https://issues.apache.org/jira/browse/NIFI-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Czaja resolved NIFI-4937.

   Resolution: Fixed
Fix Version/s: 1.6.0

OK, I checked with the current HEAD and looks it fixed and will be released in 
1.6.0

> Cannot GET /nifi-api/flow/bulletin-board behind reverse proxy
> -
>
> Key: NIFI-4937
> URL: https://issues.apache.org/jira/browse/NIFI-4937
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.5.0
> Environment: NiFi on Docker + TinyProxy
>Reporter: Damian Czaja
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: nifi_bulletin_board_error.png
>
>
> Hello,
> I have an problem, when running NiFi on Docker behind a reverse proxy 
> (TinyProxy).
> TinyProxy configuration adds three headers. NiFi is working on the same host 
> on port 8080.
> {code:java}
> tinyproxy.conf
> ...
> AddHeader "X-ProxyHost" "public.domain.com"
> AddHeader "X-ProxyContextPath" "/path/to/nifi"
> AddHeader "X-ProxyPort" "443"
> ReversePath "/" "http://localhost:8080/"{code}
> I can access NiFi through [https://public.domain.com/path/to/nifi/nifi/] and 
> NiFi works fine, but I'm getting sometimes a popup, which states, that is 
> cannot GET /nifi-api/flow/bulletin-board. For ex. I happens, when I try to 
> view configuration of a Controller Service.
> I recognized, that this request is made directly to NiFi and not through the 
> reverse proxy. It looks, that it ignores the X-Proxy headers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

73 matches

Mail list logo