[jira] [Commented] (NIFI-4866) HBase_1_1_2_ClientService performance dropped due to accessing HBase admin upon every FlowFile transfer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359267#comment-16359267
 ] 

ASF GitHub Bot commented on NIFI-4866:
--

Github user zenfenan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2464#discussion_r167390069
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -549,11 +551,7 @@ public String toTransitUri(String tableName, String 
rowKey) {
 logger.warn("Connection has not been established, could not 
create a transit URI. Returning null.");
 return null;
 }
-try {
-final String masterAddress = 
connection.getAdmin().getClusterStatus().getMaster().getHostAndPort();
-return "hbase://" + masterAddress + "/" + tableName + (rowKey 
!= null && !rowKey.isEmpty() ? "/" + rowKey : "");
-} catch (IOException e) {
-throw new RuntimeException("Failed to get HBase Admin 
interface, due to " + e, e);
-}
+final String transitUriMasterAddress = 
StringUtils.isEmpty(masterAddress) ? "unknown" : masterAddress;
--- End diff --

As far as I'm aware, such a scenario has no possibilities of happening. I 
just wanted to start the thinking in that direction so that people who know in 
and out of HBase can pitch in their thoughts. 

It looks good to me. Thanks :)


> HBase_1_1_2_ClientService performance dropped due to accessing HBase admin 
> upon every FlowFile transfer
> ---
>
> Key: NIFI-4866
> URL: https://issues.apache.org/jira/browse/NIFI-4866
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Critical
>
> NIFI-4543 added 
> "connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” call 
> from HBase_1_1_2_ClientService.toTransitUri method in order to get HBase 
> master address, which is called upon every FlowFile transfer and essentially 
> doubles the traffic through the HBase connector.
> The performance of our PutHBaseJSON processor dropped to 1/3 after deploying 
> NiFi 1.5.0.
> Those lines of code should be moved from toTransitUri to createConnection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2464: NIFI-4866: Fixed HBase performance issue

[zenfenan]

Github user zenfenan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2464#discussion_r167390069
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -549,11 +551,7 @@ public String toTransitUri(String tableName, String 
rowKey) {
 logger.warn("Connection has not been established, could not 
create a transit URI. Returning null.");
 return null;
 }
-try {
-final String masterAddress = 
connection.getAdmin().getClusterStatus().getMaster().getHostAndPort();
-return "hbase://" + masterAddress + "/" + tableName + (rowKey 
!= null && !rowKey.isEmpty() ? "/" + rowKey : "");
-} catch (IOException e) {
-throw new RuntimeException("Failed to get HBase Admin 
interface, due to " + e, e);
-}
+final String transitUriMasterAddress = 
StringUtils.isEmpty(masterAddress) ? "unknown" : masterAddress;
--- End diff --

As far as I'm aware, such a scenario has no possibilities of happening. I 
just wanted to start the thinking in that direction so that people who know in 
and out of HBase can pitch in their thoughts. 

It looks good to me. Thanks :)


---

[jira] [Commented] (NIFI-4866) HBase_1_1_2_ClientService performance dropped due to accessing HBase admin upon every FlowFile transfer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359264#comment-16359264
 ] 

ASF GitHub Bot commented on NIFI-4866:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2464#discussion_r167389731
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -549,11 +551,7 @@ public String toTransitUri(String tableName, String 
rowKey) {
 logger.warn("Connection has not been established, could not 
create a transit URI. Returning null.");
 return null;
 }
-try {
-final String masterAddress = 
connection.getAdmin().getClusterStatus().getMaster().getHostAndPort();
-return "hbase://" + masterAddress + "/" + tableName + (rowKey 
!= null && !rowKey.isEmpty() ? "/" + rowKey : "");
-} catch (IOException e) {
-throw new RuntimeException("Failed to get HBase Admin 
interface, due to " + e, e);
-}
+final String transitUriMasterAddress = 
StringUtils.isEmpty(masterAddress) ? "unknown" : masterAddress;
--- End diff --

@zenfenan Thanks for your comment. Yes, it would be a problem if this 
actually happens. But it only happens when the ControllerService was not able 
to get the master address when it was enabled. I think HBase processors will 
not be able to access HBase, too.

It depends on the programs how critical this can be, for those use the 
returned transit URI later. For example, NiFi and Atlas integration case, 
ReportLineageToAtlas reporting task will not be able to determine the right 
cluster name in Atlas.

Please let me know any situation that the ControllerServer can connect to a 
HBase, but can not get its master address. I write the code that way just to be 
more protective, but not sure if that actually happens.


> HBase_1_1_2_ClientService performance dropped due to accessing HBase admin 
> upon every FlowFile transfer
> ---
>
> Key: NIFI-4866
> URL: https://issues.apache.org/jira/browse/NIFI-4866
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Critical
>
> NIFI-4543 added 
> "connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” call 
> from HBase_1_1_2_ClientService.toTransitUri method in order to get HBase 
> master address, which is called upon every FlowFile transfer and essentially 
> doubles the traffic through the HBase connector.
> The performance of our PutHBaseJSON processor dropped to 1/3 after deploying 
> NiFi 1.5.0.
> Those lines of code should be moved from toTransitUri to createConnection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2464: NIFI-4866: Fixed HBase performance issue

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2464#discussion_r167389731
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -549,11 +551,7 @@ public String toTransitUri(String tableName, String 
rowKey) {
 logger.warn("Connection has not been established, could not 
create a transit URI. Returning null.");
 return null;
 }
-try {
-final String masterAddress = 
connection.getAdmin().getClusterStatus().getMaster().getHostAndPort();
-return "hbase://" + masterAddress + "/" + tableName + (rowKey 
!= null && !rowKey.isEmpty() ? "/" + rowKey : "");
-} catch (IOException e) {
-throw new RuntimeException("Failed to get HBase Admin 
interface, due to " + e, e);
-}
+final String transitUriMasterAddress = 
StringUtils.isEmpty(masterAddress) ? "unknown" : masterAddress;
--- End diff --

@zenfenan Thanks for your comment. Yes, it would be a problem if this 
actually happens. But it only happens when the ControllerService was not able 
to get the master address when it was enabled. I think HBase processors will 
not be able to access HBase, too.

It depends on the programs how critical this can be, for those use the 
returned transit URI later. For example, NiFi and Atlas integration case, 
ReportLineageToAtlas reporting task will not be able to determine the right 
cluster name in Atlas.

Please let me know any situation that the ControllerServer can connect to a 
HBase, but can not get its master address. I write the code that way just to be 
more protective, but not sure if that actually happens.


---

[jira] [Commented] (NIFI-4866) HBase_1_1_2_ClientService performance dropped due to accessing HBase admin upon every FlowFile transfer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359240#comment-16359240
 ] 

ASF GitHub Bot commented on NIFI-4866:
--

Github user zenfenan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2464#discussion_r167388849
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -549,11 +551,7 @@ public String toTransitUri(String tableName, String 
rowKey) {
 logger.warn("Connection has not been established, could not 
create a transit URI. Returning null.");
 return null;
 }
-try {
-final String masterAddress = 
connection.getAdmin().getClusterStatus().getMaster().getHostAndPort();
-return "hbase://" + masterAddress + "/" + tableName + (rowKey 
!= null && !rowKey.isEmpty() ? "/" + rowKey : "");
-} catch (IOException e) {
-throw new RuntimeException("Failed to get HBase Admin 
interface, due to " + e, e);
-}
+final String transitUriMasterAddress = 
StringUtils.isEmpty(masterAddress) ? "unknown" : masterAddress;
--- End diff --

We are sending back "unknown" if the master address is empty. Won't that be 
a problem, if it happens to be empty? I mean when 
`session.getProvenanceReporter().fetch(handlerFlowFile, transitUri);` gets 
executed on the HBase processors?


> HBase_1_1_2_ClientService performance dropped due to accessing HBase admin 
> upon every FlowFile transfer
> ---
>
> Key: NIFI-4866
> URL: https://issues.apache.org/jira/browse/NIFI-4866
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Critical
>
> NIFI-4543 added 
> "connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” call 
> from HBase_1_1_2_ClientService.toTransitUri method in order to get HBase 
> master address, which is called upon every FlowFile transfer and essentially 
> doubles the traffic through the HBase connector.
> The performance of our PutHBaseJSON processor dropped to 1/3 after deploying 
> NiFi 1.5.0.
> Those lines of code should be moved from toTransitUri to createConnection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2464: NIFI-4866: Fixed HBase performance issue

[zenfenan]

Github user zenfenan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2464#discussion_r167388849
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/src/main/java/org/apache/nifi/hbase/HBase_1_1_2_ClientService.java
 ---
@@ -549,11 +551,7 @@ public String toTransitUri(String tableName, String 
rowKey) {
 logger.warn("Connection has not been established, could not 
create a transit URI. Returning null.");
 return null;
 }
-try {
-final String masterAddress = 
connection.getAdmin().getClusterStatus().getMaster().getHostAndPort();
-return "hbase://" + masterAddress + "/" + tableName + (rowKey 
!= null && !rowKey.isEmpty() ? "/" + rowKey : "");
-} catch (IOException e) {
-throw new RuntimeException("Failed to get HBase Admin 
interface, due to " + e, e);
-}
+final String transitUriMasterAddress = 
StringUtils.isEmpty(masterAddress) ? "unknown" : masterAddress;
--- End diff --

We are sending back "unknown" if the master address is empty. Won't that be 
a problem, if it happens to be empty? I mean when 
`session.getProvenanceReporter().fetch(handlerFlowFile, transitUri);` gets 
executed on the HBase processors?


---

[jira] [Updated] (NIFI-4866) HBase_1_1_2_ClientService performance dropped due to accessing HBase admin upon every FlowFile transfer

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-4866:

Status: Patch Available  (was: In Progress)

> HBase_1_1_2_ClientService performance dropped due to accessing HBase admin 
> upon every FlowFile transfer
> ---
>
> Key: NIFI-4866
> URL: https://issues.apache.org/jira/browse/NIFI-4866
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Critical
>
> NIFI-4543 added 
> "connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” call 
> from HBase_1_1_2_ClientService.toTransitUri method in order to get HBase 
> master address, which is called upon every FlowFile transfer and essentially 
> doubles the traffic through the HBase connector.
> The performance of our PutHBaseJSON processor dropped to 1/3 after deploying 
> NiFi 1.5.0.
> Those lines of code should be moved from toTransitUri to createConnection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4866) HBase_1_1_2_ClientService performance dropped due to accessing HBase admin upon every FlowFile transfer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359211#comment-16359211
 ] 

ASF GitHub Bot commented on NIFI-4866:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/2464
  
Confirmed that transit URI is created correctly:
- 1.5 
hbase://0.hdp.aws.mine:16000/nifi-4866/a20f31cb-f503-4e0c-9806-73a2ed7fd43e
- 1.6 
hbase://0.hdp.aws.mine:16000/nifi-4866/93376ca2-ec3c-421f-8e4e-93cf28677b31

Confirmed performance issue has been addressed:

- Used a simple GenerateFlowFile and PutHBaseCell flow, template is 
available 
[here](https://gist.github.com/ijokarumawak/85db60ca71f1825f543c18c62bf7c3fd)
- Putting only 1 cell with 4bytes data, batched size is 25
- 1.5 saturated about at the rate of 14k FlowFIles/5min
- 1.6 saturated about at the rate of 500k FlowFiles/5min

NiFi 1.5.0

![image](https://user-images.githubusercontent.com/1107620/36057769-55896c0c-0e56-11e8-98ff-f3464f6dd736.png)

NiFi 1.6.0-SNAPSHOT with this fix

![image](https://user-images.githubusercontent.com/1107620/36057771-5d46e32a-0e56-11e8-9f95-542e835361dc.png)



> HBase_1_1_2_ClientService performance dropped due to accessing HBase admin 
> upon every FlowFile transfer
> ---
>
> Key: NIFI-4866
> URL: https://issues.apache.org/jira/browse/NIFI-4866
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Critical
>
> NIFI-4543 added 
> "connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” call 
> from HBase_1_1_2_ClientService.toTransitUri method in order to get HBase 
> master address, which is called upon every FlowFile transfer and essentially 
> doubles the traffic through the HBase connector.
> The performance of our PutHBaseJSON processor dropped to 1/3 after deploying 
> NiFi 1.5.0.
> Those lines of code should be moved from toTransitUri to createConnection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2464: NIFI-4866: Fixed HBase performance issue

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/2464
  
Confirmed that transit URI is created correctly:
- 1.5 
hbase://0.hdp.aws.mine:16000/nifi-4866/a20f31cb-f503-4e0c-9806-73a2ed7fd43e
- 1.6 
hbase://0.hdp.aws.mine:16000/nifi-4866/93376ca2-ec3c-421f-8e4e-93cf28677b31

Confirmed performance issue has been addressed:

- Used a simple GenerateFlowFile and PutHBaseCell flow, template is 
available 
[here](https://gist.github.com/ijokarumawak/85db60ca71f1825f543c18c62bf7c3fd)
- Putting only 1 cell with 4bytes data, batched size is 25
- 1.5 saturated about at the rate of 14k FlowFIles/5min
- 1.6 saturated about at the rate of 500k FlowFiles/5min

NiFi 1.5.0

![image](https://user-images.githubusercontent.com/1107620/36057769-55896c0c-0e56-11e8-98ff-f3464f6dd736.png)

NiFi 1.6.0-SNAPSHOT with this fix

![image](https://user-images.githubusercontent.com/1107620/36057771-5d46e32a-0e56-11e8-9f95-542e835361dc.png)



---

[jira] [Commented] (NIFI-4866) HBase_1_1_2_ClientService performance dropped due to accessing HBase admin upon every FlowFile transfer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359207#comment-16359207
 ] 

ASF GitHub Bot commented on NIFI-4866:
--

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/2464

NIFI-4866: Fixed HBase performance issue

HBase_1_1_2_ClientService performance dropped due to accessing HBase
admin upon every FlowFile transfer. Fixed that by getting master address
only once when the service is enabled and a connection is established.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-4866

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2464.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2464


commit 0c8bf33d6d7165e089a43adbecb6539d6f64a5f5
Author: Koji Kawamura 
Date:   2018-02-10T02:27:17Z

NIFI-4866: Fixed HBase performance issue

HBase_1_1_2_ClientService performance dropped due to accessing HBase
admin upon every FlowFile transfer. Fixed that by getting master address
only once when the service is enabled and a connection is established.




> HBase_1_1_2_ClientService performance dropped due to accessing HBase admin 
> upon every FlowFile transfer
> ---
>
> Key: NIFI-4866
> URL: https://issues.apache.org/jira/browse/NIFI-4866
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Critical
>
> NIFI-4543 added 
> "connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” call 
> from HBase_1_1_2_ClientService.toTransitUri method in order to get HBase 
> master address, which is called upon every FlowFile transfer and essentially 
> doubles the traffic through the HBase connector.
> The performance of our PutHBaseJSON processor dropped to 1/3 after deploying 
> NiFi 1.5.0.
> Those lines of code should be moved from toTransitUri to createConnection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2464: NIFI-4866: Fixed HBase performance issue

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/2464

NIFI-4866: Fixed HBase performance issue

HBase_1_1_2_ClientService performance dropped due to accessing HBase
admin upon every FlowFile transfer. Fixed that by getting master address
only once when the service is enabled and a connection is established.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-4866

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2464.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2464


commit 0c8bf33d6d7165e089a43adbecb6539d6f64a5f5
Author: Koji Kawamura 
Date:   2018-02-10T02:27:17Z

NIFI-4866: Fixed HBase performance issue

HBase_1_1_2_ClientService performance dropped due to accessing HBase
admin upon every FlowFile transfer. Fixed that by getting master address
only once when the service is enabled and a connection is established.




---

[jira] [Updated] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-3367:

   Resolution: Fixed
Fix Version/s: 1.6.0
   Status: Resolved  (was: Patch Available)

A minimum length check of 16 bytes is enforced on the token. The rate limiting 
was not implemented. 

> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
> Fix For: 1.6.0
>
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359190#comment-16359190
 ] 

ASF GitHub Bot commented on NIFI-3367:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2463


> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2463: NIFI-3367 Enforced minimum length of shared secret ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2463


---

[jira] [Commented] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359189#comment-16359189
 ] 

ASF subversion and git services commented on NIFI-3367:
---

Commit b7fdb235ee1055e24fdb3ac000cc8039751199ad in nifi's branch 
refs/heads/master from Lori Buettner
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b7fdb23 ]

NIFI-3367 Added token length check and unit test.

This closes #2463.

Signed-off-by: Andy LoPresto 


> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2463: NIFI-3367 Enforced minimum length of shared secret in TLS ...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2463
  
Ran the toolkit in client/server mode. Both rejected a token < 16 bytes 
with an informative error message. With a long token, the process succeeded. 

Ran `contrib-check` and all tests pass. +1, merging. 


---

[jira] [Commented] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359187#comment-16359187
 ] 

ASF GitHub Bot commented on NIFI-3367:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2463
  
Ran the toolkit in client/server mode. Both rejected a token < 16 bytes 
with an informative error message. With a long token, the process succeeded. 

Ran `contrib-check` and all tests pass. +1, merging. 


> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-4866) HBase_1_1_2_ClientService performance dropped due to accessing HBase admin upon every FlowFile transfer

[Koji Kawamura (JIRA)]

Koji Kawamura created NIFI-4866:
---

 Summary: HBase_1_1_2_ClientService performance dropped due to 
accessing HBase admin upon every FlowFile transfer
 Key: NIFI-4866
 URL: https://issues.apache.org/jira/browse/NIFI-4866
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Reporter: Koji Kawamura
Assignee: Koji Kawamura


NIFI-4543 added 
"connection.getAdmin().getClusterStatus().getMaster().getHostAndPort()” call 
from HBase_1_1_2_ClientService.toTransitUri method in order to get HBase master 
address, which is called upon every FlowFile transfer and essentially doubles 
the traffic through the HBase connector.

The performance of our PutHBaseJSON processor dropped to 1/3 after deploying 
NiFi 1.5.0.

Those lines of code should be moved from toTransitUri to createConnection.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359174#comment-16359174
 ] 

ASF GitHub Bot commented on NIFI-3367:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2463#discussion_r167382434
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/test/java/org/apache/nifi/toolkit/tls/util/TlsHelperTest.java
 ---
@@ -171,6 +172,43 @@ private Date inFuture(int days) {
 return new Date(System.currentTimeMillis() + 
TimeUnit.DAYS.toMillis(days));
 }
 
+@Test
+public void testTokenLengthInCalculateHmac() throws 
CertificateException, NoSuchAlgorithmException {
+List badTokens = new ArrayList<>();
+List goodTokens = new ArrayList<>();
+badTokens.add(null);
+badTokens.add("");
+badTokens.add("123");
+goodTokens.add("0123456789abcdefghijklm");
+goodTokens.add("0123456789abcdef");
+
+String dn = "CN=testDN,O=testOrg";
+X509Certificate x509Certificate = 
CertificateUtils.generateSelfSignedX509Certificate(TlsHelper.generateKeyPair(keyPairAlgorithm,
 keySize), dn, signingAlgorithm, days);
+PublicKey pubKey = x509Certificate.getPublicKey();
+
+for (String token : badTokens) {
+try {
+TlsHelper.calculateHMac(token, pubKey);
+fail("HMAC was calculated with a token that was too 
short.");
+} catch (GeneralSecurityException e) {
+assertEquals("Token does not meet minimum size of 16 
bytes.", e.getMessage());
+} catch (IllegalArgumentException e) {
+assertEquals("Token cannot be null", e.getMessage());
+}
+}
+
+for (String token : goodTokens) {
+try {
+byte[] hmac = TlsHelper.calculateHMac(token, pubKey);
+assertTrue("HMAC length ok", hmac.length > 0);
+} catch (GeneralSecurityException e) {
+fail(e.getMessage());
+}
+}
+
--- End diff --

Please remove unnecessary whitespace. 


> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359173#comment-16359173
 ] 

ASF GitHub Bot commented on NIFI-3367:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2463
  
Reviewing...


> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2463: NIFI-3367 Enforced minimum length of shared secret in TLS ...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/2463
  
Reviewing...


---

[jira] [Updated] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-3367:

Status: Patch Available  (was: In Progress)

Lori Buettner contributed two commits which enforce a minimum length on the 
token and added/fixed existing unit tests to demonstrate this. 

> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto reassigned NIFI-3367:
---

Assignee: Andy LoPresto

> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3367) TLS Toolkit should enforce minimum length restriction on CA token

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359171#comment-16359171
 ] 

ASF GitHub Bot commented on NIFI-3367:
--

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/2463

NIFI-3367 Enforced minimum length of shared secret in TLS toolkit 
client/server mode

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-3367

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2463.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2463


commit 79a0aafe51a6d806ef3952650139ebbcd007e274
Author: Lori Buettner 
Date:   2018-02-10T00:46:33Z

NIFI-3367 Added token length check and unit test.

commit 5d9680a513bc9e28067eed45b07823f250c7
Author: Lori Buettner 
Date:   2018-02-10T01:04:58Z

NIFI-3367 Resolved failing unit tests.




> TLS Toolkit should enforce minimum length restriction on CA token
> -
>
> Key: NIFI-3367
> URL: https://issues.apache.org/jira/browse/NIFI-3367
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Priority: Major
>  Labels: security, tls-toolkit
>
> The TLS Toolkit uses a shared secret "token" when running in client/server 
> mode in order to perform pre-authentication when requesting a signed 
> certificate from the CA. There is a validation that this token is *required*, 
> but not that it is of a certain length. Because the HMAC construction is 
> available in the source code, the process could easily be brute-forced if the 
> token value is short. We should enforce a minimum length of 16 bytes 
> (regardless if read from {{config.json}} or provided via command line). 
> We may also want to add exponential rate-limiting on failed HMAC values for 
> the same requested public key DN in order to mitigate malicious requests. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2463: NIFI-3367 Enforced minimum length of shared secret ...

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/2463

NIFI-3367 Enforced minimum length of shared secret in TLS toolkit 
client/server mode

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-3367

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2463.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2463


commit 79a0aafe51a6d806ef3952650139ebbcd007e274
Author: Lori Buettner 
Date:   2018-02-10T00:46:33Z

NIFI-3367 Added token length check and unit test.

commit 5d9680a513bc9e28067eed45b07823f250c7
Author: Lori Buettner 
Date:   2018-02-10T01:04:58Z

NIFI-3367 Resolved failing unit tests.




---

[GitHub] nifi issue #2180: Added GetMongoAggregation to support running Mongo aggrega...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2180
  
@mattyb149 This should be ready for merge now.


---

[GitHub] nifi issue #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2294
  
@ijokarumawak I think all of your concerns have been addressed now.


---

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359093#comment-16359093
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2294
  
@ijokarumawak I think all of your concerns have been addressed now.


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-4865) Registry API command Create Version can produce a VersionedProcessGroup that causes NiFi to NPE on Version change

[Daniel Chaffelson (JIRA)]

Daniel Chaffelson created NIFI-4865:
---

 Summary: Registry API command Create Version can produce a 
VersionedProcessGroup that causes NiFi to NPE on Version change
 Key: NIFI-4865
 URL: https://issues.apache.org/jira/browse/NIFI-4865
 Project: Apache NiFi
  Issue Type: Bug
  Components: Flow Versioning
Affects Versions: 1.5.0
 Environment: NiFi Docker 1.5.0 / NiFi-Registry Docker 0.1.0
Reporter: Daniel Chaffelson


I imported a complex (300kb) xml template into NiFi and placed it under version 
control using the typical GUI commands.

Then using the Registry API command 'create version', I instructed it to make a 
zero change new version, resulting in ver1 and ver2 being in the same bucket 
and flow object.

Returning to the GUI, I instructed it to update to the new version, which 
causes the below NPE.

I have retested this with a fresh environment and am able to reproduce it at 
will, though as the template belongs to a client and is very detailed I cannot 
in good faith share it directly on a public service - I am happy to supply it 
directly in the interests of locking down this bug.
{noformat}
2018-02-09 21:45:40,900 INFO [Version Control Update Thread-1] 
o.a.nifi.groups.StandardProcessGroup Updating 
StandardProcessGroup[identifier=420d499d-2b1d-3af8-6cc3-2722817f4d49] to 
VersionedFlowSnapshot[flowId=e799c617-0dd9-4685-b471-2987b19ad121, 
flowName=test1, version=2, comments=null]; there are 0 differences to take into 
account:

2018-02-09 21:45:40,903 INFO [Version Control Update Thread-1] 
org.apache.nifi.web.api.VersionsResource Successfully updated flow; re-enabling 
0 Controller Services
2018-02-09 21:45:40,903 INFO [Version Control Update Thread-1] 
org.apache.nifi.web.api.VersionsResource Restarting 0 Processors
2018-02-09 21:45:40,903 ERROR [Version Control Update Thread-1] 
org.apache.nifi.web.api.VersionsResource Failed to update flow to new version
java.lang.NullPointerException: null
at 
org.apache.nifi.groups.StandardProcessGroup.updateConnection(StandardProcessGroup.java:3784)
at 
org.apache.nifi.groups.StandardProcessGroup.updateProcessGroup(StandardProcessGroup.java:3656)
at 
org.apache.nifi.groups.StandardProcessGroup.updateProcessGroup(StandardProcessGroup.java:3488)
at 
org.apache.nifi.groups.StandardProcessGroup.updateFlow(StandardProcessGroup.java:3295)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.updateProcessGroupFlow(StandardProcessGroupDAO.java:279)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at 
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at 
org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:258)
at sun.reflect.GeneratedMethodAccessor644.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at 
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at 
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at 
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at 
org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$EnhancerBySpringCGLIB$$3bf4fdb8.updateProcessGroupFlow()
at 
org.apache.nifi.web.StandardNiFiServiceFacade$13.update(StandardNiFiServiceFacade.java:4304)
at 
org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:120)
at 
org.apache.nifi.web.StandardNiFiServiceFacade.updateProcessGroupContents(StandardNiFiServiceFacade.java:4300)
at 
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at 
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at 

[jira] [Updated] (MINIFICPP-396) RPG does not pass SSLContextService to site to site client in a consistent way

[marco polo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

marco polo updated MINIFICPP-396:
-
Description: 
The RPG doesn't always pass the SSLContextService in a a consistent way. The 
documentation is confusing, so we should clear this up. 

Further, there are two paths to configure TLS comms, so we should make this 
more cohesive, but also backwards compatible. 

  was:The RPG doesn't always pass the SSLContextService in a a consistent way. 
The documentation is confusing, so we should clear this up. 


> RPG does not pass SSLContextService to site to site client in a consistent way
> --
>
> Key: MINIFICPP-396
> URL: https://issues.apache.org/jira/browse/MINIFICPP-396
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: marco polo
>Assignee: marco polo
>Priority: Blocker
>
> The RPG doesn't always pass the SSLContextService in a a consistent way. The 
> documentation is confusing, so we should clear this up. 
> Further, there are two paths to configure TLS comms, so we should make this 
> more cohesive, but also backwards compatible. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4863) If changing version of flow and new flow has no connection to RPG Port, can get error saying unable to start components

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358936#comment-16358936
 ] 

ASF GitHub Bot commented on NIFI-4863:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2462


> If changing version of flow and new flow has no connection to RPG Port, can 
> get error saying unable to start components
> ---
>
> Key: NIFI-4863
> URL: https://issues.apache.org/jira/browse/NIFI-4863
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> When we have a Remote Process Group with multiple ports, we push to the flow 
> registry the ports that we have connections for. I have a connection to Port 
> A and save that to the flow registry, then create a connection to Port B. I 
> have RPG transmitting data. Then I revert local changes. This properly 
> reverts the flow but fails to restart components. This is because it is 
> attempting to start Port B, and there are no connections for that Port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4863) If changing version of flow and new flow has no connection to RPG Port, can get error saying unable to start components

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358935#comment-16358935
 ] 

ASF subversion and git services commented on NIFI-4863:
---

Commit 61c6f0305bc7b5c68456fc2d40e0536698821c47 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=61c6f03 ]

NIFI-4863: Bug fixes to the way that we handled Remote Group Ports when 
changing flow version / reverting local changes
- Everywhere that we ignore adding remote ports we should ignore removing 
remote ports as well in flow diffs

This closes #2462.

Signed-off-by: Bryan Bende 


> If changing version of flow and new flow has no connection to RPG Port, can 
> get error saying unable to start components
> ---
>
> Key: NIFI-4863
> URL: https://issues.apache.org/jira/browse/NIFI-4863
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> When we have a Remote Process Group with multiple ports, we push to the flow 
> registry the ports that we have connections for. I have a connection to Port 
> A and save that to the flow registry, then create a connection to Port B. I 
> have RPG transmitting data. Then I revert local changes. This properly 
> reverts the flow but fails to restart components. This is because it is 
> attempting to start Port B, and there are no connections for that Port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2462: NIFI-4863: Bug fixes to the way that we handled Rem...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2462


---

[GitHub] nifi issue #2462: NIFI-4863: Bug fixes to the way that we handled Remote Gro...

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/2462
  
+1 Ran through test scenarios with input and output ports on RPGs in 
versioned flows and this appears to clear up all the issues, merging to master


---

[jira] [Commented] (NIFIREG-139) Nifi registry should pick up the controller service definitions while deploying a flow

[Kevin Doran (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358918#comment-16358918
 ] 

Kevin Doran commented on NIFIREG-139:
-

{quote}If it is at a higher level then it is an external reference and will not 
be copied but rather an attempt to bind to a similar one in the other 
environment will be made.{quote}

I'm not sure this is true for controller services. [~markap14] could verify.

> Nifi registry should pick up the controller service definitions while 
> deploying a flow
> --
>
> Key: NIFIREG-139
> URL: https://issues.apache.org/jira/browse/NIFIREG-139
> Project: NiFi Registry
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: ohad israeli
>Priority: Critical
>
> I am trying to use the registry to copy a simple flow for example of 
> httprequest->joltconversion->httpresponse.
> This flow is using the internal http context service which is installed and 
> configured on the source (dev env). While deploying the flow to the target 
> (prod env) the flow cant find the service on the target (prod end)
> to summarize, on deployment of a flow if the flow is using a service 
> controller its setting should also be deployed on the target machine 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-31) Support UpdateAttribute for nifi-minifi-cpp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-31?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358881#comment-16358881
 ] 

ASF GitHub Bot commented on MINIFICPP-31:
-

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/262
  
UpdateAttributeTests fails when expression language is not built. May want 
to put the expression language specific portion of the test under Expression 
language tests and have the no-op tests where it currently exists. Just an 
idea. 


> Support UpdateAttribute for nifi-minifi-cpp
> ---
>
> Key: MINIFICPP-31
> URL: https://issues.apache.org/jira/browse/MINIFICPP-31
> Project: NiFi MiNiFi C++
>  Issue Type: New Feature
>Reporter: Randy Gelhausen
>Assignee: Andrew Christianson
>Priority: Major
>
> nifi-minifi-cpp agents can generate multiple "streams" of flowfiles.
> For instance, to monitor a host, nifi-minifi-cpp runs nmon, ps, netstat, and 
> gathers logfiles from applications.
> But, for a given flowfile, any downstream NiFi collectors wont have 
> visibility into the originating hostname, nor metadata about which "stream" 
> (ExecuteProcess(nmon), ExecuteProcess(ps), TailFile(app1), TailFile(app2)) 
> generated it.
> One solution is to use a separate InputPort for each stream. This works, but 
> burdens both the team working on agent flows, and the team managing the 
> collector- they have to be in concert.
> A simpler (better?) approach is to allow agent teams to tag flowfiles with 
> differentiating metadata via use of UpdateAttribute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp issue #262: MINIFICPP-31 Added UpdateAttribute processor

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/262
  
UpdateAttributeTests fails when expression language is not built. May want 
to put the expression language specific portion of the test under Expression 
language tests and have the no-op tests where it currently exists. Just an 
idea. 


---

[jira] [Commented] (NIFI-4863) If changing version of flow and new flow has no connection to RPG Port, can get error saying unable to start components

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358849#comment-16358849
 ] 

ASF GitHub Bot commented on NIFI-4863:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/2462
  
Reviewing...


> If changing version of flow and new flow has no connection to RPG Port, can 
> get error saying unable to start components
> ---
>
> Key: NIFI-4863
> URL: https://issues.apache.org/jira/browse/NIFI-4863
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> When we have a Remote Process Group with multiple ports, we push to the flow 
> registry the ports that we have connections for. I have a connection to Port 
> A and save that to the flow registry, then create a connection to Port B. I 
> have RPG transmitting data. Then I revert local changes. This properly 
> reverts the flow but fails to restart components. This is because it is 
> attempting to start Port B, and there are no connections for that Port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFIREG-126) Entering an invalid bucket id in a deep link causes JS error

[Kevin Doran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFIREG-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Doran resolved NIFIREG-126.
-
   Resolution: Fixed
Fix Version/s: 0.2.0

> Entering an invalid bucket id in a deep link causes JS error
> 
>
> Key: NIFIREG-126
> URL: https://issues.apache.org/jira/browse/NIFIREG-126
> Project: NiFi Registry
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Major
> Fix For: 0.2.0
>
>
> As a user when I enter an invalid deep link I want to be routed to view all 
> the items in all the buckets that I am authorized to view and to be notified 
> that the requested bucket/item id is invalid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-registry pull request #99: [NIFIREG-126] adding some polish and testing...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-registry/pull/99


---

[jira] [Commented] (NIFIREG-126) Entering an invalid bucket id in a deep link causes JS error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358847#comment-16358847
 ] 

ASF GitHub Bot commented on NIFIREG-126:


Github user asfgit closed the pull request at:

https://github.com/apache/nifi-registry/pull/99


> Entering an invalid bucket id in a deep link causes JS error
> 
>
> Key: NIFIREG-126
> URL: https://issues.apache.org/jira/browse/NIFIREG-126
> Project: NiFi Registry
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Major
>
> As a user when I enter an invalid deep link I want to be routed to view all 
> the items in all the buckets that I am authorized to view and to be notified 
> that the requested bucket/item id is invalid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #263: Minificpp 396

[phrocker]

GitHub user phrocker opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/263

Minificpp 396

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-396

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/263.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #263


commit b84a02c3e400e175c5863460f7ebe60b85872bb1
Author: Marc Parisi 
Date:   2018-02-09T18:02:45Z

MINIFICPP-396: Make ability to configure SSL in HTTP and Socket SiteToSite 
backwards compatible

commit 756c9a327a2b57849b5a7038d3039d9947abfc33
Author: Marc Parisi 
Date:   2018-02-09T19:16:26Z

MINIFICPP-396: Fix MQTT travis failures




---

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358824#comment-16358824
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167319106
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase-client-service-api/src/main/java/org/apache/nifi/hbase/HBaseClientService.java
 ---
@@ -116,6 +117,13 @@
  */
 void delete(String tableName, byte[] rowId) throws IOException;
 
+/**
+ * Deletes a list of rows in HBase. All cells are deleted.
+ *
+ */
+
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358823#comment-16358823
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167319088
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/test/java/org/apache/nifi/hbase/MockHBaseClientService.java
 ---
@@ -71,6 +73,43 @@ public void delete(String tableName, byte[] rowId) 
throws IOException {
 throw new UnsupportedOperationException();
 }
 
+@Override
+public void delete(String tableName, List rowIds) throws 
IOException {
+if (throwException) {
+throw new RuntimeException("Simulated connectivity error");
+}
+
+Random random = new Random();
+int location = 0;
+if (rowIds.size() > 1) {
+while (location == 0) {
+location = random.nextInt(rowIds.size());
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167319088
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/test/java/org/apache/nifi/hbase/MockHBaseClientService.java
 ---
@@ -71,6 +73,43 @@ public void delete(String tableName, byte[] rowId) 
throws IOException {
 throw new UnsupportedOperationException();
 }
 
+@Override
+public void delete(String tableName, List rowIds) throws 
IOException {
+if (throwException) {
+throw new RuntimeException("Simulated connectivity error");
+}
+
+Random random = new Random();
+int location = 0;
+if (rowIds.size() > 1) {
+while (location == 0) {
+location = random.nextInt(rowIds.size());
--- End diff --

Done


---

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167319106
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-hbase-client-service-api/src/main/java/org/apache/nifi/hbase/HBaseClientService.java
 ---
@@ -116,6 +117,13 @@
  */
 void delete(String tableName, byte[] rowId) throws IOException;
 
+/**
+ * Deletes a list of rows in HBase. All cells are deleted.
+ *
+ */
+
--- End diff --

Done


---

[jira] [Commented] (NIFI-4863) If changing version of flow and new flow has no connection to RPG Port, can get error saying unable to start components

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358822#comment-16358822
 ] 

ASF GitHub Bot commented on NIFI-4863:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2462

NIFI-4863: Bug fixes to the way that we handled Remote Group Ports wh…

…en changing flow version / reverting local changes

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4863

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2462.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2462


commit ef64b56f215cac51a9935500fbf553816e0ffa92
Author: Mark Payne 
Date:   2018-02-09T19:09:23Z

NIFI-4863: Bug fixes to the way that we handled Remote Group Ports when 
changing flow version / reverting local changes




> If changing version of flow and new flow has no connection to RPG Port, can 
> get error saying unable to start components
> ---
>
> Key: NIFI-4863
> URL: https://issues.apache.org/jira/browse/NIFI-4863
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> When we have a Remote Process Group with multiple ports, we push to the flow 
> registry the ports that we have connections for. I have a connection to Port 
> A and save that to the flow registry, then create a connection to Port B. I 
> have RPG transmitting data. Then I revert local changes. This properly 
> reverts the flow but fails to restart components. This is because it is 
> attempting to start Port B, and there are no connections for that Port.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2462: NIFI-4863: Bug fixes to the way that we handled Rem...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2462

NIFI-4863: Bug fixes to the way that we handled Remote Group Ports wh…

…en changing flow version / reverting local changes

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4863

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2462.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2462


commit ef64b56f215cac51a9935500fbf553816e0ffa92
Author: Mark Payne 
Date:   2018-02-09T19:09:23Z

NIFI-4863: Bug fixes to the way that we handled Remote Group Ports when 
changing flow version / reverting local changes




---

[jira] [Created] (MINIFICPP-398) Support schema config version 4

[Aldrin Piri (JIRA)]

Aldrin Piri created MINIFICPP-398:
-

 Summary: Support schema config version 4
 Key: MINIFICPP-398
 URL: https://issues.apache.org/jira/browse/MINIFICPP-398
 Project: NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Aldrin Piri


As part of MINIFI-66, and to help us carry out some initial C2 functionalities, 
a closer mapping of config schema to templates will allow us to more easily 
make use of registry functionality.  This will require that instance based 
configuration (things like repositories, security and the like) are extracted 
to other files such as the conf and/or other .properties files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-31) Support UpdateAttribute for nifi-minifi-cpp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-31?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358809#comment-16358809
 ] 

ASF GitHub Bot commented on MINIFICPP-31:
-

GitHub user achristianson opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/262

MINIFICPP-31 Added UpdateAttribute processor

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [x] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the LICENSE file?
- [x] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-31

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/262.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #262


commit fb4e92d20c578299780fcf764fc9d0636af3259d
Author: Andrew I. Christianson 
Date:   2018-02-08T18:51:08Z

MINIFICPP-31 Added UpdateAttribute processor




> Support UpdateAttribute for nifi-minifi-cpp
> ---
>
> Key: MINIFICPP-31
> URL: https://issues.apache.org/jira/browse/MINIFICPP-31
> Project: NiFi MiNiFi C++
>  Issue Type: New Feature
>Reporter: Randy Gelhausen
>Assignee: Andrew Christianson
>Priority: Major
>
> nifi-minifi-cpp agents can generate multiple "streams" of flowfiles.
> For instance, to monitor a host, nifi-minifi-cpp runs nmon, ps, netstat, and 
> gathers logfiles from applications.
> But, for a given flowfile, any downstream NiFi collectors wont have 
> visibility into the originating hostname, nor metadata about which "stream" 
> (ExecuteProcess(nmon), ExecuteProcess(ps), TailFile(app1), TailFile(app2)) 
> generated it.
> One solution is to use a separate InputPort for each stream. This works, but 
> burdens both the team working on agent flows, and the team managing the 
> collector- they have to be in concert.
> A simpler (better?) approach is to allow agent teams to tag flowfiles with 
> differentiating metadata via use of UpdateAttribute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #262: MINIFICPP-31 Added UpdateAttribute proces...

[achristianson]

GitHub user achristianson opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/262

MINIFICPP-31 Added UpdateAttribute processor

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [x] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] If applicable, have you updated the LICENSE file?
- [x] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-31

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/262.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #262


commit fb4e92d20c578299780fcf764fc9d0636af3259d
Author: Andrew I. Christianson 
Date:   2018-02-08T18:51:08Z

MINIFICPP-31 Added UpdateAttribute processor




---

[jira] [Created] (MINIFICPP-397) Implement RouteOnAttribute

[Andrew Christianson (JIRA)]

Andrew Christianson created MINIFICPP-397:
-

 Summary: Implement RouteOnAttribute
 Key: MINIFICPP-397
 URL: https://issues.apache.org/jira/browse/MINIFICPP-397
 Project: NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Andrew Christianson
Assignee: Andrew Christianson


RouteOnAttribute is notably missing from MiNiFi - C++ and should be implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358790#comment-16358790
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167307609
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttribute( attribute = "restart.index", description = "If a delete 
batch fails, it will restart from restart.index" )
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
+static final AllowableValue ROW_ID_ATTR = new AllowableValue("attr", 
"FlowFile attributes", "Get the row key from an expression language 
statement.");
+
+static final PropertyDescriptor ROW_ID_LOCATION = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-id-location")
+.displayName("Row ID Location")
+.description("The location of the row ID to use for building 
the delete. Can be from the content or an expression language statement.")
+.required(true)
+.defaultValue(ROW_ID_BODY.getValue())
+.allowableValues(ROW_ID_BODY, ROW_ID_ATTR)
+.addValidator(Validator.VALID)
+.build();
+
+static final PropertyDescriptor FLOWFILE_FETCH_COUNT = new 
PropertyDescriptor.Builder()
+.name("delete-hb-flowfile-fetch-count")
+.displayName("Flowfile Fetch Count")
+.description("The number of flowfiles to fetch per run.")
+.required(true)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("5")
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-ff-count")
+.displayName("Batch Size")
+.description("The number of deletes to send per batch.")
+.required(true)
+.defaultValue("50")
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor KEY_SEPARATOR = new 
PropertyDescriptor.Builder()
+.name("delete-hb-separator")
+.displayName("Delete Row Key Separator")
+.description("The separator character(s) that separate 
multiple row keys " +
+"when multiple row keys are provided in the flowfile 
body")
+.required(true)
+

[jira] [Commented] (NIFI-4856) ByteCountingInputStream should override available()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358607#comment-16358607
 ] 

ASF GitHub Bot commented on NIFI-4856:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2461


> ByteCountingInputStream should override available()
> ---
>
> Key: NIFI-4856
> URL: https://issues.apache.org/jira/browse/NIFI-4856
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Brandon DeVries
>Assignee: Andy LoPresto
>Priority: Major
> Fix For: 1.6.0
>
>
> ByteCountingInputStream [1] does not override the InputStream.available() 
> method as indicated in the InputStream javadocs [2].  In 0.x[3] there was 
> another implementation of ByteCountingInputStream  that correctly overrode 
> available().  However, this was replaced with the incorrect version in 1.x. 
>  
> This causes any processor that attempts to use the available() method of the 
> InputStream returned from ProcessSession.read() to erroneously believe it has 
> reached the end of the stream, as this method is supposed to return "an 
> estimate of the number of bytes that can be read (or skipped over) from this 
> input stream without blocking *or {{0}} when it reaches the end* of the input 
> stream."
>  
> The following method needs to be added:
> {code:java}
> @Override
> public int available() throws IOException {
>  return in.available();
> }{code}
>  
> [1] 
> [https://github.com/apache/nifi/blob/rel/nifi-1.5.0/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/ByteCountingInputStream.java]
> [2] [https://docs.oracle.com/javase/8/docs/api/index.html]
> [3] 
> [https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/io/ByteCountingInputStream.java]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2461: NIFI-4856 Removed deprecated ByteArrayInputStream r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2461


---

[jira] [Updated] (NIFI-4856) ByteCountingInputStream should override available()

[Kevin Doran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Doran updated NIFI-4856:
--
   Resolution: Fixed
Fix Version/s: 1.6.0
   Status: Resolved  (was: Patch Available)

> ByteCountingInputStream should override available()
> ---
>
> Key: NIFI-4856
> URL: https://issues.apache.org/jira/browse/NIFI-4856
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Brandon DeVries
>Assignee: Andy LoPresto
>Priority: Major
> Fix For: 1.6.0
>
>
> ByteCountingInputStream [1] does not override the InputStream.available() 
> method as indicated in the InputStream javadocs [2].  In 0.x[3] there was 
> another implementation of ByteCountingInputStream  that correctly overrode 
> available().  However, this was replaced with the incorrect version in 1.x. 
>  
> This causes any processor that attempts to use the available() method of the 
> InputStream returned from ProcessSession.read() to erroneously believe it has 
> reached the end of the stream, as this method is supposed to return "an 
> estimate of the number of bytes that can be read (or skipped over) from this 
> input stream without blocking *or {{0}} when it reaches the end* of the input 
> stream."
>  
> The following method needs to be added:
> {code:java}
> @Override
> public int available() throws IOException {
>  return in.available();
> }{code}
>  
> [1] 
> [https://github.com/apache/nifi/blob/rel/nifi-1.5.0/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/ByteCountingInputStream.java]
> [2] [https://docs.oracle.com/javase/8/docs/api/index.html]
> [3] 
> [https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/io/ByteCountingInputStream.java]
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (MINIFICPP-118) Dynamic Properties support for processors

[Andrew Christianson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Christianson updated MINIFICPP-118:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Dynamic Properties support for processors
> -
>
> Key: MINIFICPP-118
> URL: https://issues.apache.org/jira/browse/MINIFICPP-118
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Affects Versions: 0.1.0
>Reporter: Jeremy Dyer
>Assignee: Andrew Christianson
>Priority: Major
>
> Currently any Property read from the config.yml file that is not explicitly 
> defined in the processor's implementation will be ignored by Processor.cpp 
> when reading the configurations. This prevents any dynamic property from 
> being defined in the config.yml and passed to the processor at runtime. 
> Certain processors rely heavily on the concept of dynamic properties that are 
> passed to them at runtime to handle things like setting dynamic properties 
> based on properties that are declared. All of these possibilities cannot be 
> handled upfront so there should be a mechanism, most likely in Processor.cpp, 
> that allows for a list of dynamicProperties that are parsed form the 
> config.yml file to be stored and accessed by the underlying processor 
> implementation at runtime and use them as the processor desires.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFIREG-126) Entering an invalid bucket id in a deep link causes JS error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358584#comment-16358584
 ] 

ASF GitHub Bot commented on NIFIREG-126:


Github user scottyaslan commented on the issue:

https://github.com/apache/nifi-registry/pull/99
  
@kevdoran I have updated this PR with the new message as well as logging 
info.


> Entering an invalid bucket id in a deep link causes JS error
> 
>
> Key: NIFIREG-126
> URL: https://issues.apache.org/jira/browse/NIFIREG-126
> Project: NiFi Registry
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Major
>
> As a user when I enter an invalid deep link I want to be routed to view all 
> the items in all the buckets that I am authorized to view and to be notified 
> that the requested bucket/item id is invalid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFIREG-126) Entering an invalid bucket id in a deep link causes JS error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358585#comment-16358585
 ] 

ASF GitHub Bot commented on NIFIREG-126:


Github user kevdoran commented on the issue:

https://github.com/apache/nifi-registry/pull/99
  
Thanks! Will review


> Entering an invalid bucket id in a deep link causes JS error
> 
>
> Key: NIFIREG-126
> URL: https://issues.apache.org/jira/browse/NIFIREG-126
> Project: NiFi Registry
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Scott Aslan
>Assignee: Scott Aslan
>Priority: Major
>
> As a user when I enter an invalid deep link I want to be routed to view all 
> the items in all the buckets that I am authorized to view and to be notified 
> that the requested bucket/item id is invalid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-registry issue #99: [NIFIREG-126] adding some polish and testing around...

[scottyaslan]

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi-registry/pull/99
  
@kevdoran I have updated this PR with the new message as well as logging 
info.


---

[jira] [Commented] (NIFIREG-133) [Documentation] Explain how variables are captured when versioning a PG

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358583#comment-16358583
 ] 

Andrew Lim commented on NIFIREG-133:


Will likely document this in the NiFi User Guide.  If so, will open a separate 
Jira in NiFi project.  Am in the process of gathering use cases for a Flow 
Deployment Guide.  Variables in versioned process groups would be a good 
section for that new document.

> [Documentation] Explain how variables are captured when versioning a PG
> ---
>
> Key: NIFIREG-133
> URL: https://issues.apache.org/jira/browse/NIFIREG-133
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Pierre Villard
>Assignee: Andrew Lim
>Priority: Minor
>
> Context:
> I have a process group PG that contains an embedded process group PG1. I have 
> a variable defined at PG level that is referenced in a PG1's processor. When 
> versioning both PG and PG1 in NiFi Registry, a copy of the variable defined 
> at PG level will be created at PG1 level.
> Consequently, when importing PG in another environment, the variable needs to 
> be modified at PG1 level because the one created at PG1 level overwrites the 
> one initially defined at PG level.
> > Would be nice to add this behavior in the documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[JIRA]

 [ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakub Leś updated NIFI-4862:

Attachment: 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch

> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[JIRA]

 [ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakub Leś updated NIFI-4862:

Status: Patch Available  (was: Open)

This patch adds additional parameter "copy original attributtes" and gives 
ability to add to each flowfile original attributes.

> Add Copy original attributtes to SelectHiveQL processer
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Priority: Minor
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (NIFIREG-142) First start problem: Error creating bean with name 'flywayInitializer'

[Bryan Bende (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358548#comment-16358548
 ] 

Bryan Bende edited comment on NIFIREG-142 at 2/9/18 3:32 PM:
-

[~doaks80] sorry for the troubles here, can you try deleting the 'database' 
directory and then starting again?

It should get created relative to the base directory for NiFi registry:
{code:java}
ls -1
LICENSE
NOTICE
README
bin
conf
database
docs
flow_storage
lib
logs
run
work{code}
I'm hoping that the H2 database is just in some bad state from the first failed 
start, and by wiping it out it will start from scratch and hopefully work since 
you have the JDK now.


was (Author: bende):
[~doaks80] sorry for the troubles here, can you try deleting the 'database' 
directory in the directory and then starting again?

It should get created relative to the base directory for NiFi registry:
{code:java}
ls -1
LICENSE
NOTICE
README
bin
conf
database
docs
flow_storage
lib
logs
run
work{code}
I'm hoping that the H2 database is just in some bad state from the first failed 
start, and by wiping it out it will start from scratch and hopefully work since 
you have the JDK now.

> First start problem: Error creating bean with name 'flywayInitializer'
> --
>
> Key: NIFIREG-142
> URL: https://issues.apache.org/jira/browse/NIFIREG-142
> Project: NiFi Registry
>  Issue Type: Bug
>Affects Versions: 0.1.0
> Environment: RHEL7
> openjdk version "1.8.0_161"
>Reporter: Daniel Oakley
>Priority: Major
> Attachments: nifi-registry-app.log, nifi-registry-bootstrap.log
>
>
> Downloaded 0.1.0 tarball and and tried to run on RHEL7. No changes to any 
> config files.
> Error in log file was:
> java.lang.RuntimeException: 
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with name 'flywayInitializer' defined in class path resource 
> [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]:
>  Invocation of init method failed; nested exception is 
> org.flywaydb.core.api.FlywayException: Validate failed: Detected failed 
> migration to version 1.3 (DropBucketItemNameUniqueness)
> I could not find anything about "flyway" in the docs or config files...any 
> hints on how to get around this problem?
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-2630) Allow PublishJMS processor to create TextMessages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358556#comment-16358556
 ] 

ASF GitHub Bot commented on NIFI-2630:
--

Github user mosermw commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167258538
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

And now that I look closer, I see there is already a precedent for this in 
the code, such as in LogAttribute.


> Allow PublishJMS processor to create TextMessages
> -
>
> Key: NIFI-2630
> URL: https://issues.apache.org/jira/browse/NIFI-2630
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.0
>Reporter: James Anderson
>Assignee: Michael Moser
>Priority: Minor
>  Labels: patch
> Attachments: 
> 0001-NIFI-2630-Allow-PublishJMS-processor-to-create-TextM.patch
>
>
> Create a new configuration option for PublishJMS that allows the processor to 
> be configured to emit instances of TextMessages as well as BytesMessage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2458: NIFI-2630 Allow PublishJMS to send TextMessages

[mosermw]

Github user mosermw commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167258538
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

And now that I look closer, I see there is already a precedent for this in 
the code, such as in LogAttribute.


---

[jira] [Updated] (NIFI-4841) NPE when reverting local modifications to a versioned process group

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-4841:
-
   Resolution: Fixed
Fix Version/s: 1.6.0
   Status: Resolved  (was: Patch Available)

> NPE when reverting local modifications to a versioned process group
> ---
>
> Key: NIFI-4841
> URL: https://issues.apache.org/jira/browse/NIFI-4841
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Charlie Meyer
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: NIFI-4841.xml
>
>
> I created a process group via importing from the registry. I then made a few 
> modifications including settings properties and connecting some components. I 
> then attempted to revert my local changes so I could update the flow to a 
> newer version. When reverting the local changes, NiFi threw a NPE with the 
> following stack trace:
> {noformat}
> 2018-02-05 17:18:52,356 INFO [Version Control Update Thread-1] 
> org.apache.nifi.web.api.VersionsResource Stopping 1 Processors
> 2018-02-05 17:18:52,477 ERROR [Version Control Update Thread-1] 
> org.apache.nifi.web.api.VersionsResource Failed to update flow to new version
> java.lang.NullPointerException: null
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.scheduleComponents(StandardProcessGroupDAO.java:179)
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
>   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$EnhancerBySpringCGLIB$$bc287b8b.scheduleComponents()
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$3.update(StandardNiFiServiceFacade.java:981)
>   at 
> org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:120)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade.scheduleComponents(StandardNiFiServiceFacade.java:976)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
>   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.scheduleLock(NiFiServiceFacadeLock.java:102)
>   at sun.reflect.GeneratedMethodAccessor557.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
>   at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$8a758fa4.scheduleComponents()
>   at 
> org.apache.nifi.web.util.LocalComponentLifecycle.stopComponents(LocalComponentLifecycle.java:125)
>   at 
> org.apache.nifi.web.util.LocalComponentLifecycle.scheduleComponents(LocalComponentLifecycle.java:66)
>   at 
> 

[jira] [Commented] (NIFI-4841) NPE when reverting local modifications to a versioned process group

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358550#comment-16358550
 ] 

ASF subversion and git services commented on NIFI-4841:
---

Commit 34b678d30d25c930678b008a5db8a26a6a2938b3 in nifi's branch 
refs/heads/master from [~bbende]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=34b678d ]

NIFI-4841 Fixing NPE when reverting local changes involving remote group ports. 
This closes #2454.

Signed-off-by: Mark Payne 


> NPE when reverting local modifications to a versioned process group
> ---
>
> Key: NIFI-4841
> URL: https://issues.apache.org/jira/browse/NIFI-4841
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Charlie Meyer
>Assignee: Bryan Bende
>Priority: Major
> Attachments: NIFI-4841.xml
>
>
> I created a process group via importing from the registry. I then made a few 
> modifications including settings properties and connecting some components. I 
> then attempted to revert my local changes so I could update the flow to a 
> newer version. When reverting the local changes, NiFi threw a NPE with the 
> following stack trace:
> {noformat}
> 2018-02-05 17:18:52,356 INFO [Version Control Update Thread-1] 
> org.apache.nifi.web.api.VersionsResource Stopping 1 Processors
> 2018-02-05 17:18:52,477 ERROR [Version Control Update Thread-1] 
> org.apache.nifi.web.api.VersionsResource Failed to update flow to new version
> java.lang.NullPointerException: null
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.scheduleComponents(StandardProcessGroupDAO.java:179)
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
>   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$EnhancerBySpringCGLIB$$bc287b8b.scheduleComponents()
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$3.update(StandardNiFiServiceFacade.java:981)
>   at 
> org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:120)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade.scheduleComponents(StandardNiFiServiceFacade.java:976)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
>   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.scheduleLock(NiFiServiceFacadeLock.java:102)
>   at sun.reflect.GeneratedMethodAccessor557.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
>   at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$8a758fa4.scheduleComponents()
>   at 
> 

[jira] [Created] (NIFI-4862) Add Copy original attributtes to SelectHiveQL processer

[JIRA]

Jakub Leś created NIFI-4862:
---

 Summary: Add Copy original attributtes to SelectHiveQL processer
 Key: NIFI-4862
 URL: https://issues.apache.org/jira/browse/NIFI-4862
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.5.0
Reporter: Jakub Leś


Hi, 

Please add "Copy original attributes" to processor SelectHiveQL. Thanks to that 
we can use HttpRequest and HttpResponse to synchronize fetching query result in 
webservice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4841) NPE when reverting local modifications to a versioned process group

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358551#comment-16358551
 ] 

ASF GitHub Bot commented on NIFI-4841:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2454


> NPE when reverting local modifications to a versioned process group
> ---
>
> Key: NIFI-4841
> URL: https://issues.apache.org/jira/browse/NIFI-4841
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Charlie Meyer
>Assignee: Bryan Bende
>Priority: Major
> Attachments: NIFI-4841.xml
>
>
> I created a process group via importing from the registry. I then made a few 
> modifications including settings properties and connecting some components. I 
> then attempted to revert my local changes so I could update the flow to a 
> newer version. When reverting the local changes, NiFi threw a NPE with the 
> following stack trace:
> {noformat}
> 2018-02-05 17:18:52,356 INFO [Version Control Update Thread-1] 
> org.apache.nifi.web.api.VersionsResource Stopping 1 Processors
> 2018-02-05 17:18:52,477 ERROR [Version Control Update Thread-1] 
> org.apache.nifi.web.api.VersionsResource Failed to update flow to new version
> java.lang.NullPointerException: null
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO.scheduleComponents(StandardProcessGroupDAO.java:179)
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$FastClassBySpringCGLIB$$10a99b47.invoke()
>   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>   at 
> org.apache.nifi.web.dao.impl.StandardProcessGroupDAO$$EnhancerBySpringCGLIB$$bc287b8b.scheduleComponents()
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$3.update(StandardNiFiServiceFacade.java:981)
>   at 
> org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:120)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade.scheduleComponents(StandardNiFiServiceFacade.java:976)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
>   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>   at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>   at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
>   at 
> org.apache.nifi.web.NiFiServiceFacadeLock.scheduleLock(NiFiServiceFacadeLock.java:102)
>   at sun.reflect.GeneratedMethodAccessor557.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
>   at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
>   at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>   at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>   at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>   at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$8a758fa4.scheduleComponents()
>   at 
> org.apache.nifi.web.util.LocalComponentLifecycle.stopComponents(LocalComponentLifecycle.java:125)
>   at 
> org.apache.nifi.web.util.LocalComponentLifecycle.scheduleComponents(LocalComponentLifecycle.java:66)
>   at 
> 

[GitHub] nifi pull request #2454: NIFI-4841 Fixing NPE when reverting local changes i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2454


---

[jira] [Commented] (NIFI-2630) Allow PublishJMS processor to create TextMessages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358541#comment-16358541
 ] 

ASF GitHub Bot commented on NIFI-2630:
--

Github user mosermw commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167255119
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

Oh, I didn't think about that approach!  I like it.  I'll make the changes.


> Allow PublishJMS processor to create TextMessages
> -
>
> Key: NIFI-2630
> URL: https://issues.apache.org/jira/browse/NIFI-2630
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.0
>Reporter: James Anderson
>Assignee: Michael Moser
>Priority: Minor
>  Labels: patch
> Attachments: 
> 0001-NIFI-2630-Allow-PublishJMS-processor-to-create-TextM.patch
>
>
> Create a new configuration option for PublishJMS that allows the processor to 
> be configured to emit instances of TextMessages as well as BytesMessage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2458: NIFI-2630 Allow PublishJMS to send TextMessages

[mosermw]

Github user mosermw commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167255119
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

Oh, I didn't think about that approach!  I like it.  I'll make the changes.


---

[jira] [Commented] (NIFI-2630) Allow PublishJMS processor to create TextMessages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358513#comment-16358513
 ] 

ASF GitHub Bot commented on NIFI-2630:
--

Github user devriesb commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167252827
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

i'm not sure if this is a helpful "middle ground" suggestion or not, but 
you could add a charset property that itself defaults to 
Charset.defaultCharset().  That way the out of the box behavior uses the 
charset specified for the jvm, but can be overridden on a per processor basis 
if needed.


> Allow PublishJMS processor to create TextMessages
> -
>
> Key: NIFI-2630
> URL: https://issues.apache.org/jira/browse/NIFI-2630
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.0
>Reporter: James Anderson
>Assignee: Michael Moser
>Priority: Minor
>  Labels: patch
> Attachments: 
> 0001-NIFI-2630-Allow-PublishJMS-processor-to-create-TextM.patch
>
>
> Create a new configuration option for PublishJMS that allows the processor to 
> be configured to emit instances of TextMessages as well as BytesMessage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-registry pull request #99: [NIFIREG-126] adding some polish and testing...

[moranr]

Github user moranr commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/99#discussion_r167249634
  
--- Diff: 
nifi-registry-web-api/src/main/java/org/apache/nifi/registry/web/api/AccessPolicyResource.java
 ---
@@ -174,7 +174,7 @@ public Response getAccessPolicy(
 
 final AccessPolicy accessPolicy = 
authorizationService.getAccessPolicy(identifier);
 if (accessPolicy == null) {
-throw new ResourceNotFoundException("No access policy found 
with ID + " + identifier);
+throw new ResourceNotFoundException("The specified access 
policy does not exist for this bucket.");
--- End diff --

I think that works, we could even add _"...in this registry."_ to end of 
that. It's consistent with the other messages.


---

[jira] [Commented] (NIFI-2630) Allow PublishJMS processor to create TextMessages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358476#comment-16358476
 ] 

ASF GitHub Bot commented on NIFI-2630:
--

Github user mosermw commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167244022
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

I did think about this, and I chose this approach on purpose.  It's kind of 
the whole point of setting a default Charset in your JVM, to produce results 
that are specific to your desired environment.  Is this not right?


> Allow PublishJMS processor to create TextMessages
> -
>
> Key: NIFI-2630
> URL: https://issues.apache.org/jira/browse/NIFI-2630
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.0
>Reporter: James Anderson
>Assignee: Michael Moser
>Priority: Minor
>  Labels: patch
> Attachments: 
> 0001-NIFI-2630-Allow-PublishJMS-processor-to-create-TextM.patch
>
>
> Create a new configuration option for PublishJMS that allows the processor to 
> be configured to emit instances of TextMessages as well as BytesMessage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2458: NIFI-2630 Allow PublishJMS to send TextMessages

[mosermw]

Github user mosermw commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167244022
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

I did think about this, and I chose this approach on purpose.  It's kind of 
the whole point of setting a default Charset in your JVM, to produce results 
that are specific to your desired environment.  Is this not right?


---

[jira] [Updated] (NIFI-4853) PutMongoRecord doesn't handle nested records

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-4853:
-
   Resolution: Fixed
Fix Version/s: 1.6.0
   Status: Resolved  (was: Patch Available)

> PutMongoRecord doesn't handle nested records
> 
>
> Key: NIFI-4853
> URL: https://issues.apache.org/jira/browse/NIFI-4853
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.6.0
>
>
> PutMongoRecord works well with flat records, but when attempting to put in 
> nested records (records whose fields are arrays or records themselves), the 
> Mongo serializer doesn't know how to handle the complex NiFi record field 
> types (such as MapRecord).
> The fix is to traverse all fields in the record, transforming the fields (if 
> necessary) to Java objects for use by the Mongo serializer(s). Something very 
> similar was done for PutDruidRecord, and in fact there is a utility method 
> DataTypeUtils.convertRecordFieldtoObject() that can perform this task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4853) PutMongoRecord doesn't handle nested records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358464#comment-16358464
 ] 

ASF GitHub Bot commented on NIFI-4853:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2457


> PutMongoRecord doesn't handle nested records
> 
>
> Key: NIFI-4853
> URL: https://issues.apache.org/jira/browse/NIFI-4853
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> PutMongoRecord works well with flat records, but when attempting to put in 
> nested records (records whose fields are arrays or records themselves), the 
> Mongo serializer doesn't know how to handle the complex NiFi record field 
> types (such as MapRecord).
> The fix is to traverse all fields in the record, transforming the fields (if 
> necessary) to Java objects for use by the Mongo serializer(s). Something very 
> similar was done for PutDruidRecord, and in fact there is a utility method 
> DataTypeUtils.convertRecordFieldtoObject() that can perform this task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2457: NIFI-4853: Fixed PutMongoRecord handling of nested ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2457


---

[GitHub] nifi issue #2457: NIFI-4853: Fixed PutMongoRecord handling of nested records

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2457
  
+1, merging to master, ran some tests on a local Mongo, all looks good. 
Thanks!


---

[jira] [Commented] (NIFI-2630) Allow PublishJMS processor to create TextMessages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358457#comment-16358457
 ] 

ASF GitHub Bot commented on NIFI-2630:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167240898
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

We probably want to expose the character set as a property. At a minimum, 
we should be explicit in defining which character set would be used (UTF-8 
perhaps) instead of relying on the default charset, as this would produce 
different results from environment to environment.


> Allow PublishJMS processor to create TextMessages
> -
>
> Key: NIFI-2630
> URL: https://issues.apache.org/jira/browse/NIFI-2630
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.6.0
>Reporter: James Anderson
>Assignee: Michael Moser
>Priority: Minor
>  Labels: patch
> Attachments: 
> 0001-NIFI-2630-Allow-PublishJMS-processor-to-create-TextM.patch
>
>
> Create a new configuration option for PublishJMS that allows the processor to 
> be configured to emit instances of TextMessages as well as BytesMessage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2458: NIFI-2630 Allow PublishJMS to send TextMessages

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2458#discussion_r167240898
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/PublishJMS.java
 ---
@@ -131,4 +143,10 @@ protected JMSPublisher 
finishBuildingJmsWorker(CachingConnectionFactory connecti
 session.read(flowFile, in -> StreamUtils.fillBuffer(in, 
messageContent, true));
 return messageContent;
 }
+
+private String extractTextMessageBody(FlowFile flowFile, 
ProcessSession session) {
+final StringWriter writer = new StringWriter();
+session.read(flowFile, in -> IOUtils.copy(in, writer, 
Charset.defaultCharset()));
--- End diff --

We probably want to expose the character set as a property. At a minimum, 
we should be explicit in defining which character set would be used (UTF-8 
perhaps) instead of relying on the default charset, as this would produce 
different results from environment to environment.


---

[jira] [Commented] (NIFI-4859) Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state

[Kevin Doran (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358456#comment-16358456
 ] 

Kevin Doran commented on NIFI-4859:
---

Thanks for reporting [~chaffelson]. Will fix this the next chance I get / next 
swagger update I make.

> Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state
> 
>
> Key: NIFI-4859
> URL: https://issues.apache.org/jira/browse/NIFI-4859
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Chaffelson
>Assignee: Kevin Doran
>Priority: Minor
> Fix For: 1.6.0
>
>
> It is possible to get a Versioned Process Group into a SYNC_FAILURE state, 
> but this is not an allowable state in the code generated from the swagger.json
> This prevents versioned objects from being manipulated via the API in some 
> use cases.
> {noformat}
> @state.setter
> def state(self, state):
> """
> Sets the state of this VersionControlInformationDTO.
> The current state of the Process Group, as it relates to the Versioned Flow
> :param state: The state of this VersionControlInformationDTO.
> :type: str
> """
> allowed_values = ["LOCALLY_MODIFIED_DESCENDANT", "LOCALLY_MODIFIED", "STALE", 
> "LOCALLY_MODIFIED_AND_STALE", "UP_TO_DATE"]
> if state not in allowed_values:
> raise ValueError(
> "Invalid value for `state` ({0}), must be one of {1}"
> > .format(state, allowed_values)
> )
> E ValueError: Invalid value for `state` (SYNC_FAILURE), must be one of 
> ['LOCALLY_MODIFIED_DESCENDANT', 'LOCALLY_MODIFIED', 'STALE', 
> 'LOCALLY_MODIFIED_AND_STALE', 'UP_TO_DATE']
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4859) Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state

[Kevin Doran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Doran updated NIFI-4859:
--
Priority: Minor  (was: Major)

> Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state
> 
>
> Key: NIFI-4859
> URL: https://issues.apache.org/jira/browse/NIFI-4859
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Chaffelson
>Assignee: Kevin Doran
>Priority: Minor
> Fix For: 1.6.0
>
>
> It is possible to get a Versioned Process Group into a SYNC_FAILURE state, 
> but this is not an allowable state in the code generated from the swagger.json
> This prevents versioned objects from being manipulated via the API in some 
> use cases.
> {noformat}
> @state.setter
> def state(self, state):
> """
> Sets the state of this VersionControlInformationDTO.
> The current state of the Process Group, as it relates to the Versioned Flow
> :param state: The state of this VersionControlInformationDTO.
> :type: str
> """
> allowed_values = ["LOCALLY_MODIFIED_DESCENDANT", "LOCALLY_MODIFIED", "STALE", 
> "LOCALLY_MODIFIED_AND_STALE", "UP_TO_DATE"]
> if state not in allowed_values:
> raise ValueError(
> "Invalid value for `state` ({0}), must be one of {1}"
> > .format(state, allowed_values)
> )
> E ValueError: Invalid value for `state` (SYNC_FAILURE), must be one of 
> ['LOCALLY_MODIFIED_DESCENDANT', 'LOCALLY_MODIFIED', 'STALE', 
> 'LOCALLY_MODIFIED_AND_STALE', 'UP_TO_DATE']
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4859) Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state

[Kevin Doran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Doran updated NIFI-4859:
--
Affects Version/s: (was: 1.6.0)

> Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state
> 
>
> Key: NIFI-4859
> URL: https://issues.apache.org/jira/browse/NIFI-4859
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Chaffelson
>Assignee: Kevin Doran
>Priority: Major
> Fix For: 1.6.0
>
>
> It is possible to get a Versioned Process Group into a SYNC_FAILURE state, 
> but this is not an allowable state in the code generated from the swagger.json
> This prevents versioned objects from being manipulated via the API in some 
> use cases.
> {noformat}
> @state.setter
> def state(self, state):
> """
> Sets the state of this VersionControlInformationDTO.
> The current state of the Process Group, as it relates to the Versioned Flow
> :param state: The state of this VersionControlInformationDTO.
> :type: str
> """
> allowed_values = ["LOCALLY_MODIFIED_DESCENDANT", "LOCALLY_MODIFIED", "STALE", 
> "LOCALLY_MODIFIED_AND_STALE", "UP_TO_DATE"]
> if state not in allowed_values:
> raise ValueError(
> "Invalid value for `state` ({0}), must be one of {1}"
> > .format(state, allowed_values)
> )
> E ValueError: Invalid value for `state` (SYNC_FAILURE), must be one of 
> ['LOCALLY_MODIFIED_DESCENDANT', 'LOCALLY_MODIFIED', 'STALE', 
> 'LOCALLY_MODIFIED_AND_STALE', 'UP_TO_DATE']
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4859) Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state

[Kevin Doran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Doran updated NIFI-4859:
--
Fix Version/s: 1.6.0

> Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state
> 
>
> Key: NIFI-4859
> URL: https://issues.apache.org/jira/browse/NIFI-4859
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Chaffelson
>Assignee: Kevin Doran
>Priority: Major
> Fix For: 1.6.0
>
>
> It is possible to get a Versioned Process Group into a SYNC_FAILURE state, 
> but this is not an allowable state in the code generated from the swagger.json
> This prevents versioned objects from being manipulated via the API in some 
> use cases.
> {noformat}
> @state.setter
> def state(self, state):
> """
> Sets the state of this VersionControlInformationDTO.
> The current state of the Process Group, as it relates to the Versioned Flow
> :param state: The state of this VersionControlInformationDTO.
> :type: str
> """
> allowed_values = ["LOCALLY_MODIFIED_DESCENDANT", "LOCALLY_MODIFIED", "STALE", 
> "LOCALLY_MODIFIED_AND_STALE", "UP_TO_DATE"]
> if state not in allowed_values:
> raise ValueError(
> "Invalid value for `state` ({0}), must be one of {1}"
> > .format(state, allowed_values)
> )
> E ValueError: Invalid value for `state` (SYNC_FAILURE), must be one of 
> ['LOCALLY_MODIFIED_DESCENDANT', 'LOCALLY_MODIFIED', 'STALE', 
> 'LOCALLY_MODIFIED_AND_STALE', 'UP_TO_DATE']
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-4859) Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state

[Kevin Doran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Doran reassigned NIFI-4859:
-

Assignee: Kevin Doran

> Swagger Spec VersionControlInformationDTO missing SYNC_FAILURE state
> 
>
> Key: NIFI-4859
> URL: https://issues.apache.org/jira/browse/NIFI-4859
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Flow Versioning
>Affects Versions: 1.5.0
>Reporter: Daniel Chaffelson
>Assignee: Kevin Doran
>Priority: Major
> Fix For: 1.6.0
>
>
> It is possible to get a Versioned Process Group into a SYNC_FAILURE state, 
> but this is not an allowable state in the code generated from the swagger.json
> This prevents versioned objects from being manipulated via the API in some 
> use cases.
> {noformat}
> @state.setter
> def state(self, state):
> """
> Sets the state of this VersionControlInformationDTO.
> The current state of the Process Group, as it relates to the Versioned Flow
> :param state: The state of this VersionControlInformationDTO.
> :type: str
> """
> allowed_values = ["LOCALLY_MODIFIED_DESCENDANT", "LOCALLY_MODIFIED", "STALE", 
> "LOCALLY_MODIFIED_AND_STALE", "UP_TO_DATE"]
> if state not in allowed_values:
> raise ValueError(
> "Invalid value for `state` ({0}), must be one of {1}"
> > .format(state, allowed_values)
> )
> E ValueError: Invalid value for `state` (SYNC_FAILURE), must be one of 
> ['LOCALLY_MODIFIED_DESCENDANT', 'LOCALLY_MODIFIED', 'STALE', 
> 'LOCALLY_MODIFIED_AND_STALE', 'UP_TO_DATE']
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4845) Add JanusGraph put processor

[Matt Gilman (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358447#comment-16358447
 ] 

Matt Gilman commented on NIFI-4845:
---

[~liufucai-inspur] Should be good to go.

> Add JanusGraph put processor
> 
>
> Key: NIFI-4845
> URL: https://issues.apache.org/jira/browse/NIFI-4845
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Fred Liu
>Assignee: Fred Liu
>Priority: Major
>
> Create processor for Reading records from an incoming FlowFile using the 
> provided Record Reader, and writting those records to JanusGraph. And using a 
> JanusGraphControllerService is good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-4845) Add JanusGraph put processor

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman reassigned NIFI-4845:
-

Assignee: Fred Liu

> Add JanusGraph put processor
> 
>
> Key: NIFI-4845
> URL: https://issues.apache.org/jira/browse/NIFI-4845
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Fred Liu
>Assignee: Fred Liu
>Priority: Major
>
> Create processor for Reading records from an incoming FlowFile using the 
> provided Record Reader, and writting those records to JanusGraph. And using a 
> JanusGraphControllerService is good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (MINIFICPP-396) RPG does not pass SSLContextService to site to site client in a consistent way

[marco polo (JIRA)]

marco polo created MINIFICPP-396:


 Summary: RPG does not pass SSLContextService to site to site 
client in a consistent way
 Key: MINIFICPP-396
 URL: https://issues.apache.org/jira/browse/MINIFICPP-396
 Project: NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: marco polo
Assignee: marco polo


The RPG doesn't always pass the SSLContextService in a a consistent way. The 
documentation is confusing, so we should clear this up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-4861) store sql response code in Flowfile attributes

[Alois Gruber (JIRA)]

Alois Gruber created NIFI-4861:
--

 Summary: store sql response code in Flowfile attributes
 Key: NIFI-4861
 URL: https://issues.apache.org/jira/browse/NIFI-4861
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Alois Gruber


for any processor performing SQL transations, it would be helpful to get JDBC 
response codes and if available, the number of affected rows, 

sql.state

sql.errorcode

sql.message

sql.affected_rows



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (NIFI-2575) HiveQL Processors Fail due to invalid JDBC URI resolution when using Zookeeper URI

[Gardella Juan Pablo (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356883#comment-16356883
 ] 

Gardella Juan Pablo edited comment on NIFI-2575 at 2/9/18 12:06 PM:


I had the same issue. It happens in version Nifi 1.3.0 and also in Nifi 1.5.0. 
The problem is the driver, I've updated to 2.1.0 and it works. Another simple 
workaround is using  Hortonworks' 
[nifi-hive-nar|http://nexus-private.hortonworks.com/nexus/content/groups/public/org/apache/nifi/nifi-hive-nar/1.5.0.3.2.0.0-4/],
 it seems Hortonworks is using a patched version.

The root cause I believe is https://issues.apache.org/jira/browse/HIVE-11875, 
so to solve the issue it is required to upgrade Hive driver which include that 
fix.

Notice that affects any processor that uses HiveConnectionPool.

 


was (Author: gardellajuanpablo):
I had the same issue. It happens in version Nifi 1.3.0 and also in Nifi 1.5.0. 
The problem is the driver, I've updated to 2.1.0 and it works. Another simple 
workaround is using  Hortonworks' 
[nifi-hive-nar|http://nexus-private.hortonworks.com/nexus/content/groups/public/org/apache/nifi/nifi-hive-nar/1.5.0.3.2.0.0-4/],
 it seems Hortonworks is using a patched version.

The root cause I believe is https://issues.apache.org/jira/browse/HIVE-11875, 
so to solve the issue it is required to upgrade Hive driver which include that 
fix.

> HiveQL Processors Fail due to invalid JDBC URI resolution when using 
> Zookeeper URI
> --
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Yolanda M. Davis
>Priority: Major
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g. 
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
>  it appears that the JDBC driver does not properly build the the uri in the 
> expected format.  This is because HS2 is storing JDBC parameters in ZK 
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the 
> driver to be able to parse and use those values to configure the connection. 
> However it appears the driver is expecting zookeeper to simply return the 
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was 
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2] 
> org.apache.hive.jdbc.Utils Resolved authority: 
> hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2] 
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with 
> JDBC Uri: 
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2] 
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC 
> Uri: 
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=1;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2] 
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread] 
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:45:14,182 INFO [Curator-Framework-0] 
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] 
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due 
> to java.lang.reflect.UndeclaredThrowableException; rolling back session: 
> 

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358306#comment-16358306
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167210043
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/AbstractDeleteHBase.java
 ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public abstract class AbstractDeleteHBase extends AbstractProcessor {
+protected static final PropertyDescriptor HBASE_CLIENT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("HBase Client Service")
+.description("Specifies the Controller Service to use for 
accessing HBase.")
+.required(true)
+.identifiesControllerService(HBaseClientService.class)
+.build();
+protected static final PropertyDescriptor TABLE_NAME = new 
PropertyDescriptor.Builder()
+.name("Table Name")
+.description("The name of the HBase Table to put data into")
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358305#comment-16358305
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167210016
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/AbstractDeleteHBase.java
 ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public abstract class AbstractDeleteHBase extends AbstractProcessor {
+protected static final PropertyDescriptor HBASE_CLIENT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("HBase Client Service")
+.description("Specifies the Controller Service to use for 
accessing HBase.")
+.required(true)
+.identifiesControllerService(HBaseClientService.class)
+.build();
+protected static final PropertyDescriptor TABLE_NAME = new 
PropertyDescriptor.Builder()
+.name("Table Name")
+.description("The name of the HBase Table to put data into")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+protected static final PropertyDescriptor ROW_ID = new 
PropertyDescriptor.Builder()
+.name("Row Identifier")
+.description("Specifies the Row ID to use when inserting data 
into HBase")
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167210043
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/AbstractDeleteHBase.java
 ---
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public abstract class AbstractDeleteHBase extends AbstractProcessor {
+protected static final PropertyDescriptor HBASE_CLIENT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("HBase Client Service")
+.description("Specifies the Controller Service to use for 
accessing HBase.")
+.required(true)
+.identifiesControllerService(HBaseClientService.class)
+.build();
+protected static final PropertyDescriptor TABLE_NAME = new 
PropertyDescriptor.Builder()
+.name("Table Name")
+.description("The name of the HBase Table to put data into")
--- End diff --

Done


---

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358302#comment-16358302
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167209940
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttribute( attribute = "restart.index", description = "If a delete 
batch fails, it will restart from restart.index" )
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2294: NIFI-3538 Added DeleteHBaseRow

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167209979
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttribute( attribute = "restart.index", description = "If a delete 
batch fails, it will restart from restart.index" )
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
+static final AllowableValue ROW_ID_ATTR = new AllowableValue("attr", 
"FlowFile attributes", "Get the row key from an expression language 
statement.");
+
+static final PropertyDescriptor ROW_ID_LOCATION = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-id-location")
+.displayName("Row ID Location")
+.description("The location of the row ID to use for building 
the delete. Can be from the content or an expression language statement.")
+.required(true)
+.defaultValue(ROW_ID_BODY.getValue())
+.allowableValues(ROW_ID_BODY, ROW_ID_ATTR)
+.addValidator(Validator.VALID)
+.build();
+
+static final PropertyDescriptor FLOWFILE_FETCH_COUNT = new 
PropertyDescriptor.Builder()
+.name("delete-hb-flowfile-fetch-count")
+.displayName("Flowfile Fetch Count")
+.description("The number of flowfiles to fetch per run.")
+.required(true)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("5")
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-ff-count")
+.displayName("Batch Size")
+.description("The number of deletes to send per batch.")
+.required(true)
+.defaultValue("50")
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor KEY_SEPARATOR = new 
PropertyDescriptor.Builder()
+.name("delete-hb-separator")
+.displayName("Delete Row Key Separator")
+.description("The separator character(s) that separate 
multiple row keys " +
+"when multiple row keys are provided in the flowfile 
body")
+.required(true)
+.defaultValue(",")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+static final PropertyDescriptor CHARSET = new 

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358304#comment-16358304
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167209979
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttribute( attribute = "restart.index", description = "If a delete 
batch fails, it will restart from restart.index" )
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
+static final AllowableValue ROW_ID_ATTR = new AllowableValue("attr", 
"FlowFile attributes", "Get the row key from an expression language 
statement.");
+
+static final PropertyDescriptor ROW_ID_LOCATION = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-id-location")
+.displayName("Row ID Location")
+.description("The location of the row ID to use for building 
the delete. Can be from the content or an expression language statement.")
+.required(true)
+.defaultValue(ROW_ID_BODY.getValue())
+.allowableValues(ROW_ID_BODY, ROW_ID_ATTR)
+.addValidator(Validator.VALID)
+.build();
+
+static final PropertyDescriptor FLOWFILE_FETCH_COUNT = new 
PropertyDescriptor.Builder()
+.name("delete-hb-flowfile-fetch-count")
+.displayName("Flowfile Fetch Count")
+.description("The number of flowfiles to fetch per run.")
+.required(true)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("5")
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-ff-count")
+.displayName("Batch Size")
+.description("The number of deletes to send per batch.")
+.required(true)
+.defaultValue("50")
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor KEY_SEPARATOR = new 
PropertyDescriptor.Builder()
+.name("delete-hb-separator")
+.displayName("Delete Row Key Separator")
+.description("The separator character(s) that separate 
multiple row keys " +
+"when multiple row keys are provided in the flowfile 
body")
+.required(true)
+

[jira] [Commented] (NIFI-3538) Add DeleteHBase processor(s)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358303#comment-16358303
 ] 

ASF GitHub Bot commented on NIFI-3538:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2294#discussion_r167209961
  
--- Diff: 
nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors/src/main/java/org/apache/nifi/hbase/DeleteHBaseRow.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.hbase;
+
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@WritesAttribute( attribute = "restart.index", description = "If a delete 
batch fails, it will restart from restart.index" )
+@Tags({ "delete", "hbase" })
+@CapabilityDescription(
+"Delete HBase records individually or in batches. The input can be 
a single row ID in the body, one ID per line, " +
+"row IDs separated by commas or a combination of the two. ")
+public class DeleteHBaseRow extends AbstractDeleteHBase {
+static final AllowableValue ROW_ID_BODY = new AllowableValue("body", 
"FlowFile content", "Get the row key(s) from the flowfile content.");
+static final AllowableValue ROW_ID_ATTR = new AllowableValue("attr", 
"FlowFile attributes", "Get the row key from an expression language 
statement.");
+
+static final PropertyDescriptor ROW_ID_LOCATION = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-id-location")
+.displayName("Row ID Location")
+.description("The location of the row ID to use for building 
the delete. Can be from the content or an expression language statement.")
+.required(true)
+.defaultValue(ROW_ID_BODY.getValue())
+.allowableValues(ROW_ID_BODY, ROW_ID_ATTR)
+.addValidator(Validator.VALID)
+.build();
+
+static final PropertyDescriptor FLOWFILE_FETCH_COUNT = new 
PropertyDescriptor.Builder()
+.name("delete-hb-flowfile-fetch-count")
+.displayName("Flowfile Fetch Count")
+.description("The number of flowfiles to fetch per run.")
+.required(true)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("5")
+.expressionLanguageSupported(false)
+.build();
+
+static final PropertyDescriptor BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("delete-hb-row-ff-count")
--- End diff --

Done


> Add DeleteHBase processor(s)
> 
>
> Key: NIFI-3538
> URL: https://issues.apache.org/jira/browse/NIFI-3538
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Mike Thomsen
>Priority: Major
>
> NiFi currently has processors for storing and retrieving cells/rows in HBase, 
> but there is no mechanism for deleting records and/or tables.
> I'm not sure if a single DeleteHBase processor could accomplish both, that 
> can be discussed under this Jira (and can be split out if necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)