[jira] [Commented] (NIFI-5075) Funnels with no outgoing relationship error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440421#comment-16440421
 ] 

ASF GitHub Bot commented on NIFI-5075:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/2634
  
@ottobackwards Thanks for mentioning about tests. I've added tests for 
Funnels.
@markap14 I've removed caching `hasNonLoopConnection` because it can be 
left unchanged when connections are added while a component is running.
@markobean I grouped some conditions to clearly express that a component is 
treated as a 'source' component, I hope it becomes more understandable now.

Thank you all for giving your comments. I've tested adding brand-new 
funnels and other components as well as loading existing components by 
restarting NiFi. It looks work as expected. Hope it is ready to be merged.


> Funnels with no outgoing relationship error
> ---
>
> Key: NIFI-5075
> URL: https://issues.apache.org/jira/browse/NIFI-5075
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Peter Wicks
>Assignee: Koji Kawamura
>Priority: Major
>
> If a Funnel has no outgoing relationships it will throw an exception when it 
> tries to send FlowFile's to that non-existent relationship.
> Replicate by creating a GenerateFlowFile processor to a Funnel, start the 
> GenerateFlowFile processor and check your log file.
>  
> 2018-04-11 23:53:28,066 ERROR [Timer-Driven Process Thread-31] 
> o.apache.nifi.controller.StandardFunnel 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] failed to process 
> session due to java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known; Processor 
> Administratively Yielded for 1 sec: java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Relationship 
> '' is not known
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:365)
>     at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
>     at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>     at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Relationship '' is not known
>     at 
> org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1935)
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:379)
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:358)
>     ... 9 common frames omitted



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2634: NIFI-5075: Do not execute Funnels with no outgoing connect...

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/2634
  
@ottobackwards Thanks for mentioning about tests. I've added tests for 
Funnels.
@markap14 I've removed caching `hasNonLoopConnection` because it can be 
left unchanged when connections are added while a component is running.
@markobean I grouped some conditions to clearly express that a component is 
treated as a 'source' component, I hope it becomes more understandable now.

Thank you all for giving your comments. I've tested adding brand-new 
funnels and other components as well as loading existing components by 
restarting NiFi. It looks work as expected. Hope it is ready to be merged.


---

[jira] [Commented] (NIFI-5075) Funnels with no outgoing relationship error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440382#comment-16440382
 ] 

ASF GitHub Bot commented on NIFI-5075:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2634#discussion_r181951840
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/ConnectableTask.java
 ---
@@ -103,8 +104,24 @@ private boolean isYielded() {
 return connectable.getYieldExpiration() > 
System.currentTimeMillis();
 }
 
+/**
+ * Make sure processor has work to do. This means that it meets one of 
these criteria:
+ * 
+ * It is annotated with @TriggerWhenEmpty
+ * It has no incoming connections
+ * All incoming connections are self-loops
+ * It has data in an incoming Connection
+ * AND It is not a Funnel without outgoing connections
+ * 
+ * 
+ * @return true if there is work to do, otherwise false
+ */
 private boolean isWorkToDo() {
-return connectable.isTriggerWhenEmpty() || 
!connectable.hasIncomingConnection() || !hasNonLoopConnection || 
Connectables.flowFilesQueued(connectable);
+return connectable.isTriggerWhenEmpty()
+|| !connectable.hasIncomingConnection()
+|| !hasNonLoopConnection
+|| (Connectables.flowFilesQueued(connectable)
--- End diff --

@markap14 Thank you for testing.

I understand what went wrong. I tested with an existing flow including 
Funnel which had an incoming connection from GenerateFlowFile already. Since 
Funnels are not stopped/scheduled, the `hasNonLoopConnection` field only gets 
populated when the Funnel is added. 
`hasNonLoopConnection` becomes `false` for a newly created Funnels, and 
stays false even incoming connections are added later. ConnectableTasks for 
other components than Funnel can be recreated when the component is 
re-scheduled, however, those components should also NOT cache 
`hasNonLoopConnection` because NiFi allows new connections to be added while a 
component is running, without requiring stop/start.

I will update the PR.


> Funnels with no outgoing relationship error
> ---
>
> Key: NIFI-5075
> URL: https://issues.apache.org/jira/browse/NIFI-5075
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Peter Wicks
>Assignee: Koji Kawamura
>Priority: Major
>
> If a Funnel has no outgoing relationships it will throw an exception when it 
> tries to send FlowFile's to that non-existent relationship.
> Replicate by creating a GenerateFlowFile processor to a Funnel, start the 
> GenerateFlowFile processor and check your log file.
>  
> 2018-04-11 23:53:28,066 ERROR [Timer-Driven Process Thread-31] 
> o.apache.nifi.controller.StandardFunnel 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] failed to process 
> session due to java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known; Processor 
> Administratively Yielded for 1 sec: java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Relationship 
> '' is not known
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:365)
>     at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
>     at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>     at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Relationship '' is not known
>     at 
> org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1935)
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:379)
>     

[GitHub] nifi pull request #2634: NIFI-5075: Do not execute Funnels with no outgoing ...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2634#discussion_r181951840
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/ConnectableTask.java
 ---
@@ -103,8 +104,24 @@ private boolean isYielded() {
 return connectable.getYieldExpiration() > 
System.currentTimeMillis();
 }
 
+/**
+ * Make sure processor has work to do. This means that it meets one of 
these criteria:
+ * 
+ * It is annotated with @TriggerWhenEmpty
+ * It has no incoming connections
+ * All incoming connections are self-loops
+ * It has data in an incoming Connection
+ * AND It is not a Funnel without outgoing connections
+ * 
+ * 
+ * @return true if there is work to do, otherwise false
+ */
 private boolean isWorkToDo() {
-return connectable.isTriggerWhenEmpty() || 
!connectable.hasIncomingConnection() || !hasNonLoopConnection || 
Connectables.flowFilesQueued(connectable);
+return connectable.isTriggerWhenEmpty()
+|| !connectable.hasIncomingConnection()
+|| !hasNonLoopConnection
+|| (Connectables.flowFilesQueued(connectable)
--- End diff --

@markap14 Thank you for testing.

I understand what went wrong. I tested with an existing flow including 
Funnel which had an incoming connection from GenerateFlowFile already. Since 
Funnels are not stopped/scheduled, the `hasNonLoopConnection` field only gets 
populated when the Funnel is added. 
`hasNonLoopConnection` becomes `false` for a newly created Funnels, and 
stays false even incoming connections are added later. ConnectableTasks for 
other components than Funnel can be recreated when the component is 
re-scheduled, however, those components should also NOT cache 
`hasNonLoopConnection` because NiFi allows new connections to be added while a 
component is running, without requiring stop/start.

I will update the PR.


---

[jira] [Commented] (NIFI-5085) InvokeHttp does not close the response in all cases

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440294#comment-16440294
 ] 

ASF GitHub Bot commented on NIFI-5085:
--

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi/pull/2637

NIFI-5085 In InvokeHttp, moving the OkHttp Response object to a try w…

…ith resources

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [X] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [X] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [X] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi 
NIFI-5085_closing_OkHttpResponse_in_InvokeHttp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2637


commit a320601fbd20f9ae77044c9babe2728dc89d6426
Author: Joe Percivall 
Date:   2018-04-17T02:36:26Z

NIFI-5085 In InvokeHttp, moving the OkHttp Response object to a try with 
resources




> InvokeHttp does not close the response in all cases
> ---
>
> Key: NIFI-5085
> URL: https://issues.apache.org/jira/browse/NIFI-5085
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0, 1.5.0, 1.6.0
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), 
> the response needs to be closed for every instance of the response. 
> InvokeHttp currently only closes the underlying body stream in the instance 
> of there existing a body[2][3]. The proper way to do it is how the 
> HttpNotificationService does, utilizing a try with resources on the 
> response[4]. 
>  
> I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. 
> I'm still not a 100% sure this is the root cause but one common thing between 
> those OOM errors is repeated socket timeout exceptions in InvokeHttp (and 
> even if it's not, it should still be fixed).
>  
> [1] https://github.com/square/okhttp/issues/2311 
> [2] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822
>  
> [3] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894
>  
> [4] 
> https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5085) InvokeHttp does not close the response in all cases

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-5085:
---
Status: Patch Available  (was: Open)

> InvokeHttp does not close the response in all cases
> ---
>
> Key: NIFI-5085
> URL: https://issues.apache.org/jira/browse/NIFI-5085
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.6.0, 1.5.0, 1.4.0
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), 
> the response needs to be closed for every instance of the response. 
> InvokeHttp currently only closes the underlying body stream in the instance 
> of there existing a body[2][3]. The proper way to do it is how the 
> HttpNotificationService does, utilizing a try with resources on the 
> response[4]. 
>  
> I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. 
> I'm still not a 100% sure this is the root cause but one common thing between 
> those OOM errors is repeated socket timeout exceptions in InvokeHttp (and 
> even if it's not, it should still be fixed).
>  
> [1] https://github.com/square/okhttp/issues/2311 
> [2] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822
>  
> [3] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894
>  
> [4] 
> https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2637: NIFI-5085 In InvokeHttp, moving the OkHttp Response...

[JPercivall]

GitHub user JPercivall opened a pull request:

https://github.com/apache/nifi/pull/2637

NIFI-5085 In InvokeHttp, moving the OkHttp Response object to a try w…

…ith resources

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [X] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [X] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [X] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JPercivall/nifi 
NIFI-5085_closing_OkHttpResponse_in_InvokeHttp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2637


commit a320601fbd20f9ae77044c9babe2728dc89d6426
Author: Joe Percivall 
Date:   2018-04-17T02:36:26Z

NIFI-5085 In InvokeHttp, moving the OkHttp Response object to a try with 
resources




---

[jira] [Commented] (NIFI-4952) JettyWebSocketClient websocket-uri property missing evaluateAttributeExpressions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440117#comment-16440117
 ] 

ASF GitHub Bot commented on NIFI-4952:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2572
  
@claudiu-stanciu I think this will be ready to merge once you get that 
change in.


> JettyWebSocketClient websocket-uri property missing 
> evaluateAttributeExpressions
> 
>
> Key: NIFI-4952
> URL: https://issues.apache.org/jira/browse/NIFI-4952
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Sven Van Hemel
>Priority: Major
>
> The {{websocket-uri property}} of the 
> {{org.apache.nifi.websocket.jetty.JettyWebSocketClient}} class is marked as 
> EL enabled, but its evaluation in the {{startClient()}} method seems to be 
> missing a {{evaluateAttributeExpressions()}} call.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2572: NIFI-4952 Add EL support for JettyWebSocketClient and Jett...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2572
  
@claudiu-stanciu I think this will be ready to merge once you get that 
change in.


---

[jira] [Commented] (NIFI-4975) Add support for MongoDB GridFS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440114#comment-16440114
 ] 

ASF GitHub Bot commented on NIFI-4975:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2546
  
@mattyb149 when I rebased this commit, I added fixes for the integration 
tests so they could run again with the new way of handling EL. 


> Add support for MongoDB GridFS
> --
>
> Key: NIFI-4975
> URL: https://issues.apache.org/jira/browse/NIFI-4975
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> [An overview |https://docs.mongodb.com/manual/core/gridfs/]of what GridFS is.
> Basic CRUD processors for handling GridFS should be added to the MongoDB NAR.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2546: NIFI-4975 Add GridFS processors

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2546
  
@mattyb149 when I rebased this commit, I added fixes for the integration 
tests so they could run again with the new way of handling EL. 


---

[jira] [Updated] (NIFI-5085) InvokeHttp does not close the response in all cases

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-5085:
---
Affects Version/s: (was: 0.7.4)
   (was: 1.3.0)
   (was: 1.0.1)
   (was: 1.1.1)
   (was: 1.2.0)
   (was: 0.7.1)
   (was: 1.1.0)
   (was: 0.6.1)
   (was: 0.7.0)
   (was: 0.5.1)
   (was: 0.4.1)
   (was: 0.6.0)
   (was: 0.5.0)
   (was: 0.4.0)
   (was: 1.0.0)

> InvokeHttp does not close the response in all cases
> ---
>
> Key: NIFI-5085
> URL: https://issues.apache.org/jira/browse/NIFI-5085
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0, 1.5.0, 1.6.0
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), 
> the response needs to be closed for every instance of the response. 
> InvokeHttp currently only closes the underlying body stream in the instance 
> of there existing a body[2][3]. The proper way to do it is how the 
> HttpNotificationService does, utilizing a try with resources on the 
> response[4]. 
>  
> I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. 
> I'm still not a 100% sure this is the root cause but one common thing between 
> those OOM errors is repeated socket timeout exceptions in InvokeHttp (and 
> even if it's not, it should still be fixed).
>  
> [1] https://github.com/square/okhttp/issues/2311 
> [2] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822
>  
> [3] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894
>  
> [4] 
> https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5085) InvokeHttp does not close the response in all cases

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440082#comment-16440082
 ] 

Joseph Percivall commented on NIFI-5085:


Removed the "affectsVersion" for all prior to 1.4.0 as I believe this is a 
regression hit as part of NIFI-2162

> InvokeHttp does not close the response in all cases
> ---
>
> Key: NIFI-5085
> URL: https://issues.apache.org/jira/browse/NIFI-5085
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0, 1.5.0, 1.6.0
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), 
> the response needs to be closed for every instance of the response. 
> InvokeHttp currently only closes the underlying body stream in the instance 
> of there existing a body[2][3]. The proper way to do it is how the 
> HttpNotificationService does, utilizing a try with resources on the 
> response[4]. 
>  
> I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. 
> I'm still not a 100% sure this is the root cause but one common thing between 
> those OOM errors is repeated socket timeout exceptions in InvokeHttp (and 
> even if it's not, it should still be fixed).
>  
> [1] https://github.com/square/okhttp/issues/2311 
> [2] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822
>  
> [3] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894
>  
> [4] 
> https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5085) InvokeHttp does not close the response in all cases

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440074#comment-16440074
 ] 

Joseph Percivall commented on NIFI-5085:


Was curious why I didn't do this in the first place, turns out "closeable" 
wasn't added to the response until after I refactored it[1]. Not sure when the 
requirement for always closing it was added though. At the very least, the try 
with resources should've been added in NIFI-2162 when we updated to OkHttp 3.x.

 

[1] 
https://github.com/square/okhttp/commit/3699d5c9fd0ad78fc52e3ea317951f9d485f656f

> InvokeHttp does not close the response in all cases
> ---
>
> Key: NIFI-5085
> URL: https://issues.apache.org/jira/browse/NIFI-5085
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.4.0, 0.5.0, 0.6.0, 0.4.1, 0.5.1, 0.7.0, 0.6.1, 
> 1.1.0, 0.7.1, 1.2.0, 1.1.1, 1.0.1, 1.3.0, 1.4.0, 0.7.4, 1.5.0, 1.6.0
>Reporter: Joseph Percivall
>Assignee: Joseph Percivall
>Priority: Major
>
> As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), 
> the response needs to be closed for every instance of the response. 
> InvokeHttp currently only closes the underlying body stream in the instance 
> of there existing a body[2][3]. The proper way to do it is how the 
> HttpNotificationService does, utilizing a try with resources on the 
> response[4]. 
>  
> I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. 
> I'm still not a 100% sure this is the root cause but one common thing between 
> those OOM errors is repeated socket timeout exceptions in InvokeHttp (and 
> even if it's not, it should still be fixed).
>  
> [1] https://github.com/square/okhttp/issues/2311 
> [2] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822
>  
> [3] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894
>  
> [4] 
> https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5086) Many usages of AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch aren't closed properly

[Joseph Percivall (JIRA)]

Joseph Percivall created NIFI-5086:
--

 Summary: Many usages of 
AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch aren't closed 
properly
 Key: NIFI-5086
 URL: https://issues.apache.org/jira/browse/NIFI-5086
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Joseph Percivall


Similar to NIFI-5085, all OkHttp responses must be closed[1]. 
"AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch"  returns the 
actual OkHttp response. All implementing processors should properly close this. 
Processors that need to be updated:
 * FetchElasticsearchHttp[2]
 * PutElasticsearchHttp[3]
 * PutElasticsearchHttpRecord[4]
 * QueryElasticsearchHttp[5]
 * ScrollElasticsearchHtt[6][7]

[1] [https://github.com/square/okhttp/issues/2311] 
[2] 
[https://github.com/apache/nifi/blob/9736cb9d338b611ff3a096d97fd439cf8b8bbac3/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java#L223
][3][https://github.com/apache/nifi/blob/e916594b69601bce58e045ba8ae2acf6af66eb46/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L337]
[4] 
[https://github.com/apache/nifi/blob/6b34d3bea9a1fd0923024018d73c3c0b0a807a67/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L400]
[5] 
[https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L287
][6] 
[https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L256
][7][https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L269]




 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5086) Many usages of AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch aren't closed properly

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-5086:
---
Description: 
Similar to NIFI-5085, all OkHttp responses must be closed[1]. 
"AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch"  returns the 
actual OkHttp response. All implementing processors should properly close this. 
Processors that need to be updated:
 * FetchElasticsearchHttp[2]
 * PutElasticsearchHttp[3]
 * PutElasticsearchHttpRecord[4]
 * QueryElasticsearchHttp[5]
 * ScrollElasticsearchHtt[6][7]

[1] [https://github.com/square/okhttp/issues/2311] 
 [2] 
[[https://github.com/apache/nifi/blob/9736cb9d338b611ff3a096d97fd439cf8b8bbac3/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java#L223]
[3][https://github.com/apache/nifi/blob/e916594b69601bce58e045ba8ae2acf6af66eb46/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L337]
 [4] 
[https://github.com/apache/nifi/blob/6b34d3bea9a1fd0923024018d73c3c0b0a807a67/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L400]
 [5] 
[[https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L287]
[6] 
[[https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L256]
[7][https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L269]

 

  was:
Similar to NIFI-5085, all OkHttp responses must be closed[1]. 
"AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch"  returns the 
actual OkHttp response. All implementing processors should properly close this. 
Processors that need to be updated:
 * FetchElasticsearchHttp[2]
 * PutElasticsearchHttp[3]
 * PutElasticsearchHttpRecord[4]
 * QueryElasticsearchHttp[5]
 * ScrollElasticsearchHtt[6][7]

[1] [https://github.com/square/okhttp/issues/2311] 
[2] 
[https://github.com/apache/nifi/blob/9736cb9d338b611ff3a096d97fd439cf8b8bbac3/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java#L223
][3][https://github.com/apache/nifi/blob/e916594b69601bce58e045ba8ae2acf6af66eb46/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L337]
[4] 
[https://github.com/apache/nifi/blob/6b34d3bea9a1fd0923024018d73c3c0b0a807a67/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L400]
[5] 
[https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L287
][6] 
[https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L256
][7][https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L269]




 


> Many usages of AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch 
> aren't closed properly
> ---
>
> Key: NIFI-5086
> URL: https://issues.apache.org/jira/browse/NIFI-5086
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Percivall
>Priority: Major
>
> Similar to NIFI-5085, all OkHttp responses must be closed[1]. 
> "AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch"  returns the 
> actual OkHttp response. All implementing processors should properly close 
> this. Processors that need to be updated:
>  * FetchElasticsearchHttp[2]
>  * PutElasticsearchHttp[3]
>  * PutElasticsearchHttpRecord[4]
>  * QueryElasticsearchHttp[5]
>  * ScrollElasticsearchHtt[6][7]
> [1] 

[jira] [Created] (NIFI-5085) InvokeHttp does not close the response in all cases

[Joseph Percivall (JIRA)]

Joseph Percivall created NIFI-5085:
--

 Summary: InvokeHttp does not close the response in all cases
 Key: NIFI-5085
 URL: https://issues.apache.org/jira/browse/NIFI-5085
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 1.6.0, 1.5.0, 0.7.4, 1.4.0, 1.3.0, 1.0.1, 1.1.1, 1.2.0, 
0.7.1, 1.1.0, 0.6.1, 0.7.0, 0.5.1, 0.4.1, 0.6.0, 0.5.0, 0.4.0, 1.0.0
Reporter: Joseph Percivall
Assignee: Joseph Percivall


As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), the 
response needs to be closed for every instance of the response. InvokeHttp 
currently only closes the underlying body stream in the instance of there 
existing a body[2][3]. The proper way to do it is how the 
HttpNotificationService does, utilizing a try with resources on the 
response[4]. 

 

I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. I'm 
still not a 100% sure this is the root cause but one common thing between those 
OOM errors is repeated socket timeout exceptions in InvokeHttp (and even if 
it's not, it should still be fixed).

 

[1] https://github.com/square/okhttp/issues/2311 
[2] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822
 
[3] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894
 
[4] 
https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5062) Remove hbase-client from nifi-hbase-bundle pom

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439984#comment-16439984
 ] 

ASF GitHub Bot commented on NIFI-5062:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2636
  
@zenfenan have you had a chance to run this against a live HBase cluster?


> Remove hbase-client from nifi-hbase-bundle pom
> --
>
> Key: NIFI-5062
> URL: https://issues.apache.org/jira/browse/NIFI-5062
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Bryan Bende
>Assignee: Sivaprasanna Sethuraman
>Priority: Minor
>
> Since the hbase-client dependency should be coming from the client service 
> implementation, we shouldn't need to specify it here:
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hbase-bundle/pom.xml#L43]
> We should also make hbase.version a property that can be overridden at build 
> time, rather than hard-coding it here:
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/pom.xml#L73



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2636: NIFI-5062: Removed hbase-client dependecy from hbase bundl...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2636
  
@zenfenan have you had a chance to run this against a live HBase cluster?


---

[jira] [Commented] (NIFI-4516) Add FetchSolr processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439936#comment-16439936
 ] 

ASF GitHub Bot commented on NIFI-4516:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2517
  
Have you had a chance to add the missing provenance tracking?


> Add FetchSolr processor
> ---
>
> Key: NIFI-4516
> URL: https://issues.apache.org/jira/browse/NIFI-4516
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Johannes Peter
>Assignee: Johannes Peter
>Priority: Major
>  Labels: features
>
> The processor shall be capable 
> * to query Solr within a workflow,
> * to make use of standard functionalities of Solr such as faceting, 
> highlighting, result grouping, etc.,
> * to make use of NiFis expression language to build Solr queries, 
> * to handle results as records.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2517: NIFI-4516 FetchSolr Processor

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2517
  
Have you had a chance to add the missing provenance tracking?


---

[jira] [Updated] (NIFI-4862) Copy original FlowFile attributes to output FlowFiles at SelectHiveQL processor

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-4862:
---
Status: Resolved  (was: Closed)

> Copy original FlowFile attributes to output FlowFiles at SelectHiveQL 
> processor
> ---
>
> Key: NIFI-4862
> URL: https://issues.apache.org/jira/browse/NIFI-4862
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Jakub Leś
>Assignee: Matt Burgess
>Priority: Minor
> Fix For: 1.7.0
>
> Attachments: 
> 0001-NIFI-4862-Add-Copy-original-attributtes-to-SelectHiv.patch
>
>
> Hi, 
> Please add "Copy original attributes" to processor SelectHiveQL. Thanks to 
> that we can use HttpRequest and HttpResponse to synchronize fetching query 
> result in webservice.
>  
> UPDATED:
> SelectHiveQL creates new FlowFiles from Hive query result sets. When it also 
> has incoming FlowFiles, it should create new FlowFiles from the input 
> FlowFile, so that it can copy original FlowFile attributes to output 
> FlowFiles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4185) Add XML record reader & writer services

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439880#comment-16439880
 ] 

ASF GitHub Bot commented on NIFI-4185:
--

Github user JohannesDaniel commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2587#discussion_r181851731
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordReader.java
 ---
@@ -0,0 +1,502 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.IOException;
+import java.io.InputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.function.Supplier;
+
+public class XMLRecordReader implements RecordReader {
+
+private final ComponentLog logger;
+private final RecordSchema schema;
+private final String recordName;
+private final String attributePrefix;
+private final String contentFieldName;
+
+// thread safety required?
+private StartElement currentRecordStartTag;
+
+private final XMLEventReader xmlEventReader;
+
+private final Supplier LAZY_DATE_FORMAT;
+private final Supplier LAZY_TIME_FORMAT;
+private final Supplier LAZY_TIMESTAMP_FORMAT;
+
+public XMLRecordReader(InputStream in, RecordSchema schema, String 
rootName, String recordName, String attributePrefix, String contentFieldName,
+   final String dateFormat, final String 
timeFormat, final String timestampFormat, final ComponentLog logger) throws 
MalformedRecordException {
+this.schema = schema;
+this.recordName = recordName;
+this.attributePrefix = attributePrefix;
+this.contentFieldName = contentFieldName;
+this.logger = logger;
+
+final DateFormat df = dateFormat == null ? null : 
DataTypeUtils.getDateFormat(dateFormat);
+final DateFormat tf = timeFormat == null ? null : 
DataTypeUtils.getDateFormat(timeFormat);
+final DateFormat tsf = timestampFormat == null ? null : 
DataTypeUtils.getDateFormat(timestampFormat);
+
+LAZY_DATE_FORMAT = () -> df;
+LAZY_TIME_FORMAT = () -> tf;
+LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+try {
+final XMLInputFactory xmlInputFactory = 
XMLInputFactory.newInstance();
+
+// Avoid namespace replacements
+
xmlInputFactory.setProperty(XMLInputFactory.IS_NAMESPACE_AWARE, false);
+
+xmlEventReader = xmlInputFactory.createXMLEventReader(in);
+final StartElement rootTag = 

[GitHub] nifi pull request #2587: NIFI-4185 Add XML Record Reader

[JohannesDaniel]

Github user JohannesDaniel commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2587#discussion_r181851731
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordReader.java
 ---
@@ -0,0 +1,502 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.IOException;
+import java.io.InputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.function.Supplier;
+
+public class XMLRecordReader implements RecordReader {
+
+private final ComponentLog logger;
+private final RecordSchema schema;
+private final String recordName;
+private final String attributePrefix;
+private final String contentFieldName;
+
+// thread safety required?
+private StartElement currentRecordStartTag;
+
+private final XMLEventReader xmlEventReader;
+
+private final Supplier LAZY_DATE_FORMAT;
+private final Supplier LAZY_TIME_FORMAT;
+private final Supplier LAZY_TIMESTAMP_FORMAT;
+
+public XMLRecordReader(InputStream in, RecordSchema schema, String 
rootName, String recordName, String attributePrefix, String contentFieldName,
+   final String dateFormat, final String 
timeFormat, final String timestampFormat, final ComponentLog logger) throws 
MalformedRecordException {
+this.schema = schema;
+this.recordName = recordName;
+this.attributePrefix = attributePrefix;
+this.contentFieldName = contentFieldName;
+this.logger = logger;
+
+final DateFormat df = dateFormat == null ? null : 
DataTypeUtils.getDateFormat(dateFormat);
+final DateFormat tf = timeFormat == null ? null : 
DataTypeUtils.getDateFormat(timeFormat);
+final DateFormat tsf = timestampFormat == null ? null : 
DataTypeUtils.getDateFormat(timestampFormat);
+
+LAZY_DATE_FORMAT = () -> df;
+LAZY_TIME_FORMAT = () -> tf;
+LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+try {
+final XMLInputFactory xmlInputFactory = 
XMLInputFactory.newInstance();
+
+// Avoid namespace replacements
+
xmlInputFactory.setProperty(XMLInputFactory.IS_NAMESPACE_AWARE, false);
+
+xmlEventReader = xmlInputFactory.createXMLEventReader(in);
+final StartElement rootTag = getNextStartTag();
+
+// root tag validation
+if (rootName != null && 
!rootName.equals(rootTag.getName().toString())) {
+final StringBuffer message = new StringBuffer();
+

[jira] [Commented] (MINIFICPP-454) Builds don't effectively pull system OpenSSL for rpi

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439847#comment-16439847
 ] 

ASF GitHub Bot commented on MINIFICPP-454:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/299


> Builds don't effectively pull system OpenSSL for rpi
> 
>
> Key: MINIFICPP-454
> URL: https://issues.apache.org/jira/browse/MINIFICPP-454
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: marco polo
>Assignee: marco polo
>Priority: Blocker
> Fix For: 0.5.0
>
>
> Builds don't effectively pull system OpenSSLfor rpi



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (MINIFICPP-454) Builds don't effectively pull system OpenSSL for rpi

[Aldrin Piri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aldrin Piri resolved MINIFICPP-454.
---
Resolution: Fixed

> Builds don't effectively pull system OpenSSL for rpi
> 
>
> Key: MINIFICPP-454
> URL: https://issues.apache.org/jira/browse/MINIFICPP-454
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: marco polo
>Assignee: marco polo
>Priority: Blocker
> Fix For: 0.5.0
>
>
> Builds don't effectively pull system OpenSSLfor rpi



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #299: MINIFICPP-454: Fix apt-get install statem...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/299


---

[jira] [Commented] (MINIFICPP-454) Builds don't effectively pull system OpenSSL for rpi

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439836#comment-16439836
 ] 

ASF GitHub Bot commented on MINIFICPP-454:
--

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/299
  
Was able to verify in a docker container, looks good.


> Builds don't effectively pull system OpenSSL for rpi
> 
>
> Key: MINIFICPP-454
> URL: https://issues.apache.org/jira/browse/MINIFICPP-454
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: marco polo
>Assignee: marco polo
>Priority: Blocker
> Fix For: 0.5.0
>
>
> Builds don't effectively pull system OpenSSLfor rpi



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp issue #299: MINIFICPP-454: Fix apt-get install statement

[apiri]

Github user apiri commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/299
  
Was able to verify in a docker container, looks good.


---

[jira] [Commented] (MINIFICPP-454) Builds don't effectively pull system OpenSSL for rpi

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439795#comment-16439795
 ] 

ASF GitHub Bot commented on MINIFICPP-454:
--

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/299#discussion_r181833527
  
--- Diff: debian.sh ---
@@ -31,11 +27,12 @@ bootstrap_cmake(){
   sudo apt-get -y install cmake
 }
 build_deps(){
+  sudo apt-get -y update 
   ## need to account for debian
-  sudo apt-get install libssl1.0-dev > /dev/null
+  sudo apt-get install -y libssl1.0-dev > /dev/null
   RETVAL=$?
-  if ["$RETVAL" -ne "0"]; then  
- sudo apt-get install libssl-dev > /dev/null
+  if [ "$RETVAL" -ne "0"]; then  
--- End diff --

Missing a space before the closing bracket


> Builds don't effectively pull system OpenSSL for rpi
> 
>
> Key: MINIFICPP-454
> URL: https://issues.apache.org/jira/browse/MINIFICPP-454
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: marco polo
>Assignee: marco polo
>Priority: Blocker
> Fix For: 0.5.0
>
>
> Builds don't effectively pull system OpenSSLfor rpi



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-446) Implement escape/unescape HTML3 EL functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439796#comment-16439796
 ] 

ASF GitHub Bot commented on MINIFICPP-446:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/297


> Implement escape/unescape HTML3 EL functions
> 
>
> Key: MINIFICPP-446
> URL: https://issues.apache.org/jira/browse/MINIFICPP-446
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> [Encode/Decode 
> Functions|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#encode]
>  * 
> [escapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapehtml3]
>  * 
> [unescapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapehtml3]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #297: MINIFICPP-446 Add escape/unescape HTML3 E...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/297


---

[GitHub] nifi-minifi-cpp pull request #299: MINIFICPP-454: Fix apt-get install statem...

[apiri]

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/299#discussion_r181833527
  
--- Diff: debian.sh ---
@@ -31,11 +27,12 @@ bootstrap_cmake(){
   sudo apt-get -y install cmake
 }
 build_deps(){
+  sudo apt-get -y update 
   ## need to account for debian
-  sudo apt-get install libssl1.0-dev > /dev/null
+  sudo apt-get install -y libssl1.0-dev > /dev/null
   RETVAL=$?
-  if ["$RETVAL" -ne "0"]; then  
- sudo apt-get install libssl-dev > /dev/null
+  if [ "$RETVAL" -ne "0"]; then  
--- End diff --

Missing a space before the closing bracket


---

[jira] [Commented] (MINIFICPP-446) Implement escape/unescape HTML3 EL functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439792#comment-16439792
 ] 

ASF GitHub Bot commented on MINIFICPP-446:
--

Github user minifirocks commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/297
  
looks good


> Implement escape/unescape HTML3 EL functions
> 
>
> Key: MINIFICPP-446
> URL: https://issues.apache.org/jira/browse/MINIFICPP-446
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> [Encode/Decode 
> Functions|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#encode]
>  * 
> [escapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapehtml3]
>  * 
> [unescapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapehtml3]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-454) Builds don't effectively pull system OpenSSL for rpi

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439790#comment-16439790
 ] 

ASF GitHub Bot commented on MINIFICPP-454:
--

Github user minifirocks commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/299
  
@phrocker looks good, please merge to 1 commit


> Builds don't effectively pull system OpenSSL for rpi
> 
>
> Key: MINIFICPP-454
> URL: https://issues.apache.org/jira/browse/MINIFICPP-454
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: marco polo
>Assignee: marco polo
>Priority: Blocker
> Fix For: 0.5.0
>
>
> Builds don't effectively pull system OpenSSLfor rpi



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp issue #297: MINIFICPP-446 Add escape/unescape HTML3 EL funct...

[minifirocks]

Github user minifirocks commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/297
  
looks good


---

[GitHub] nifi-minifi-cpp issue #299: MINIFICPP-454: Fix apt-get install statement

[minifirocks]

Github user minifirocks commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/299
  
@phrocker looks good, please merge to 1 commit


---

[GitHub] nifi-minifi-cpp pull request #298: MINIFICPP-453 Added external build of zli...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/298


---

[jira] [Commented] (MINIFICPP-453) Support external build/static link of zlib

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439789#comment-16439789
 ] 

ASF GitHub Bot commented on MINIFICPP-453:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/298


> Support external build/static link of zlib
> --
>
> Key: MINIFICPP-453
> URL: https://issues.apache.org/jira/browse/MINIFICPP-453
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> To improve portability in certain cases, support building and 
> statically-linking zlib as an external project.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5066) Support enable and disable component action when multiple components selected or when selecting a process group.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439708#comment-16439708
 ] 

ASF GitHub Bot commented on NIFI-5066:
--

Github user andrewmlim commented on the issue:

https://github.com/apache/nifi/pull/2633
  
Changes look good.  Enable/disable and start/stop buttons behave 
consistently.  This is minor, but it would be nice if all of these buttons were 
greyed out (or unavailable in the context menu) when there were no components 
available i.e. when the process group is empty.


> Support enable and disable component action when multiple components selected 
> or when selecting a process group.
> 
>
> Key: NIFI-5066
> URL: https://issues.apache.org/jira/browse/NIFI-5066
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.5.0
>Reporter: Matthew Clarke
>Assignee: Matt Gilman
>Priority: Major
>
> Currently NiFi validates all processors that are in a STOPPED state.  To 
> reduce impact when flows contain very large numbers of STOPPED processors, 
> users should be disabling these STOPPED processors.  NiFi's "Enable" and 
> "Disable" buttons do not support being used when more then one processor is 
> selected.  When needing to enable or disable large numbers of processors, 
> this is less then ideal. The Enable and Disable buttons should work similar 
> to how the Start and Stop buttons work.
> Have multiple components selected or a process group selected.  Select 
> "Enable" or "Disabled" button.  Any eligible component (those that are not 
> running) should be either enabled or disabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2633: NIFI-5066: Allow global enable/disable component requests

[andrewmlim]

Github user andrewmlim commented on the issue:

https://github.com/apache/nifi/pull/2633
  
Changes look good.  Enable/disable and start/stop buttons behave 
consistently.  This is minor, but it would be nice if all of these buttons were 
greyed out (or unavailable in the context menu) when there were no components 
available i.e. when the process group is empty.


---

[jira] [Created] (MINIFICPP-456) Docker build currently broken for master

[Aldrin Piri (JIRA)]

Aldrin Piri created MINIFICPP-456:
-

 Summary: Docker build currently broken for master
 Key: MINIFICPP-456
 URL: https://issues.apache.org/jira/browse/MINIFICPP-456
 Project: NiFi MiNiFi C++
  Issue Type: Bug
Affects Versions: 0.5.0
Reporter: Aldrin Piri
 Fix For: 0.5.0


The docker build is currently not working against master (commit: 
ea6bc178cf63d7436f44fd3f814e7a1c1422ec18, applied 11 April)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4997) Actions taken on process groups do not appear in flow configuration history

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-4997:
--
   Resolution: Fixed
Fix Version/s: 1.7.0
   Status: Resolved  (was: Patch Available)

> Actions taken on process groups do not appear in flow configuration history
> ---
>
> Key: NIFI-4997
> URL: https://issues.apache.org/jira/browse/NIFI-4997
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.4.0
> Environment: CentOS 7, Docker
>Reporter: Craig Becker
>Assignee: Matt Gilman
>Priority: Major
> Fix For: 1.7.0
>
>
> Selecting a process group and executing a stop or start action does not cause 
> anything to be logged in the flow configuration history. The current behavior 
> makes it impossible to trace who/when a process group was turned off. 
>  
> The expected/desired behavior would be to either add a log entry per 
> processor that was stopped/started, or to log that the process group was 
> stopped/started.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5075) Funnels with no outgoing relationship error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439617#comment-16439617
 ] 

ASF GitHub Bot commented on NIFI-5075:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2634#discussion_r181792251
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/ConnectableTask.java
 ---
@@ -103,8 +104,24 @@ private boolean isYielded() {
 return connectable.getYieldExpiration() > 
System.currentTimeMillis();
 }
 
+/**
+ * Make sure processor has work to do. This means that it meets one of 
these criteria:
+ * 
+ * It is annotated with @TriggerWhenEmpty
+ * It has no incoming connections
+ * All incoming connections are self-loops
+ * It has data in an incoming Connection
+ * AND It is not a Funnel without outgoing connections
+ * 
+ * 
+ * @return true if there is work to do, otherwise false
+ */
 private boolean isWorkToDo() {
-return connectable.isTriggerWhenEmpty() || 
!connectable.hasIncomingConnection() || !hasNonLoopConnection || 
Connectables.flowFilesQueued(connectable);
+return connectable.isTriggerWhenEmpty()
+|| !connectable.hasIncomingConnection()
+|| !hasNonLoopConnection
+|| (Connectables.flowFilesQueued(connectable)
--- End diff --

@ijokarumawak unfortunately, this doesn't appear to fully address the 
issue. It appears that the clause above this one, "!hasNonLoopConnection" is 
evaluating to true in my case (simple GenerateFlowFile -> Funnel).

I think I would actually recommend just adding a block of code to the 
beginning of this method like:
```
if (connectable.getConnectableType() == ConnectableType.FUNNEL) {
  return !connectable.getConnections().isEmpty() && 
connectable.hasIncomingConnection() && hasNonLoopConnection;
} else {
  // do what we already do
}
```

I recommend this because Funnels are a special case, since they are really 
intended to be 'notional' and never stopped or started. They are always 
running. And for Funnels, in particular, it only really makes sense to run if 
there is in fact an incoming connection (rather than processors where we want 
to run even if there isn't an incoming connection), AND If there is at least 
one outgoing connection (funnels don't support auto-terminating relationships).


> Funnels with no outgoing relationship error
> ---
>
> Key: NIFI-5075
> URL: https://issues.apache.org/jira/browse/NIFI-5075
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Peter Wicks
>Assignee: Koji Kawamura
>Priority: Major
>
> If a Funnel has no outgoing relationships it will throw an exception when it 
> tries to send FlowFile's to that non-existent relationship.
> Replicate by creating a GenerateFlowFile processor to a Funnel, start the 
> GenerateFlowFile processor and check your log file.
>  
> 2018-04-11 23:53:28,066 ERROR [Timer-Driven Process Thread-31] 
> o.apache.nifi.controller.StandardFunnel 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] failed to process 
> session due to java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known; Processor 
> Administratively Yielded for 1 sec: java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Relationship 
> '' is not known
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:365)
>     at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
>     at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>     at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:748)
> 

[GitHub] nifi pull request #2634: NIFI-5075: Do not execute Funnels with no outgoing ...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2634#discussion_r181792251
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/ConnectableTask.java
 ---
@@ -103,8 +104,24 @@ private boolean isYielded() {
 return connectable.getYieldExpiration() > 
System.currentTimeMillis();
 }
 
+/**
+ * Make sure processor has work to do. This means that it meets one of 
these criteria:
+ * 
+ * It is annotated with @TriggerWhenEmpty
+ * It has no incoming connections
+ * All incoming connections are self-loops
+ * It has data in an incoming Connection
+ * AND It is not a Funnel without outgoing connections
+ * 
+ * 
+ * @return true if there is work to do, otherwise false
+ */
 private boolean isWorkToDo() {
-return connectable.isTriggerWhenEmpty() || 
!connectable.hasIncomingConnection() || !hasNonLoopConnection || 
Connectables.flowFilesQueued(connectable);
+return connectable.isTriggerWhenEmpty()
+|| !connectable.hasIncomingConnection()
+|| !hasNonLoopConnection
+|| (Connectables.flowFilesQueued(connectable)
--- End diff --

@ijokarumawak unfortunately, this doesn't appear to fully address the 
issue. It appears that the clause above this one, "!hasNonLoopConnection" is 
evaluating to true in my case (simple GenerateFlowFile -> Funnel).

I think I would actually recommend just adding a block of code to the 
beginning of this method like:
```
if (connectable.getConnectableType() == ConnectableType.FUNNEL) {
  return !connectable.getConnections().isEmpty() && 
connectable.hasIncomingConnection() && hasNonLoopConnection;
} else {
  // do what we already do
}
```

I recommend this because Funnels are a special case, since they are really 
intended to be 'notional' and never stopped or started. They are always 
running. And for Funnels, in particular, it only really makes sense to run if 
there is in fact an incoming connection (rather than processors where we want 
to run even if there isn't an incoming connection), AND If there is at least 
one outgoing connection (funnels don't support auto-terminating relationships).


---

[jira] [Commented] (NIFI-5062) Remove hbase-client from nifi-hbase-bundle pom

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439567#comment-16439567
 ] 

ASF GitHub Bot commented on NIFI-5062:
--

GitHub user zenfenan opened a pull request:

https://github.com/apache/nifi/pull/2636

NIFI-5062: Removed hbase-client dependecy from hbase bundle

Since the `MockHBaseClientService` and `TestPutHBaseRecord` extensively 
uses `Bytes` class, I have created a new class here and literally copied the 
relevant methods from the actual `Bytes` class to this one.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zenfenan/nifi NIFI-5062

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2636


commit a34ca612464a115d03e7e7ca6ef7e08c06045d4f
Author: zenfenan 
Date:   2018-04-14T06:04:07Z

NIFI-5062: Removed hbase-client dependecy from hbase bundle




> Remove hbase-client from nifi-hbase-bundle pom
> --
>
> Key: NIFI-5062
> URL: https://issues.apache.org/jira/browse/NIFI-5062
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Bryan Bende
>Assignee: Sivaprasanna Sethuraman
>Priority: Minor
>
> Since the hbase-client dependency should be coming from the client service 
> implementation, we shouldn't need to specify it here:
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hbase-bundle/pom.xml#L43]
> We should also make hbase.version a property that can be overridden at build 
> time, rather than hard-coding it here:
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/pom.xml#L73



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2636: NIFI-5062: Removed hbase-client dependecy from hbas...

[zenfenan]

GitHub user zenfenan opened a pull request:

https://github.com/apache/nifi/pull/2636

NIFI-5062: Removed hbase-client dependecy from hbase bundle

Since the `MockHBaseClientService` and `TestPutHBaseRecord` extensively 
uses `Bytes` class, I have created a new class here and literally copied the 
relevant methods from the actual `Bytes` class to this one.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zenfenan/nifi NIFI-5062

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2636


commit a34ca612464a115d03e7e7ca6ef7e08c06045d4f
Author: zenfenan 
Date:   2018-04-14T06:04:07Z

NIFI-5062: Removed hbase-client dependecy from hbase bundle




---

[jira] [Commented] (NIFI-5075) Funnels with no outgoing relationship error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439558#comment-16439558
 ] 

ASF GitHub Bot commented on NIFI-5075:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2634#discussion_r181775911
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/ConnectableTask.java
 ---
@@ -103,8 +104,24 @@ private boolean isYielded() {
 return connectable.getYieldExpiration() > 
System.currentTimeMillis();
 }
 
+/**
+ * Make sure processor has work to do. This means that it meets one of 
these criteria:
+ * 
+ * It is annotated with @TriggerWhenEmpty
+ * It has no incoming connections
+ * All incoming connections are self-loops
+ * It has data in an incoming Connection
+ * AND It is not a Funnel without outgoing connections
+ * 
+ * 
+ * @return true if there is work to do, otherwise false
+ */
 private boolean isWorkToDo() {
-return connectable.isTriggerWhenEmpty() || 
!connectable.hasIncomingConnection() || !hasNonLoopConnection || 
Connectables.flowFilesQueued(connectable);
+return connectable.isTriggerWhenEmpty()
+|| !connectable.hasIncomingConnection()
--- End diff --

If connectable.hasIncomingConnection() is false, then that indicates that 
the component is a 'source' component, and doesn't depend on incoming data to 
be triggered, so it is said to have work to do. Similarly, if all incoming 
connections are self-looping connections, then it is a source component and is 
said to have work to do.


> Funnels with no outgoing relationship error
> ---
>
> Key: NIFI-5075
> URL: https://issues.apache.org/jira/browse/NIFI-5075
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Peter Wicks
>Assignee: Koji Kawamura
>Priority: Major
>
> If a Funnel has no outgoing relationships it will throw an exception when it 
> tries to send FlowFile's to that non-existent relationship.
> Replicate by creating a GenerateFlowFile processor to a Funnel, start the 
> GenerateFlowFile processor and check your log file.
>  
> 2018-04-11 23:53:28,066 ERROR [Timer-Driven Process Thread-31] 
> o.apache.nifi.controller.StandardFunnel 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] failed to process 
> session due to java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known; Processor 
> Administratively Yielded for 1 sec: java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Relationship 
> '' is not known
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:365)
>     at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
>     at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>     at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Relationship '' is not known
>     at 
> org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1935)
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:379)
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:358)
>     ... 9 common frames omitted



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2634: NIFI-5075: Do not execute Funnels with no outgoing ...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2634#discussion_r181775911
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/ConnectableTask.java
 ---
@@ -103,8 +104,24 @@ private boolean isYielded() {
 return connectable.getYieldExpiration() > 
System.currentTimeMillis();
 }
 
+/**
+ * Make sure processor has work to do. This means that it meets one of 
these criteria:
+ * 
+ * It is annotated with @TriggerWhenEmpty
+ * It has no incoming connections
+ * All incoming connections are self-loops
+ * It has data in an incoming Connection
+ * AND It is not a Funnel without outgoing connections
+ * 
+ * 
+ * @return true if there is work to do, otherwise false
+ */
 private boolean isWorkToDo() {
-return connectable.isTriggerWhenEmpty() || 
!connectable.hasIncomingConnection() || !hasNonLoopConnection || 
Connectables.flowFilesQueued(connectable);
+return connectable.isTriggerWhenEmpty()
+|| !connectable.hasIncomingConnection()
--- End diff --

If connectable.hasIncomingConnection() is false, then that indicates that 
the component is a 'source' component, and doesn't depend on incoming data to 
be triggered, so it is said to have work to do. Similarly, if all incoming 
connections are self-looping connections, then it is a source component and is 
said to have work to do.


---

[GitHub] nifi-minifi-cpp issue #295: MINFICPP-403: Flow Meta tagging

[minifirocks]

Github user minifirocks commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/295
  
@phrocker please let me know whether you have more comments. Thanks.


---

[jira] [Updated] (MINIFICPP-455) Implement escape/unescape HTML4 EL functions

[Andrew Christianson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Christianson updated MINIFICPP-455:
--
Description: 
[Encode/Decode 
Functions|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#encode]
 * 
[escapeHtml4|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapehtml4]
 * 
[unescapeHtml4|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapehtml4]

  was:
[Encode/Decode 
Functions|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#encode]
 * 
[escapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapehtml3]
 * 
[unescapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapehtml3]


> Implement escape/unescape HTML4 EL functions
> 
>
> Key: MINIFICPP-455
> URL: https://issues.apache.org/jira/browse/MINIFICPP-455
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> [Encode/Decode 
> Functions|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#encode]
>  * 
> [escapeHtml4|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapehtml4]
>  * 
> [unescapeHtml4|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapehtml4]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (MINIFICPP-455) Implement escape/unescape HTML4 EL functions

[Andrew Christianson (JIRA)]

Andrew Christianson created MINIFICPP-455:
-

 Summary: Implement escape/unescape HTML4 EL functions
 Key: MINIFICPP-455
 URL: https://issues.apache.org/jira/browse/MINIFICPP-455
 Project: NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Andrew Christianson
Assignee: Andrew Christianson


[Encode/Decode 
Functions|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#encode]
 * 
[escapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapehtml3]
 * 
[unescapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapehtml3]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5041) Add convenient SPNEGO/Kerberos authentication support to LivySessionController

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439487#comment-16439487
 ] 

ASF GitHub Bot commented on NIFI-5041:
--

Github user peter-toth commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2630#discussion_r181746461
  
--- Diff: pom.xml ---
@@ -94,6 +94,7 @@
 1.7.25
 0.7.1
 9.4.3.v20170317
+4.5.5
--- End diff --

@mattyb149, could you please review my change and let me know if I did the 
right thing? Any suggestions are welcome.

I changed the httpclient dependency to provided so it won't get into all 
NARs depending on nifi-hadoop-utils. Unfortunately httpclient seems to change a 
lot from version to version and the minimum required version that my changes 
can work with is 4.4.1.

All my additions to hadoop-utils are in new classes, so I believe even a 
hadoop-utils dependent NAR that specifies a lower version of httpclient will be 
fine as long as it doesn't start to use my classes directly.


> Add convenient SPNEGO/Kerberos authentication support to LivySessionController
> --
>
> Key: NIFI-5041
> URL: https://issues.apache.org/jira/browse/NIFI-5041
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.5.0
>Reporter: Peter Toth
>Priority: Minor
>
> Livy requires SPNEGO/Kerberos authentication on a secured cluster. Initiating 
> such an authentication from NiFi is a viable by providing a 
> java.security.auth.login.config system property 
> (https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/lab/part6.html),
>  but this is a bit cumbersome and needs kinit running outside of NiFi.
> An alternative and more sophisticated solution would be to do the SPNEGO 
> negotiation programmatically.
>  * This solution would add some new properties to the LivySessionController 
> to fetch kerberos principal and password/keytab
>  * Add the required HTTP Negotiate header (with an SPNEGO token) to the 
> HttpURLConnection to do the authentication programmatically 
> (https://tools.ietf.org/html/rfc4559)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2630: NIFI-5041 Adds SPNEGO authentication to LivySession...

[peter-toth]

Github user peter-toth commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2630#discussion_r181746461
  
--- Diff: pom.xml ---
@@ -94,6 +94,7 @@
 1.7.25
 0.7.1
 9.4.3.v20170317
+4.5.5
--- End diff --

@mattyb149, could you please review my change and let me know if I did the 
right thing? Any suggestions are welcome.

I changed the httpclient dependency to provided so it won't get into all 
NARs depending on nifi-hadoop-utils. Unfortunately httpclient seems to change a 
lot from version to version and the minimum required version that my changes 
can work with is 4.4.1.

All my additions to hadoop-utils are in new classes, so I believe even a 
hadoop-utils dependent NAR that specifies a lower version of httpclient will be 
fine as long as it doesn't start to use my classes directly.


---

[jira] [Assigned] (NIFI-5062) Remove hbase-client from nifi-hbase-bundle pom

[Sivaprasanna Sethuraman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sivaprasanna Sethuraman reassigned NIFI-5062:
-

Assignee: Sivaprasanna Sethuraman

> Remove hbase-client from nifi-hbase-bundle pom
> --
>
> Key: NIFI-5062
> URL: https://issues.apache.org/jira/browse/NIFI-5062
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.6.0
>Reporter: Bryan Bende
>Assignee: Sivaprasanna Sethuraman
>Priority: Minor
>
> Since the hbase-client dependency should be coming from the client service 
> implementation, we shouldn't need to specify it here:
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hbase-bundle/pom.xml#L43]
> We should also make hbase.version a property that can be overridden at build 
> time, rather than hard-coding it here:
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service/pom.xml#L73



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5075) Funnels with no outgoing relationship error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439451#comment-16439451
 ] 

ASF GitHub Bot commented on NIFI-5075:
--

Github user markobean commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2634#discussion_r181735366
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/ConnectableTask.java
 ---
@@ -103,8 +104,24 @@ private boolean isYielded() {
 return connectable.getYieldExpiration() > 
System.currentTimeMillis();
 }
 
+/**
+ * Make sure processor has work to do. This means that it meets one of 
these criteria:
+ * 
+ * It is annotated with @TriggerWhenEmpty
+ * It has no incoming connections
+ * All incoming connections are self-loops
+ * It has data in an incoming Connection
+ * AND It is not a Funnel without outgoing connections
+ * 
+ * 
+ * @return true if there is work to do, otherwise false
+ */
 private boolean isWorkToDo() {
-return connectable.isTriggerWhenEmpty() || 
!connectable.hasIncomingConnection() || !hasNonLoopConnection || 
Connectables.flowFilesQueued(connectable);
+return connectable.isTriggerWhenEmpty()
+|| !connectable.hasIncomingConnection()
--- End diff --

This logic seems inverted unnecessarily. Why is 
'connectable.hasIncomingConnection()' negated? This leads to the method 
returning true when there are no incoming connections. Similarly, is it the 
intent to indicate no work to do if the only incoming connections are looped 
connections?


> Funnels with no outgoing relationship error
> ---
>
> Key: NIFI-5075
> URL: https://issues.apache.org/jira/browse/NIFI-5075
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Peter Wicks
>Assignee: Koji Kawamura
>Priority: Major
>
> If a Funnel has no outgoing relationships it will throw an exception when it 
> tries to send FlowFile's to that non-existent relationship.
> Replicate by creating a GenerateFlowFile processor to a Funnel, start the 
> GenerateFlowFile processor and check your log file.
>  
> 2018-04-11 23:53:28,066 ERROR [Timer-Driven Process Thread-31] 
> o.apache.nifi.controller.StandardFunnel 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] 
> StandardFunnel[id=b868231c-0162-1000-571c-ae3e7d15d848] failed to process 
> session due to java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known; Processor 
> Administratively Yielded for 1 sec: java.lang.RuntimeException: 
> java.lang.IllegalArgumentException: Relationship '' is not known
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Relationship 
> '' is not known
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:365)
>     at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
>     at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>     at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Relationship '' is not known
>     at 
> org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1935)
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:379)
>     at 
> org.apache.nifi.controller.StandardFunnel.onTrigger(StandardFunnel.java:358)
>     ... 9 common frames omitted



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2634: NIFI-5075: Do not execute Funnels with no outgoing ...

[markobean]

Github user markobean commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2634#discussion_r181735366
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/tasks/ConnectableTask.java
 ---
@@ -103,8 +104,24 @@ private boolean isYielded() {
 return connectable.getYieldExpiration() > 
System.currentTimeMillis();
 }
 
+/**
+ * Make sure processor has work to do. This means that it meets one of 
these criteria:
+ * 
+ * It is annotated with @TriggerWhenEmpty
+ * It has no incoming connections
+ * All incoming connections are self-loops
+ * It has data in an incoming Connection
+ * AND It is not a Funnel without outgoing connections
+ * 
+ * 
+ * @return true if there is work to do, otherwise false
+ */
 private boolean isWorkToDo() {
-return connectable.isTriggerWhenEmpty() || 
!connectable.hasIncomingConnection() || !hasNonLoopConnection || 
Connectables.flowFilesQueued(connectable);
+return connectable.isTriggerWhenEmpty()
+|| !connectable.hasIncomingConnection()
--- End diff --

This logic seems inverted unnecessarily. Why is 
'connectable.hasIncomingConnection()' negated? This leads to the method 
returning true when there are no incoming connections. Similarly, is it the 
intent to indicate no work to do if the only incoming connections are looped 
connections?


---

[jira] [Commented] (NIFI-4185) Add XML record reader & writer services

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439106#comment-16439106
 ] 

ASF GitHub Bot commented on NIFI-4185:
--

Github user JohannesDaniel commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2587#discussion_r181653767
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordReader.java
 ---
@@ -0,0 +1,502 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.IOException;
+import java.io.InputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.function.Supplier;
+
+public class XMLRecordReader implements RecordReader {
+
+private final ComponentLog logger;
+private final RecordSchema schema;
+private final String recordName;
+private final String attributePrefix;
+private final String contentFieldName;
+
+// thread safety required?
+private StartElement currentRecordStartTag;
+
+private final XMLEventReader xmlEventReader;
+
+private final Supplier LAZY_DATE_FORMAT;
+private final Supplier LAZY_TIME_FORMAT;
+private final Supplier LAZY_TIMESTAMP_FORMAT;
+
+public XMLRecordReader(InputStream in, RecordSchema schema, String 
rootName, String recordName, String attributePrefix, String contentFieldName,
+   final String dateFormat, final String 
timeFormat, final String timestampFormat, final ComponentLog logger) throws 
MalformedRecordException {
+this.schema = schema;
+this.recordName = recordName;
+this.attributePrefix = attributePrefix;
+this.contentFieldName = contentFieldName;
+this.logger = logger;
+
+final DateFormat df = dateFormat == null ? null : 
DataTypeUtils.getDateFormat(dateFormat);
+final DateFormat tf = timeFormat == null ? null : 
DataTypeUtils.getDateFormat(timeFormat);
+final DateFormat tsf = timestampFormat == null ? null : 
DataTypeUtils.getDateFormat(timestampFormat);
+
+LAZY_DATE_FORMAT = () -> df;
+LAZY_TIME_FORMAT = () -> tf;
+LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+try {
+final XMLInputFactory xmlInputFactory = 
XMLInputFactory.newInstance();
+
+// Avoid namespace replacements
+
xmlInputFactory.setProperty(XMLInputFactory.IS_NAMESPACE_AWARE, false);
+
+xmlEventReader = xmlInputFactory.createXMLEventReader(in);
+final StartElement rootTag = 

[GitHub] nifi pull request #2587: NIFI-4185 Add XML Record Reader

[JohannesDaniel]

Github user JohannesDaniel commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2587#discussion_r181653767
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/xml/XMLRecordReader.java
 ---
@@ -0,0 +1,502 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.xml;
+
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.type.ArrayDataType;
+import org.apache.nifi.serialization.record.type.RecordDataType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import javax.xml.stream.XMLEventReader;
+import javax.xml.stream.XMLInputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.events.Attribute;
+import javax.xml.stream.events.Characters;
+import javax.xml.stream.events.StartElement;
+import javax.xml.stream.events.XMLEvent;
+import java.io.IOException;
+import java.io.InputStream;
+import java.text.DateFormat;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.function.Supplier;
+
+public class XMLRecordReader implements RecordReader {
+
+private final ComponentLog logger;
+private final RecordSchema schema;
+private final String recordName;
+private final String attributePrefix;
+private final String contentFieldName;
+
+// thread safety required?
+private StartElement currentRecordStartTag;
+
+private final XMLEventReader xmlEventReader;
+
+private final Supplier LAZY_DATE_FORMAT;
+private final Supplier LAZY_TIME_FORMAT;
+private final Supplier LAZY_TIMESTAMP_FORMAT;
+
+public XMLRecordReader(InputStream in, RecordSchema schema, String 
rootName, String recordName, String attributePrefix, String contentFieldName,
+   final String dateFormat, final String 
timeFormat, final String timestampFormat, final ComponentLog logger) throws 
MalformedRecordException {
+this.schema = schema;
+this.recordName = recordName;
+this.attributePrefix = attributePrefix;
+this.contentFieldName = contentFieldName;
+this.logger = logger;
+
+final DateFormat df = dateFormat == null ? null : 
DataTypeUtils.getDateFormat(dateFormat);
+final DateFormat tf = timeFormat == null ? null : 
DataTypeUtils.getDateFormat(timeFormat);
+final DateFormat tsf = timestampFormat == null ? null : 
DataTypeUtils.getDateFormat(timestampFormat);
+
+LAZY_DATE_FORMAT = () -> df;
+LAZY_TIME_FORMAT = () -> tf;
+LAZY_TIMESTAMP_FORMAT = () -> tsf;
+
+try {
+final XMLInputFactory xmlInputFactory = 
XMLInputFactory.newInstance();
+
+// Avoid namespace replacements
+
xmlInputFactory.setProperty(XMLInputFactory.IS_NAMESPACE_AWARE, false);
+
+xmlEventReader = xmlInputFactory.createXMLEventReader(in);
+final StartElement rootTag = getNextStartTag();
+
+// root tag validation
+if (rootName != null && 
!rootName.equals(rootTag.getName().toString())) {
+final StringBuffer message = new StringBuffer();
+