[jira] [Commented] (NIFI-5024) Deadlock in ExecuteStreamCommand processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473785#comment-16473785
 ] 

ASF GitHub Bot commented on NIFI-5024:
--

Github user nsanglar commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2594#discussion_r187841413
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
 ---
@@ -382,10 +389,10 @@ public void onTrigger(ProcessContext context, final 
ProcessSession session) thro
 Map attributes = new HashMap<>();
 
 final StringBuilder strBldr = new StringBuilder();
-try {
-String line;
-while ((line = bufferedReader.readLine()) != null) {
-strBldr.append(line).append("\n");
+try (final InputStream is = new FileInputStream(errorOut)) {
--- End diff --

Strange, the link is not working for me either anymore. There are some 
pointers on this stackoverflow thread: 

https://stackoverflow.com/questions/16983372/why-does-process-hang-if-the-parent-does-not-consume-stdout-stderr-in-java
The difference is that by writing to a file, you bypass the OS pipe for 
stderr, and therefore it does not block.
By the way if you execute `TestExecuteStreamCommand` with the old code, you 
will be able to reproduce the deadlock even with unit tests.


> Deadlock in ExecuteStreamCommand processor
> --
>
> Key: NIFI-5024
> URL: https://issues.apache.org/jira/browse/NIFI-5024
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Nicolas Sanglard
>Priority: Minor
> Attachments: Screen Shot 2018-03-28 at 15.34.36.png, Screen Shot 
> 2018-03-28 at 15.36.02.png
>
>
> Whenever a process is producing too much output on stderr, the current 
> implementation will run into a deadlock between the JVM and the unix process 
> started by the ExecuteStreamCommand.
> This is a known issue that is fully described here: 
> [http://java-monitor.com/forum/showthread.php?t=4067]
> In short:
>  * If the process produces too much stderr that is not consumed by 
> ExecuteStreamCommand, it will block until data is read.
>  * The current processor implementation is reading from stderr only after 
> having called process.waitFor()
>  * Thus, the two processes are waiting for each other and fall into a deadlock
>  
>  
> The following setup will lead to a deadlock:
>  
> A jar containing the following Main application:
> {code:java}
> object Main extends App {
>   import scala.collection.JavaConverters._
>   val str = 
> Source.fromInputStream(this.getClass.getResourceAsStream("/1mb.txt")).mkString
>   System.err.println(str)
> }
> {code}
> The following NiFi Flow:
>  
> !Screen Shot 2018-03-28 at 15.34.36.png!
>  
> Configuration for ExecuteStreamCommand:
>  
> !Screen Shot 2018-03-28 at 15.36.02.png!
>  
> The script is simply containing a call to the jar: 
> {code:java}
> java -jar stderr.jar
> {code}
>  
> Once the processor calls the script, it appears as "processing" indefinitely 
> and can only be stopped by restarting NiFi.
>  
> I already have a running solution that I will publish as soon as possible.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5024) Deadlock in ExecuteStreamCommand processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473786#comment-16473786
 ] 

ASF GitHub Bot commented on NIFI-5024:
--

Github user nsanglar commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2594#discussion_r187841438
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
 ---
@@ -351,6 +350,16 @@ public void onTrigger(ProcessContext context, final 
ProcessSession session) thro
 builder.directory(dir);
 builder.redirectInput(Redirect.PIPE);
 builder.redirectOutput(Redirect.PIPE);
+final File errorOut;
+try {
+errorOut = File.createTempFile("out", null);
+errorOut.deleteOnExit();
--- End diff --

Good point, I will do that


> Deadlock in ExecuteStreamCommand processor
> --
>
> Key: NIFI-5024
> URL: https://issues.apache.org/jira/browse/NIFI-5024
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Nicolas Sanglard
>Priority: Minor
> Attachments: Screen Shot 2018-03-28 at 15.34.36.png, Screen Shot 
> 2018-03-28 at 15.36.02.png
>
>
> Whenever a process is producing too much output on stderr, the current 
> implementation will run into a deadlock between the JVM and the unix process 
> started by the ExecuteStreamCommand.
> This is a known issue that is fully described here: 
> [http://java-monitor.com/forum/showthread.php?t=4067]
> In short:
>  * If the process produces too much stderr that is not consumed by 
> ExecuteStreamCommand, it will block until data is read.
>  * The current processor implementation is reading from stderr only after 
> having called process.waitFor()
>  * Thus, the two processes are waiting for each other and fall into a deadlock
>  
>  
> The following setup will lead to a deadlock:
>  
> A jar containing the following Main application:
> {code:java}
> object Main extends App {
>   import scala.collection.JavaConverters._
>   val str = 
> Source.fromInputStream(this.getClass.getResourceAsStream("/1mb.txt")).mkString
>   System.err.println(str)
> }
> {code}
> The following NiFi Flow:
>  
> !Screen Shot 2018-03-28 at 15.34.36.png!
>  
> Configuration for ExecuteStreamCommand:
>  
> !Screen Shot 2018-03-28 at 15.36.02.png!
>  
> The script is simply containing a call to the jar: 
> {code:java}
> java -jar stderr.jar
> {code}
>  
> Once the processor calls the script, it appears as "processing" indefinitely 
> and can only be stopped by restarting NiFi.
>  
> I already have a running solution that I will publish as soon as possible.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2594: NIFI-5024 Resolves deadlock in ExecuteStreamCommand...

[nsanglar]

Github user nsanglar commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2594#discussion_r187841438
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
 ---
@@ -351,6 +350,16 @@ public void onTrigger(ProcessContext context, final 
ProcessSession session) thro
 builder.directory(dir);
 builder.redirectInput(Redirect.PIPE);
 builder.redirectOutput(Redirect.PIPE);
+final File errorOut;
+try {
+errorOut = File.createTempFile("out", null);
+errorOut.deleteOnExit();
--- End diff --

Good point, I will do that


---

[GitHub] nifi pull request #2594: NIFI-5024 Resolves deadlock in ExecuteStreamCommand...

[nsanglar]

Github user nsanglar commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2594#discussion_r187841413
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
 ---
@@ -382,10 +389,10 @@ public void onTrigger(ProcessContext context, final 
ProcessSession session) thro
 Map attributes = new HashMap<>();
 
 final StringBuilder strBldr = new StringBuilder();
-try {
-String line;
-while ((line = bufferedReader.readLine()) != null) {
-strBldr.append(line).append("\n");
+try (final InputStream is = new FileInputStream(errorOut)) {
--- End diff --

Strange, the link is not working for me either anymore. There are some 
pointers on this stackoverflow thread: 

https://stackoverflow.com/questions/16983372/why-does-process-hang-if-the-parent-does-not-consume-stdout-stderr-in-java
The difference is that by writing to a file, you bypass the OS pipe for 
stderr, and therefore it does not block.
By the way if you execute `TestExecuteStreamCommand` with the old code, you 
will be able to reproduce the deadlock even with unit tests.


---

[jira] [Comment Edited] (NIFI-5044) SelectHiveQL accept only one statement

[Ed Berezitsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473374#comment-16473374
 ] 

Ed Berezitsky edited comment on NIFI-5044 at 5/14/18 12:51 AM:
---

[~disoardi], [~pvillard] and [~mattyb149],

PR created, take a look please, and let me know if everything is OK.


was (Author: bdesert):
[~disoardi], [~pvillard] and [~mattyb149],

PR create, take a look please, and let me know if everything is OK.

> SelectHiveQL accept only one statement
> --
>
> Key: NIFI-5044
> URL: https://issues.apache.org/jira/browse/NIFI-5044
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Davide Isoardi
>Assignee: Ed Berezitsky
>Priority: Critical
>  Labels: features, patch, pull-request-available
> Attachments: 
> 0001-NIFI-5044-SelectHiveQL-accept-only-one-statement.patch
>
>
> In [this 
> |[https://github.com/apache/nifi/commit/bbc714e73ba245de7bc32fd9958667c847101f7d]
>  ] commit claims to add support to running multiple statements both on 
> SelectHiveQL and PutHiveQL; instead, it adds only the support to PutHiveQL, 
> so SelectHiveQL still lacks this important feature. @Matt Burgess, I saw that 
> you worked on that, is there any reason for this? If not, can we support it?
> If I try to execute this query:
> {quote}set hive.vectorized.execution.enabled = false; SELECT * FROM table_name
> {quote}
> I have this error:
>  
> {quote}2018-04-05 13:35:40,572 ERROR [Timer-Driven Process Thread-146] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=243d4c17-b1fe-14af--ee8ce15e] Unable to execute 
> HiveQL select query set hive.vectorized.execution.enabled = false; SELECT * 
> FROM table_name for 
> StandardFlowFileRecord[uuid=0e035558-07ce-473b-b0d4-ac00b8b1df93,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1522824912161-2753, 
> container=default, section=705], offset=838441, 
> length=25],offset=0,name=cliente_attributi.csv,size=25] due to 
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!; routing to failure: {}
>  org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:305)
>  at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2529)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:275)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
>  Caused by: java.sql.SQLException: The query did not generate a result set!
>  at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:438)
>  at 
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
>  at 
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:293)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4914) Implement record model processor for Pulsar, i.e. ConsumePulsarRecord, PublishPulsarRecord

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473608#comment-16473608
 ] 

ASF GitHub Bot commented on NIFI-4914:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2614
  
@david-streamlio Ok, I think I figured out what happened. At some point, it 
looks like you accidentally did a pull on upstream master into your branch. The 
fact that you keep having over 200 commits even with rebasing against master 
very strongly suggests that. What I did to verify was I checked out your 
branch, pushed it to my fork and saw 210ish commits with a merge conflict into 
my master. So I locally rebased against master, did a forced push and it 
dropped it down to ~7 commits.

So carefully follow the four steps I gave you:

1. git checkout master
2. git pull upstream master
3. git checkout NIFI-4914
4. git rebase master

**Make sure** that `upstream` is changed to whatever you call `apache/nifi` 
on github.com. The do:

`git push origin --force NIFI-4914` and it should all clear up to a few 
commits.


> Implement record model processor for Pulsar, i.e. ConsumePulsarRecord, 
> PublishPulsarRecord
> --
>
> Key: NIFI-4914
> URL: https://issues.apache.org/jira/browse/NIFI-4914
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: David Kjerrumgaard
>Priority: Minor
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Create record-based processors for Apache Pulsar 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2614: Added Apache Pulsar Processors and Controller Service

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2614
  
@david-streamlio Ok, I think I figured out what happened. At some point, it 
looks like you accidentally did a pull on upstream master into your branch. The 
fact that you keep having over 200 commits even with rebasing against master 
very strongly suggests that. What I did to verify was I checked out your 
branch, pushed it to my fork and saw 210ish commits with a merge conflict into 
my master. So I locally rebased against master, did a forced push and it 
dropped it down to ~7 commits.

So carefully follow the four steps I gave you:

1. git checkout master
2. git pull upstream master
3. git checkout NIFI-4914
4. git rebase master

**Make sure** that `upstream` is changed to whatever you call `apache/nifi` 
on github.com. The do:

`git push origin --force NIFI-4914` and it should all clear up to a few 
commits.


---

[jira] [Commented] (NIFI-4914) Implement record model processor for Pulsar, i.e. ConsumePulsarRecord, PublishPulsarRecord

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473605#comment-16473605
 ] 

ASF GitHub Bot commented on NIFI-4914:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2614
  
Huh, I just checked out your latest branch and it's clean as a whistle WRT 
rebasing and appears 100% up to date w/ the upstream master. So not sure what 
the heck is going on here. Try doing a `git push origin --force NIFI-4914`. 
That should forcefully knock out whatever is going wrong.


> Implement record model processor for Pulsar, i.e. ConsumePulsarRecord, 
> PublishPulsarRecord
> --
>
> Key: NIFI-4914
> URL: https://issues.apache.org/jira/browse/NIFI-4914
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: David Kjerrumgaard
>Priority: Minor
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Create record-based processors for Apache Pulsar 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2614: Added Apache Pulsar Processors and Controller Service

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2614
  
Huh, I just checked out your latest branch and it's clean as a whistle WRT 
rebasing and appears 100% up to date w/ the upstream master. So not sure what 
the heck is going on here. Try doing a `git push origin --force NIFI-4914`. 
That should forcefully knock out whatever is going wrong.


---

[GitHub] nifi issue #2614: Added Apache Pulsar Processors and Controller Service

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2614
  
Ok, I'll take a look. Worst comes to worst, we cherry-pick the heck out of 
your commits and play a game of hacky sack with Git and commits to get you back 
on track.


---

[jira] [Commented] (NIFI-4907) Provenance authorization refactoring

[Mark Bean (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473570#comment-16473570
 ] 

Mark Bean commented on NIFI-4907:
-

WIP code is at: https://github.com/markobean/nifi/tree/NIFI-4907

> Provenance authorization refactoring
> 
>
> Key: NIFI-4907
> URL: https://issues.apache.org/jira/browse/NIFI-4907
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Mark Bean
>Assignee: Mark Bean
>Priority: Major
>
> Currently, the 'view the data' component policy is too tightly coupled with 
> Provenance queries. The 'query provenance' policy should be the only policy 
> required for viewing Provenance query results. Both 'view the component' and 
> 'view the data' policies should be used to refine the appropriate visibility 
> of event details - but not the event itself.
> 1) Component Visibility
> The authorization of Provenance events is inconsistent with the behavior of 
> the graph. For example, if a user does not have 'view the component' policy, 
> the graph shows this component as a "black box" (no details such as name, 
> UUID, etc.) However, when querying Provenance, this component will show up 
> including the Component Type and the Component Name. This is in effect a 
> violation of the policy. These component details should be obscured in the 
> Provenance event displayed if user does not have the appropriate 'view the 
> component' policy.
> 2) Data Visibility
> For a Provenance query, all events should be visible as long as the user 
> performing the query belongs to the 'query provenance' global policy. As 
> mentioned above, some information about the component may be obscured 
> depending on 'view the component' policy, but the event itself should be 
> visible. Additionally, details of the event (clicking the View Details "i" 
> icon) should only be accessible if the user belongs to the 'view the data' 
> policy for the affected component. If the user is not in the appropriate 
> 'view the data' policy, a popup warning should be displayed indicating the 
> reason details are not visible with more specific detail than the current 
> "Contact the system administrator".
> 3) Lineage Graphs
> As with the Provenance table view recommendation above, the lineage graph 
> should display all events. Currently, if the lineage graph includes an event 
> belonging to a component which the user does not have 'view the data', it is 
> shown on the graph as "UNKNOWN". As with Data Visibility mentioned above, the 
> graph should indicate the event type as long as the user is in the 'view the 
> component'. Subsequent "View Details" on the event should only be visible if 
> the user is in the 'view the data' policy.
> In summary, for Provenance query results and lineage graphs, all events 
> should be shown. Component Name and Component Type information should be 
> conditionally visible depending on the corresponding component policy 'view 
> the component' policy. Event details including Provenance event type and 
> FlowFile information should be conditionally available depending on the 
> corresponding component policy 'view the data'. Inability to display event 
> details should provide feedback to the user indicating the reason.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4907) Provenance authorization refactoring

[Mark Bean (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473565#comment-16473565
 ] 

Mark Bean commented on NIFI-4907:
-

I'm returning to NIFI-4907. It's been on the back burner for a while, and I'm 
ready to get it wrapped up now. I ran into an issue though. I based the new 
'view provenance' Component Policy on the 'view the data' policy. However, the 
authorization chain is not implemented (correctly). And, I'm wondering if it 
needs to be implemented at all.
 
In general, I'm not entirely clear on the authorization chain. By usage, I know 
'view the data' requires both the user and the cluster nodes (if clustered) in 
the policy. I believe this is part of the authorization chain. Can you explain 
why this is necessary? Secondly, would this be required for the 'view 
provenance' Component Policy as well?
 
I have not tested my code in a cluster environment, but it is working just fine 
in a single instance. It's the ProvenanceDataAuthorizableTest unit tests 
(replicated in large part from DataAuthorizableTest class) that are failing. 
And, I suspect if I put this in a cluster environment, it may not behave as 
needed - if authorization chaining is required.

> Provenance authorization refactoring
> 
>
> Key: NIFI-4907
> URL: https://issues.apache.org/jira/browse/NIFI-4907
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Mark Bean
>Priority: Major
>
> Currently, the 'view the data' component policy is too tightly coupled with 
> Provenance queries. The 'query provenance' policy should be the only policy 
> required for viewing Provenance query results. Both 'view the component' and 
> 'view the data' policies should be used to refine the appropriate visibility 
> of event details - but not the event itself.
> 1) Component Visibility
> The authorization of Provenance events is inconsistent with the behavior of 
> the graph. For example, if a user does not have 'view the component' policy, 
> the graph shows this component as a "black box" (no details such as name, 
> UUID, etc.) However, when querying Provenance, this component will show up 
> including the Component Type and the Component Name. This is in effect a 
> violation of the policy. These component details should be obscured in the 
> Provenance event displayed if user does not have the appropriate 'view the 
> component' policy.
> 2) Data Visibility
> For a Provenance query, all events should be visible as long as the user 
> performing the query belongs to the 'query provenance' global policy. As 
> mentioned above, some information about the component may be obscured 
> depending on 'view the component' policy, but the event itself should be 
> visible. Additionally, details of the event (clicking the View Details "i" 
> icon) should only be accessible if the user belongs to the 'view the data' 
> policy for the affected component. If the user is not in the appropriate 
> 'view the data' policy, a popup warning should be displayed indicating the 
> reason details are not visible with more specific detail than the current 
> "Contact the system administrator".
> 3) Lineage Graphs
> As with the Provenance table view recommendation above, the lineage graph 
> should display all events. Currently, if the lineage graph includes an event 
> belonging to a component which the user does not have 'view the data', it is 
> shown on the graph as "UNKNOWN". As with Data Visibility mentioned above, the 
> graph should indicate the event type as long as the user is in the 'view the 
> component'. Subsequent "View Details" on the event should only be visible if 
> the user is in the 'view the data' policy.
> In summary, for Provenance query results and lineage graphs, all events 
> should be shown. Component Name and Component Type information should be 
> conditionally visible depending on the corresponding component policy 'view 
> the component' policy. Event details including Provenance event type and 
> FlowFile information should be conditionally available depending on the 
> corresponding component policy 'view the data'. Inability to display event 
> details should provide feedback to the user indicating the reason.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-4907) Provenance authorization refactoring

[Mark Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Bean reassigned NIFI-4907:
---

Assignee: Mark Bean

> Provenance authorization refactoring
> 
>
> Key: NIFI-4907
> URL: https://issues.apache.org/jira/browse/NIFI-4907
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Mark Bean
>Assignee: Mark Bean
>Priority: Major
>
> Currently, the 'view the data' component policy is too tightly coupled with 
> Provenance queries. The 'query provenance' policy should be the only policy 
> required for viewing Provenance query results. Both 'view the component' and 
> 'view the data' policies should be used to refine the appropriate visibility 
> of event details - but not the event itself.
> 1) Component Visibility
> The authorization of Provenance events is inconsistent with the behavior of 
> the graph. For example, if a user does not have 'view the component' policy, 
> the graph shows this component as a "black box" (no details such as name, 
> UUID, etc.) However, when querying Provenance, this component will show up 
> including the Component Type and the Component Name. This is in effect a 
> violation of the policy. These component details should be obscured in the 
> Provenance event displayed if user does not have the appropriate 'view the 
> component' policy.
> 2) Data Visibility
> For a Provenance query, all events should be visible as long as the user 
> performing the query belongs to the 'query provenance' global policy. As 
> mentioned above, some information about the component may be obscured 
> depending on 'view the component' policy, but the event itself should be 
> visible. Additionally, details of the event (clicking the View Details "i" 
> icon) should only be accessible if the user belongs to the 'view the data' 
> policy for the affected component. If the user is not in the appropriate 
> 'view the data' policy, a popup warning should be displayed indicating the 
> reason details are not visible with more specific detail than the current 
> "Contact the system administrator".
> 3) Lineage Graphs
> As with the Provenance table view recommendation above, the lineage graph 
> should display all events. Currently, if the lineage graph includes an event 
> belonging to a component which the user does not have 'view the data', it is 
> shown on the graph as "UNKNOWN". As with Data Visibility mentioned above, the 
> graph should indicate the event type as long as the user is in the 'view the 
> component'. Subsequent "View Details" on the event should only be visible if 
> the user is in the 'view the data' policy.
> In summary, for Provenance query results and lineage graphs, all events 
> should be shown. Component Name and Component Type information should be 
> conditionally visible depending on the corresponding component policy 'view 
> the component' policy. Event details including Provenance event type and 
> FlowFile information should be conditionally available depending on the 
> corresponding component policy 'view the data'. Inability to display event 
> details should provide feedback to the user indicating the reason.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2614: Added Apache Pulsar Processors and Controller Service

[david-streamlio]

Github user david-streamlio commented on the issue:

https://github.com/apache/nifi/pull/2614
  
Well, that didn't go as well as we had hoped.


---

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473473#comment-16473473
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2672
  
@pvillard31 can you take a look?


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2672: NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguag...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2672
  
@pvillard31 can you take a look?


---

[jira] [Commented] (NIFI-5170) Update Grok to 0.1.9

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473470#comment-16473470
 ] 

ASF subversion and git services commented on NIFI-5170:
---

Commit 61fe493786425826e46cab737be9def11300dd19 in nifi's branch 
refs/heads/master from [~ottobackwards]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=61fe493 ]

NIFI-5170 Upgrad Grok to version 0.1.9

This closes #2691

Signed-off-by: Mike Thomsen 


> Update Grok to 0.1.9
> 
>
> Key: NIFI-5170
> URL: https://issues.apache.org/jira/browse/NIFI-5170
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Grok 0.1.9 has been released, including work for empty capture support.
>  
> https://github.com/thekrakken/java-grok#maven-repository



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-5170) Update Grok to 0.1.9

[Mike Thomsen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Thomsen resolved NIFI-5170.

Resolution: Fixed

> Update Grok to 0.1.9
> 
>
> Key: NIFI-5170
> URL: https://issues.apache.org/jira/browse/NIFI-5170
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Grok 0.1.9 has been released, including work for empty capture support.
>  
> https://github.com/thekrakken/java-grok#maven-repository



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5170) Update Grok to 0.1.9

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473471#comment-16473471
 ] 

ASF GitHub Bot commented on NIFI-5170:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2691


> Update Grok to 0.1.9
> 
>
> Key: NIFI-5170
> URL: https://issues.apache.org/jira/browse/NIFI-5170
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Grok 0.1.9 has been released, including work for empty capture support.
>  
> https://github.com/thekrakken/java-grok#maven-repository



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2691: NIFI-5170 Upgrad Grok to version 0.1.9

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2691


---

[jira] [Commented] (NIFI-5170) Update Grok to 0.1.9

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473431#comment-16473431
 ] 

ASF GitHub Bot commented on NIFI-5170:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2691#discussion_r187795885
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -80,18 +84,21 @@
 public static final String FLOWFILE_ATTRIBUTE = "flowfile-attribute";
 public static final String FLOWFILE_CONTENT = "flowfile-content";
 private static final String APPLICATION_JSON = "application/json";
+private static final String DEFAULT_PATTERN_NAME = 
"/default-grok-patterns.txt";
 
 public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor.Builder()
 .name("Grok Expression")
-.description("Grok expression")
+.description("Grok expression. If other Grok expressions are 
referenced in this expression, they must be provided "
++ "in the Grok Pattern File if set or exist in the default Grok 
patterns")
 .required(true)
-.addValidator(validateGrokExpression())
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
--- End diff --

Yeah. That seems like a behavior they really should have from the start.


> Update Grok to 0.1.9
> 
>
> Key: NIFI-5170
> URL: https://issues.apache.org/jira/browse/NIFI-5170
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Grok 0.1.9 has been released, including work for empty capture support.
>  
> https://github.com/thekrakken/java-grok#maven-repository



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2691: NIFI-5170 Upgrad Grok to version 0.1.9

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2691#discussion_r187795885
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractGrok.java
 ---
@@ -80,18 +84,21 @@
 public static final String FLOWFILE_ATTRIBUTE = "flowfile-attribute";
 public static final String FLOWFILE_CONTENT = "flowfile-content";
 private static final String APPLICATION_JSON = "application/json";
+private static final String DEFAULT_PATTERN_NAME = 
"/default-grok-patterns.txt";
 
 public static final PropertyDescriptor GROK_EXPRESSION = new 
PropertyDescriptor.Builder()
 .name("Grok Expression")
-.description("Grok expression")
+.description("Grok expression. If other Grok expressions are 
referenced in this expression, they must be provided "
++ "in the Grok Pattern File if set or exist in the default Grok 
patterns")
 .required(true)
-.addValidator(validateGrokExpression())
+.addValidator(StandardValidators.NON_BLANK_VALIDATOR)
--- End diff --

Yeah. That seems like a behavior they really should have from the start.


---

[jira] [Commented] (NIFI-5170) Update Grok to 0.1.9

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473430#comment-16473430
 ] 

ASF GitHub Bot commented on NIFI-5170:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2691#discussion_r187795865
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/resources/default-grok-patterns.txt
 ---
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
--- End diff --

Ok, so it's a known quantity.


> Update Grok to 0.1.9
> 
>
> Key: NIFI-5170
> URL: https://issues.apache.org/jira/browse/NIFI-5170
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Grok 0.1.9 has been released, including work for empty capture support.
>  
> https://github.com/thekrakken/java-grok#maven-repository



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2691: NIFI-5170 Upgrad Grok to version 0.1.9

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2691#discussion_r187795865
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/resources/default-grok-patterns.txt
 ---
@@ -0,0 +1,115 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
--- End diff --

Ok, so it's a known quantity.


---

[jira] [Commented] (NIFI-5073) JMSConnectionFactory doesn't resolve 'variables' properly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473395#comment-16473395
 ] 

ASF GitHub Bot commented on NIFI-5073:
--

Github user zenfenan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2653#discussion_r187790872
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/cf/JMSConnectionFactoryProvider.java
 ---
@@ -159,13 +159,15 @@ public void enable(ConfigurationContext context) 
throws InitializationException
 if (logger.isInfoEnabled()) {
 logger.info("Configuring " + 
this.getClass().getSimpleName() + " for '"
 + 
context.getProperty(CONNECTION_FACTORY_IMPL).evaluateAttributeExpressions().getValue()
 + "' to be connected to '"
-+ BROKER_URI + "'");
++ 
context.getProperty(BROKER_URI).evaluateAttributeExpressions().getValue() + 
"'");
 }
+
 // will load user provided libraries/resources on the 
classpath
-
Utils.addResourcesToClasspath(context.getProperty(CLIENT_LIB_DIR_PATH).evaluateAttributeExpressions().getValue());
+final String clientLibPath = 
context.getProperty(CLIENT_LIB_DIR_PATH).evaluateAttributeExpressions().getValue();
+ClassLoader customClassLoader = 
ClassLoaderUtils.getCustomClassLoader(clientLibPath, 
this.getClass().getClassLoader(), null);
+
Thread.currentThread().setContextClassLoader(customClassLoader);
--- End diff --

Understood. I have updated `CLIENT_LIB_DIR_PATH` to include 
`.dynamicallyModifiesClasspath(true)`


> JMSConnectionFactory doesn't resolve 'variables' properly
> -
>
> Key: NIFI-5073
> URL: https://issues.apache.org/jira/browse/NIFI-5073
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Matthew Clarke
>Assignee: Sivaprasanna Sethuraman
>Priority: Major
> Attachments: 
> 0001-NIFI-5073-JMSConnectionFactoryProvider-now-resolves-.patch
>
>
> Create a new process Group.
> Add "Variables" to the process group:
> for example:
> broker_uri=tcp://localhost:4141
> client_libs=/NiFi/custom-lib-dir/MQlib
> con_factory=blah
> Then while that process group is selected, create  a controller service.
> Create JMSConnectionFactory.
> Configure this controller service to use EL for PG defined variables above:
> ${con_factory}, ${con_factory}, and ${broker_uri}
> The controller service will remain invalid because the EL statements are not 
> properly resolved to their set values.
> Doing the exact same thing above using the external NiFi registry file works 
> as expected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2653: NIFI-5073: JMSConnectionFactoryProvider now resolve...

[zenfenan]

Github user zenfenan commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2653#discussion_r187790872
  
--- Diff: 
nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/cf/JMSConnectionFactoryProvider.java
 ---
@@ -159,13 +159,15 @@ public void enable(ConfigurationContext context) 
throws InitializationException
 if (logger.isInfoEnabled()) {
 logger.info("Configuring " + 
this.getClass().getSimpleName() + " for '"
 + 
context.getProperty(CONNECTION_FACTORY_IMPL).evaluateAttributeExpressions().getValue()
 + "' to be connected to '"
-+ BROKER_URI + "'");
++ 
context.getProperty(BROKER_URI).evaluateAttributeExpressions().getValue() + 
"'");
 }
+
 // will load user provided libraries/resources on the 
classpath
-
Utils.addResourcesToClasspath(context.getProperty(CLIENT_LIB_DIR_PATH).evaluateAttributeExpressions().getValue());
+final String clientLibPath = 
context.getProperty(CLIENT_LIB_DIR_PATH).evaluateAttributeExpressions().getValue();
+ClassLoader customClassLoader = 
ClassLoaderUtils.getCustomClassLoader(clientLibPath, 
this.getClass().getClassLoader(), null);
+
Thread.currentThread().setContextClassLoader(customClassLoader);
--- End diff --

Understood. I have updated `CLIENT_LIB_DIR_PATH` to include 
`.dynamicallyModifiesClasspath(true)`


---