[jira] [Commented] (NIFI-12590) Add an option to add a prefix on Kubernetes resources

[Michael Hogue (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804791#comment-17804791
 ] 

Michael Hogue commented on NIFI-12590:
--

Slack thread: https://apachenifi.slack.com/archives/C0L9VCD47/p1704727400482319

> Add an option to add a prefix on Kubernetes resources
> -
>
> Key: NIFI-12590
> URL: https://issues.apache.org/jira/browse/NIFI-12590
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 2.0.0
>Reporter: Julien G.
>Priority: Major
>
> In case of using Kubernetes as a *Cluster Leader Election* and *State 
> Mangement*, NiFi will create two types of resources: *ConfigMaps* and 
> *Leases*.
> 2 *Leases* for the _Cluster Leader Election_ and multiple *ConfigMaps* 
> depending the number of components which their state to be stored.
> Their names are normalized:
> - *Leases*: {{cluster-coordinator}} and {{primary-node}}.
> - *ConfigMap*: {{nifi-component-%s}} with {{%s}} the component identifier.
> But currently, if someone wants to deploy more than one cluster in one 
> namespace, conflicts will arise. Because the resources will share their 
> names. It has been done this way to prevent security concerns where a cluster 
> will manipulate resources of another cluster.
> But as discussed on Slack, these concerns can't prevented with 100% certainty.
> So users should be able to configure a prefix to add to these resources to 
> prevent conflict and assume the security concerns.
> It can be done by replicating the logic done with ZooKeeper by adding a 
> property in {{nifi.properties}} and a new field in the 
> {{state_management.xml}} file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-10273) MergeContent tar compression fails on FlowFiles > 8.5GB

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-10273:
-
Description: 
When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the 
following error:
{noformat}
 java.lang.RuntimeException: entry size '' is too big (> 
8589934591){noformat}
This is because apache commons-compress library has a hard maximum :

[https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]

0777 octal bytes is 8589934591 decimal bytes (~8.5GB).

This can be addressed by configuring a BigNumberMode on the 
{{{}TarArchiveOutputStream{}}}:

[https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]



{{Here's an example of another project doing this:}}

{{[https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files]}}

  was:
When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the 
following error:
{noformat}
 java.lang.RuntimeException: entry size '' is too big (> 
8589934591){noformat}
This is because apache commons-compress library has a hard maximum :

[https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]

0777 octal bytes is 8589934591 decimal bytes (~8.5GB).

This can be addressed by configuring a BigNumberMode on the 
{{{}TarArchiveOutputStream{}}}:

[https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
}}

{{Here's an example of another project doing this:}}

{{[https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files]}}


> MergeContent tar compression fails on FlowFiles > 8.5GB
> ---
>
> Key: NIFI-10273
> URL: https://issues.apache.org/jira/browse/NIFI-10273
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.16.3
>Reporter: Michael Hogue
>Priority: Minor
>
> When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with 
> the following error:
> {noformat}
>  java.lang.RuntimeException: entry size '' is too big (> 
> 8589934591){noformat}
> This is because apache commons-compress library has a hard maximum :
> [https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]
> 0777 octal bytes is 8589934591 decimal bytes (~8.5GB).
> This can be addressed by configuring a BigNumberMode on the 
> {{{}TarArchiveOutputStream{}}}:
> [https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
> {{Here's an example of another project doing this:}}
> {{[https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files]}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-10273) MergeContent tar compression fails on FlowFiles > 8.5GB

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-10273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-10273:
-
Description: 
When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the 
following error:
{noformat}
 java.lang.RuntimeException: entry size '' is too big (> 
8589934591){noformat}
This is because apache commons-compress library has a hard maximum :

[https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]

0777 octal bytes is 8589934591 decimal bytes (~8.5GB).

This can be addressed by configuring a BigNumberMode on the 
{{{}TarArchiveOutputStream{}}}:

[https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
}}

{{Here's an example of another project doing this:}}

{{[https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files]}}

  was:
When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the 
following error:
{noformat}
 java.lang.RuntimeException: entry size '' is too big (> 
8589934591){noformat}
This is because apache commons-compress library has a hard maximum :

[https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]

0777 octal bytes is 8589934591 decimal bytes (~8.5GB).

This can be addressed by configuring a BigNumberMode on the 
{{TarArchiveOutputStream:

[https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
}}

{{Here's an example of another project doing this:}}

{{https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files}}


> MergeContent tar compression fails on FlowFiles > 8.5GB
> ---
>
> Key: NIFI-10273
> URL: https://issues.apache.org/jira/browse/NIFI-10273
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.16.3
>Reporter: Michael Hogue
>Priority: Minor
>
> When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with 
> the following error:
> {noformat}
>  java.lang.RuntimeException: entry size '' is too big (> 
> 8589934591){noformat}
> This is because apache commons-compress library has a hard maximum :
> [https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]
> 0777 octal bytes is 8589934591 decimal bytes (~8.5GB).
> This can be addressed by configuring a BigNumberMode on the 
> {{{}TarArchiveOutputStream{}}}:
> [https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
> }}
> {{Here's an example of another project doing this:}}
> {{[https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files]}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (NIFI-10273) MergeContent tar compression fails on FlowFiles > 8.5GB

[Michael Hogue (Jira)]

Michael Hogue created NIFI-10273:


 Summary: MergeContent tar compression fails on FlowFiles > 8.5GB
 Key: NIFI-10273
 URL: https://issues.apache.org/jira/browse/NIFI-10273
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Affects Versions: 1.16.3
Reporter: Michael Hogue


When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the 
following error:
{noformat}
 java.lang.RuntimeException: entry size '' is too big (> 
8589934591){noformat}
This is because apache commons-compress library has a hard maximum :

[https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]

0777 octal bytes is 8589934591 decimal bytes (~8.5GB).

This can be addressed by configuring a BigNumberMode on the 
{{TarArchiveOutputStream:

[https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
}}

{{Here's an example of another project doing this:}}

{{https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (NIFI-9218) Extend ExtractEmailAttachments to optionally extract email content

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-9218:

Description: The email processing bundle supports extracting email headers 
and attachments but not the content, if that's desired. Add a flag to the 
ExtractEmailAttachments processor to optionally write the email content to its 
own relationship.  (was: The email processing bundle supports extracting email 
headers and attachments but not only the content. Add a flag to the 
ExtractEmailAttachments processor to optionally write the email content to its 
own relationship.)

> Extend ExtractEmailAttachments to optionally extract email content
> --
>
> Key: NIFI-9218
> URL: https://issues.apache.org/jira/browse/NIFI-9218
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michael Hogue
>Priority: Minor
>
> The email processing bundle supports extracting email headers and attachments 
> but not the content, if that's desired. Add a flag to the 
> ExtractEmailAttachments processor to optionally write the email content to 
> its own relationship.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-9218) Extend ExtractEmailAttachments to optionally extract email content

[Michael Hogue (Jira)]

Michael Hogue created NIFI-9218:
---

 Summary: Extend ExtractEmailAttachments to optionally extract 
email content
 Key: NIFI-9218
 URL: https://issues.apache.org/jira/browse/NIFI-9218
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Michael Hogue


The email processing bundle supports extracting email headers and attachments 
but not only the content. Add a flag to the ExtractEmailAttachments processor 
to optionally write the email content to its own relationship.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-8926) NiFi docs incorrectly state enabling/disabling is not a local change

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-8926:

Labels: easyfix help-wanted newbie  (was: easyfix newbie)

> NiFi docs incorrectly state enabling/disabling is not a local change
> 
>
> Key: NIFI-8926
> URL: https://issues.apache.org/jira/browse/NIFI-8926
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Reporter: Michael Hogue
>Priority: Minor
>  Labels: easyfix, help-wanted, newbie
>
> According to the NiFi documentation under "Managing Local Changes" [1] it 
> states:
> {quote}The following actions are not considered local changes:
> {quote}
> {quote} * disabling/enabling processors and controller services
>  * ...{quote}
> However, according to the actual behavior and the comment linked in PR #3546 
> [2], i believe disabling/enabling processors & controller services is 
> considered a local change.
>  
> [1] 
> [https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#managing_local_changes]
> [2] https://github.com/apache/nifi/pull/3546#issuecomment-507755482



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-8926) NiFi docs incorrectly state enabling/disabling is not a local change

[Michael Hogue (Jira)]

Michael Hogue created NIFI-8926:
---

 Summary: NiFi docs incorrectly state enabling/disabling is not a 
local change
 Key: NIFI-8926
 URL: https://issues.apache.org/jira/browse/NIFI-8926
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Documentation  Website
Reporter: Michael Hogue


According to the NiFi documentation under "Managing Local Changes" [1] it 
states:
{quote}The following actions are not considered local changes:
{quote}
{quote} * disabling/enabling processors and controller services

 * ...{quote}
However, according to the actual behavior and the comment linked in PR #3546 
[2], i believe disabling/enabling processors & controller services is 
considered a local change.

 

[1] 
[https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#managing_local_changes]

[2] https://github.com/apache/nifi/pull/3546#issuecomment-507755482



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-5925) Controller Service values should be searchable

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-5925:

Status: Patch Available  (was: In Progress)

> Controller Service values should be searchable
> --
>
> Key: NIFI-5925
> URL: https://issues.apache.org/jira/browse/NIFI-5925
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Brandon Rhys DeVries
>Assignee: Michael Hogue
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently processor name, type, property name and property value are all 
> searchable.  The same should be true for controller services.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (NIFI-5925) Controller Service values should be searchable

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-5925:
---

Assignee: Michael Hogue

> Controller Service values should be searchable
> --
>
> Key: NIFI-5925
> URL: https://issues.apache.org/jira/browse/NIFI-5925
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Brandon Rhys DeVries
>Assignee: Michael Hogue
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently processor name, type, property name and property value are all 
> searchable.  The same should be true for controller services.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-5925) Controller Service values should be searchable

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-5925:

Component/s: Core UI

> Controller Service values should be searchable
> --
>
> Key: NIFI-5925
> URL: https://issues.apache.org/jira/browse/NIFI-5925
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Brandon Rhys DeVries
>Priority: Minor
>
> Currently processor name, type, property name and property value are all 
> searchable.  The same should be true for controller services.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7073) PutHDFS silently ignores exceptions when closing temporary file output stream

[Michael Hogue (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026891#comment-17026891
 ] 

Michael Hogue commented on NIFI-7073:
-

Agree. Unfortunately i don't have a handle on the stack trace. I think instead 
catching {{Throwable}} will suffice since the only recourse in this case is to 
try and clean up and exit.

> PutHDFS silently ignores exceptions when closing temporary file output stream
> -
>
> Key: NIFI-7073
> URL: https://issues.apache.org/jira/browse/NIFI-7073
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Michael Hogue
>Assignee: Matt Burgess
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The PutHDFS processor can silently fail when closing an output stream when 
> it's finished writing FlowFiles to a temporary space in HDFS. 
> {{DFSOutputStream}} can throw an {{IOException}} when it's closed and that's 
> ignored by the processor. This results in the FlowFile getting dropped (e.g. 
> a loss of data) when it should be routed to the failure relationship.
> Source: 
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java#L365]
> Instead, the exception should be caught & thrown and the FlowFile should be 
> routed to the failure relationship. There's probably an argument to avoid 
> ignoring _any_ exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-7073) PutHDFS silently ignores exceptions when closing temporary file output stream

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-7073:

Description: 
The PutHDFS processor can silently fail when closing an output stream when it's 
finished writing FlowFiles to a temporary space in HDFS. {{DFSOutputStream}} 
can throw an {{IOException}} when it's closed and that's ignored by the 
processor. This results in the FlowFile getting dropped (e.g. a loss of data) 
when it should be routed to the failure relationship.

Source: 
[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java#L365]

Instead, the exception should be caught & thrown and the FlowFile should be 
routed to the failure relationship. There's probably an argument to avoid 
ignoring _any_ exception.

  was:
The PutHDFS processor can silently fail when closing an output stream when it's 
finished writing FlowFiles to a temporary space in HDFS. {{DFSOutputStream}} 
can throw an {{IOException}} when it's closed and that's ignored by the 
processor. This results in the FlowFile getting dropped (e.g. a loss of data) 
when it should be routed to the failure relationship.

Source: 
[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java#L365]

Instead, the exception should be caught & thrown and the FlowFile should be 
routed to the failure relationship.


> PutHDFS silently ignores exceptions when closing temporary file output stream
> -
>
> Key: NIFI-7073
> URL: https://issues.apache.org/jira/browse/NIFI-7073
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.11.0
>Reporter: Michael Hogue
>Priority: Major
>
> The PutHDFS processor can silently fail when closing an output stream when 
> it's finished writing FlowFiles to a temporary space in HDFS. 
> {{DFSOutputStream}} can throw an {{IOException}} when it's closed and that's 
> ignored by the processor. This results in the FlowFile getting dropped (e.g. 
> a loss of data) when it should be routed to the failure relationship.
> Source: 
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java#L365]
> Instead, the exception should be caught & thrown and the FlowFile should be 
> routed to the failure relationship. There's probably an argument to avoid 
> ignoring _any_ exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-7073) PutHDFS silently ignores exceptions when closing temporary file output stream

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-7073:

Summary: PutHDFS silently ignores exceptions when closing temporary file 
output stream  (was: PutHDFS silently ignores exceptions when writing temporary 
file)

> PutHDFS silently ignores exceptions when closing temporary file output stream
> -
>
> Key: NIFI-7073
> URL: https://issues.apache.org/jira/browse/NIFI-7073
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.11.0
>Reporter: Michael Hogue
>Priority: Major
>
> The PutHDFS processor can silently fail when closing an output stream when 
> it's finished writing FlowFiles to a temporary space in HDFS. 
> {{DFSOutputStream}} can throw an {{IOException}} when it's closed and that's 
> ignored by the processor. This results in the FlowFile getting dropped (e.g. 
> a loss of data) when it should be routed to the failure relationship.
>  
> Instead, the exception should be caught & thrown and the FlowFile should be 
> routed to the failure relationship.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-7073) PutHDFS silently ignores exceptions when closing temporary file output stream

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-7073:

Description: 
The PutHDFS processor can silently fail when closing an output stream when it's 
finished writing FlowFiles to a temporary space in HDFS. {{DFSOutputStream}} 
can throw an {{IOException}} when it's closed and that's ignored by the 
processor. This results in the FlowFile getting dropped (e.g. a loss of data) 
when it should be routed to the failure relationship.

Source: 
[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java#L365]

Instead, the exception should be caught & thrown and the FlowFile should be 
routed to the failure relationship.

  was:
The PutHDFS processor can silently fail when closing an output stream when it's 
finished writing FlowFiles to a temporary space in HDFS. {{DFSOutputStream}} 
can throw an {{IOException}} when it's closed and that's ignored by the 
processor. This results in the FlowFile getting dropped (e.g. a loss of data) 
when it should be routed to the failure relationship.

 

Instead, the exception should be caught & thrown and the FlowFile should be 
routed to the failure relationship.


> PutHDFS silently ignores exceptions when closing temporary file output stream
> -
>
> Key: NIFI-7073
> URL: https://issues.apache.org/jira/browse/NIFI-7073
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.11.0
>Reporter: Michael Hogue
>Priority: Major
>
> The PutHDFS processor can silently fail when closing an output stream when 
> it's finished writing FlowFiles to a temporary space in HDFS. 
> {{DFSOutputStream}} can throw an {{IOException}} when it's closed and that's 
> ignored by the processor. This results in the FlowFile getting dropped (e.g. 
> a loss of data) when it should be routed to the failure relationship.
> Source: 
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java#L365]
> Instead, the exception should be caught & thrown and the FlowFile should be 
> routed to the failure relationship.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-7073) PutHDFS silently ignores exceptions when writing temporary file

[Michael Hogue (Jira)]

Michael Hogue created NIFI-7073:
---

 Summary: PutHDFS silently ignores exceptions when writing 
temporary file
 Key: NIFI-7073
 URL: https://issues.apache.org/jira/browse/NIFI-7073
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Affects Versions: 1.11.0
Reporter: Michael Hogue


The PutHDFS processor can silently fail when closing an output stream when it's 
finished writing FlowFiles to a temporary space in HDFS. {{DFSOutputStream}} 
can throw an {{IOException}} when it's closed and that's ignored by the 
processor. This results in the FlowFile getting dropped (e.g. a loss of data) 
when it should be routed to the failure relationship.

 

Instead, the exception should be caught & thrown and the FlowFile should be 
routed to the failure relationship.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-6719) Changing JMS clientID causes durable subscription problems

[Michael Hogue (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010820#comment-17010820
 ] 

Michael Hogue commented on NIFI-6719:
-

I suspect this is the same as NIFI-6915, which already has an open PR

> Changing JMS clientID causes durable subscription problems
> --
>
> Key: NIFI-6719
> URL: https://issues.apache.org/jira/browse/NIFI-6719
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Extensions
>Affects Versions: 1.8.0
> Environment: Linux CENTOS 7
>Reporter: Tim Chermak
>Priority: Minor
>  Labels: patch
>
> Using ConsumeJMS we set clientID = nificid, durable = true and subscription = 
> nifidur. After several tests where we received messages and stopped/started 
> the processor between tests. We had the following durables for TIBCO:
>  * infrastructure.itx.monitoring nificid-5:nifidur N  40 8.8 kb
>  * infrastructure.itx.monitoring nificid-6:nifidur N  20 4.4 kb
>  * infrastructure.itx.monitoring nificid-4:nifidur N  60 13.2 kb
> The clientID used was modified with an appended "-", and the digit 
> increased with each restart. The test with ActiveMQ had the same problem.
>  
> The resulting offline durables collect messages. The durable name is a 
> combination of the clientID and the subscription name and NiFi has to 
> remember and reuse the same durable name after a restart of the processor. 
> The same problem was found and fixed in an older version before 1.x.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-6915) Jms Durable non shared subscription is broken

[Michael Hogue (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-6915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010818#comment-17010818
 ] 

Michael Hogue commented on NIFI-6915:
-

I think this could be a duplicate of NIFI-6719, but we could close that one in 
favor of this if it's the same.

> Jms Durable non shared subscription is broken
> -
>
> Key: NIFI-6915
> URL: https://issues.apache.org/jira/browse/NIFI-6915
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0, 1.9.1, 1.9.2
> Environment: All
>Reporter: Gardella Juan Pablo
>Assignee: Gardella Juan Pablo
>Priority: Critical
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The enhancement NIFI-4834 broke JMS non shared Durable subscriptions. It may 
> lose messages after stopping and starting a {{ConsumeJMS}} processor as 
> [client ID is always 
> different|https://github.com/apache/nifi/pull/2445/commits/cd091103e4a76e7b54e00257e5e18eaab3d389ec#diff-4ce7e53c92829e48b85959da41653f2bR189]
>  since NIFI-4834.
> Using different client identifiers, makes the consumer missing messages after 
> it is restarted. The problem is when some messages were published to the 
> topic during the consumer is not running.
> A simple solution is to keep old behavior if it is a durable subscriber.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-6851) Improve Provenance RECEIVE and FETCH events to allow for specifics on how flowfile was RECEIVE'ed/FETCH'ed

[Michael Hogue (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009965#comment-17009965
 ] 

Michael Hogue commented on NIFI-6851:
-

I've verified backward compatibility via unit tests [1], so running a new nifi 
with these changes can read old provenance events. I can't write unit tests to 
verify forward compatibility (e.g. old nifi reading new provenance events, or 
rollback scenario), but it looks like it'll work fine too. Here's why:
 * {{RecordSchema#readFrom(InputStream)}} reads arbitrary schemas and populates 
a map of fieldName -> fieldValue pairs. [2]
 * The provenance event deserialization implementations all pull explicit 
fieldNames out of the {{RecordSchema}} map to build 
{{StandardProvenanceRecord}} objects [3]

So in a rollback scenario, events written with this new 
{{FlowFileAcquisitionMethod}} field will just be constructed with that field 
ignored, which is okay. 

 
[1] 
https://github.com/apache/nifi/pull/3930/files#diff-973cdae5e5e4d369ce7c2f9d321781e8R141
 (note new method to create prov event w/ flow file acquisition method)
[2] 
[https://github.com/apache/nifi/blob/master/nifi-commons/nifi-schema-utils/src/main/java/org/apache/nifi/repository/schema/RecordSchema.java#L103]
[3] 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/schema/LookupTableEventRecord.java#L241

> Improve Provenance RECEIVE and FETCH events to allow for specifics on how 
> flowfile was RECEIVE'ed/FETCH'ed
> --
>
> Key: NIFI-6851
> URL: https://issues.apache.org/jira/browse/NIFI-6851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.10.0
>Reporter: Nissim Shiman
>Assignee: Michael Hogue
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently there is no way to know if a Provenance RECEIVE or FETCH event is 
> the result of an active (i.e. GetFTP, FetchFTP, GetHTTP), passive (i.e. 
> ListenHTTP)  or query (i.e. GetMongo) event.
> There was a discussion on the dev mailing lists regarding the most graceful 
> way to do this [1] [2], 
>  
> At this time the smoothest path forward is to keep the current method 
> signatures of receive() and fetch() as is in ProvenanceReporter.java [3]
> and to add new fetch() and receive()'s with a new parameter:
> receive(...,ACTIVE|PASSIVE|QUERY)
> If not specified it would be UNSPECIFIED. 
>  (Thank You [~joewitt] for this solution)
>  
>  
> [1] [https://mail-archives.apache.org/mod_mbox/nifi-dev/201910.mbox/browser] 
> , topic of PULL ProvenanceEvent
> [2] [https://mail-archives.apache.org/mod_mbox/nifi-dev/201911.mbox/browser] 
> , topic of Re: PULL ProvenanceEvent
> [3] 
> [https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-6919) Add relationship attribute to flowfile in DistributeLoad

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-6919:

Status: Patch Available  (was: Open)

> Add relationship attribute to flowfile in DistributeLoad
> 
>
> Key: NIFI-6919
> URL: https://issues.apache.org/jira/browse/NIFI-6919
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Add the following to the DistributeLoad processor:
>  * Flowfile attribute capturing which relationship a particular flowfile was 
> routed through
>  * Feature flag processor configuration on DistributeLoad controlling whether 
> or not to set the relationship attribute
> The reason behind this change is to potentially use the relationship that a 
> flowfile was routed through in DistributeLoad in later processors for routing 
> efficiencies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-6851) Improve Provenance RECEIVE and FETCH events to allow for specifics on how flowfile was RECEIVE'ed/FETCH'ed

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-6851:

Status: Patch Available  (was: In Progress)

> Improve Provenance RECEIVE and FETCH events to allow for specifics on how 
> flowfile was RECEIVE'ed/FETCH'ed
> --
>
> Key: NIFI-6851
> URL: https://issues.apache.org/jira/browse/NIFI-6851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.10.0
>Reporter: Nissim Shiman
>Assignee: Michael Hogue
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently there is no way to know if a Provenance RECEIVE or FETCH event is 
> the result of an active (i.e. GetFTP, FetchFTP, GetHTTP), passive (i.e. 
> ListenHTTP)  or query (i.e. GetMongo) event.
> There was a discussion on the dev mailing lists regarding the most graceful 
> way to do this [1] [2], 
>  
> At this time the smoothest path forward is to keep the current method 
> signatures of receive() and fetch() as is in ProvenanceReporter.java [3]
> and to add new fetch() and receive()'s with a new parameter:
> receive(...,ACTIVE|PASSIVE|QUERY)
> If not specified it would be UNSPECIFIED. 
>  (Thank You [~joewitt] for this solution)
>  
>  
> [1] [https://mail-archives.apache.org/mod_mbox/nifi-dev/201910.mbox/browser] 
> , topic of PULL ProvenanceEvent
> [2] [https://mail-archives.apache.org/mod_mbox/nifi-dev/201911.mbox/browser] 
> , topic of Re: PULL ProvenanceEvent
> [3] 
> [https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (NIFI-6851) Improve Provenance RECEIVE and FETCH events to allow for specifics on how flowfile was RECEIVE'ed/FETCH'ed

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-6851:
---

Assignee: Michael Hogue

> Improve Provenance RECEIVE and FETCH events to allow for specifics on how 
> flowfile was RECEIVE'ed/FETCH'ed
> --
>
> Key: NIFI-6851
> URL: https://issues.apache.org/jira/browse/NIFI-6851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.10.0
>Reporter: Nissim Shiman
>Assignee: Michael Hogue
>Priority: Major
>
> Currently there is no way to know if a Provenance RECEIVE or FETCH event is 
> the result of an active (i.e. GetFTP, FetchFTP, GetHTTP), passive (i.e. 
> ListenHTTP)  or query (i.e. GetMongo) event.
> There was a discussion on the dev mailing lists regarding the most graceful 
> way to do this [1] [2], 
>  
> At this time the smoothest path forward is to keep the current method 
> signatures of receive() and fetch() as is in ProvenanceReporter.java [3]
> and to add new fetch() and receive()'s with a new parameter:
> receive(...,ACTIVE|PASSIVE|QUERY)
> If not specified it would be UNSPECIFIED. 
>  (Thank You [~joewitt] for this solution)
>  
>  
> [1] [https://mail-archives.apache.org/mod_mbox/nifi-dev/201910.mbox/browser] 
> , topic of PULL ProvenanceEvent
> [2] [https://mail-archives.apache.org/mod_mbox/nifi-dev/201911.mbox/browser] 
> , topic of Re: PULL ProvenanceEvent
> [3] 
> [https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (NIFI-6919) Add relationship attribute to flowfile in DistributeLoad

[Michael Hogue (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-6919:
---

Assignee: Michael Hogue

> Add relationship attribute to flowfile in DistributeLoad
> 
>
> Key: NIFI-6919
> URL: https://issues.apache.org/jira/browse/NIFI-6919
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
>
> Add the following to the DistributeLoad processor:
>  * Flowfile attribute capturing which relationship a particular flowfile was 
> routed through
>  * Feature flag processor configuration on DistributeLoad controlling whether 
> or not to set the relationship attribute
> The reason behind this change is to potentially use the relationship that a 
> flowfile was routed through in DistributeLoad in later processors for routing 
> efficiencies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-6919) Add relationship attribute to flowfile in DistributeLoad

[Michael Hogue (Jira)]

Michael Hogue created NIFI-6919:
---

 Summary: Add relationship attribute to flowfile in DistributeLoad
 Key: NIFI-6919
 URL: https://issues.apache.org/jira/browse/NIFI-6919
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Reporter: Michael Hogue


Add the following to the DistributeLoad processor:
 * Flowfile attribute capturing which relationship a particular flowfile was 
routed through
 * Feature flag processor configuration on DistributeLoad controlling whether 
or not to set the relationship attribute

The reason behind this change is to potentially use the relationship that a 
flowfile was routed through in DistributeLoad in later processors for routing 
efficiencies.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-4700) PostHTTP: close client

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-4700:

Status: Open  (was: Patch Available)

> PostHTTP: close client
> --
>
> Key: NIFI-4700
> URL: https://issues.apache.org/jira/browse/NIFI-4700
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Major
> Fix For: 1.6.0
>
>
> In PostHTTP, the CloseableHttpClient never actually appears to be closed...
> Additionally, we could leverage CloseableHttpResponse to close responses.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4700) PostHTTP: close client

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353042#comment-16353042
 ] 

Michael Hogue commented on NIFI-4700:
-

No problem. It's closed.

> PostHTTP: close client
> --
>
> Key: NIFI-4700
> URL: https://issues.apache.org/jira/browse/NIFI-4700
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Major
>
> In PostHTTP, the CloseableHttpClient never actually appears to be closed...
> Additionally, we could leverage CloseableHttpResponse to close responses.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4700) PostHTTP: close client

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-4700:

Fix Version/s: (was: 1.6.0)

> PostHTTP: close client
> --
>
> Key: NIFI-4700
> URL: https://issues.apache.org/jira/browse/NIFI-4700
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Major
>
> In PostHTTP, the CloseableHttpClient never actually appears to be closed...
> Additionally, we could leverage CloseableHttpResponse to close responses.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-4700) PostHTTP: close client

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue resolved NIFI-4700.
-
Resolution: Won't Fix

> PostHTTP: close client
> --
>
> Key: NIFI-4700
> URL: https://issues.apache.org/jira/browse/NIFI-4700
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Major
>
> In PostHTTP, the CloseableHttpClient never actually appears to be closed...
> Additionally, we could leverage CloseableHttpResponse to close responses.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4700) PostHTTP: close client

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-4700:

Description: 
In PostHTTP, the CloseableHttpClient never actually appears to be closed...

Additionally, we could leverage CloseableHttpResponse to close responses.

  was:In PostHTTP, the CloseableHttpClient never actaully appears to be 
closed...


> PostHTTP: close client
> --
>
> Key: NIFI-4700
> URL: https://issues.apache.org/jira/browse/NIFI-4700
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Major
> Fix For: 1.6.0
>
>
> In PostHTTP, the CloseableHttpClient never actually appears to be closed...
> Additionally, we could leverage CloseableHttpResponse to close responses.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4700) PostHTTP: close client

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-4700:

Fix Version/s: 1.6.0
   Status: Patch Available  (was: In Progress)

> PostHTTP: close client
> --
>
> Key: NIFI-4700
> URL: https://issues.apache.org/jira/browse/NIFI-4700
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Major
> Fix For: 1.6.0
>
>
> In PostHTTP, the CloseableHttpClient never actaully appears to be closed...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-4700) PostHTTP: close client

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-4700:
---

Assignee: Michael Hogue

> PostHTTP: close client
> --
>
> Key: NIFI-4700
> URL: https://issues.apache.org/jira/browse/NIFI-4700
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Major
>
> In PostHTTP, the CloseableHttpClient never actaully appears to be closed...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4697) PostHTTP: correct documentation

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317643#comment-16317643
 ] 

Michael Hogue commented on NIFI-4697:
-

No problem, [~mosermw]! Thanks for picking it up.

> PostHTTP: correct documentation
> ---
>
> Key: NIFI-4697
> URL: https://issues.apache.org/jira/browse/NIFI-4697
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Moser
>Priority: Minor
>
> The description of the URL property of PostHTTP says 
> "The URL to POST to. The first part of the URL must be static. However, the 
> path of the URL may be defined using the Attribute Expression Language. "  
> This does not appear to be true.  We should modify to something like "The URL 
> to POST to."
> ...or, someone can point out to me why I'm wrong in saying it's wrong...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4697) PostHTTP: correct documentation

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-4697:
---

Assignee: Michael Moser  (was: Michael Hogue)

> PostHTTP: correct documentation
> ---
>
> Key: NIFI-4697
> URL: https://issues.apache.org/jira/browse/NIFI-4697
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Moser
>Priority: Minor
>
> The description of the URL property of PostHTTP says 
> "The URL to POST to. The first part of the URL must be static. However, the 
> path of the URL may be defined using the Attribute Expression Language. "  
> This does not appear to be true.  We should modify to something like "The URL 
> to POST to."
> ...or, someone can point out to me why I'm wrong in saying it's wrong...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-4697) PostHTTP: correct documentation

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293957#comment-16293957
 ] 

Michael Hogue edited comment on NIFI-4697 at 12/16/17 10:15 PM:


How about "The URL to POST to. The URL may be defined using the Attribute 
Expression Language."?

Or just drop the latter sentence entirely?


was (Author: m-hogue):
How about "The URL to POST to. The URL may be defined using the Attribute 
Expression Language."?

> PostHTTP: correct documentation
> ---
>
> Key: NIFI-4697
> URL: https://issues.apache.org/jira/browse/NIFI-4697
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Minor
>
> The description of the URL property of PostHTTP says 
> "The URL to POST to. The first part of the URL must be static. However, the 
> path of the URL may be defined using the Attribute Expression Language. "  
> This does not appear to be true.  We should modify to something like "The URL 
> to POST to."
> ...or, someone can point out to me why I'm wrong in saying it's wrong...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4697) PostHTTP: correct documentation

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293957#comment-16293957
 ] 

Michael Hogue commented on NIFI-4697:
-

How about "The URL to POST to. The URL may be defined using the Attribute 
Expression Language."

> PostHTTP: correct documentation
> ---
>
> Key: NIFI-4697
> URL: https://issues.apache.org/jira/browse/NIFI-4697
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Minor
>
> The description of the URL property of PostHTTP says 
> "The URL to POST to. The first part of the URL must be static. However, the 
> path of the URL may be defined using the Attribute Expression Language. "  
> This does not appear to be true.  We should modify to something like "The URL 
> to POST to."
> ...or, someone can point out to me why I'm wrong in saying it's wrong...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-4697) PostHTTP: correct documentation

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293957#comment-16293957
 ] 

Michael Hogue edited comment on NIFI-4697 at 12/16/17 10:14 PM:


How about "The URL to POST to. The URL may be defined using the Attribute 
Expression Language."?


was (Author: m-hogue):
How about "The URL to POST to. The URL may be defined using the Attribute 
Expression Language."

> PostHTTP: correct documentation
> ---
>
> Key: NIFI-4697
> URL: https://issues.apache.org/jira/browse/NIFI-4697
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Minor
>
> The description of the URL property of PostHTTP says 
> "The URL to POST to. The first part of the URL must be static. However, the 
> path of the URL may be defined using the Attribute Expression Language. "  
> This does not appear to be true.  We should modify to something like "The URL 
> to POST to."
> ...or, someone can point out to me why I'm wrong in saying it's wrong...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4697) PostHTTP: correct documentation

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-4697:
---

Assignee: Michael Hogue

> PostHTTP: correct documentation
> ---
>
> Key: NIFI-4697
> URL: https://issues.apache.org/jira/browse/NIFI-4697
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Minor
>
> The description of the URL property of PostHTTP says 
> "The URL to POST to. The first part of the URL must be static. However, the 
> path of the URL may be defined using the Attribute Expression Language. "  
> This does not appear to be true.  We should modify to something like "The URL 
> to POST to."
> ...or, someone can point out to me why I'm wrong in saying it's wrong...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3869:
---

Assignee: (was: Michael Hogue)

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4584) Add Support for @EventDriven to InvokeGRPC

[Michael Hogue (JIRA)]

Michael Hogue created NIFI-4584:
---

 Summary: Add Support for @EventDriven to InvokeGRPC
 Key: NIFI-4584
 URL: https://issues.apache.org/jira/browse/NIFI-4584
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Michael Hogue
Assignee: Michael Hogue
Priority: Trivial


InvokeGRPC is a logical candidate for supporting EventDriven processing on flow 
files. It could be useful to add support for it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3515) Consider adding @EventDriven to PutFile

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3515:
---

Assignee: Michael Hogue

> Consider adding @EventDriven to PutFile
> ---
>
> Key: NIFI-3515
> URL: https://issues.apache.org/jira/browse/NIFI-3515
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Trivial
>
> PutFile seems like it could potentially be a candidate for supporting Event 
> Driven scheduling...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-4394) gRPC processor (Listen) CPU usage while listening processor is started

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue resolved NIFI-4394.
-
   Resolution: Fixed
Fix Version/s: 1.4.0

> gRPC processor (Listen) CPU usage while listening processor is started
> --
>
> Key: NIFI-4394
> URL: https://issues.apache.org/jira/browse/NIFI-4394
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.4.0
> Environment: All
>Reporter: Sébastien Bouchex Bellomié
>Priority: Minor
> Fix For: 1.4.0
>
>
> * Create a gRPC (ListenGRPC) processor in the interface
> * Start the processor
> * Check the CPU usage on the machine
> Problem : CPU usage is high



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3510) log when ControllerService is ENABLED

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163609#comment-16163609
 ] 

Michael Hogue commented on NIFI-3510:
-

PR submitted for 1.x: https://github.com/apache/nifi/pull/2151


> log when ControllerService is ENABLED
> -
>
> Key: NIFI-3510
> URL: https://issues.apache.org/jira/browse/NIFI-3510
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Trivial
>
> We currently have logging for when we begin "enabling" a controller service, 
> but none for when it has actually been "enabled".  I suggest adding a line 
> like 
> {code}
> LOG.debug("Successfully Enabled " + this);
> {code}
> to StandardControllerServiceNode.enable()...
> 0.x:
> {code}
> if (shouldEnable) {
> heartbeater.heartbeat();
> LOG.debug("Successfully Enabled " + this);
> } else {...
> {code}
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java#L313-L315
> 1.x:
> {code}
> ...
> }
> else{
> LOG.debug("Successfully Enabled " + this);
> }
> {code}
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java#L357
> [~mosermw] [~jskora]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3510) log when ControllerService is ENABLED

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3510:
---

Assignee: Michael Hogue

> log when ControllerService is ENABLED
> -
>
> Key: NIFI-3510
> URL: https://issues.apache.org/jira/browse/NIFI-3510
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Trivial
>
> We currently have logging for when we begin "enabling" a controller service, 
> but none for when it has actually been "enabled".  I suggest adding a line 
> like 
> {code}
> LOG.debug("Successfully Enabled " + this);
> {code}
> to StandardControllerServiceNode.enable()...
> 0.x:
> {code}
> if (shouldEnable) {
> heartbeater.heartbeat();
> LOG.debug("Successfully Enabled " + this);
> } else {...
> {code}
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java#L313-L315
> 1.x:
> {code}
> ...
> }
> else{
> LOG.debug("Successfully Enabled " + this);
> }
> {code}
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceNode.java#L357
> [~mosermw] [~jskora]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3402) Add ETag Support to InvokeHTTP

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163497#comment-16163497
 ] 

Michael Hogue commented on NIFI-3402:
-

PR submitted: https://github.com/apache/nifi/pull/2150

> Add ETag Support to InvokeHTTP
> --
>
> Key: NIFI-3402
> URL: https://issues.apache.org/jira/browse/NIFI-3402
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Trivial
>
> Unlike GetHTTP, when running in "source" mode InvokeHTTP doesn't support 
> ETags.  It will pull from a URL as often as it is scheduled to run.  When 
> running with an input relationship, it would potentially make sense to not 
> use the ETag.  But at least in "source" mode, it seems like it should at 
> least be an option.
> To maintain backwards compatibility and support the non-"source" usage, I'd 
> suggest creating a new "Use ETag" property that defaults to false...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3402) Add ETag Support to InvokeHTTP

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3402:
---

Assignee: Michael Hogue

> Add ETag Support to InvokeHTTP
> --
>
> Key: NIFI-3402
> URL: https://issues.apache.org/jira/browse/NIFI-3402
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Brandon DeVries
>Assignee: Michael Hogue
>Priority: Trivial
>
> Unlike GetHTTP, when running in "source" mode InvokeHTTP doesn't support 
> ETags.  It will pull from a URL as often as it is scheduled to run.  When 
> running with an input relationship, it would potentially make sense to not 
> use the ETag.  But at least in "source" mode, it seems like it should at 
> least be an option.
> To maintain backwards compatibility and support the non-"source" usage, I'd 
> suggest creating a new "Use ETag" property that defaults to false...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-3810) Fix API response type for /process-groups/{id}/process-groups

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue resolved NIFI-3810.
-
Resolution: Implemented

> Fix API response type for /process-groups/{id}/process-groups
> -
>
> Key: NIFI-3810
> URL: https://issues.apache.org/jira/browse/NIFI-3810
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Joe Skora
>Priority: Minor
>  Labels: easyfix, newbie
>
> The response type for {{nifi-api/process-groups/\{id}/process-groups}} is 
> defined as {{ProcessorsEntity.class}} but should be 
> {{ProcessGroupEntity.class}}.
> This doesn't affect NiFi itself, as far as I know, but it does impact APIs 
> defined based on the Swagger annotations, making it impossible to use the API 
> to list process groups within parent process group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3810) Fix API response type for /process-groups/{id}/process-groups

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163045#comment-16163045
 ] 

Michael Hogue commented on NIFI-3810:
-

This was fixed in NIFI-3939.

> Fix API response type for /process-groups/{id}/process-groups
> -
>
> Key: NIFI-3810
> URL: https://issues.apache.org/jira/browse/NIFI-3810
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Joe Skora
>Priority: Minor
>  Labels: easyfix, newbie
>
> The response type for {{nifi-api/process-groups/\{id}/process-groups}} is 
> defined as {{ProcessorsEntity.class}} but should be 
> {{ProcessGroupEntity.class}}.
> This doesn't affect NiFi itself, as far as I know, but it does impact APIs 
> defined based on the Swagger annotations, making it impossible to use the API 
> to list process groups within parent process group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (NIFI-3939) Fix entity type mismatches in web API resource response types

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue resolved NIFI-3939.
-
   Resolution: Fixed
Fix Version/s: 1.4.0

> Fix entity type mismatches in web API resource response types 
> --
>
> Key: NIFI-3939
> URL: https://issues.apache.org/jira/browse/NIFI-3939
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
> Fix For: 1.4.0
>
>
> In {{ProcessGroupResource.java}} annotations on the 
> {{/process-groups/\{id}/getProcessGroups}} endpoint incorrectly identifies 
> the response as a {{ProcessorsEntity}} when it should be a 
> {{ProcessGroupsEntity}}.  Similar mismatches may exist in 
> {{FlowFileQueueResource}}, {{FlowResource}}, and possibly other classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3218) MockProcessSession should prevent transferring new FlowFile to input queue

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157274#comment-16157274
 ] 

Michael Hogue commented on NIFI-3218:
-

Set fix version to 1.4.0.

> MockProcessSession should prevent transferring new FlowFile to input queue
> --
>
> Key: NIFI-3218
> URL: https://issues.apache.org/jira/browse/NIFI-3218
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0, 0.8.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
> Fix For: 1.4.0
>
>
> StandardProcessSession.transfer() throws an exception if called with a newly 
> created FlowFile and no relationship.  MockProcessionSession should behave 
> similarly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-3218) MockProcessSession should prevent transferring new FlowFile to input queue

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-3218:

Fix Version/s: 1.4.0

> MockProcessSession should prevent transferring new FlowFile to input queue
> --
>
> Key: NIFI-3218
> URL: https://issues.apache.org/jira/browse/NIFI-3218
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0, 0.8.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
> Fix For: 1.4.0
>
>
> StandardProcessSession.transfer() throws an exception if called with a newly 
> created FlowFile and no relationship.  MockProcessionSession should behave 
> similarly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135852#comment-16135852
 ] 

Michael Hogue edited comment on NIFI-3869 at 8/31/17 3:45 PM:
--

After changes to {{ListenHTTP}}, i wrote up a really quick performance test to 
see if there were any performance gains/losses for non-secure communications. I 
published the code and results on github:

https://github.com/m-hogue/nifi-http2

http2 is faster across the board in my testing for messages ranging in size 
between [10B, 100MB].


was (Author: m-hogue):
After changes to {{ListenHTTP}}, i wrote up a really quick performance test to 
see if there were any performance gains/losses. I published the code and 
results on github:

https://github.com/m-hogue/nifi-http2

http2 is faster across the board in my testing for messages ranging in size 
between [10B, 100MB].

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4335) Update Listen* components to refer to RestrictedSSLContextService

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149018#comment-16149018
 ] 

Michael Hogue commented on NIFI-4335:
-

NOTE: I've changed {{HandleHTTPRequest}} to refer to the 
RestrictedSSLContextService in PR 1985: https://github.com/apache/nifi/pull/1985

> Update Listen* components to refer to RestrictedSSLContextService
> -
>
> Key: NIFI-4335
> URL: https://issues.apache.org/jira/browse/NIFI-4335
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Extensions
>Affects Versions: 1.3.0
>Reporter: Andy LoPresto
>Assignee: Michael Hogue
>  Labels: jetty, security, tls
>
> [~m-hogue] added a {{RestrictedSSLContextService}} in NIFI-2528 due to 
> discussions regarding the baseline TLS protocol versions that are supported 
> as of NiFi 1.2.0. The {{ListenHTTP}} processor was updated to require this 
> new interface, but other {{Listen*}} processors and services need to be 
> updated as well. 
> From discussion on PR 1986: 
> {quote}
> Also, as @joewitt noted earlier, we should change the available interface for 
> other "listener" processors. Here's a preliminary list I put together, but I 
> would like confirmation from another member:
> * `HandleHTTPRequest`
> * `ListenBeats`
> * 
> `DistributedCacheServer`/`DistributedSetCacheServer`/`DistributedMapCacheServer`
> * `ListenSMTP`
> * `ListenGRPC`
> * `ListenLumberjack` (*Deprecated*)
> * `ListenRELP`
> * `ListenSyslog`
> * `ListenTCP`/`ListenTCPRecord`
> Also:
> * `AbstractSiteToSiteReportingTask`
> * `org.apache.nifi.processors.slack.TestServer`
> * `WebSocketService`/`JettyWebSocketService`
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4335) Update Listen* components to refer to RestrictedSSLContextService

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-4335:
---

Assignee: Michael Hogue

> Update Listen* components to refer to RestrictedSSLContextService
> -
>
> Key: NIFI-4335
> URL: https://issues.apache.org/jira/browse/NIFI-4335
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Extensions
>Affects Versions: 1.3.0
>Reporter: Andy LoPresto
>Assignee: Michael Hogue
>  Labels: jetty, security, tls
>
> [~m-hogue] added a {{RestrictedSSLContextService}} in NIFI-2528 due to 
> discussions regarding the baseline TLS protocol versions that are supported 
> as of NiFi 1.2.0. The {{ListenHTTP}} processor was updated to require this 
> new interface, but other {{Listen*}} processors and services need to be 
> updated as well. 
> From discussion on PR 1986: 
> {quote}
> Also, as @joewitt noted earlier, we should change the available interface for 
> other "listener" processors. Here's a preliminary list I put together, but I 
> would like confirmation from another member:
> * `HandleHTTPRequest`
> * `ListenBeats`
> * 
> `DistributedCacheServer`/`DistributedSetCacheServer`/`DistributedMapCacheServer`
> * `ListenSMTP`
> * `ListenGRPC`
> * `ListenLumberjack` (*Deprecated*)
> * `ListenRELP`
> * `ListenSyslog`
> * `ListenTCP`/`ListenTCPRecord`
> Also:
> * `AbstractSiteToSiteReportingTask`
> * `org.apache.nifi.processors.slack.TestServer`
> * `WebSocketService`/`JettyWebSocketService`
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4335) Update Listen* components to refer to RestrictedSSLContextService

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148401#comment-16148401
 ] 

Michael Hogue commented on NIFI-4335:
-

I can knock this out.

> Update Listen* components to refer to RestrictedSSLContextService
> -
>
> Key: NIFI-4335
> URL: https://issues.apache.org/jira/browse/NIFI-4335
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Extensions
>Affects Versions: 1.3.0
>Reporter: Andy LoPresto
>Assignee: Michael Hogue
>  Labels: jetty, security, tls
>
> [~m-hogue] added a {{RestrictedSSLContextService}} in NIFI-2528 due to 
> discussions regarding the baseline TLS protocol versions that are supported 
> as of NiFi 1.2.0. The {{ListenHTTP}} processor was updated to require this 
> new interface, but other {{Listen*}} processors and services need to be 
> updated as well. 
> From discussion on PR 1986: 
> {quote}
> Also, as @joewitt noted earlier, we should change the available interface for 
> other "listener" processors. Here's a preliminary list I put together, but I 
> would like confirmation from another member:
> * `HandleHTTPRequest`
> * `ListenBeats`
> * 
> `DistributedCacheServer`/`DistributedSetCacheServer`/`DistributedMapCacheServer`
> * `ListenSMTP`
> * `ListenGRPC`
> * `ListenLumberjack` (*Deprecated*)
> * `ListenRELP`
> * `ListenSyslog`
> * `ListenTCP`/`ListenTCPRecord`
> Also:
> * `AbstractSiteToSiteReportingTask`
> * `org.apache.nifi.processors.slack.TestServer`
> * `WebSocketService`/`JettyWebSocketService`
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135852#comment-16135852
 ] 

Michael Hogue edited comment on NIFI-3869 at 8/21/17 9:17 PM:
--

After changes to {{ListenHTTP}}, i wrote up a really quick performance test to 
see if there were any performance gains/losses. I published the code and 
results on github:

https://github.com/m-hogue/nifi-http2

http2 is faster across the board in my testing for messages ranging in size 
between [10B, 100MB].


was (Author: m-hogue):
After changes to {{ListenHTTP}}, i wrote up a really quick performance test to 
see if there were any changes. I published the code and results on github:

https://github.com/m-hogue/nifi-http2

http2 is faster across the board in my testing for messages ranging in size 
between [10B, 100MB].

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135852#comment-16135852
 ] 

Michael Hogue commented on NIFI-3869:
-

After changes to {{ListenHTTP}}, i wrote up a really quick performance test to 
see if there were any changes. I published the code and results on github:

https://github.com/m-hogue/nifi-http2

http2 is faster across the board in my testing for messages ranging in size 
between [10B, 100MB].

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127720#comment-16127720
 ] 

Michael Hogue commented on NIFI-3869:
-

Work here: https://github.com/m-hogue/nifi/tree/NIFI-3869

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127703#comment-16127703
 ] 

Michael Hogue commented on NIFI-3869:
-

[~trk...@gmail.com] : At a quick glance, i see two HTTP listening processors: 
{{ListenHTTP}} and {{HandleHTTPRequest}}. Do you know of more?

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3869) add http/2 support to HTTP listening processors

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3869:
---

Assignee: Michael Hogue

> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3376) Content repository disk usage is not close to reported size in Status Bar

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3376:
---

Assignee: (was: Michael Hogue)

> Content repository disk usage is not close to reported size in Status Bar
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
> Attachments: NIFI-3376_Content_Repo_size_demo.xml
>
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3376) Implement content repository ResourceClaim compaction

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096339#comment-16096339
 ] 

Michael Hogue commented on NIFI-3376:
-

[~markap14]: I'm at a point in implementation here where I know when a resource 
claim should be compacted, but i'm not entirely sure on how to rewrite a 
resource claim file. It looks like content claims are appended to a file 
without demarcation, which makes reading them out of an existing file pretty 
difficult. Any advice on constructing a new resource claim with only active 
content claims? 

My current WIP is pushed to the branch linked in my previous comment.

> Implement content repository ResourceClaim compaction
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
>Assignee: Michael Hogue
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3376) Implement content repository ResourceClaim compaction

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3376:
---

Assignee: Michael Hogue

> Implement content repository ResourceClaim compaction
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
>Assignee: Michael Hogue
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3376) Implement content repository ResourceClaim compaction

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094986#comment-16094986
 ] 

Michael Hogue commented on NIFI-3376:
-

compaction work here: https://github.com/m-hogue/nifi/tree/NIFI-3376

> Implement content repository ResourceClaim compaction
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3376) Implement content repository ResourceClaim compaction

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093086#comment-16093086
 ] 

Michael Hogue commented on NIFI-3376:
-

[~mosermw] : I've opened a PR for NIFI-3736 and NIFI-1603 
(https://github.com/apache/nifi/pull/2010), which leaves only resource claim 
compaction to be explored here. I've started looking into that.

> Implement content repository ResourceClaim compaction
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3736) NiFi not honoring the "nifi.content.claim.max.appendable.size" and "nifi.content.claim.max.flow.files" properties

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087572#comment-16087572
 ] 

Michael Hogue commented on NIFI-3736:
-

Work here: https://github.com/m-hogue/nifi/tree/NIFI-3736

> NiFi not honoring the "nifi.content.claim.max.appendable.size" and 
> "nifi.content.claim.max.flow.files" properties
> -
>
> Key: NIFI-3736
> URL: https://issues.apache.org/jira/browse/NIFI-3736
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Michael Hogue
>
> The nifi.properties file has two properties for controlling how many 
> FlowFiles to jam into one Content Claim. Unfortunately, it looks like this is 
> no longer honored in FileSystemRepository.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-1603) Remove deprecated content repo properties

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-1603:
---

Assignee: Michael Hogue

> Remove deprecated content repo properties
> -
>
> Key: NIFI-1603
> URL: https://issues.apache.org/jira/browse/NIFI-1603
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Configuration, Core Framework
>Affects Versions: 0.5.1
>Reporter: Aldrin Piri
>Assignee: Michael Hogue
>Priority: Trivial
>
> Both nifi.content.claim.max.appendable.size & 
> nifi.content.claim.max.flow.files no longer appear to be used with defaults 
> specified for the associated properties in FileSystemRepository's 
> maxAppendClaimLength and the configured writableClaimQueue length.
> These should either be removed from both the codebase and documentation or 
> configured to use the associated properties and have their documentation 
> updated to reflect current configuration defaults.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3736) NiFi not honoring the "nifi.content.claim.max.appendable.size" and "nifi.content.claim.max.flow.files" properties

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3736:
---

Assignee: Michael Hogue

> NiFi not honoring the "nifi.content.claim.max.appendable.size" and 
> "nifi.content.claim.max.flow.files" properties
> -
>
> Key: NIFI-3736
> URL: https://issues.apache.org/jira/browse/NIFI-3736
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Michael Hogue
>
> The nifi.properties file has two properties for controlling how many 
> FlowFiles to jam into one Content Claim. Unfortunately, it looks like this is 
> no longer honored in FileSystemRepository.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3376) Implement content repository ResourceClaim compaction

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087466#comment-16087466
 ] 

Michael Hogue commented on NIFI-3376:
-

Great. I think i can knock those two out and then return to this one. Thanks!

> Implement content repository ResourceClaim compaction
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-3376) Implement content repository ResourceClaim compaction

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087440#comment-16087440
 ] 

Michael Hogue edited comment on NIFI-3376 at 7/14/17 3:07 PM:
--

[~mosermw] : Would both of the solutions be viable? Exposing 
{{MAX_APPENDABLE_CLAIM_LENGTH}} seems fairly straightforward and a potential 
quick win, but would both compaction and exposing of this property be useful? 
If so, we might consider breaking this issue into two since compaction will 
require a bit more thought & care.

Meanwhile, i'm investigating adding resource claim compaction.


was (Author: m-hogue):
[~mosermw] : Would both of the solutions be viable? Exposing 
{{MAX_APPENDABLE_CLAIM_LENGTH}} seems fairly straightforward and a potential 
quick win, but would both compaction and exposing of this property be useful? 
If so, we might consider breaking this issue into two since compaction will 
require a bit more thought & care.

> Implement content repository ResourceClaim compaction
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3376) Implement content repository ResourceClaim compaction

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16087440#comment-16087440
 ] 

Michael Hogue commented on NIFI-3376:
-

[~mosermw] : Would both of the solutions be viable? Exposing 
{{MAX_APPENDABLE_CLAIM_LENGTH}} seems fairly straightforward and a potential 
quick win, but would both compaction and exposing of this property be useful? 
If so, we might consider breaking this issue into two since compaction will 
require a bit more thought & care.

> Implement content repository ResourceClaim compaction
> -
>
> Key: NIFI-3376
> URL: https://issues.apache.org/jira/browse/NIFI-3376
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.7.1, 1.1.1
>Reporter: Michael Moser
>
> On NiFi systems that deal with many files whose size is less than 1 MB, we 
> often see that the actual disk usage of the content_repository is much 
> greater than the size of flowfiles that NiFi reports are in its queues.  As 
> an example, NiFi may report "50,000 / 12.5 GB" but the content_repository 
> takes up 240 GB of its file system.  This leads to scenarios where a 500 GB 
> content_repository file system gets 100% full, but "I only had 40 GB of data 
> in my NiFi!"
> When several content claims exist in a single resource claim, and most but 
> not all content claims are terminated, the entire resource claim is still not 
> eligible for deletion or archive.  This could mean that only one 10 KB 
> content claim out of a 1 MB resource claim is counted by NiFi as existing in 
> its queues.
> If a particular flow has a slow egress point where flowfiles could back up 
> and remain on the system longer than expected, this problem is exacerbated.
> A potential solution is to compact resource claim files on disk. A background 
> thread could examine all resource claims, and for those that get "old" and 
> whose active content claim usage drops below a threshold, then rewrite the 
> resource claim file.
> A potential work-around is to allow modification of the FileSystemRepository 
> MAX_APPENDABLE_CLAIM_LENGTH to make it a smaller number.  This would increase 
> the probability that the content claims reference count in a resource claim 
> would reach 0 and the resource claim becomes eligible for deletion/archive.  
> Let users trade-off performance for more accurate accounting of NiFi queue 
> size to content repository size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4172) Typo in nifi-web-api Entity class name

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-4172:
---

Assignee: Michael Hogue

> Typo in nifi-web-api Entity class name
> --
>
> Key: NIFI-4172
> URL: https://issues.apache.org/jira/browse/NIFI-4172
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Trivial
>
> There's an Entity in {{nifi-web-api}} named {{ClusteSummaryEntity}}. This 
> should be {{ClusterSummaryEntity}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (NIFI-4172) Typo in nifi-web-api Entity class name

[Michael Hogue (JIRA)]

Michael Hogue created NIFI-4172:
---

 Summary: Typo in nifi-web-api Entity class name
 Key: NIFI-4172
 URL: https://issues.apache.org/jira/browse/NIFI-4172
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Michael Hogue
Priority: Trivial


There's an Entity in {{nifi-web-api}} named {{ClusteSummaryEntity}}. This 
should be {{ClusterSummaryEntity}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-3939) Fix entity type mismatches in web API resource response types

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-3939:

Description: In {{ProcessGroupResource.java}} annotations on the 
{{/process-groups/\{id}/getProcessGroups}} endpoint incorrectly identifies the 
response as a {{ProcessorsEntity}} when it should be a {{ProcessGroupsEntity}}. 
 Similar mismatches may exist in {{FlowFileQueueResource}}, {{FlowResource}}, 
and possibly other classes.  (was: In {{ProcessGroupResource.java}} annotations 
on the {{/process-groups/\{id}/getProcessGroups}} endpoint incorrectly 
identifies the response as a {{ProcessorsEntity}} when it should be a 
{{ProcessGroupsEntity}}.  Similar mismatches may exist in 
{{FlowFileQueueResponse}}, {{FlowResource}}, and possibly other classes.)

> Fix entity type mismatches in web API resource response types 
> --
>
> Key: NIFI-3939
> URL: https://issues.apache.org/jira/browse/NIFI-3939
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> In {{ProcessGroupResource.java}} annotations on the 
> {{/process-groups/\{id}/getProcessGroups}} endpoint incorrectly identifies 
> the response as a {{ProcessorsEntity}} when it should be a 
> {{ProcessGroupsEntity}}.  Similar mismatches may exist in 
> {{FlowFileQueueResource}}, {{FlowResource}}, and possibly other classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3939) Fix entity type mismatches in web API resource response types

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3939:
---

Assignee: Michael Hogue

> Fix entity type mismatches in web API resource response types 
> --
>
> Key: NIFI-3939
> URL: https://issues.apache.org/jira/browse/NIFI-3939
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.2.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> In {{ProcessGroupResource.java}} annotations on the 
> {{/process-groups/\{id}/getProcessGroups}} endpoint incorrectly identifies 
> the response as a {{ProcessorsEntity}} when it should be a 
> {{ProcessGroupsEntity}}.  Similar mismatches may exist in 
> {{FlowFileQueueResponse}}, {{FlowResource}}, and possibly other classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-3218) MockProcessSession should prevent transferring new FlowFile to input queue

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076815#comment-16076815
 ] 

Michael Hogue edited comment on NIFI-3218 at 7/6/17 4:28 PM:
-

I made an assumption that `currentVersions` in `MockProcessSession` would hold 
files created in "this session" since that's the data structure files are 
placed in on `create()`

[~jskora] : Please let me know if this meets the need.


was (Author: m-hogue):
Made an assumption that `currentVersions` in `MockProcessSession` would hold 
files created in "this session" since that's the data structure files are 
placed in on `create()`

> MockProcessSession should prevent transferring new FlowFile to input queue
> --
>
> Key: NIFI-3218
> URL: https://issues.apache.org/jira/browse/NIFI-3218
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0, 0.8.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> StandardProcessSession.transfer() throws an exception if called with a newly 
> created FlowFile and no relationship.  MockProcessionSession should behave 
> similarly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-3218) MockProcessSession should prevent transferring new FlowFile to input queue

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076815#comment-16076815
 ] 

Michael Hogue edited comment on NIFI-3218 at 7/6/17 4:28 PM:
-

I made an assumption that `currentVersions` in `MockProcessSession` would hold 
files created in "this session" since that's the data structure that FlowFiles 
are placed in on `create()`

[~jskora] : Please let me know if this meets the need.


was (Author: m-hogue):
I made an assumption that `currentVersions` in `MockProcessSession` would hold 
files created in "this session" since that's the data structure files are 
placed in on `create()`

[~jskora] : Please let me know if this meets the need.

> MockProcessSession should prevent transferring new FlowFile to input queue
> --
>
> Key: NIFI-3218
> URL: https://issues.apache.org/jira/browse/NIFI-3218
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0, 0.8.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> StandardProcessSession.transfer() throws an exception if called with a newly 
> created FlowFile and no relationship.  MockProcessionSession should behave 
> similarly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3218) MockProcessSession should prevent transferring new FlowFile to input queue

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076815#comment-16076815
 ] 

Michael Hogue commented on NIFI-3218:
-

Made an assumption that `currentVersions` in `MockProcessSession` would hold 
files created in "this session" since that's the data structure files are 
placed in on `create()`

> MockProcessSession should prevent transferring new FlowFile to input queue
> --
>
> Key: NIFI-3218
> URL: https://issues.apache.org/jira/browse/NIFI-3218
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0, 0.8.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> StandardProcessSession.transfer() throws an exception if called with a newly 
> created FlowFile and no relationship.  MockProcessionSession should behave 
> similarly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3218) MockProcessSession should prevent transferring new FlowFile to input queue

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076800#comment-16076800
 ] 

Michael Hogue commented on NIFI-3218:
-

I'll take a whack at this. Work here: 
https://github.com/m-hogue/nifi/tree/NIFI-3218

> MockProcessSession should prevent transferring new FlowFile to input queue
> --
>
> Key: NIFI-3218
> URL: https://issues.apache.org/jira/browse/NIFI-3218
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0, 0.8.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> StandardProcessSession.transfer() throws an exception if called with a newly 
> created FlowFile and no relationship.  MockProcessionSession should behave 
> similarly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-3218) MockProcessSession should prevent transferring new FlowFile to input queue

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-3218:
---

Assignee: Michael Hogue

> MockProcessSession should prevent transferring new FlowFile to input queue
> --
>
> Key: NIFI-3218
> URL: https://issues.apache.org/jira/browse/NIFI-3218
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.1.0, 0.8.0
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> StandardProcessSession.transfer() throws an exception if called with a newly 
> created FlowFile and no relationship.  MockProcessionSession should behave 
> similarly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-2528) Update ListenHTTP to honor SSLContextService Protocols

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076771#comment-16076771
 ] 

Michael Hogue commented on NIFI-2528:
-

Work here: https://github.com/m-hogue/nifi/tree/NIFI-2528

> Update ListenHTTP to honor SSLContextService Protocols
> --
>
> Key: NIFI-2528
> URL: https://issues.apache.org/jira/browse/NIFI-2528
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.8.0, 0.7.1
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> Update ListenHTTP to honor SSLContextService Protocols as [NIFI-1688] did for 
> PostHTTP.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-2528) Update ListenHTTP to honor SSLContextService Protocols

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-2528:
---

Assignee: Michael Hogue

> Update ListenHTTP to honor SSLContextService Protocols
> --
>
> Key: NIFI-2528
> URL: https://issues.apache.org/jira/browse/NIFI-2528
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.8.0, 0.7.1
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> Update ListenHTTP to honor SSLContextService Protocols as [NIFI-1688] did for 
> PostHTTP.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-2529) Update HandleHttpRequest to honor SSLContextService Protocols

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-2529:
---

Assignee: Michael Hogue

> Update HandleHttpRequest to honor SSLContextService Protocols
> -
>
> Key: NIFI-2529
> URL: https://issues.apache.org/jira/browse/NIFI-2529
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.8.0, 0.7.1
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> Update HandleHttpRequest to honor SSLContextService Protocols as [NIFI-1688] 
> did for PostHTTP.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-2529) Update HandleHttpRequest to honor SSLContextService Protocols

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076638#comment-16076638
 ] 

Michael Hogue commented on NIFI-2529:
-

Work here: https://github.com/m-hogue/nifi/tree/NIFI-2529

> Update HandleHttpRequest to honor SSLContextService Protocols
> -
>
> Key: NIFI-2529
> URL: https://issues.apache.org/jira/browse/NIFI-2529
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.8.0, 0.7.1
>Reporter: Joe Skora
>Assignee: Michael Hogue
>
> Update HandleHttpRequest to honor SSLContextService Protocols as [NIFI-1688] 
> did for PostHTTP.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-552) Improve LogAttribute property matching

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075441#comment-16075441
 ] 

Michael Hogue commented on NIFI-552:


I've elected to add properties to support regex matching rather than replace 
the existing ones. The way i've implemented it should be backward compatible 
and shouldn't break anyone who used the existing properties. 

> Improve LogAttribute property matching
> --
>
> Key: NIFI-552
> URL: https://issues.apache.org/jira/browse/NIFI-552
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.0.2
>Reporter: Michael Moser
>Assignee: Michael Hogue
>Priority: Minor
>
> The LogAttribute properties "Attributes to Log" and "Attributes to Ignore" 
> currently accept a comma separated list of attributes.  This becomes unwieldy 
> when you want to ignore a long list of attributes.
> Modify these properties or create new properties allowing users to specify a 
> regular expression.  Any attribute which matches the regular expression will 
> add that attribute to the appropriate "include" list or "ignore" list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-552) Improve LogAttribute property matching

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075440#comment-16075440
 ] 

Michael Hogue commented on NIFI-552:


Work here: https://github.com/m-hogue/nifi/tree/NIFI-552

> Improve LogAttribute property matching
> --
>
> Key: NIFI-552
> URL: https://issues.apache.org/jira/browse/NIFI-552
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.0.2
>Reporter: Michael Moser
>Assignee: Michael Hogue
>Priority: Minor
>
> The LogAttribute properties "Attributes to Log" and "Attributes to Ignore" 
> currently accept a comma separated list of attributes.  This becomes unwieldy 
> when you want to ignore a long list of attributes.
> Modify these properties or create new properties allowing users to specify a 
> regular expression.  Any attribute which matches the regular expression will 
> add that attribute to the appropriate "include" list or "ignore" list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-552) Improve LogAttribute property matching

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-552:
--

Assignee: Michael Hogue

> Improve LogAttribute property matching
> --
>
> Key: NIFI-552
> URL: https://issues.apache.org/jira/browse/NIFI-552
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.0.2
>Reporter: Michael Moser
>Assignee: Michael Hogue
>Priority: Minor
>
> The LogAttribute properties "Attributes to Log" and "Attributes to Ignore" 
> currently accept a comma separated list of attributes.  This becomes unwieldy 
> when you want to ignore a long list of attributes.
> Modify these properties or create new properties allowing users to specify a 
> regular expression.  Any attribute which matches the regular expression will 
> add that attribute to the appropriate "include" list or "ignore" list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3193) Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071731#comment-16071731
 ] 

Michael Hogue commented on NIFI-3193:
-

[~bdeep] : Good deal. I'll wrap this up then and get a PR in ASAP. Thanks for 
the response!

> Update ConsumeAMQP and PublishAMQP to retrieve username from certificate 
> common name
> 
>
> Key: NIFI-3193
> URL: https://issues.apache.org/jira/browse/NIFI-3193
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0, 0.7.1
>Reporter: Brian
>Assignee: Michael Hogue
>
> At the moment the NiFi AMQP processors can establish a SSL connection to 
> RabbitMQ but still user a user defined username and password to authenticate. 
> When using certificates RabbitMQ allows you to use to COMMON_NAME from the 
> certificate to authenticate instead of providing a username and password. 
> Unfortunately the NiFi processors do not support this so I would like to 
> request an update to the processors to enable this functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3193) Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065415#comment-16065415
 ] 

Michael Hogue commented on NIFI-3193:
-

Work here: https://github.com/m-hogue/nifi/tree/NIFI-3193

> Update ConsumeAMQP and PublishAMQP to retrieve username from certificate 
> common name
> 
>
> Key: NIFI-3193
> URL: https://issues.apache.org/jira/browse/NIFI-3193
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0, 0.7.1
>Reporter: Brian
>Assignee: Michael Hogue
>
> At the moment the NiFi AMQP processors can establish a SSL connection to 
> RabbitMQ but still user a user defined username and password to authenticate. 
> When using certificates RabbitMQ allows you to use to COMMON_NAME from the 
> certificate to authenticate instead of providing a username and password. 
> Unfortunately the NiFi processors do not support this so I would like to 
> request an update to the processors to enable this functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-3193) Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065392#comment-16065392
 ] 

Michael Hogue edited comment on NIFI-3193 at 6/27/17 8:04 PM:
--

[~bdeep] : A couple of things. In thinking about your feature request to go 
ahead and declare a queue if it doesn't exist, that's only possible for NiFi to 
do if you're not using exchanges (i.e. you're just using the default exchange). 
The reason is that RabbitMQ maintains a mapping of  which 
the exchanges leverage to route messages to the appropriate queue. In that 
case, NiFi would have no knowledge of which queue the message would be routed 
to.

Per one of the RabbitMQ docs [1], in the case that the exchange provided is 
empty, then the routing key is assumed to be the queue name. In that case, we 
could declare it if it doesn't exist. The issue is that there are a number of 
arguments in queue declaration that i feel would need to be explicit (See 
Queues in [1]). Perhaps it'd be better if the queue declaration were left to 
RabbitMQ admins? Or maybe just have sensible defaults? Thoughts?

> For example, when you declare a queue with the name of 
> "search-indexing-online", the AMQP 0-9-1 broker will bind it to the default 
> exchange using "search-indexing-online" as the routing key. Therefore, a 
> message published to the default exchange with the routing key 
> "search-indexing-online" will be routed to the queue 
> "search-indexing-online". In other words, the default exchange makes it seem 
> like it is possible to deliver messages directly to queues, even though that 
> is not technically what is happening.

[1] https://www.rabbitmq.com/tutorials/amqp-concepts.html -- See 'Default 
Exchange', 'Queues'


was (Author: m-hogue):
[~bdeep] : A couple of things. In thinking about your feature request to go 
ahead and declare a queue if it doesn't exist, that's only possible for NiFi to 
do if you're not using exchanges (i.e. you're just using the default exchange). 
The reason is that RabbitMQ maintains a mapping of  which 
the exchanges leverage to route messages to the appropriate queue. In that 
case, NiFi would have no knowledge of which queue the message would be routed 
to.

Per one of the RabbitMQ docs [1], in the case that the exchange provided is 
empty, then the routing key is assumed to be the queue name. In that case, we 
could declare it if it doesn't exist. The issue is that there are a number of 
arguments in queue declaration that i feel would need to be explicit (See 
Queues in [1]). Perhaps it'd be better if the queue declaration were left to 
RabbitMQ admins? Thoughts?

> For example, when you declare a queue with the name of 
> "search-indexing-online", the AMQP 0-9-1 broker will bind it to the default 
> exchange using "search-indexing-online" as the routing key. Therefore, a 
> message published to the default exchange with the routing key 
> "search-indexing-online" will be routed to the queue 
> "search-indexing-online". In other words, the default exchange makes it seem 
> like it is possible to deliver messages directly to queues, even though that 
> is not technically what is happening.

[1] https://www.rabbitmq.com/tutorials/amqp-concepts.html -- See 'Default 
Exchange', 'Queues'

> Update ConsumeAMQP and PublishAMQP to retrieve username from certificate 
> common name
> 
>
> Key: NIFI-3193
> URL: https://issues.apache.org/jira/browse/NIFI-3193
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0, 0.7.1
>Reporter: Brian
>Assignee: Michael Hogue
>
> At the moment the NiFi AMQP processors can establish a SSL connection to 
> RabbitMQ but still user a user defined username and password to authenticate. 
> When using certificates RabbitMQ allows you to use to COMMON_NAME from the 
> certificate to authenticate instead of providing a username and password. 
> Unfortunately the NiFi processors do not support this so I would like to 
> request an update to the processors to enable this functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3193) Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065392#comment-16065392
 ] 

Michael Hogue commented on NIFI-3193:
-

[~bdeep] : A couple of things. In thinking about your feature request to go 
ahead and declare a queue if it doesn't exist, that's only possible for NiFi to 
do if you're not using exchanges (i.e. you're just using the default exchange). 
The reason is that RabbitMQ maintains a mapping of  which 
the exchanges leverage to route messages to the appropriate queue. In that 
case, NiFi would have no knowledge of which queue the message would be routed 
to.

Per one of the RabbitMQ docs [1], in the case that the exchange provided is 
empty, then the routing key is assumed to be the queue name. In that case, we 
could declare it if it doesn't exist. The issue is that there are a number of 
arguments in queue declaration that i feel would need to be explicit (See 
Queues in [1]). Perhaps it'd be better if the queue declaration were left to 
RabbitMQ admins? Thoughts?

> For example, when you declare a queue with the name of 
> "search-indexing-online", the AMQP 0-9-1 broker will bind it to the default 
> exchange using "search-indexing-online" as the routing key. Therefore, a 
> message published to the default exchange with the routing key 
> "search-indexing-online" will be routed to the queue 
> "search-indexing-online". In other words, the default exchange makes it seem 
> like it is possible to deliver messages directly to queues, even though that 
> is not technically what is happening.

[1] https://www.rabbitmq.com/tutorials/amqp-concepts.html -- See 'Default 
Exchange', 'Queues'

> Update ConsumeAMQP and PublishAMQP to retrieve username from certificate 
> common name
> 
>
> Key: NIFI-3193
> URL: https://issues.apache.org/jira/browse/NIFI-3193
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0, 0.7.1
>Reporter: Brian
>Assignee: Michael Hogue
>
> At the moment the NiFi AMQP processors can establish a SSL connection to 
> RabbitMQ but still user a user defined username and password to authenticate. 
> When using certificates RabbitMQ allows you to use to COMMON_NAME from the 
> certificate to authenticate instead of providing a username and password. 
> Unfortunately the NiFi processors do not support this so I would like to 
> request an update to the processors to enable this functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-3193) Update ConsumeAMQP and PublishAMQP to retrieve username from certificate common name

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065166#comment-16065166
 ] 

Michael Hogue commented on NIFI-3193:
-

I'm on it.

> Update ConsumeAMQP and PublishAMQP to retrieve username from certificate 
> common name
> 
>
> Key: NIFI-3193
> URL: https://issues.apache.org/jira/browse/NIFI-3193
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0, 0.7.1
>Reporter: Brian
>Assignee: Michael Hogue
>
> At the moment the NiFi AMQP processors can establish a SSL connection to 
> RabbitMQ but still user a user defined username and password to authenticate. 
> When using certificates RabbitMQ allows you to use to COMMON_NAME from the 
> certificate to authenticate instead of providing a username and password. 
> Unfortunately the NiFi processors do not support this so I would like to 
> request an update to the processors to enable this functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4038) Create gRPC server processor

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-4038:

Attachment: listen_and_invoke_grpc.xml

> Create gRPC server processor
> 
>
> Key: NIFI-4038
> URL: https://issues.apache.org/jira/browse/NIFI-4038
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Attachments: listen_and_invoke_grpc.xml
>
>
> Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
> listens for RPCs from the gRPC processor created in NIFI-4037.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4038) Create gRPC server processor

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063802#comment-16063802
 ] 

Michael Hogue commented on NIFI-4038:
-

PR opened: https://github.com/apache/nifi/pull/1947

> Create gRPC server processor
> 
>
> Key: NIFI-4038
> URL: https://issues.apache.org/jira/browse/NIFI-4038
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
>
> Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
> listens for RPCs from the gRPC processor created in NIFI-4037.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (NIFI-4037) Create gRPC client processor

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054223#comment-16054223
 ] 

Michael Hogue edited comment on NIFI-4037 at 6/19/17 3:37 PM:
--

I tested this by creating a flow with two processors: {{GenerateFlowFile}} and 
{{InvokeGRPC}} with a local gRPC server running that dumps the messages: 
https://github.com/m-hogue/nifi-grpc-server

See attached image.


was (Author: m-hogue):
I tested this by creating a flow with two processors: {{GenerateFlowFile}} and 
{{InvokeGRPC}} with a local gRPC server running that dumps the messages: 
https://github.com/m-hogue/nifi-grpc-server

!nifi_invoke_grpc.png|thumbnail!

> Create gRPC client processor 
> -
>
> Key: NIFI-4037
> URL: https://issues.apache.org/jira/browse/NIFI-4037
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Attachments: nifi_invoke_grpc.png
>
>
> Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
> allow  the sharing of flow files to a configured remote gRPC server.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4037) Create gRPC client processor

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054223#comment-16054223
 ] 

Michael Hogue commented on NIFI-4037:
-

I tested this by creating a flow with two processors: {{GenerateFlowFile}} and 
{{InvokeGRPC}} with a local gRPC server running that dumps the messages: 
https://github.com/m-hogue/nifi-grpc-server

!nifi_invoke_grpc.png|thumbnail!

> Create gRPC client processor 
> -
>
> Key: NIFI-4037
> URL: https://issues.apache.org/jira/browse/NIFI-4037
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Attachments: nifi_invoke_grpc.png
>
>
> Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
> allow  the sharing of flow files to a configured remote gRPC server.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (NIFI-4037) Create gRPC client processor

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-4037:

Attachment: nifi_invoke_grpc.png

> Create gRPC client processor 
> -
>
> Key: NIFI-4037
> URL: https://issues.apache.org/jira/browse/NIFI-4037
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
> Attachments: nifi_invoke_grpc.png
>
>
> Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
> allow  the sharing of flow files to a configured remote gRPC server.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4037) Create gRPC client processor

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049848#comment-16049848
 ] 

Michael Hogue commented on NIFI-4037:
-

Work is on this branch: 
https://github.com/m-hogue/nifi/tree/NIFI-4037-grpc-client-proc

Used this to test outside of unit tests initially: 
https://github.com/m-hogue/nifi-grpc-server

> Create gRPC client processor 
> -
>
> Key: NIFI-4037
> URL: https://issues.apache.org/jira/browse/NIFI-4037
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
>
> Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
> allow  the sharing of flow files to a configured remote gRPC server.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (NIFI-4038) Create gRPC server processor

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue reassigned NIFI-4038:
---

Assignee: Michael Hogue

> Create gRPC server processor
> 
>
> Key: NIFI-4038
> URL: https://issues.apache.org/jira/browse/NIFI-4038
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Assignee: Michael Hogue
>Priority: Minor
>
> Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
> listens for RPCs from the gRPC processor created in NIFI-4037.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-4037) Create gRPC client processor

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-4037:

Description: 
Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
allow  the sharing of flow files to a configured remote gRPC server.

[1] http://www.grpc.io/about/

  was:Create a simple gRPC client processor similar to InvokeHTTP that would 
allow  the sharing of flow files to a configured remote gRPC server.


> Create gRPC client processor 
> -
>
> Key: NIFI-4037
> URL: https://issues.apache.org/jira/browse/NIFI-4037
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Priority: Minor
>
> Create a simple gRPC [1] client processor similar to InvokeHTTP that would 
> allow  the sharing of flow files to a configured remote gRPC server.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (NIFI-4038) Create gRPC server processor

[Michael Hogue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Hogue updated NIFI-4038:

Description: 
Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
listens for RPCs from the gRPC processor created in NIFI-4037.

[1] http://www.grpc.io/about/

  was:Create a simple gRPC server processor similar to `HandleHttpRequest` that 
listens for RPCs from the gRPC processor created in NIFI-4037.


> Create gRPC server processor
> 
>
> Key: NIFI-4038
> URL: https://issues.apache.org/jira/browse/NIFI-4038
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Michael Hogue
>Priority: Minor
>
> Create a simple gRPC [1] server processor similar to `HandleHttpRequest` that 
> listens for RPCs from the gRPC processor created in NIFI-4037.
> [1] http://www.grpc.io/about/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)