[jira] [Updated] (NIFI-5582) Integrate legacy behavior of HashAttribute into CryptographicHashAttribute
[
https://issues.apache.org/jira/browse/NIFI-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy LoPresto updated NIFI-5582:
Description:
There has been discussion on the mailing lists regarding the use of the
existing {{HashAttribute}} processor and the introduction of
{{CryptographicHashAttribute}}. The behavior of these processors does not
currently overlap, but {{CHA}} can be made to include {{HA}}'s functionality in
the documented use cases (to generate a unique identifier over a set of
specific attributes and values), if not its exact behavior.
*From discussion*
{quote}
Given your well-described use cases for HA, I think I may be able to provide
that in CHA as well. I would expect to add a dropdown PD for “attribute
enumeration style” and offer “individual” (each hash is generated on a single
attribute), “list” (each hash is generated over an ordered, delimited list of
literal matches), and “regex” (each hash is generated over an ordered list of
all attribute names matching the provided regex). Then the dynamic properties
would describe the output, as happens in the existing PR. Maybe a custom
delimiter property is needed too, but for now ‘’ could be used to join the
values. I’ll write up a Jira for this, and hopefully you can both let me know
if this meets your requirements.
Example:
*Incoming Flowfile*
attributes: [username: “alopresto”, role: “security”, email:
“alopre...@apache.org”, git_account: “alopresto”]
*CHA Properties (Individual)*
attribute_enumeration_style: “individual”
(dynamic) username_sha256: “username”
(dynamic) git_account_sha256: “git_account”
*Behavior (Individual)*
username_sha256 = git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) =
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23
*Resulting Flowfile (Individual)*
attributes: [username: “alopresto”, role: “security”, email:
“alopre...@apache.org”, git_account: “alopresto”, username_sha256:
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”,
git_account_sha256:
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23"]
*CHA Properties (List)*
attribute_enumeration_style: “list”
(dynamic) username_and_email_sha256: “username, email”
(dynamic) git_account_sha256: “git_account”
*Behavior (List)*
username_and_email_sha256 = $(echo -n "aloprestoalopre...@apache.org" | shasum
-a 256) = 22a11b7b3173f95c23a1f434949ec2a2e66455b9cb26b7ebc90afca25d91333f
git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) =
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23
*Resulting Flowfile (List)*
attributes: [username: “alopresto”, role: “security”, email:
“alopre...@apache.org”, git_account: “alopresto”, username_email_sha256: “
22a11b7b3173f95c23a1f434949ec2a2e66455b9cb26b7ebc90afca25d91333f”,
git_account_sha256:
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”]
*CHA Properties (Regex)*
attribute_enumeration_style: “regex”
(dynamic) all_sha256: “.*”
(dynamic) git_account_sha256: “git_account”
*Behavior (Regex)*
all_sha256 = sort(attributes_that_match_regex) = [email, git_account, role,
username] = $(echo -n "alopresto@apache.orgaloprestosecurityalopresto" | shasum
-a 256) = b370fdf0132933cea76e3daa3d4a437bb8c571dd0cd0e79ee5d7759cf64efced
git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) =
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23
*Resulting Flowfile (Regex)*
attributes: [username: “alopresto”, role: “security”, email:
“alopre...@apache.org”, git_account: “alopresto”, all_sha256: “
b370fdf0132933cea76e3daa3d4a437bb8c571dd0cd0e79ee5d7759cf64efced”,
git_account_sha256:
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”]
{quote}
This will necessitate switching the "order" of dynamic properties in
{{CryptographicHashAttribute}} -- rather than a dynamic property
*existing_attribute_name* - {{new_attribute_name_containing_hash}}, the
ordering will be *new_attribute_name_containing_hash* -
{{existing_attribute_name}} to allow for other values like {{attribute_.*}} or
{{attribute_1, attribute_2}}.
There will also be a boolean flag to include the attribute name in the hashed
value:
Example:
*existing_attribute_name* - "some value"
If _true_, the value in *new_attribute_name_containing_hash* would be
{{hash("existing_attribute_namesome value")}}. If _false_, it would be
{{hash("some value")}}. As no one is using the new
{{CryptographicHashAttribute}} in the field yet, this change can only be made
now.
[Mailing list
discussion|https://lists.apache.org/thread.html/7defc9dbcb5e900a66bfc58b3e96f4860397dab0f0859c27f2e72061@%3Cusers.nifi.apache.org%3E]
was:
There has been discussion on the mailing lists regarding the use of the
existing {{HashAttribute}} processor and the introduction of
{{CryptographicHashAttribute}}. The behavior of these processors does not
currently
[jira] [Commented] (NIFI-5566) Bring HashContent inline with HashService and rename legacy components
[
https://issues.apache.org/jira/browse/NIFI-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607899#comment-16607899
]
ASF GitHub Bot commented on NIFI-5566:
--
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2983
Based on @thenatog 's comment on [PR
2980](https://github.com/apache/nifi/pull/2980#issuecomment-419574743) which
captured a Java Big Endian Byte Order Mark (BOM) being inserted in UTF-16
strings pre-hash, I added some logic to prevent that here. I closed PR 2980,
and this PR can be considered as a standalone solution to
[NIFI-5147](https://issues.apache.org/jira/browse/NIFI-5147) and
[NIFI-5566](https://issues.apache.org/jira/browse/NIFI-5566). I also opened
[NIFI-5582](https://issues.apache.org/jira/browse/NIFI-5582) which depends on
this PR.
> Bring HashContent inline with HashService and rename legacy components
> --
>
> Key: NIFI-5566
> URL: https://issues.apache.org/jira/browse/NIFI-5566
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: backwards-compatibility, hash, security
>
> As documented in [NIFI-5147|https://issues.apache.org/jira/browse/NIFI-5147]
> and [PR 2980|https://github.com/apache/nifi/pull/2980], the {{HashAttribute}}
> processor and {{HashContent}} processor are lacking some features, do not
> offer consistent algorithms across platforms, etc.
> I propose the following:
> * Rename {{HashAttribute}} (which does not provide the service of calculating
> a hash over one or more attributes) to {{HashAttributeLegacy}}
> * Renamed {{CalculateAttributeHash}} to {{HashAttribute}} to make semantic
> sense
> * Rename {{HashContent}} to {{HashContentLegacy}} for users who need obscure
> digest algorithms which may or may not have been offered on their platform
> * Implement a processor {{HashContent}} with similar semantics to the
> existing processor but with consistent algorithm offerings and using the
> common {{HashService}} offering
> With the new component versioning features provided as part of the flow
> versioning behavior, silently disrupting existing flows which use these
> processors is no longer a concern. Rather, Any flow currently using the
> existing processors will either:
> 1. continue normal operation
> 1. require flow manager interaction and provide documentation about the change
> 1. migration notes and upgrade instructions will be provided
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2983: NIFI-5566 Improve HashContent processor and standardize Ha...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2983 Based on @thenatog 's comment on [PR 2980](https://github.com/apache/nifi/pull/2980#issuecomment-419574743) which captured a Java Big Endian Byte Order Mark (BOM) being inserted in UTF-16 strings pre-hash, I added some logic to prevent that here. I closed PR 2980, and this PR can be considered as a standalone solution to [NIFI-5147](https://issues.apache.org/jira/browse/NIFI-5147) and [NIFI-5566](https://issues.apache.org/jira/browse/NIFI-5566). I also opened [NIFI-5582](https://issues.apache.org/jira/browse/NIFI-5582) which depends on this PR. ---
[jira] [Commented] (NIFI-5147) Improve HashAttribute processor
[
https://issues.apache.org/jira/browse/NIFI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607897#comment-16607897
]
ASF GitHub Bot commented on NIFI-5147:
--
Github user alopresto closed the pull request at:
https://github.com/apache/nifi/pull/2980
> Improve HashAttribute processor
> ---
>
> Key: NIFI-5147
> URL: https://issues.apache.org/jira/browse/NIFI-5147
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: hash, security
> Fix For: 1.8.0
>
>
> The {{HashAttribute}} processor currently has surprising behavior. Barring
> familiarity with the processor, a user would expect {{HashAttribute}} to
> generate a hash value over one or more attributes. Instead, the processor as
> it is implemented "groups" incoming flowfiles into groups based on regular
> expressions which match attribute values, and then generates a
> (non-configurable) MD5 hash over the concatenation of the matching attribute
> keys and values.
> In addition:
> * the processor throws an error and routes to failure any incoming flowfile
> which does not have all attributes specified in the processor
> * the use of MD5 is vastly deprecated
> * no other hash algorithms are available
> I am unaware of community use of this processor, but I do not want to break
> backward compatibility. I propose the following steps:
> * Implement a new {{CalculateAttributeHash}} processor (awkward name, but
> this processor already has the desired name)
> ** This processor will perform the "standard" use case -- identify an
> attribute, calculate the specified hash over the value, and write it to an
> output attribute
> ** This processor will have a required property descriptor allowing a
> dropdown menu of valid hash algorithms
> ** This processor will accept arbitrary dynamic properties identifying the
> attributes to be hashed as a key, and the resulting attribute name as a value
> ** Example: I want to generate a SHA-512 hash on the attribute {{username}},
> and a flowfile enters the processor with {{username}} value {{alopresto}}. I
> configure {{algorithm}} with {{SHA-512}} and add a dynamic property
> {{username}} -- {{username_SHA512}}. The resulting flowfile will have
> attribute {{username_SHA512}} with value
> {{739b4f6722fb5de20125751c7a1a358b2a7eb8f07e530e4bf18561fbff93234908aa9d250c876bca9ede5ba784d5ce6081dbbdfe5ddd446678f223b8d632}}
> * Improve the documentation of this processor to explain the goal/expected
> use case (?)
> * Link in processor documentation to new processor for standard use cases
> * Remove the error alert when an incoming flowfile does not contain all
> expected attributes. I propose changing the severity to INFO and still
> routing to failure
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5147) Improve HashAttribute processor
[
https://issues.apache.org/jira/browse/NIFI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607896#comment-16607896
]
ASF GitHub Bot commented on NIFI-5147:
--
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2980
Thanks for discovering this @thenatog . This is an excellent catch.
I've added behavior to catch this, better documentation, and unit tests.
However, I added them on the branch that includes [PR
2983](https://github.com/apache/nifi/pull/2983). Let's mark this PR as closed
and just review the other one, as it is more complete and addresses this issue.
```
2018-09-07 21:21:19,784 WARN [Timer-Driven Process Thread-6]
o.a.n.security.util.crypto.HashService The charset provided was UTF-16, but
Java will insert a Big Endian BOM in the decoded message before hashing, so
switching to UTF-16BE
2018-09-07 21:21:19,797 INFO [Timer-Driven Process Thread-9]
o.a.n.processors.standard.LogAttribute
LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file
StandardFlowFileRecord[uuid=a4a223fb-aa11-43b9-93a3-d7675c44593c,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default,
section=1], offset=56, length=4],offset=0,name=33467912436349,size=4]
[SUCCESS]
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Fri Sep 07 21:21:19 PDT 2018'
Key: 'lineageStartDate'
Value: 'Fri Sep 07 21:21:19 PDT 2018'
Key: 'fileSize'
Value: '4'
FlowFile Attribute Map Content
Key: 'filename'
Value: '33467912436349'
Key: 'path'
Value: './'
Key: 'test_attribute'
Value: 'hehe'
Key: 'test_attribute_md5_utf16le'
Value: '2db0ecc27f7abd29ba95412feb3b5e07'
Key: 'uuid'
Value: 'a4a223fb-aa11-43b9-93a3-d7675c44593c'
[SUCCESS]
hehe
2018-09-07 21:21:19,799 INFO [Timer-Driven Process Thread-9]
o.a.n.processors.standard.LogAttribute
LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file
StandardFlowFileRecord[uuid=b7459e40-500b-488d-a0dc-3e09ebc6b86e,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default,
section=1], offset=56, length=4],offset=0,name=33467912436349,size=4]
[SUCCESS]
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Fri Sep 07 21:21:19 PDT 2018'
Key: 'lineageStartDate'
Value: 'Fri Sep 07 21:21:19 PDT 2018'
Key: 'fileSize'
Value: '4'
FlowFile Attribute Map Content
Key: 'filename'
Value: '33467912436349'
Key: 'path'
Value: './'
Key: 'test_attribute'
Value: 'hehe'
Key: 'test_attribute_md5_utf16'
Value: 'b0ed26b524e0b0606551d78e42b5b7bc'
Key: 'uuid'
Value: 'b7459e40-500b-488d-a0dc-3e09ebc6b86e'
[SUCCESS]
hehe
2018-09-07 21:21:19,801 INFO [Timer-Driven Process Thread-9]
o.a.n.processors.standard.LogAttribute
LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file
StandardFlowFileRecord[uuid=25c5d1b1-faa4-418d-911c-5c0cea399b83,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default,
section=1], offset=56, length=4],offset=0,name=33467912436349,size=4]
[SUCCESS]
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Fri Sep 07 21:21:19 PDT 2018'
Key: 'lineageStartDate'
Value: 'Fri Sep 07 21:21:19 PDT 2018'
Key: 'fileSize'
Value: '4'
FlowFile Attribute Map Content
Key: 'filename'
Value: '33467912436349'
Key: 'path'
Value: './'
Key: 'test_attribute'
Value: 'hehe'
Key: 'test_attribute_md5_utf16be'
Value: 'b0ed26b524e0b0606551d78e42b5b7bc'
Key: 'uuid'
Value: '25c5d1b1-faa4-418d-911c-5c0cea399b83'
[SUCCESS]
hehe
```
> Improve HashAttribute processor
> ---
>
> Key: NIFI-5147
> URL: https://issues.apache.org/jira/browse/NIFI-5147
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: hash, security
> Fix For: 1.8.0
>
>
> The {{HashAttribute}} processor currently has surprising behavior. Barring
> familiarity with the processor, a user would expect {{HashAttribute}} to
> generate a hash value over one or more attributes. Instead, the processor as
> it is implemented "groups" incoming flowfiles into groups based
[GitHub] nifi pull request #2980: NIFI-5147 Implement CalculateAttributeHash processo...
Github user alopresto closed the pull request at: https://github.com/apache/nifi/pull/2980 ---
[GitHub] nifi issue #2980: NIFI-5147 Implement CalculateAttributeHash processor
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2980 Thanks for discovering this @thenatog . This is an excellent catch. I've added behavior to catch this, better documentation, and unit tests. However, I added them on the branch that includes [PR 2983](https://github.com/apache/nifi/pull/2983). Let's mark this PR as closed and just review the other one, as it is more complete and addresses this issue. ``` 2018-09-07 21:21:19,784 WARN [Timer-Driven Process Thread-6] o.a.n.security.util.crypto.HashService The charset provided was UTF-16, but Java will insert a Big Endian BOM in the decoded message before hashing, so switching to UTF-16BE 2018-09-07 21:21:19,797 INFO [Timer-Driven Process Thread-9] o.a.n.processors.standard.LogAttribute LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file StandardFlowFileRecord[uuid=a4a223fb-aa11-43b9-93a3-d7675c44593c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default, section=1], offset=56, length=4],offset=0,name=33467912436349,size=4] [SUCCESS] Standard FlowFile Attributes Key: 'entryDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'lineageStartDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'fileSize' Value: '4' FlowFile Attribute Map Content Key: 'filename' Value: '33467912436349' Key: 'path' Value: './' Key: 'test_attribute' Value: 'hehe' Key: 'test_attribute_md5_utf16le' Value: '2db0ecc27f7abd29ba95412feb3b5e07' Key: 'uuid' Value: 'a4a223fb-aa11-43b9-93a3-d7675c44593c' [SUCCESS] hehe 2018-09-07 21:21:19,799 INFO [Timer-Driven Process Thread-9] o.a.n.processors.standard.LogAttribute LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file StandardFlowFileRecord[uuid=b7459e40-500b-488d-a0dc-3e09ebc6b86e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default, section=1], offset=56, length=4],offset=0,name=33467912436349,size=4] [SUCCESS] Standard FlowFile Attributes Key: 'entryDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'lineageStartDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'fileSize' Value: '4' FlowFile Attribute Map Content Key: 'filename' Value: '33467912436349' Key: 'path' Value: './' Key: 'test_attribute' Value: 'hehe' Key: 'test_attribute_md5_utf16' Value: 'b0ed26b524e0b0606551d78e42b5b7bc' Key: 'uuid' Value: 'b7459e40-500b-488d-a0dc-3e09ebc6b86e' [SUCCESS] hehe 2018-09-07 21:21:19,801 INFO [Timer-Driven Process Thread-9] o.a.n.processors.standard.LogAttribute LogAttribute[id=b15f3209-344d-10a6-4a7b-454530bb72fc] logging for flow file StandardFlowFileRecord[uuid=25c5d1b1-faa4-418d-911c-5c0cea399b83,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1536378604366-1, container=default, section=1], offset=56, length=4],offset=0,name=33467912436349,size=4] [SUCCESS] Standard FlowFile Attributes Key: 'entryDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'lineageStartDate' Value: 'Fri Sep 07 21:21:19 PDT 2018' Key: 'fileSize' Value: '4' FlowFile Attribute Map Content Key: 'filename' Value: '33467912436349' Key: 'path' Value: './' Key: 'test_attribute' Value: 'hehe' Key: 'test_attribute_md5_utf16be' Value: 'b0ed26b524e0b0606551d78e42b5b7bc' Key: 'uuid' Value: '25c5d1b1-faa4-418d-911c-5c0cea399b83' [SUCCESS] hehe ``` ---
[jira] [Commented] (NIFI-5147) Improve HashAttribute processor
[
https://issues.apache.org/jira/browse/NIFI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607894#comment-16607894
]
Andy LoPresto commented on NIFI-5147:
-
Noting here that I have opened
[NIFI-5582|https://issues.apache.org/jira/browse/NIFI-5582] in response to
mailing list discussion and this will switch the behavior of the dynamic
properties described above, as this processor has not yet been released.
> Improve HashAttribute processor
> ---
>
> Key: NIFI-5147
> URL: https://issues.apache.org/jira/browse/NIFI-5147
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: hash, security
> Fix For: 1.8.0
>
>
> The {{HashAttribute}} processor currently has surprising behavior. Barring
> familiarity with the processor, a user would expect {{HashAttribute}} to
> generate a hash value over one or more attributes. Instead, the processor as
> it is implemented "groups" incoming flowfiles into groups based on regular
> expressions which match attribute values, and then generates a
> (non-configurable) MD5 hash over the concatenation of the matching attribute
> keys and values.
> In addition:
> * the processor throws an error and routes to failure any incoming flowfile
> which does not have all attributes specified in the processor
> * the use of MD5 is vastly deprecated
> * no other hash algorithms are available
> I am unaware of community use of this processor, but I do not want to break
> backward compatibility. I propose the following steps:
> * Implement a new {{CalculateAttributeHash}} processor (awkward name, but
> this processor already has the desired name)
> ** This processor will perform the "standard" use case -- identify an
> attribute, calculate the specified hash over the value, and write it to an
> output attribute
> ** This processor will have a required property descriptor allowing a
> dropdown menu of valid hash algorithms
> ** This processor will accept arbitrary dynamic properties identifying the
> attributes to be hashed as a key, and the resulting attribute name as a value
> ** Example: I want to generate a SHA-512 hash on the attribute {{username}},
> and a flowfile enters the processor with {{username}} value {{alopresto}}. I
> configure {{algorithm}} with {{SHA-512}} and add a dynamic property
> {{username}} -- {{username_SHA512}}. The resulting flowfile will have
> attribute {{username_SHA512}} with value
> {{739b4f6722fb5de20125751c7a1a358b2a7eb8f07e530e4bf18561fbff93234908aa9d250c876bca9ede5ba784d5ce6081dbbdfe5ddd446678f223b8d632}}
> * Improve the documentation of this processor to explain the goal/expected
> use case (?)
> * Link in processor documentation to new processor for standard use cases
> * Remove the error alert when an incoming flowfile does not contain all
> expected attributes. I propose changing the severity to INFO and still
> routing to failure
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5582) Integrate legacy behavior of HashAttribute into CryptographicHashAttribute
Andy LoPresto created NIFI-5582:
---
Summary: Integrate legacy behavior of HashAttribute into
CryptographicHashAttribute
Key: NIFI-5582
URL: https://issues.apache.org/jira/browse/NIFI-5582
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 1.7.1
Reporter: Andy LoPresto
Assignee: Andy LoPresto
There has been discussion on the mailing lists regarding the use of the
existing {{HashAttribute}} processor and the introduction of
{{CryptographicHashAttribute}}. The behavior of these processors does not
currently overlap, but {{CHA}} can be made to include {{HA}}'s functionality in
the documented use cases (to generate a unique identifier over a set of
specific attributes and values), if not its exact behavior.
*From discussion*
{quote}
Given your well-described use cases for HA, I think I may be able to provide
that in CHA as well. I would expect to add a dropdown PD for “attribute
enumeration style” and offer “individual” (each hash is generated on a single
attribute), “list” (each hash is generated over an ordered, delimited list of
literal matches), and “regex” (each hash is generated over an ordered list of
all attribute names matching the provided regex). Then the dynamic properties
would describe the output, as happens in the existing PR. Maybe a custom
delimiter property is needed too, but for now ‘’ could be used to join the
values. I’ll write up a Jira for this, and hopefully you can both let me know
if this meets your requirements.
Example:
*Incoming Flowfile*
attributes: [username: “alopresto”, role: “security”, email:
“alopre...@apache.org”, git_account: “alopresto”]
*CHA Properties (Individual)*
attribute_enumeration_style: “individual”
(dynamic) username_sha256: “username”
(dynamic) git_account_sha256: “git_account”
*Behavior (Individual)*
username_sha256 = git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) =
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23
*Resulting Flowfile (Individual)*
attributes: [username: “alopresto”, role: “security”, email:
“alopre...@apache.org”, git_account: “alopresto”, username_sha256:
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”,
git_account_sha256:
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23"]
*CHA Properties (List)*
attribute_enumeration_style: “list”
(dynamic) username_and_email_sha256: “username, email”
(dynamic) git_account_sha256: “git_account”
*Behavior (List)*
username_and_email_sha256 = $(echo -n "aloprestoalopre...@apache.org" | shasum
-a 256) = 22a11b7b3173f95c23a1f434949ec2a2e66455b9cb26b7ebc90afca25d91333f
git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) =
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23
*Resulting Flowfile (List)*
attributes: [username: “alopresto”, role: “security”, email:
“alopre...@apache.org”, git_account: “alopresto”, username_email_sha256: “
22a11b7b3173f95c23a1f434949ec2a2e66455b9cb26b7ebc90afca25d91333f”,
git_account_sha256:
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”]
*CHA Properties (Regex)*
attribute_enumeration_style: “regex”
(dynamic) all_sha256: “.*”
(dynamic) git_account_sha256: “git_account”
*Behavior (Regex)*
all_sha256 = sort(attributes_that_match_regex) = [email, git_account, role,
username] = $(echo -n "alopresto@apache.orgaloprestosecurityalopresto" | shasum
-a 256) = b370fdf0132933cea76e3daa3d4a437bb8c571dd0cd0e79ee5d7759cf64efced
git_account_sha256 = $(echo -n "alopresto" | shasum -a 256) =
600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23
*Resulting Flowfile (Regex)*
attributes: [username: “alopresto”, role: “security”, email:
“alopre...@apache.org”, git_account: “alopresto”, all_sha256: “
b370fdf0132933cea76e3daa3d4a437bb8c571dd0cd0e79ee5d7759cf64efced”,
git_account_sha256:
“600973dc8f2b7bb2a20651ebefe4bf91c5295afef19f4d5b9994d581f5a68a23”]
{quote}
This will necessitate switching the "order" of dynamic properties in
{{CryptographicHashAttribute}} -- rather than a dynamic property
**existing_attribute_name** - {{new_attribute_name_containing_hash}}, the
ordering will be **new_attribute_name_containing_hash** -
{{existing_attribute_name}} to allow for other values like {{attribute_.*}} or
{{attribute_1, attribute_2}}.
There will also be a boolean flag to include the attribute name in the hashed
value:
Example:
*existing_attribute_name* - "some value"
If **true**, the value in *new_attribute_name_containing_hash* would be
{{hash("existing_attribute_namesome value")}}. If **false**, it would be
{{hash("some value")}}. As no one is using the new
{{CryptographicHashAttribute}} in the field yet, this change can only be made
now.
[Mailing list
[jira] [Assigned] (NIFI-5147) Improve HashAttribute processor
[
https://issues.apache.org/jira/browse/NIFI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy LoPresto reassigned NIFI-5147:
---
Assignee: Andy LoPresto (was: Otto Fowler)
> Improve HashAttribute processor
> ---
>
> Key: NIFI-5147
> URL: https://issues.apache.org/jira/browse/NIFI-5147
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
> Labels: hash, security
> Fix For: 1.8.0
>
>
> The {{HashAttribute}} processor currently has surprising behavior. Barring
> familiarity with the processor, a user would expect {{HashAttribute}} to
> generate a hash value over one or more attributes. Instead, the processor as
> it is implemented "groups" incoming flowfiles into groups based on regular
> expressions which match attribute values, and then generates a
> (non-configurable) MD5 hash over the concatenation of the matching attribute
> keys and values.
> In addition:
> * the processor throws an error and routes to failure any incoming flowfile
> which does not have all attributes specified in the processor
> * the use of MD5 is vastly deprecated
> * no other hash algorithms are available
> I am unaware of community use of this processor, but I do not want to break
> backward compatibility. I propose the following steps:
> * Implement a new {{CalculateAttributeHash}} processor (awkward name, but
> this processor already has the desired name)
> ** This processor will perform the "standard" use case -- identify an
> attribute, calculate the specified hash over the value, and write it to an
> output attribute
> ** This processor will have a required property descriptor allowing a
> dropdown menu of valid hash algorithms
> ** This processor will accept arbitrary dynamic properties identifying the
> attributes to be hashed as a key, and the resulting attribute name as a value
> ** Example: I want to generate a SHA-512 hash on the attribute {{username}},
> and a flowfile enters the processor with {{username}} value {{alopresto}}. I
> configure {{algorithm}} with {{SHA-512}} and add a dynamic property
> {{username}} -- {{username_SHA512}}. The resulting flowfile will have
> attribute {{username_SHA512}} with value
> {{739b4f6722fb5de20125751c7a1a358b2a7eb8f07e530e4bf18561fbff93234908aa9d250c876bca9ede5ba784d5ce6081dbbdfe5ddd446678f223b8d632}}
> * Improve the documentation of this processor to explain the goal/expected
> use case (?)
> * Link in processor documentation to new processor for standard use cases
> * Remove the error alert when an incoming flowfile does not contain all
> expected attributes. I propose changing the severity to INFO and still
> routing to failure
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFIREG-196) Upgrade lodash, parsejson, https-proxy-agent
[ https://issues.apache.org/jira/browse/NIFIREG-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFIREG-196: Fix Version/s: 0.3.0 > Upgrade lodash, parsejson, https-proxy-agent > > > Key: NIFIREG-196 > URL: https://issues.apache.org/jira/browse/NIFIREG-196 > Project: NiFi Registry > Issue Type: Bug >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 0.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5581) Seeing timeouts when trying to replicate requests across the cluster
[ https://issues.apache.org/jira/browse/NIFI-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607760#comment-16607760 ] ASF GitHub Bot commented on NIFI-5581: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/2996 NIFI-5581: Disable connection pooling for OkHttpReplicationClient. We… … should revisit this in the future, but for now, it appears that Jetty is having problems with the connections if they are reused. By disabling the Connection Pooling, we address the concern, but for secure connections this means that every request results in a TLS handshake - and for a mutable request, both the verification and the 'performance' stages require the TLS handshake. But it's better than timing out, which is the currently observed behavior Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5581 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2996.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2996 commit 9b2ead1013f90d1909f1c00a26ac0e86218f96fe Author: Mark Payne Date: 2018-09-07T23:00:55Z NIFI-5581: Disable connection pooling for OkHttpReplicationClient. We should revisit this in the future, but for now, it appears that Jetty is having problems with the connections if they are reused. By disabling the Connection Pooling, we address the concern, but for secure connections this means that every request results in a TLS handshake - and for a mutable request, both the verification and the 'performance' stages require the TLS handshake. But it's better than timing out, which is the currently observed behavior > Seeing timeouts when trying to replicate requests across the cluster > > > Key: NIFI-5581 > URL: https://issues.apache.org/jira/browse/NIFI-5581 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.8.0 > > > When trying to replicate requests across the cluster on the current master > branch, I see everything go smoothly for GET requests, but all mutable > requests timeout. > This issue appears to have been introduced by the upgrade to a new version of > Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2996: NIFI-5581: Disable connection pooling for OkHttpRep...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/2996 NIFI-5581: Disable connection pooling for OkHttpReplicationClient. We⦠⦠should revisit this in the future, but for now, it appears that Jetty is having problems with the connections if they are reused. By disabling the Connection Pooling, we address the concern, but for secure connections this means that every request results in a TLS handshake - and for a mutable request, both the verification and the 'performance' stages require the TLS handshake. But it's better than timing out, which is the currently observed behavior Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5581 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2996.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2996 commit 9b2ead1013f90d1909f1c00a26ac0e86218f96fe Author: Mark Payne Date: 2018-09-07T23:00:55Z NIFI-5581: Disable connection pooling for OkHttpReplicationClient. We should revisit this in the future, but for now, it appears that Jetty is having problems with the connections if they are reused. By disabling the Connection Pooling, we address the concern, but for secure connections this means that every request results in a TLS handshake - and for a mutable request, both the verification and the 'performance' stages require the TLS handshake. But it's better than timing out, which is the currently observed behavior ---
[jira] [Created] (NIFI-5581) Seeing timeouts when trying to replicate requests across the cluster
Mark Payne created NIFI-5581: Summary: Seeing timeouts when trying to replicate requests across the cluster Key: NIFI-5581 URL: https://issues.apache.org/jira/browse/NIFI-5581 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.8.0 Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.8.0 When trying to replicate requests across the cluster on the current master branch, I see everything go smoothly for GET requests, but all mutable requests timeout. This issue appears to have been introduced by the upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5147) Improve HashAttribute processor
[
https://issues.apache.org/jira/browse/NIFI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607701#comment-16607701
]
ASF GitHub Bot commented on NIFI-5147:
--
Github user thenatog commented on the issue:
https://github.com/apache/nifi/pull/2980
Tested out the HashAttribute processor. This all worked fine:
- MD5 and creating a new attribute
- MD5 and overwriting the attribute with hashed value
- SHA256 and creating a new attribute
- MD5 of chinese characters using UTF-8 (matched web tool hasher and
command line md5 utility)
UTF-16 is where I came unstuck:
- MD5 of simple string using "UTF-16" encoding, I get a different hash to
what I expect.
- MD5 of simple string using "UTF-16BE" and "UTF-16LE" encoding DO match
what I expect.
Test input string in all cases: “hehe”
NiFi CalculateAttributeHash:
UTF-8:MD5 = 529ca8050a00180790cf88b63468826a
UTF-16BE:MD5 = b0ed26b524e0b0606551d78e42b5b7bc
UTF-16LE:MD5 = 2db0ecc27f7abd29ba95412feb3b5e07
UTF-16:MD5 = 9b6dcd3887ebdb43d66fb4b3ef9c259b
CyberChef
(https://gchq.github.io/CyberChef/#recipe=Encode_text('UTF16BE%20(1201)')MD5()=aGVoZQ):
UTF-8:MD5 = 529ca8050a00180790cf88b63468826a
UTF-16BE:MD5 = b0ed26b524e0b0606551d78e42b5b7bc
UTF-16LE:MD5 = 2db0ecc27f7abd29ba95412feb3b5e07
I found that “UTF-16” is different because when encoding, Java adds a
big-endian BOM: _“When decoding, the UTF-16 charset interprets the byte-order
mark at the beginning of the input stream to indicate the byte-order of the
stream but defaults to big-endian if there is no byte-order mark; when
encoding, it uses big-endian byte order and writes a big-endian byte-order
mark.”_ As expected, adding the BOM changes the output bytes which are then
hashed, resulting in a different hash to “UTF-16BE” encoding. Is this a problem
or is this simply expected behaviour - ie. should the user realize that there
will be a difference between UTF-16 and UTF-16BE encoding and the resulting
hash?
> Improve HashAttribute processor
> ---
>
> Key: NIFI-5147
> URL: https://issues.apache.org/jira/browse/NIFI-5147
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Assignee: Otto Fowler
>Priority: Major
> Labels: hash, security
> Fix For: 1.8.0
>
>
> The {{HashAttribute}} processor currently has surprising behavior. Barring
> familiarity with the processor, a user would expect {{HashAttribute}} to
> generate a hash value over one or more attributes. Instead, the processor as
> it is implemented "groups" incoming flowfiles into groups based on regular
> expressions which match attribute values, and then generates a
> (non-configurable) MD5 hash over the concatenation of the matching attribute
> keys and values.
> In addition:
> * the processor throws an error and routes to failure any incoming flowfile
> which does not have all attributes specified in the processor
> * the use of MD5 is vastly deprecated
> * no other hash algorithms are available
> I am unaware of community use of this processor, but I do not want to break
> backward compatibility. I propose the following steps:
> * Implement a new {{CalculateAttributeHash}} processor (awkward name, but
> this processor already has the desired name)
> ** This processor will perform the "standard" use case -- identify an
> attribute, calculate the specified hash over the value, and write it to an
> output attribute
> ** This processor will have a required property descriptor allowing a
> dropdown menu of valid hash algorithms
> ** This processor will accept arbitrary dynamic properties identifying the
> attributes to be hashed as a key, and the resulting attribute name as a value
> ** Example: I want to generate a SHA-512 hash on the attribute {{username}},
> and a flowfile enters the processor with {{username}} value {{alopresto}}. I
> configure {{algorithm}} with {{SHA-512}} and add a dynamic property
> {{username}} -- {{username_SHA512}}. The resulting flowfile will have
> attribute {{username_SHA512}} with value
> {{739b4f6722fb5de20125751c7a1a358b2a7eb8f07e530e4bf18561fbff93234908aa9d250c876bca9ede5ba784d5ce6081dbbdfe5ddd446678f223b8d632}}
> * Improve the documentation of this processor to explain the goal/expected
> use case (?)
> * Link in processor documentation to new processor for standard use cases
> * Remove the error alert when an incoming flowfile does not contain all
> expected attributes. I propose changing the severity to INFO and still
> routing to failure
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2980: NIFI-5147 Implement CalculateAttributeHash processor
Github user thenatog commented on the issue:
https://github.com/apache/nifi/pull/2980
Tested out the HashAttribute processor. This all worked fine:
- MD5 and creating a new attribute
- MD5 and overwriting the attribute with hashed value
- SHA256 and creating a new attribute
- MD5 of chinese characters using UTF-8 (matched web tool hasher and
command line md5 utility)
UTF-16 is where I came unstuck:
- MD5 of simple string using "UTF-16" encoding, I get a different hash to
what I expect.
- MD5 of simple string using "UTF-16BE" and "UTF-16LE" encoding DO match
what I expect.
Test input string in all cases: âheheâ
NiFi CalculateAttributeHash:
UTF-8:MD5 = 529ca8050a00180790cf88b63468826a
UTF-16BE:MD5 = b0ed26b524e0b0606551d78e42b5b7bc
UTF-16LE:MD5 = 2db0ecc27f7abd29ba95412feb3b5e07
UTF-16:MD5 = 9b6dcd3887ebdb43d66fb4b3ef9c259b
CyberChef
(https://gchq.github.io/CyberChef/#recipe=Encode_text('UTF16BE%20(1201)')MD5()=aGVoZQ):
UTF-8:MD5 = 529ca8050a00180790cf88b63468826a
UTF-16BE:MD5 = b0ed26b524e0b0606551d78e42b5b7bc
UTF-16LE:MD5 = 2db0ecc27f7abd29ba95412feb3b5e07
I found that âUTF-16â is different because when encoding, Java adds a
big-endian BOM: _âWhen decoding, the UTF-16 charset interprets the byte-order
mark at the beginning of the input stream to indicate the byte-order of the
stream but defaults to big-endian if there is no byte-order mark; when
encoding, it uses big-endian byte order and writes a big-endian byte-order
mark.â_ As expected, adding the BOM changes the output bytes which are then
hashed, resulting in a different hash to âUTF-16BEâ encoding. Is this a
problem or is this simply expected behaviour - ie. should the user realize that
there will be a difference between UTF-16 and UTF-16BE encoding and the
resulting hash?
---
[jira] [Updated] (NIFI-5580) PostHTTP always performs HEAD request to check acceptance of Flow Files even when 'Send as FlowFile' is false
[ https://issues.apache.org/jira/browse/NIFI-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Leon updated NIFI-5580: - Description: The PostHTTP processor performs a HEAD Request to the specified REST endpoint to check if the endpoint accepts FlowFiles. This check should occur if "Send As FlowFile" is set to false; this will throw a 404 exception if the endpoint does not accept HEAD requests. Encountered while using PostHTTP processor to send a POST request to /_bulk endpoint of Elasticsearch 5.3 2018-09-07 16:31:34,133 WARN [Timer-Driven Process Thread-42] o.a.nifi.processors.standard.PostHTTP PostHTTP[id=38eb43d1-8be7-18e0--348fb09c] Unable to communicate with destination; when attempting to perform an HTTP HEAD, got unexpected response code of 404: Not Found was:The PostHTTP processor performs a HEAD Request to the specified REST endpoint to check if the endpoint accepts FlowFiles. This check should occur if "Send As FlowFile" is set to false; this will throw a 404 exception if the endpoint does not accept HEAD requests. > PostHTTP always performs HEAD request to check acceptance of Flow Files even > when 'Send as FlowFile' is false > - > > Key: NIFI-5580 > URL: https://issues.apache.org/jira/browse/NIFI-5580 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Aaron Leon >Priority: Major > > The PostHTTP processor performs a HEAD Request to the specified REST endpoint > to check if the endpoint accepts FlowFiles. This check should occur if "Send > As FlowFile" is set to false; this will throw a 404 exception if the endpoint > does not accept HEAD requests. > Encountered while using PostHTTP processor to send a POST request to /_bulk > endpoint of Elasticsearch 5.3 > 2018-09-07 16:31:34,133 WARN [Timer-Driven Process Thread-42] > o.a.nifi.processors.standard.PostHTTP > PostHTTP[id=38eb43d1-8be7-18e0--348fb09c] Unable to communicate with > destination; when attempting to perform an HTTP HEAD, got unexpected response > code of 404: Not Found -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5580) PostHTTP always performs HEAD request to check acceptance of Flow Files even when 'Send as FlowFile' is false
Aaron Leon created NIFI-5580: Summary: PostHTTP always performs HEAD request to check acceptance of Flow Files even when 'Send as FlowFile' is false Key: NIFI-5580 URL: https://issues.apache.org/jira/browse/NIFI-5580 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.8.0 Reporter: Aaron Leon The PostHTTP processor performs a HEAD Request to the specified REST endpoint to check if the endpoint accepts FlowFiles. This check should occur if "Send As FlowFile" is set to false; this will throw a 404 exception if the endpoint does not accept HEAD requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5579) Allow ExecuteGroovyScript to take a SQL property that is a DBCPConnectionPoolLookup
[
https://issues.apache.org/jira/browse/NIFI-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607672#comment-16607672
]
Colin Dean commented on NIFI-5579:
--
NIFI-5519 is the better solution for my particular use case!
> Allow ExecuteGroovyScript to take a SQL property that is a
> DBCPConnectionPoolLookup
> ---
>
> Key: NIFI-5579
> URL: https://issues.apache.org/jira/browse/NIFI-5579
> Project: Apache NiFi
> Issue Type: Wish
> Components: Core Framework
>Affects Versions: 1.7.0, 1.7.1
> Environment: Any
>Reporter: Colin Dean
>Priority: Major
> Labels: groovy
>
> I would like to reference a {{DBCPConnectionPoolLookup}} controller service
> from within an {{ExecuteGroovyScript}} processor. If I create a property
> {{SQL.lookup}} and set its value to an existing {{DBCPConnectionPoolLookup}}
> controller service, when I start the processor, it fails immediately because
> the creation of the "connection" by the {{DBCPConnectionPoolLookup}}
> controller service fails because it was not passed the expected
> {{database.name}} attribute.
> {code}
> 2018-09-07 16:04:28,229 ERROR [Timer-Driven Process Thread-227]
> o.a.n.p.groovyx.ExecuteGroovyScript
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13]
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name':
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name'
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name'
> at
> org.apache.nifi.dbcp.DBCPConnectionPoolLookup.getConnection(DBCPConnectionPoolLookup.java:124)
> at sun.reflect.GeneratedMethodAccessor507.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy150.getConnection(Unknown Source)
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onInitSQL(ExecuteGroovyScript.java:339)
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onTrigger(ExecuteGroovyScript.java:439)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> 2018-09-07 16:04:28,232 ERROR [Timer-Driven Process Thread-227]
> o.a.n.p.groovyx.ExecuteGroovyScript
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13]
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13] failed to
> process session due to java.lang.ClassCastException: com.sun.proxy.$Proxy150
> cannot be cast to org.apache.nifi.processors.groovyx.sql.OSql; Processor
> Administratively Yielded for 1 sec: java.lang.ClassCastException:
> com.sun.proxy.$Proxy150 cannot be cast to
> org.apache.nifi.processors.groovyx.sql.OSql
> java.lang.ClassCastException: com.sun.proxy.$Proxy150 cannot be cast to
> org.apache.nifi.processors.groovyx.sql.OSql
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onFinitSQL(ExecuteGroovyScript.java:371)
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onTrigger(ExecuteGroovyScript.java:464)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
>
[jira] [Commented] (NIFI-5579) Allow ExecuteGroovyScript to take a SQL property that is a DBCPConnectionPoolLookup
[
https://issues.apache.org/jira/browse/NIFI-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607661#comment-16607661
]
Colin Dean commented on NIFI-5579:
--
Interestingly enough, based on what I see
[here|https://github.com/apache/nifi/blob/fc1461298a0a137d0fae3cd3b494fc6dec25070a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java#L429],
it looks like the controller service itself is added to the {{SQL}} map
binding just fine, but the {{onInitSQL}} method needs to look at other
properties for what to put in the collection it passes to {{getConnection}}.
A quick fix could be at the call to {{onInitSQL}}, which could be in a trycatch
that logs the initialization exceptions but continues going. {{onFinitSQL}}
would need to check if the value of an entry in {{SQL}} is actionable, too,
[here|https://github.com/apache/nifi/blob/fc1461298a0a137d0fae3cd3b494fc6dec25070a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/main/java/org/apache/nifi/processors/groovyx/ExecuteGroovyScript.java#L371]
> Allow ExecuteGroovyScript to take a SQL property that is a
> DBCPConnectionPoolLookup
> ---
>
> Key: NIFI-5579
> URL: https://issues.apache.org/jira/browse/NIFI-5579
> Project: Apache NiFi
> Issue Type: Wish
> Components: Core Framework
>Affects Versions: 1.7.0, 1.7.1
> Environment: Any
>Reporter: Colin Dean
>Priority: Major
> Labels: groovy
>
> I would like to reference a {{DBCPConnectionPoolLookup}} controller service
> from within an {{ExecuteGroovyScript}} processor. If I create a property
> {{SQL.lookup}} and set its value to an existing {{DBCPConnectionPoolLookup}}
> controller service, when I start the processor, it fails immediately because
> the creation of the "connection" by the {{DBCPConnectionPoolLookup}}
> controller service fails because it was not passed the expected
> {{database.name}} attribute.
> {code}
> 2018-09-07 16:04:28,229 ERROR [Timer-Driven Process Thread-227]
> o.a.n.p.groovyx.ExecuteGroovyScript
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13]
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name':
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name'
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name'
> at
> org.apache.nifi.dbcp.DBCPConnectionPoolLookup.getConnection(DBCPConnectionPoolLookup.java:124)
> at sun.reflect.GeneratedMethodAccessor507.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy150.getConnection(Unknown Source)
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onInitSQL(ExecuteGroovyScript.java:339)
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onTrigger(ExecuteGroovyScript.java:439)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> 2018-09-07 16:04:28,232 ERROR [Timer-Driven Process Thread-227]
> o.a.n.p.groovyx.ExecuteGroovyScript
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13]
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13] failed to
> process session due to java.lang.ClassCastException: com.sun.proxy.$Proxy150
> cannot be cast to
[jira] [Commented] (NIFI-5579) Allow ExecuteGroovyScript to take a SQL property that is a DBCPConnectionPoolLookup
[
https://issues.apache.org/jira/browse/NIFI-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607645#comment-16607645
]
Colin Dean commented on NIFI-5579:
--
This may actually be more of bug, but JIRA wouldn't let me switch to *bug*
after I'd chosen *wish*…
> Allow ExecuteGroovyScript to take a SQL property that is a
> DBCPConnectionPoolLookup
> ---
>
> Key: NIFI-5579
> URL: https://issues.apache.org/jira/browse/NIFI-5579
> Project: Apache NiFi
> Issue Type: Wish
> Components: Core Framework
>Affects Versions: 1.7.0, 1.7.1
> Environment: Any
>Reporter: Colin Dean
>Priority: Major
> Labels: groovy
>
> I would like to reference a {{DBCPConnectionPoolLookup}} controller service
> from within an {{ExecuteGroovyScript}} processor. If I create a property
> {{SQL.lookup}} and set its value to an existing {{DBCPConnectionPoolLookup}}
> controller service, when I start the processor, it fails immediately because
> the creation of the "connection" by the {{DBCPConnectionPoolLookup}}
> controller service fails because it was not passed the expected
> {{database.name}} attribute.
> {code}
> 2018-09-07 16:04:28,229 ERROR [Timer-Driven Process Thread-227]
> o.a.n.p.groovyx.ExecuteGroovyScript
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13]
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name':
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name'
> org.apache.nifi.processor.exception.ProcessException: Attributes must contain
> an attribute name 'database.name'
> at
> org.apache.nifi.dbcp.DBCPConnectionPoolLookup.getConnection(DBCPConnectionPoolLookup.java:124)
> at sun.reflect.GeneratedMethodAccessor507.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
> at com.sun.proxy.$Proxy150.getConnection(Unknown Source)
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onInitSQL(ExecuteGroovyScript.java:339)
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onTrigger(ExecuteGroovyScript.java:439)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> 2018-09-07 16:04:28,232 ERROR [Timer-Driven Process Thread-227]
> o.a.n.p.groovyx.ExecuteGroovyScript
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13]
> ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13] failed to
> process session due to java.lang.ClassCastException: com.sun.proxy.$Proxy150
> cannot be cast to org.apache.nifi.processors.groovyx.sql.OSql; Processor
> Administratively Yielded for 1 sec: java.lang.ClassCastException:
> com.sun.proxy.$Proxy150 cannot be cast to
> org.apache.nifi.processors.groovyx.sql.OSql
> java.lang.ClassCastException: com.sun.proxy.$Proxy150 cannot be cast to
> org.apache.nifi.processors.groovyx.sql.OSql
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onFinitSQL(ExecuteGroovyScript.java:371)
> at
> org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onTrigger(ExecuteGroovyScript.java:464)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
[jira] [Created] (NIFI-5579) Allow ExecuteGroovyScript to take a SQL property that is a DBCPConnectionPoolLookup
Colin Dean created NIFI-5579:
Summary: Allow ExecuteGroovyScript to take a SQL property that is
a DBCPConnectionPoolLookup
Key: NIFI-5579
URL: https://issues.apache.org/jira/browse/NIFI-5579
Project: Apache NiFi
Issue Type: Wish
Components: Core Framework
Affects Versions: 1.7.1, 1.7.0
Environment: Any
Reporter: Colin Dean
I would like to reference a {{DBCPConnectionPoolLookup}} controller service
from within an {{ExecuteGroovyScript}} processor. If I create a property
{{SQL.lookup}} and set its value to an existing {{DBCPConnectionPoolLookup}}
controller service, when I start the processor, it fails immediately because
the creation of the "connection" by the {{DBCPConnectionPoolLookup}} controller
service fails because it was not passed the expected {{database.name}}
attribute.
{code}
2018-09-07 16:04:28,229 ERROR [Timer-Driven Process Thread-227]
o.a.n.p.groovyx.ExecuteGroovyScript
ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13]
org.apache.nifi.processor.exception.ProcessException: Attributes must contain
an attribute name 'database.name':
org.apache.nifi.processor.exception.ProcessException: Attributes must contain
an attribute name 'database.name'
org.apache.nifi.processor.exception.ProcessException: Attributes must contain
an attribute name 'database.name'
at
org.apache.nifi.dbcp.DBCPConnectionPoolLookup.getConnection(DBCPConnectionPoolLookup.java:124)
at sun.reflect.GeneratedMethodAccessor507.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:84)
at com.sun.proxy.$Proxy150.getConnection(Unknown Source)
at
org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onInitSQL(ExecuteGroovyScript.java:339)
at
org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onTrigger(ExecuteGroovyScript.java:439)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
2018-09-07 16:04:28,232 ERROR [Timer-Driven Process Thread-227]
o.a.n.p.groovyx.ExecuteGroovyScript
ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13]
ExecuteGroovyScript[id=684100f5-78cf-35f9-28db-0fa4d1d30c13] failed to process
session due to java.lang.ClassCastException: com.sun.proxy.$Proxy150 cannot be
cast to org.apache.nifi.processors.groovyx.sql.OSql; Processor Administratively
Yielded for 1 sec: java.lang.ClassCastException: com.sun.proxy.$Proxy150 cannot
be cast to org.apache.nifi.processors.groovyx.sql.OSql
java.lang.ClassCastException: com.sun.proxy.$Proxy150 cannot be cast to
org.apache.nifi.processors.groovyx.sql.OSql
at
org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onFinitSQL(ExecuteGroovyScript.java:371)
at
org.apache.nifi.processors.groovyx.ExecuteGroovyScript.onTrigger(ExecuteGroovyScript.java:464)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
[jira] [Commented] (MINIFICPP-602) Support a CSV within RPG URL
[ https://issues.apache.org/jira/browse/MINIFICPP-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607626#comment-16607626 ] ASF GitHub Bot commented on MINIFICPP-602: -- GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/395 MINIFICPP-602: Add ability to use CSV within url field Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-602 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/395.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #395 commit 79a3dda285466e6fb134abd11ca601ee204508b1 Author: Marc Parisi Date: 2018-09-07T20:29:07Z MINIFICPP-602: Add ability to use CSV within url field > Support a CSV within RPG URL > > > Key: MINIFICPP-602 > URL: https://issues.apache.org/jira/browse/MINIFICPP-602 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #395: MINIFICPP-602: Add ability to use CSV wit...
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/395 MINIFICPP-602: Add ability to use CSV within url field Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-602 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/395.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #395 commit 79a3dda285466e6fb134abd11ca601ee204508b1 Author: Marc Parisi Date: 2018-09-07T20:29:07Z MINIFICPP-602: Add ability to use CSV within url field ---
[jira] [Resolved] (NIFI-5438) Declare volumes for the NiFi docker container for persistence by default
[ https://issues.apache.org/jira/browse/NIFI-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Moser resolved NIFI-5438. - Resolution: Fixed marking this as Resolved > Declare volumes for the NiFi docker container for persistence by default > > > Key: NIFI-5438 > URL: https://issues.apache.org/jira/browse/NIFI-5438 > Project: Apache NiFi > Issue Type: Improvement > Components: Docker >Affects Versions: 1.8.0 >Reporter: Peter Wilcsinszky >Assignee: Peter Wilcsinszky >Priority: Major > Fix For: 1.8.0 > > > Volume declarations are missing from the NiFi Dockerfiles. > *Without* that all data get lost after removing the container when there was > no explicit volume declarations defined. > *With* volume declarations all the data on the declared volumes will get > persisted on the host without any explicit volume declarations required by > the user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5578) Expression Language Documentation typo in Date Manipulation examples
[
https://issues.apache.org/jira/browse/NIFI-5578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607510#comment-16607510
]
Andrew Lim edited comment on NIFI-5578 at 9/7/18 7:25 PM:
--
Have noticed other formatting errors/typos. Example:
For example, ${filename} will return the value of the ``filename'' attribute.
which should be:
For example, ${filename} will return the value of the {{filename}} attribute.
was (Author: andrewmlim):
Have noticed other formatting errors/typos. Example:
For example, ${filename} will return the value of the ``filename'' attribute.
which should be:
For example, ${filename} will return the value of the "filename'' attribute.
> Expression Language Documentation typo in Date Manipulation examples
>
>
> Key: NIFI-5578
> URL: https://issues.apache.org/jira/browse/NIFI-5578
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation Website
>Affects Versions: 1.7.0, 1.7.1
> Environment: Chrome browser
>Reporter: William Brooks
>Assignee: Andrew Lim
>Priority: Major
>
> Documentation for NiFi expression language contains several examples with
> erroneous backticks instead of single quotes.
> [https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#dates]
> Section: Date Manipulation
> Table 18. format Examples
> Example:
> ${time:format("/MM/dd HH:mm:ss.SSS’Z'", "GMT")}
> (backtick is between and Z)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5578) Expression Language Documentation typo in Date Manipulation examples
[
https://issues.apache.org/jira/browse/NIFI-5578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607510#comment-16607510
]
Andrew Lim commented on NIFI-5578:
--
Have noticed other formatting errors/typos. Example:
For example, ${filename} will return the value of the ``filename'' attribute.
which should be:
For example, ${filename} will return the value of the "filename'' attribute.
> Expression Language Documentation typo in Date Manipulation examples
>
>
> Key: NIFI-5578
> URL: https://issues.apache.org/jira/browse/NIFI-5578
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation Website
>Affects Versions: 1.7.0, 1.7.1
> Environment: Chrome browser
>Reporter: William Brooks
>Assignee: Andrew Lim
>Priority: Major
>
> Documentation for NiFi expression language contains several examples with
> erroneous backticks instead of single quotes.
> [https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#dates]
> Section: Date Manipulation
> Table 18. format Examples
> Example:
> ${time:format("/MM/dd HH:mm:ss.SSS’Z'", "GMT")}
> (backtick is between and Z)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5578) Expression Language Documentation typo in Date Manipulation examples
[
https://issues.apache.org/jira/browse/NIFI-5578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Lim reassigned NIFI-5578:
Assignee: Andrew Lim
> Expression Language Documentation typo in Date Manipulation examples
>
>
> Key: NIFI-5578
> URL: https://issues.apache.org/jira/browse/NIFI-5578
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation Website
>Affects Versions: 1.7.0, 1.7.1
> Environment: Chrome browser
>Reporter: William Brooks
>Assignee: Andrew Lim
>Priority: Major
>
> Documentation for NiFi expression language contains several examples with
> erroneous backticks instead of single quotes.
> [https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#dates]
> Section: Date Manipulation
> Table 18. format Examples
> Example:
> ${time:format("/MM/dd HH:mm:ss.SSS’Z'", "GMT")}
> (backtick is between and Z)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
[
https://issues.apache.org/jira/browse/NIFI-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607428#comment-16607428
]
ASF GitHub Bot commented on NIFI-5557:
--
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2971#discussion_r216031690
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
---
@@ -269,13 +272,15 @@ public Object run() {
}
changeOwner(context, hdfs, configuredRootDirPath,
flowFile);
} catch (IOException e) {
-if (!Strings.isNullOrEmpty(e.getMessage()) &&
e.getMessage().contains(String.format("Couldn't setup connection for %s",
ugi.getUserName( {
- getLogger().error(String.format("An error
occured while connecting to HDFS. Rolling back session, and penalizing flowfile
%s",
-
flowFile.getAttribute(CoreAttributes.UUID.key(;
- session.rollback(true);
-} else {
- throw e;
-}
+ boolean tgtExpired = hasCause(e, GSSException.class,
gsse -> "Failed to find any Kerberos tgt".equals(gsse.getMinorString()));
+ if (tgtExpired) {
+getLogger().error(String.format("An error occured
while connecting to HDFS. Rolling back session, and penalizing flow file %s",
+
putFlowFile.getAttribute(CoreAttributes.UUID.key(;
+session.rollback(true);
+ } else {
+getLogger().error("Failed to access HDFS due to
{}", new Object[]{e});
+session.transfer(session.penalize(putFlowFile),
REL_FAILURE);
--- End diff --
@ekovacs I don't think we need to penalize on the transfer to failure here.
> PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
> -
>
> Key: NIFI-5557
> URL: https://issues.apache.org/jira/browse/NIFI-5557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Major
>
> when using *PutHDFS* processor in a kerberized environment, with a flow
> "traffic" which approximately matches or less frequent then the lifetime of
> the ticket of the principal, we see this in the log:
> {code:java}
> INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
> Exception while invoking getFileInfo of class
> ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for princi...@example.com to host2.example.com/ip2:8020;
> Host Details : local host is: "host1.example.com/ip1"; destination host is:
> "host2.example.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:254)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
>
[jira] [Commented] (NIFI-5557) PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
[
https://issues.apache.org/jira/browse/NIFI-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607429#comment-16607429
]
ASF GitHub Bot commented on NIFI-5557:
--
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2971#discussion_r216037639
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
---
@@ -269,13 +272,15 @@ public Object run() {
}
changeOwner(context, hdfs, configuredRootDirPath,
flowFile);
} catch (IOException e) {
-if (!Strings.isNullOrEmpty(e.getMessage()) &&
e.getMessage().contains(String.format("Couldn't setup connection for %s",
ugi.getUserName( {
- getLogger().error(String.format("An error
occured while connecting to HDFS. Rolling back session, and penalizing flowfile
%s",
-
flowFile.getAttribute(CoreAttributes.UUID.key(;
- session.rollback(true);
-} else {
- throw e;
-}
+ boolean tgtExpired = hasCause(e, GSSException.class,
gsse -> "Failed to find any Kerberos tgt".equals(gsse.getMinorString()));
--- End diff --
@ekovacs After seeing the use of getMinorString here, I looked at
GSSException, and it looks like there's some error codes that could be used to
detect the actual cause, rather than string matching. Are getMajor and
getMinor returning ints when these exceptions happen?
> PutHDFS "GSSException: No valid credentials provided" when krb ticket expires
> -
>
> Key: NIFI-5557
> URL: https://issues.apache.org/jira/browse/NIFI-5557
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
>Reporter: Endre Kovacs
>Assignee: Endre Kovacs
>Priority: Major
>
> when using *PutHDFS* processor in a kerberized environment, with a flow
> "traffic" which approximately matches or less frequent then the lifetime of
> the ticket of the principal, we see this in the log:
> {code:java}
> INFO [Timer-Driven Process Thread-4] o.a.h.io.retry.RetryInvocationHandler
> Exception while invoking getFileInfo of class
> ClientNamenodeProtocolTranslatorPB over host2/ip2:8020 after 13 fail over
> attempts. Trying to fail over immediately.
> java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
> setup connection for princi...@example.com to host2.example.com/ip2:8020;
> Host Details : local host is: "host1.example.com/ip1"; destination host is:
> "host2.example.com":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy134.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> at sun.reflect.GeneratedMethodAccessor344.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy135.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:254)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:222)
> {code}
> and the flowfile is routed to failure relationship.
> *To reproduce:*
> Create a principal in your KDC with two minutes ticket lifetime,
> and set up a similar flow:
> {code:java}
> GetFile => putHDFS - success- ->
[jira] [Commented] (NIFI-5147) Improve HashAttribute processor
[
https://issues.apache.org/jira/browse/NIFI-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607426#comment-16607426
]
ASF GitHub Bot commented on NIFI-5147:
--
Github user thenatog commented on the issue:
https://github.com/apache/nifi/pull/2980
Reviewing..
> Improve HashAttribute processor
> ---
>
> Key: NIFI-5147
> URL: https://issues.apache.org/jira/browse/NIFI-5147
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Assignee: Otto Fowler
>Priority: Major
> Labels: hash, security
> Fix For: 1.8.0
>
>
> The {{HashAttribute}} processor currently has surprising behavior. Barring
> familiarity with the processor, a user would expect {{HashAttribute}} to
> generate a hash value over one or more attributes. Instead, the processor as
> it is implemented "groups" incoming flowfiles into groups based on regular
> expressions which match attribute values, and then generates a
> (non-configurable) MD5 hash over the concatenation of the matching attribute
> keys and values.
> In addition:
> * the processor throws an error and routes to failure any incoming flowfile
> which does not have all attributes specified in the processor
> * the use of MD5 is vastly deprecated
> * no other hash algorithms are available
> I am unaware of community use of this processor, but I do not want to break
> backward compatibility. I propose the following steps:
> * Implement a new {{CalculateAttributeHash}} processor (awkward name, but
> this processor already has the desired name)
> ** This processor will perform the "standard" use case -- identify an
> attribute, calculate the specified hash over the value, and write it to an
> output attribute
> ** This processor will have a required property descriptor allowing a
> dropdown menu of valid hash algorithms
> ** This processor will accept arbitrary dynamic properties identifying the
> attributes to be hashed as a key, and the resulting attribute name as a value
> ** Example: I want to generate a SHA-512 hash on the attribute {{username}},
> and a flowfile enters the processor with {{username}} value {{alopresto}}. I
> configure {{algorithm}} with {{SHA-512}} and add a dynamic property
> {{username}} -- {{username_SHA512}}. The resulting flowfile will have
> attribute {{username_SHA512}} with value
> {{739b4f6722fb5de20125751c7a1a358b2a7eb8f07e530e4bf18561fbff93234908aa9d250c876bca9ede5ba784d5ce6081dbbdfe5ddd446678f223b8d632}}
> * Improve the documentation of this processor to explain the goal/expected
> use case (?)
> * Link in processor documentation to new processor for standard use cases
> * Remove the error alert when an incoming flowfile does not contain all
> expected attributes. I propose changing the severity to INFO and still
> routing to failure
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2971: NIFI-5557: handling expired ticket by rollback and ...
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2971#discussion_r216031690
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
---
@@ -269,13 +272,15 @@ public Object run() {
}
changeOwner(context, hdfs, configuredRootDirPath,
flowFile);
} catch (IOException e) {
-if (!Strings.isNullOrEmpty(e.getMessage()) &&
e.getMessage().contains(String.format("Couldn't setup connection for %s",
ugi.getUserName( {
- getLogger().error(String.format("An error
occured while connecting to HDFS. Rolling back session, and penalizing flowfile
%s",
-
flowFile.getAttribute(CoreAttributes.UUID.key(;
- session.rollback(true);
-} else {
- throw e;
-}
+ boolean tgtExpired = hasCause(e, GSSException.class,
gsse -> "Failed to find any Kerberos tgt".equals(gsse.getMinorString()));
+ if (tgtExpired) {
+getLogger().error(String.format("An error occured
while connecting to HDFS. Rolling back session, and penalizing flow file %s",
+
putFlowFile.getAttribute(CoreAttributes.UUID.key(;
+session.rollback(true);
+ } else {
+getLogger().error("Failed to access HDFS due to
{}", new Object[]{e});
+session.transfer(session.penalize(putFlowFile),
REL_FAILURE);
--- End diff --
@ekovacs I don't think we need to penalize on the transfer to failure here.
---
[GitHub] nifi pull request #2971: NIFI-5557: handling expired ticket by rollback and ...
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2971#discussion_r216037639
--- Diff:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
---
@@ -269,13 +272,15 @@ public Object run() {
}
changeOwner(context, hdfs, configuredRootDirPath,
flowFile);
} catch (IOException e) {
-if (!Strings.isNullOrEmpty(e.getMessage()) &&
e.getMessage().contains(String.format("Couldn't setup connection for %s",
ugi.getUserName( {
- getLogger().error(String.format("An error
occured while connecting to HDFS. Rolling back session, and penalizing flowfile
%s",
-
flowFile.getAttribute(CoreAttributes.UUID.key(;
- session.rollback(true);
-} else {
- throw e;
-}
+ boolean tgtExpired = hasCause(e, GSSException.class,
gsse -> "Failed to find any Kerberos tgt".equals(gsse.getMinorString()));
--- End diff --
@ekovacs After seeing the use of getMinorString here, I looked at
GSSException, and it looks like there's some error codes that could be used to
detect the actual cause, rather than string matching. Are getMajor and
getMinor returning ints when these exceptions happen?
---
[GitHub] nifi issue #2980: NIFI-5147 Implement CalculateAttributeHash processor
Github user thenatog commented on the issue: https://github.com/apache/nifi/pull/2980 Reviewing.. ---
[jira] [Commented] (NIFIREG-198) VersionedRemoteProcessGroup targetUri and targetUris return incorrect derived value
[ https://issues.apache.org/jira/browse/NIFIREG-198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607391#comment-16607391 ] ASF GitHub Bot commented on NIFIREG-198: GitHub user kevdoran opened a pull request: https://github.com/apache/nifi-registry/pull/139 NIFIREG-198 Fix VersionedRemoteProcessGroup targetUri bug VersionedRemoteProcessGroup has two fields: targetUri and targetUris. They do not have simple getters, but rather each will fallback to using the other. Given how they are described, for certain combinations of targetUri and targetUris, they do not work correctly (mainly, when targetUri is not set and targetUris contains only a single value). Furthermore, in NiFi only TargetUris is used/needed. It was a replacement for targetUri when the requirement for multiple URIs was added. Therefore, in addition to fixing the bug descirbed above, targetUri can be deprecated and dropped in future versions of NiFi Registry. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevdoran/nifi-registry NIFIREG-198-vrpg-targetUri Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-registry/pull/139.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #139 commit 79af0dbe771e14b3e38e713c0ae9d860a7e83eb2 Author: Kevin Doran Date: 2018-09-06T23:10:50Z NIFIREG-198 Add unit test that reproduces bug commit bc138f1d8649eba3a7daf4ef8acd3e2e2f733622 Author: Kevin Doran Date: 2018-09-07T00:26:06Z NIFIREG-198 Fix VersionedRemoteProcesGroup targetUri commit 73ac36f678a3d82acdc935e18c5b0cf08b4064f0 Author: Kevin Doran Date: 2018-09-07T14:53:09Z NIFIREG-198 Deprecate VersionedRemoteProcessGroup targetUri > VersionedRemoteProcessGroup targetUri and targetUris return incorrect derived > value > --- > > Key: NIFIREG-198 > URL: https://issues.apache.org/jira/browse/NIFIREG-198 > Project: NiFi Registry > Issue Type: Bug >Affects Versions: 0.1.0, 0.2.0 >Reporter: Kevin Doran >Assignee: Kevin Doran >Priority: Major > Fix For: 0.3.0 > > > VersionedRemoteProcessGroup has two fields: targetUri and targetUris. They do > not have simple getters, but rather each will fallback to using the other. > Given how they are described, for certain combinations of targetUri and > targetUris, they do not work correctly (mainly, when targetUri is not set and > targetUris contains only a single value). > Furthermore, in NiFi only TargetUris is used/needed. It was a replacement for > targetUri when the requirement for multiple URIs was added. Therefore, in > addition to fixing the bug descirbed above, targetUri can be deprecated and > dropped in future versions of NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #139: NIFIREG-198 Fix VersionedRemoteProcessGroup...
GitHub user kevdoran opened a pull request: https://github.com/apache/nifi-registry/pull/139 NIFIREG-198 Fix VersionedRemoteProcessGroup targetUri bug VersionedRemoteProcessGroup has two fields: targetUri and targetUris. They do not have simple getters, but rather each will fallback to using the other. Given how they are described, for certain combinations of targetUri and targetUris, they do not work correctly (mainly, when targetUri is not set and targetUris contains only a single value). Furthermore, in NiFi only TargetUris is used/needed. It was a replacement for targetUri when the requirement for multiple URIs was added. Therefore, in addition to fixing the bug descirbed above, targetUri can be deprecated and dropped in future versions of NiFi Registry. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevdoran/nifi-registry NIFIREG-198-vrpg-targetUri Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-registry/pull/139.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #139 commit 79af0dbe771e14b3e38e713c0ae9d860a7e83eb2 Author: Kevin Doran Date: 2018-09-06T23:10:50Z NIFIREG-198 Add unit test that reproduces bug commit bc138f1d8649eba3a7daf4ef8acd3e2e2f733622 Author: Kevin Doran Date: 2018-09-07T00:26:06Z NIFIREG-198 Fix VersionedRemoteProcesGroup targetUri commit 73ac36f678a3d82acdc935e18c5b0cf08b4064f0 Author: Kevin Doran Date: 2018-09-07T14:53:09Z NIFIREG-198 Deprecate VersionedRemoteProcessGroup targetUri ---
[jira] [Created] (NIFIREG-198) VersionedRemoteProcessGroup targetUri and targetUris return incorrect derived value
Kevin Doran created NIFIREG-198: --- Summary: VersionedRemoteProcessGroup targetUri and targetUris return incorrect derived value Key: NIFIREG-198 URL: https://issues.apache.org/jira/browse/NIFIREG-198 Project: NiFi Registry Issue Type: Bug Affects Versions: 0.2.0, 0.1.0 Reporter: Kevin Doran Assignee: Kevin Doran Fix For: 0.3.0 VersionedRemoteProcessGroup has two fields: targetUri and targetUris. They do not have simple getters, but rather each will fallback to using the other. Given how they are described, for certain combinations of targetUri and targetUris, they do not work correctly (mainly, when targetUri is not set and targetUris contains only a single value). Furthermore, in NiFi only TargetUris is used/needed. It was a replacement for targetUri when the requirement for multiple URIs was added. Therefore, in addition to fixing the bug descirbed above, targetUri can be deprecated and dropped in future versions of NiFi Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2992: NIFI-5510: Introducing PutCassandraRecord processor
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2992 @MikeThomsen No complex setup is required. A Cassandra cluster has to be up and running. That's all. ---
[jira] [Commented] (NIFI-5510) Allow records to be put directly into Cassandra
[ https://issues.apache.org/jira/browse/NIFI-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607366#comment-16607366 ] ASF GitHub Bot commented on NIFI-5510: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2992 @MikeThomsen No complex setup is required. A Cassandra cluster has to be up and running. That's all. > Allow records to be put directly into Cassandra > --- > > Key: NIFI-5510 > URL: https://issues.apache.org/jira/browse/NIFI-5510 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Sivaprasanna Sethuraman >Priority: Major > > Currently the standard way of getting data into Cassandra through NiFi is to > use PutCassandraQL, which often means raw data needs to be converted to CQL > statements, usually done (with modifications) via ConvertJSONToSQL. > It would be better to have something closer to PutDatabaseRecord, a processor > called PutCassandraRecord perhaps, that would take the raw data and input it > into Cassandra "directly", without the need for the user to convert the data > into CQL statements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-375) New user role: Operator who can start and stop components
[ https://issues.apache.org/jira/browse/NIFI-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607268#comment-16607268 ] ASF GitHub Bot commented on NIFI-375: - Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2990 @mcgilman Updated how API exposes required data to operate components. Although I need to do the similar refactoring with RGPs, the most complex ControllerService implementation is done. I updated the PR description, too, to illustrate how ControllerService enable/disable operations are authorized. It'd be great if you can review this while I'm working on the remaining RPG stuffs. Thanks! > New user role: Operator who can start and stop components > - > > Key: NIFI-375 > URL: https://issues.apache.org/jira/browse/NIFI-375 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Reporter: Daniel Ueberfluss >Assignee: Koji Kawamura >Priority: Major > > Would like to have a user role that allows a user to stop/start processors > but perform no other changes to the dataflow. > This would allow users to address simple problems without providing full > access to modifying a data flow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2990: NIFI-375: Added operation policy
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2990 @mcgilman Updated how API exposes required data to operate components. Although I need to do the similar refactoring with RGPs, the most complex ControllerService implementation is done. I updated the PR description, too, to illustrate how ControllerService enable/disable operations are authorized. It'd be great if you can review this while I'm working on the remaining RPG stuffs. Thanks! ---
[jira] [Created] (MINIFICPP-602) Support a CSV within RPG URL
Mr TheSegfault created MINIFICPP-602: Summary: Support a CSV within RPG URL Key: MINIFICPP-602 URL: https://issues.apache.org/jira/browse/MINIFICPP-602 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Mr TheSegfault -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5479) Upgrade version of Jetty
[
https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607187#comment-16607187
]
Nathan Gough edited comment on NIFI-5479 at 9/7/18 2:36 PM:
Found that the 'ASM implementation' warning:
{code:java}
WARN [main] o.e.jetty.annotations.AnnotationParser Unknown asm implementation
version, assuming version 393216{code}
is a known issue with ASM v6.0+ and Java 8:
[https://github.com/eclipse/jetty.project/issues/2834]
[https://gitlab.ow2.org/asm/asm/issues/317847]
was (Author: thenatog):
Found that the 'ASM implementation' warning:
{code:java}
WARN [main] o.e.jetty.annotations.AnnotationParser Unknown asm implementation
version, assuming version 393216{code}
is a known issue with ASM v6.0+ and Java 8:
[https://github.com/eclipse/jetty.project/issues/2834]
> Upgrade version of Jetty
>
>
> Key: NIFI-5479
> URL: https://issues.apache.org/jira/browse/NIFI-5479
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.8.0
>
>
> Upgrade to a new version of Jetty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5479) Upgrade version of Jetty
[
https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607187#comment-16607187
]
Nathan Gough edited comment on NIFI-5479 at 9/7/18 2:35 PM:
Found that the 'ASM implementation' warning:
{code:java}
WARN [main] o.e.jetty.annotations.AnnotationParser Unknown asm implementation
version, assuming version 393216{code}
is a known issue with ASM v6.0+ and Java 8:
[https://github.com/eclipse/jetty.project/issues/2834]
was (Author: thenatog):
Found that the 'ASM implementation' warning:
{code:java}
WARN [main] o.e.jetty.annotations.AnnotationParser Unknown asm implementation
version, assuming version 393216{code}
is a known issue with ASM v6.0+ and Java 8:
[https://github.com/eclipse/jetty.project/issues/2834]
> Upgrade version of Jetty
>
>
> Key: NIFI-5479
> URL: https://issues.apache.org/jira/browse/NIFI-5479
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.8.0
>
>
> Upgrade to a new version of Jetty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[
https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607187#comment-16607187
]
Nathan Gough commented on NIFI-5479:
Found that the 'ASM implementation' warning:
{code:java}
WARN [main] o.e.jetty.annotations.AnnotationParser Unknown asm implementation
version, assuming version 393216{code}
is a known issue with ASM v6.0+ and Java 8:
[https://github.com/eclipse/jetty.project/issues/2834]
> Upgrade version of Jetty
>
>
> Key: NIFI-5479
> URL: https://issues.apache.org/jira/browse/NIFI-5479
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Andy LoPresto
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.8.0
>
>
> Upgrade to a new version of Jetty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607050#comment-16607050 ] Matt Gilman commented on NIFI-5479: --- Also, it appears that running this version of Jetty with the updated request replication model using OkHttp fails for any request that uses two phase commit. Upgraded Jetty with the old request replication model using Jersey does not have these issues. It's not clear what the cause it but there appears to be some incompatibility. Between the new Jetty and how we are issuing each request. They appear to just time out. > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-5479: -- Priority: Blocker (was: Major) > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
