[jira] [Updated] (NIFI-5997) If swap file written but FlowFile Repository fails to update, connection queue counts wrong and flowfiles are duplicated upon restart
[ https://issues.apache.org/jira/browse/NIFI-5997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5997: --- Affects Version/s: 1.8.0 1.7.1 > If swap file written but FlowFile Repository fails to update, connection > queue counts wrong and flowfiles are duplicated upon restart > - > > Key: NIFI-5997 > URL: https://issues.apache.org/jira/browse/NIFI-5997 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0, 1.7.1 >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.9.0 > > Time Spent: 1h > Remaining Estimate: 0h > > If a queue writes out a Swap File but then the FlowFile Repository throws an > Exception when attempting to update, we end up with a scenario where the size > of the queue increases by 10,000 FlowFiles (the number of FlowFiles to be > written to the swap file) as well as the corresponding size of the FlowFiles. > We also have a Swap File that is written out to disk but the FlowFile Repo > didn't get updated so on restart we have those FlowFiles in the FlowFile Repo > as well as in the Swap File, so we end up with two of the same FlowFile. This > can then cause some odd behavior because two FlowFiles exist with the same ID > and the counts on the queues are very wrong, which also causes a lot of > confusion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5997) If swap file written but FlowFile Repository fails to update, connection queue counts wrong and flowfiles are duplicated upon restart
[ https://issues.apache.org/jira/browse/NIFI-5997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807789#comment-16807789 ] Joseph Percivall commented on NIFI-5997: Sounds good thanks [~markap14]. That aligns with the testing we've done. We saw the issue with 1.8 and 1.7.1, and we were able to manually backport the fix. I'll go ahead and mark affects version for the versions that we were able to reproduce and observe the fix on (1.8 and 1.7.1). > If swap file written but FlowFile Repository fails to update, connection > queue counts wrong and flowfiles are duplicated upon restart > - > > Key: NIFI-5997 > URL: https://issues.apache.org/jira/browse/NIFI-5997 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.9.0 > > Time Spent: 1h > Remaining Estimate: 0h > > If a queue writes out a Swap File but then the FlowFile Repository throws an > Exception when attempting to update, we end up with a scenario where the size > of the queue increases by 10,000 FlowFiles (the number of FlowFiles to be > written to the swap file) as well as the corresponding size of the FlowFiles. > We also have a Swap File that is written out to disk but the FlowFile Repo > didn't get updated so on restart we have those FlowFiles in the FlowFile Repo > as well as in the Swap File, so we end up with two of the same FlowFile. This > can then cause some odd behavior because two FlowFiles exist with the same ID > and the counts on the queues are very wrong, which also causes a lot of > confusion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5997) If swap file written but FlowFile Repository fails to update, connection queue counts wrong and flowfiles are duplicated upon restart
[ https://issues.apache.org/jira/browse/NIFI-5997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801183#comment-16801183 ] Joseph Percivall commented on NIFI-5997: [~markap14] do you know which versions this affects? > If swap file written but FlowFile Repository fails to update, connection > queue counts wrong and flowfiles are duplicated upon restart > - > > Key: NIFI-5997 > URL: https://issues.apache.org/jira/browse/NIFI-5997 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.9.0 > > Time Spent: 1h > Remaining Estimate: 0h > > If a queue writes out a Swap File but then the FlowFile Repository throws an > Exception when attempting to update, we end up with a scenario where the size > of the queue increases by 10,000 FlowFiles (the number of FlowFiles to be > written to the swap file) as well as the corresponding size of the FlowFiles. > We also have a Swap File that is written out to disk but the FlowFile Repo > didn't get updated so on restart we have those FlowFiles in the FlowFile Repo > as well as in the Swap File, so we end up with two of the same FlowFile. This > can then cause some odd behavior because two FlowFiles exist with the same ID > and the counts on the queues are very wrong, which also causes a lot of > confusion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-6096) ValidateRecord does not handle nested Map type correctly
Joseph Percivall created NIFI-6096: -- Summary: ValidateRecord does not handle nested Map type correctly Key: NIFI-6096 URL: https://issues.apache.org/jira/browse/NIFI-6096 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.9.0 Reporter: Joseph Percivall Attachments: Nested_map_record_failing_validation.xml When attempting to validate a map that was nested as such top-level record -> array of records -> value in record is a map, I hit the following error: "Value is of type org.apache.nifi.serialization.record.MapRecord but was expected to be of type MAP" This is the same error in NIFI-5678. Attached is a template to reproduce the error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782454#comment-16782454
]
Joseph Percivall commented on NIFI-5678:
[~joewitt] yeah I debated about creating a new one but figured it may be that
the issue wasn't fixed and may need to be reopened. I'll go ahead and create a
new one.
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
> Attachments: Nested_map_record_failing_validation.xml
>
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFIREG-232) Support in-place migration between flow persistence providers
[ https://issues.apache.org/jira/browse/NIFIREG-232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFIREG-232: Assignee: Joseph Percivall (was: Bryan Rosander) > Support in-place migration between flow persistence providers > - > > Key: NIFIREG-232 > URL: https://issues.apache.org/jira/browse/NIFIREG-232 > Project: NiFi Registry > Issue Type: New Feature >Reporter: Bryan Rosander >Assignee: Joseph Percivall >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to be able to switch flow persistence providers without > losing state. > > Current documentation says that this is not possible: > "In order to switch the Persistence Provider to use, it is necessary to reset > NiFi Registry." > > I think we should be able to support this via a nifi-registry toolkit that > reads from the old provider, writes to a new one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFIREG-232) Support in-place migration between flow persistence providers
[ https://issues.apache.org/jira/browse/NIFIREG-232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFIREG-232: Assignee: Bryan Rosander > Support in-place migration between flow persistence providers > - > > Key: NIFIREG-232 > URL: https://issues.apache.org/jira/browse/NIFIREG-232 > Project: NiFi Registry > Issue Type: New Feature >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to be able to switch flow persistence providers without > losing state. > > Current documentation says that this is not possible: > "In order to switch the Persistence Provider to use, it is necessary to reset > NiFi Registry." > > I think we should be able to support this via a nifi-registry toolkit that > reads from the old provider, writes to a new one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFIREG-232) Support in-place migration between flow persistence providers
[ https://issues.apache.org/jira/browse/NIFIREG-232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFIREG-232: Assignee: Bryan Rosander (was: Joseph Percivall) > Support in-place migration between flow persistence providers > - > > Key: NIFIREG-232 > URL: https://issues.apache.org/jira/browse/NIFIREG-232 > Project: NiFi Registry > Issue Type: New Feature >Reporter: Bryan Rosander >Assignee: Bryan Rosander >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to be able to switch flow persistence providers without > losing state. > > Current documentation says that this is not possible: > "In order to switch the Persistence Provider to use, it is necessary to reset > NiFi Registry." > > I think we should be able to support this via a nifi-registry toolkit that > reads from the old provider, writes to a new one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781299#comment-16781299
]
Joseph Percivall commented on NIFI-5678:
[~mattyb149] I just attempted to do this with a v1.9.0 instance and it hits the
same error message ("Value is of type
org.apache.nifi.serialization.record.MapRecord but was expected to be of type
MAP"). Attached is a template to recreate it. A potential reason is
that the map is nested within an array of records.
[^Nested_map_record_failing_validation.xml]
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
> Attachments: Nested_map_record_failing_validation.xml
>
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Percivall updated NIFI-5678:
---
Attachment: Nested_map_record_failing_validation.xml
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
> Attachments: Nested_map_record_failing_validation.xml
>
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-6036) If a Processor is intended to be running but is invalid when NiFi starts, it cannot be udpated.
[ https://issues.apache.org/jira/browse/NIFI-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775967#comment-16775967 ] Joseph Percivall commented on NIFI-6036: [~markap14] do we know the affects version here? Assuming 1.8 is affected, I'm looking at backport this fix. The PR is a combination of multiple different issues, not all of which I want to backport. What changes in the PR are for fixing this issue? > If a Processor is intended to be running but is invalid when NiFi starts, it > cannot be udpated. > --- > > Key: NIFI-6036 > URL: https://issues.apache.org/jira/browse/NIFI-6036 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Blocker > Fix For: 1.9.0 > > > To reproduce, create a GetFile processor and point it to `./data` and create > that directory. Start the processor. Stop NiFi and rename the directory. > Start NiFi again. On restart, the Processor will be invalid. If you rename > the directory back to `./data` then the processor will become valid and run. > However, if instead you attempt to update the Processor to point to a > different directory, it will give an error saying that the Processor cannot > be updated because it is not stopped. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5172) PutElasticsearchHttpRecord does not handle failed records individually
[ https://issues.apache.org/jira/browse/NIFI-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5172: --- Summary: PutElasticsearchHttpRecord does not handle failed records individually (was: PutElasticSearchRecord does to fail individual recods) > PutElasticsearchHttpRecord does not handle failed records individually > -- > > Key: NIFI-5172 > URL: https://issues.apache.org/jira/browse/NIFI-5172 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Juan C. Sequeiros >Assignee: Joseph Percivall >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > My observation and not sure if working as expected but when I send my output > from MergeRecord ( set to 1 ) max number of records and one of those > records has an invalid timestamp "bogusdata" value ES rejects it, rightly so > since on ES we have a schema template more granular and is expecting > timestamp as type "date". > > From USER forums Matt Burgess: > Yes the current behavior is to move the entire input flowfile to > failure if any errors occur. Some other record-aware processors create > separate flow files for failed and successful records, but > PutElasticsearchHttpRecord does not (yet) do that. Please feel free to > write a Jira for this improvement. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5172) PutElasticSearchRecord does to fail individual recods
[ https://issues.apache.org/jira/browse/NIFI-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5172: --- Status: Patch Available (was: Open) PR up here: https://github.com/apache/nifi/pull/3299 > PutElasticSearchRecord does to fail individual recods > - > > Key: NIFI-5172 > URL: https://issues.apache.org/jira/browse/NIFI-5172 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Juan C. Sequeiros >Assignee: Joseph Percivall >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > My observation and not sure if working as expected but when I send my output > from MergeRecord ( set to 1 ) max number of records and one of those > records has an invalid timestamp "bogusdata" value ES rejects it, rightly so > since on ES we have a schema template more granular and is expecting > timestamp as type "date". > > From USER forums Matt Burgess: > Yes the current behavior is to move the entire input flowfile to > failure if any errors occur. Some other record-aware processors create > separate flow files for failed and successful records, but > PutElasticsearchHttpRecord does not (yet) do that. Please feel free to > write a Jira for this improvement. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5172) PutElasticSearchRecord does to fail individual recods
[ https://issues.apache.org/jira/browse/NIFI-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFI-5172: -- Assignee: Joseph Percivall > PutElasticSearchRecord does to fail individual recods > - > > Key: NIFI-5172 > URL: https://issues.apache.org/jira/browse/NIFI-5172 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Juan C. Sequeiros >Assignee: Joseph Percivall >Priority: Minor > > My observation and not sure if working as expected but when I send my output > from MergeRecord ( set to 1 ) max number of records and one of those > records has an invalid timestamp "bogusdata" value ES rejects it, rightly so > since on ES we have a schema template more granular and is expecting > timestamp as type "date". > > From USER forums Matt Burgess: > Yes the current behavior is to move the entire input flowfile to > failure if any errors occur. Some other record-aware processors create > separate flow files for failed and successful records, but > PutElasticsearchHttpRecord does not (yet) do that. Please feel free to > write a Jira for this improvement. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5922) NiFi-Fn, an alternative runtime for NiFi flows
[ https://issues.apache.org/jira/browse/NIFI-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753689#comment-16753689 ] Joseph Percivall commented on NIFI-5922: For tracking, an initial discussion thread is here: https://lists.apache.org/thread.html/%3c1217956179.5384158.1546477466...@mail.yahoo.com%3E > NiFi-Fn, an alternative runtime for NiFi flows > -- > > Key: NIFI-5922 > URL: https://issues.apache.org/jira/browse/NIFI-5922 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Sam Hjelmfelt >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > NiFi-Fn is a library for running NiFi flows as stateless functions. It > provides similar delivery guarantees as NiFi without the need for on-disk > repositories by waiting to confirm receipt ofincoming data until it has been > written to the destination. This is similar to Storm’s acking mechanism and > Spark’s interface for committing Kafka offsets, except that in NiFi-Fn, this > is completely handled by the framework while still supporting all NiFi > processors and controller services natively without change.This results in > the ability to run NiFi flows as ephemeral, stateless functions and should be > able to rival MirrorMaker, Distcp, and Scoop for performance,efficiency, and > scalability while leveraging the vast library of NiFi processors and the NiFi > UI for building custom flows. > By leveraging container engines (e.g.YARN, Kubernetes), long-running NiFi-Fn > flows can be deployed that take full advantage of the platform’s scale and > multi-tenancy features. By leveraging Function as a Service engines (FaaS) > (e.g. AWS Lambda, Apache OpenWhisk), NiFi-Fn flows can be attached to event > sources (or just cron) for event-driven data movement where flows only run > when triggered and pricing is measured at the 100ms granularity. By combining > the two, large-scale batch processing could also be performed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5936) MockProcessSession remove() does not report a "DROP" provenance event
Joseph Percivall created NIFI-5936: -- Summary: MockProcessSession remove() does not report a "DROP" provenance event Key: NIFI-5936 URL: https://issues.apache.org/jira/browse/NIFI-5936 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Joseph Percivall The StandardProcessSession remove method emits a "DROPPED" provenance event[1] whereas the MockProcessSession does not[2]. MockProcessSession should mimic the Standard as closely as possible. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java#L2002 [2] https://github.com/apache/nifi/blob/master/nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java#L620 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5811) Need Scallable Open Source Platform
[ https://issues.apache.org/jira/browse/NIFI-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-5811. Resolution: Invalid > Need Scallable Open Source Platform > --- > > Key: NIFI-5811 > URL: https://issues.apache.org/jira/browse/NIFI-5811 > Project: Apache NiFi > Issue Type: Task > Components: SDLC >Affects Versions: 1.7.1 >Reporter: SalesSupport >Priority: Critical > > Need scallable solution for Kafka and Apache Spark Integration Design > Solutions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5172) PutElasticSearchRecord does to fail individual recods
[ https://issues.apache.org/jira/browse/NIFI-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700967#comment-16700967 ] Joseph Percivall commented on NIFI-5172: I ran into this issue today and I assumed this to be a "bug" given that it differs from the logic in for all the other record based processors and the PutElasticsearchHttp processor most of the core logic was copied from(including the comment stating the "correct" logic[1]). Addressing [~mike.thomsen]'s comment, at least on the HTTP endpoint, ES will tell you which one failed as that's the logic currently in PutElasticsearchHttp[2]. Talking with [~mattyb149], the logic for adding it would be something along the lines of a merging of the error handling of PutElasticsearchHttp and with the record creation of SplitRecord[3]. I would argue that this is higher than a minor priority as behaves opposite what we've trained users to expect and potentially leads to duplicate data being ingested. [1] https://issues.apache.org/jira/browse/NIFI-5172 [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L345 [3] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitRecord.java#L144 > PutElasticSearchRecord does to fail individual recods > - > > Key: NIFI-5172 > URL: https://issues.apache.org/jira/browse/NIFI-5172 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Juan C. Sequeiros >Priority: Minor > > My observation and not sure if working as expected but when I send my output > from MergeRecord ( set to 1 ) max number of records and one of those > records has an invalid timestamp "bogusdata" value ES rejects it, rightly so > since on ES we have a schema template more granular and is expecting > timestamp as type "date". > > From USER forums Matt Burgess: > Yes the current behavior is to move the entire input flowfile to > failure if any errors occur. Some other record-aware processors create > separate flow files for failed and successful records, but > PutElasticsearchHttpRecord does not (yet) do that. Please feel free to > write a Jira for this improvement. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5775) DataTypeUtils "toString" incorrectly treats value as a "byte" when passing an array leading to ClassCastException
[
https://issues.apache.org/jira/browse/NIFI-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679155#comment-16679155
]
Joseph Percivall commented on NIFI-5775:
[~sivaprasanna] this can be replicated in the unit tests by adding the String
data type to the linked unit test.
To be more exact on when it'll occur, if you have a field with the ability to
be multiple types (e.g. array + string + null) and have string as first in the
schema type list, this will occur.
> DataTypeUtils "toString" incorrectly treats value as a "byte" when passing an
> array leading to ClassCastException
> -
>
> Key: NIFI-5775
> URL: https://issues.apache.org/jira/browse/NIFI-5775
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Joseph Percivall
>Priority: Major
>
> To reproduce, change this line[1] to either put "String" as the first choice
> of record type or just set the key to use string.
> The resulting error:
> {noformat}
> java.lang.ClassCastException: java.lang.String cannot be cast to
> java.lang.Byte
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.toString(DataTypeUtils.java:530)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:147)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:115)
> at
> org.apache.nifi.json.WriteJsonResult.writeValue(WriteJsonResult.java:284)
> at
> org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:187)
> at
> org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:136)
> at
> org.apache.nifi.json.TestWriteJsonResult.testChoiceArray(TestWriteJsonResult.java:494)
> {noformat}
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/test/java/org/apache/nifi/json/TestWriteJsonResult.java#L479
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5775) DataTypeUtils "toString" incorrectly treats value as a "byte" when passing an array leading to ClassCastException
Joseph Percivall created NIFI-5775:
--
Summary: DataTypeUtils "toString" incorrectly treats value as a
"byte" when passing an array leading to ClassCastException
Key: NIFI-5775
URL: https://issues.apache.org/jira/browse/NIFI-5775
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Joseph Percivall
To reproduce, change this line[1] to either put "String" as the first choice of
record type or just set the key to use string.
The resulting error:
{noformat}
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Byte
at
org.apache.nifi.serialization.record.util.DataTypeUtils.toString(DataTypeUtils.java:530)
at
org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:147)
at
org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:115)
at
org.apache.nifi.json.WriteJsonResult.writeValue(WriteJsonResult.java:284)
at
org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:187)
at
org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:136)
at
org.apache.nifi.json.TestWriteJsonResult.testChoiceArray(TestWriteJsonResult.java:494)
{noformat}
[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/test/java/org/apache/nifi/json/TestWriteJsonResult.java#L479
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
[ https://issues.apache.org/jira/browse/NIFI-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFI-5765: -- Assignee: Joseph Percivall > WriteJsonResult fails with Class Cast Exception when Choice data type > resolves to Array > --- > > Key: NIFI-5765 > URL: https://issues.apache.org/jira/browse/NIFI-5765 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, > WriteJsonResult_Choice_Array_example.xml > > > The problem is this line[1]. For the casting, it uses the passed in value > instead of the chosen data type. > A template demonstrating the problem is attached. The corner case is hit when > there is a choice of data types, and the data type chosen is Array. It > properly does everything else but fails when casting it due to: > org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to > org.apache.nifi.serialization.record.type.ArrayDataType. > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
Joseph Percivall created NIFI-5765: -- Summary: WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array Key: NIFI-5765 URL: https://issues.apache.org/jira/browse/NIFI-5765 Project: Apache NiFi Issue Type: Bug Reporter: Joseph Percivall Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, WriteJsonResult_Choice_Array_example.xml The problem is this line[1]. For the casting, it uses the passed in value instead of the chosen data type. A template demonstrating the problem is attached. The corner case is hit when there is a choice of data types, and the data type chosen is Array. It properly does everything else but fails when casting it due to: org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to org.apache.nifi.serialization.record.type.ArrayDataType. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5765) WriteJsonResult fails with Class Cast Exception when Choice data type resolves to Array
[ https://issues.apache.org/jira/browse/NIFI-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5765: --- Affects Version/s: 1.8.0 > WriteJsonResult fails with Class Cast Exception when Choice data type > resolves to Array > --- > > Key: NIFI-5765 > URL: https://issues.apache.org/jira/browse/NIFI-5765 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Joseph Percivall >Priority: Major > Attachments: Screen Shot 2018-10-29 at 2.35.50 PM.png, > WriteJsonResult_Choice_Array_example.xml > > > The problem is this line[1]. For the casting, it uses the passed in value > instead of the chosen data type. > A template demonstrating the problem is attached. The corner case is hit when > there is a choice of data types, and the data type chosen is Array. It > properly does everything else but fails when casting it due to: > org.apache.nifi.serialization.record.type.ChoiceDataType cannot be cast to > org.apache.nifi.serialization.record.type.ArrayDataType. > [1] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/WriteJsonResult.java#L379 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3792) A processor to facilitate retrying FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-3792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653813#comment-16653813 ] Joseph Percivall commented on NIFI-3792: Hey [~patricker], I actually have a working processor that we've been using but just never got around to putting up a PR. It has some handling to try to avoid the livelock but wasn't able to 100% fix due to limitations on knowing about the destination queues from the ontrigger. I'll put up a branch with the Processor this week. > A processor to facilitate retrying FlowFiles > > > Key: NIFI-3792 > URL: https://issues.apache.org/jira/browse/NIFI-3792 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > When dealing with processor failures in production, potentially related to > network issues, the current best practice is to retry the flowfile a couple > times before declaring it a failure[1]. This currently requires multiple > processors and penalizing the flowfile isn't possible. Also if the flow is > not fast enough, back-pressure can cause a livelocked state which requires > manual intervention. > A new processor should be added to not only encourage best practices but also > offer configuration options to deal with un-optimal situations. > [1] > https://community.hortonworks.com/questions/77336/nifi-best-practices-for-error-handling.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5703) SyslogParser regex fails to handle messages that end with Windows style new line
Joseph Percivall created NIFI-5703: -- Summary: SyslogParser regex fails to handle messages that end with Windows style new line Key: NIFI-5703 URL: https://issues.apache.org/jira/browse/NIFI-5703 Project: Apache NiFi Issue Type: Bug Reporter: Joseph Percivall The common SyslogParser class, which is used by ListenSyslog and ParseSyslog, has two different regexes to match the RFCs for syslog. Both of which end in a "(.*)$"[1]. There is also special handling if the last character is "\n"[2]. Note that "." does not match any of the newline characters[3]. The endline handling should be expanded to handle "\r\n". [1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-syslog-utils/src/main/java/org/apache/nifi/syslog/parsers/SyslogParser.java#L49 [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-syslog-utils/src/main/java/org/apache/nifi/syslog/parsers/SyslogParser.java#L128 [3] https://stackoverflow.com/questions/3445326/regex-in-java-how-to-deal-with-newline -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5694) StandardSSLContextService references configContext in a non-thread safe manner
[
https://issues.apache.org/jira/browse/NIFI-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648112#comment-16648112
]
Joseph Percivall commented on NIFI-5694:
FYI, I originally had another comment in the description[1] but turns out this
wasn't a root cause. I've not super well versed on why/how but I have a PKCS12
file which is also valid as a JKS file (verified using openssl + keytool).
Something weird with the generation of the trust factory leads it to a state
where it successfully created the trust factory when treating the file as JKS
but not as PKCS12.
This ticket is still an issue though and should be addressed.
[1]
{noformat}
I believe this is the cause of the weirdness I'm seeing where I have the SSL
context "successfully" configured with a truststore of type "JKS" and am able
to use it with an InovokeHttp processor. The problem is that the truststore is
actually P12 (verified on the command line). I believe the issue came about
because I wasn't sure if the type/password was correct and was
enabling+disabling+reconfiguring it in rapid succession. {noformat}
> StandardSSLContextService references configContext in a non-thread safe manner
> --
>
> Key: NIFI-5694
> URL: https://issues.apache.org/jira/browse/NIFI-5694
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.1
>Reporter: Joseph Percivall
>Priority: Major
>
> configContext is a variable which is accessed from many different threads
> (validate, enable, and any processor which calls "createSSLContext"). It is
> not declared with any thread safe modifier[1]. Potentially leading to odd
> behavior.
>
> The other shared variables should be marked with a thread-safe modifier as
> well.
> [1]
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service/src/main/java/org/apache/nifi/ssl/StandardSSLContextService.java#L121]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5694) StandardSSLContextService references configContext in a non-thread safe manner
[ https://issues.apache.org/jira/browse/NIFI-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5694: --- Description: configContext is a variable which is accessed from many different threads (validate, enable, and any processor which calls "createSSLContext"). It is not declared with any thread safe modifier[1]. Potentially leading to odd behavior. The other shared variables should be marked with a thread-safe modifier as well. [1] [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service/src/main/java/org/apache/nifi/ssl/StandardSSLContextService.java#L121] was: configContext is a variable which is accessed from many different threads (validate, enable, and any processor which calls "createSSLContext"). It is not declared with any thread safe modifier[1]. Potentially leading to odd behavior. I believe this is the cause of the weirdness I'm seeing where I have the SSL context "successfully" configured with a truststore of type "JKS" and am able to use it with an InovokeHttp processor. The problem is that the truststore is actually P12 (verified on the command line). I believe the issue came about because I wasn't sure if the type/password was correct and was enabling+disabling+reconfiguring it in rapid succession. The other shared variables should be marked with a thread-safe modifier as well. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service/src/main/java/org/apache/nifi/ssl/StandardSSLContextService.java#L121 > StandardSSLContextService references configContext in a non-thread safe manner > -- > > Key: NIFI-5694 > URL: https://issues.apache.org/jira/browse/NIFI-5694 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.7.1 >Reporter: Joseph Percivall >Priority: Major > > configContext is a variable which is accessed from many different threads > (validate, enable, and any processor which calls "createSSLContext"). It is > not declared with any thread safe modifier[1]. Potentially leading to odd > behavior. > > The other shared variables should be marked with a thread-safe modifier as > well. > [1] > [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service/src/main/java/org/apache/nifi/ssl/StandardSSLContextService.java#L121] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5694) StandardSSLContextService references configContext in a non-thread safe manner
Joseph Percivall created NIFI-5694: -- Summary: StandardSSLContextService references configContext in a non-thread safe manner Key: NIFI-5694 URL: https://issues.apache.org/jira/browse/NIFI-5694 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.7.1 Reporter: Joseph Percivall configContext is a variable which is accessed from many different threads (validate, enable, and any processor which calls "createSSLContext"). It is not declared with any thread safe modifier[1]. Potentially leading to odd behavior. I believe this is the cause of the weirdness I'm seeing where I have the SSL context "successfully" configured with a truststore of type "JKS" and am able to use it with an InovokeHttp processor. The problem is that the truststore is actually P12 (verified on the command line). I believe the issue came about because I wasn't sure if the type/password was correct and was enabling+disabling+reconfiguring it in rapid succession. The other shared variables should be marked with a thread-safe modifier as well. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle/nifi-ssl-context-service/src/main/java/org/apache/nifi/ssl/StandardSSLContextService.java#L121 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5692) InvokeHttp fails to initialize if SSL context doesn't have truststore set
Joseph Percivall created NIFI-5692:
--
Summary: InvokeHttp fails to initialize if SSL context doesn't
have truststore set
Key: NIFI-5692
URL: https://issues.apache.org/jira/browse/NIFI-5692
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.7.1
Reporter: Joseph Percivall
Impact: not able to use InvokeHttp to talk over HTTPS without using a
truststore and verifying the server.
To reproduce, create an InvokeHttp configured to use a
StandardRestrictedSSLContextService. Configure a keystore in the SSL context
but no truststore. Then enable the context. Attempting to run the processor
will fail with the following bulletin and log message:
{noformat}
InvokeHTTP[id=6875554d-0166-1000-5f09-c0e320896bfb] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to
java.lang.reflect.InvocationTargetException:
java.lang.reflect.InvocationTargetException
{noformat}
{noformat}
2018-10-12 10:30:38,384 ERROR [Timer-Driven Process Thread-1]
o.a.nifi.processors.standard.InvokeHTTP
InvokeHTTP[id=6875554d-0166-1000-5f09-c0e320896bfb] Failed to properly
initialize Processor. If still scheduled to run, NiFi will attempt to
initialize and run the Processor again after the 'Administrative Yield
Duration' has elapsed. Failure is due to
java.lang.reflect.InvocationTargetException:
java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException: null at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
at
org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$4(StandardProcessorNode.java:1499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.lang.IllegalStateException: TrustManagerFactoryImpl is not initialized
at
sun.security.ssl.TrustManagerFactoryImpl.engineGetTrustManagers(TrustManagerFactoryImpl.java:100)
at
javax.net.ssl.TrustManagerFactory.getTrustManagers(TrustManagerFactory.java:285)
at
org.apache.nifi.processors.standard.InvokeHTTP.setSslSocketFactory(InvokeHTTP.java:699)
at
org.apache.nifi.processors.standard.InvokeHTTP.setUpClient(InvokeHTTP.java:631)
... 15 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5606) UpdateRecord doesn't allow population of nested fields if input parent is null
Joseph Percivall created NIFI-5606:
--
Summary: UpdateRecord doesn't allow population of nested fields if
input parent is null
Key: NIFI-5606
URL: https://issues.apache.org/jira/browse/NIFI-5606
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.7.1
Reporter: Joseph Percivall
To reproduce, open the TestUpdateRecord.java processor and change the dynamic
properties in testFieldValuesInEL[1] to the following:
{noformat}
runner.setProperty("/name/last", "NiFi");
runner.setProperty("/name/first", "Apache");{noformat}
Also, change person.json[2] to have no name field:
{noformat}
{
"id": 485,
"mother": {
"last": "Doe",
"first": "Jane"
}
}{noformat}
After running, the output is:
{noformat}
[ { "id" : 485, "name" : null } ]{noformat}
Where the expected output would be:
{noformat}
{
"id": 485,
"name": {
"last": "NiFi",
"first": "Apache"
}
}
{noformat}
[1][https://github.com/apache/nifi/blob/4c787799ff7d971eb924df1e496da8338e6ab192/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestUpdateRecord.java#L303]
[2]
[https://github.com/apache/nifi/blob/9ebf2cfaf1fdb1a28427aed5a8168004071efd12/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestUpdateRecord/input/person.json#L3]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5084) Add GenerateRecord processor to create dummy data
[ https://issues.apache.org/jira/browse/NIFI-5084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5084: --- Summary: Add GenerateRecord processor to create dummy data (was: Add GenerateRecord processor) > Add GenerateRecord processor to create dummy data > - > > Key: NIFI-5084 > URL: https://issues.apache.org/jira/browse/NIFI-5084 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Birender Saini >Assignee: Mike Thomsen >Priority: Major > > A GenerateRecord processor (cousin of GenerateFlowFile) that generates dummy > data based on schema in the SR will be very useful in a number of scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5509) Changing an RPG URL does not log a config change event
Joseph Percivall created NIFI-5509: -- Summary: Changing an RPG URL does not log a config change event Key: NIFI-5509 URL: https://issues.apache.org/jira/browse/NIFI-5509 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.7.1, 1.7.0, 1.6.0, 1.5.0 Reporter: Joseph Percivall With NIFI-4526, the user is able to change the URL of a stopped RPG. This event is not logged in the Flow Configuration history and is not auditable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5474) ReplaceText RegexReplace evaluates payload as Expression language
[
https://issues.apache.org/jira/browse/NIFI-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564030#comment-16564030
]
Joseph Percivall commented on NIFI-5474:
Yup, thanks [~ottobackwards] !
> ReplaceText RegexReplace evaluates payload as Expression language
> -
>
> Key: NIFI-5474
> URL: https://issues.apache.org/jira/browse/NIFI-5474
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.0, 1.7.1
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Major
>
> To reproduce, add "${this will fail}" to the ReplaceTest unit test resource
> "hello.txt" and run one of the tests (like testRegexWithExpressionLanguage).
> You'll end up seeing an error message like this:
> {quote}java.lang.AssertionError:
> org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
> Invalid Expression: ${replaceValue}, World! ${this will fail} due to
> Unexpected token 'will' at line 1, column 7. Query: ${this will fail}
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:201)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:160)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:155)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:150)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:145)
> at
> org.apache.nifi.processors.standard.TestReplaceText.testRegexWithExpressionLanguage(TestReplaceText.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by:
> org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
> Invalid Expression: ${replaceValue}, World! ${this will fail} due to
> Unexpected token 'will' at line 1, column 7. Query: ${this will fail}
> at
> org.apache.nifi.attribute.expression.language.InvalidPreparedQuery.evaluateExpressions(InvalidPreparedQuery.java:49)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.util.MockPropertyValue.evaluateAttributeExpressions(MockPropertyValue.java:257)
> at
> org.apache.nifi.util.MockPropertyValue.evaluateAttributeExpressions(MockPropertyValue.java:244)
> at
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:564)
> at
> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:299)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:251)
> at
> org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:245)
> at
[jira] [Commented] (NIFI-5474) ReplaceText RegexReplace evaluates payload as Expression language
[
https://issues.apache.org/jira/browse/NIFI-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564014#comment-16564014
]
Joseph Percivall commented on NIFI-5474:
Yup, the "embedded" EL in the content being replaced was what I was trying to
convey in the ticket. Apologies if it was unclear.
I'm in the camp of "no embedded expressions". As it's something that isn't
quite intuitive and there hasn't been a need come up for it yet (no feature
requests).
> ReplaceText RegexReplace evaluates payload as Expression language
> -
>
> Key: NIFI-5474
> URL: https://issues.apache.org/jira/browse/NIFI-5474
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.7.0, 1.7.1
>Reporter: Joseph Percivall
>Priority: Major
>
> To reproduce, add "${this will fail}" to the ReplaceTest unit test resource
> "hello.txt" and run one of the tests (like testRegexWithExpressionLanguage).
> You'll end up seeing an error message like this:
> {quote}java.lang.AssertionError:
> org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
> Invalid Expression: ${replaceValue}, World! ${this will fail} due to
> Unexpected token 'will' at line 1, column 7. Query: ${this will fail}
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:201)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:160)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:155)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:150)
> at
> org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:145)
> at
> org.apache.nifi.processors.standard.TestReplaceText.testRegexWithExpressionLanguage(TestReplaceText.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by:
> org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
> Invalid Expression: ${replaceValue}, World! ${this will fail} due to
> Unexpected token 'will' at line 1, column 7. Query: ${this will fail}
> at
> org.apache.nifi.attribute.expression.language.InvalidPreparedQuery.evaluateExpressions(InvalidPreparedQuery.java:49)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.util.MockPropertyValue.evaluateAttributeExpressions(MockPropertyValue.java:257)
> at
> org.apache.nifi.util.MockPropertyValue.evaluateAttributeExpressions(MockPropertyValue.java:244)
> at
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:564)
> at
> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:299)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
>
[jira] [Created] (NIFI-5474) ReplaceText RegexReplace evaluates payload as Expression language
Joseph Percivall created NIFI-5474:
--
Summary: ReplaceText RegexReplace evaluates payload as Expression
language
Key: NIFI-5474
URL: https://issues.apache.org/jira/browse/NIFI-5474
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.7.1, 1.7.0
Reporter: Joseph Percivall
To reproduce, add "${this will fail}" to the ReplaceTest unit test resource
"hello.txt" and run one of the tests (like testRegexWithExpressionLanguage).
You'll end up seeing an error message like this:
{quote}java.lang.AssertionError:
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
Invalid Expression: ${replaceValue}, World! ${this will fail} due to
Unexpected token 'will' at line 1, column 7. Query: ${this will fail}
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:201)
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:160)
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:155)
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:150)
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:145)
at
org.apache.nifi.processors.standard.TestReplaceText.testRegexWithExpressionLanguage(TestReplaceText.java:382)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by:
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
Invalid Expression: ${replaceValue}, World! ${this will fail} due to
Unexpected token 'will' at line 1, column 7. Query: ${this will fail}
at
org.apache.nifi.attribute.expression.language.InvalidPreparedQuery.evaluateExpressions(InvalidPreparedQuery.java:49)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
at
org.apache.nifi.util.MockPropertyValue.evaluateAttributeExpressions(MockPropertyValue.java:257)
at
org.apache.nifi.util.MockPropertyValue.evaluateAttributeExpressions(MockPropertyValue.java:244)
at
org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:564)
at
org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:299)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:251)
at
org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:245)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
[jira] [Updated] (NIFI-4362) Prometheus Reporting Task
[ https://issues.apache.org/jira/browse/NIFI-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4362: --- Priority: Minor (was: Trivial) > Prometheus Reporting Task > - > > Key: NIFI-4362 > URL: https://issues.apache.org/jira/browse/NIFI-4362 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: matt price >Assignee: matt price >Priority: Minor > Labels: features, newbie > > Right now Datadog is one of the few external monitoring systems that is > supported by Nifi via a reporting task. We are building a Prometheus > reporting task that will report similar metrics as Datadog/processor status > history and wanted to contribute this back to the community. > This is my first contribution to Nifi so please correct me if I'm doing > something incorrectly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4362) Prometheus Reporting Task
[ https://issues.apache.org/jira/browse/NIFI-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4362: --- Issue Type: New Feature (was: Task) > Prometheus Reporting Task > - > > Key: NIFI-4362 > URL: https://issues.apache.org/jira/browse/NIFI-4362 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: matt price >Assignee: matt price >Priority: Trivial > Labels: features, newbie > > Right now Datadog is one of the few external monitoring systems that is > supported by Nifi via a reporting task. We are building a Prometheus > reporting task that will report similar metrics as Datadog/processor status > history and wanted to contribute this back to the community. > This is my first contribution to Nifi so please correct me if I'm doing > something incorrectly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4362) Prometheus Reporting Task
[ https://issues.apache.org/jira/browse/NIFI-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4362: --- Affects Version/s: (was: 1.3.0) > Prometheus Reporting Task > - > > Key: NIFI-4362 > URL: https://issues.apache.org/jira/browse/NIFI-4362 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: matt price >Assignee: matt price >Priority: Trivial > Labels: features, newbie > > Right now Datadog is one of the few external monitoring systems that is > supported by Nifi via a reporting task. We are building a Prometheus > reporting task that will report similar metrics as Datadog/processor status > history and wanted to contribute this back to the community. > This is my first contribution to Nifi so please correct me if I'm doing > something incorrectly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-5457. Resolution: Fixed Not a problem, thanks for the assist [~aldrin] > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556402#comment-16556402 ] Joseph Percivall commented on NIFI-5457: [~aldrin] done > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556399#comment-16556399 ] Joseph Percivall commented on NIFI-5457: Ah yup, can do > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-5457. Resolution: Fixed > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556350#comment-16556350 ] Joseph Percivall commented on NIFI-5457: [~aldrin] I checked out rel/nifi-1.7.1, changed the Dockerfile, then built and pushed the 1.7.1 tag. I then verified by pulling the tagged image down on another box. That said, I did misunderstand the push command at first which caused the 1.4.0 image to be overwritten. To make sure I didn't mess anything up, rebuilt using the tagged rel/nifi-1.4.0 commit and pushed out the resulting image. The only difference is that now 1.4.0 is listed as updated recently. Let me know if there's anything for me to fix but I'll be closing this issue as done. > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFI-5457: -- Assignee: Joseph Percivall > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556311#comment-16556311 ] Joseph Percivall commented on NIFI-5457: Ok cool, I'll work on overwriting the image > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
[ https://issues.apache.org/jira/browse/NIFI-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556269#comment-16556269 ] Joseph Percivall commented on NIFI-5457: Looking at the release guide, it's not as simple as just rebuilding and pushing to dockerhub. It's based on a tagged git push to ASF[1]. [~aldrin] any thoughts to fix it? [1] https://nifi.apache.org/release-guide.html > apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 > - > > Key: NIFI-5457 > URL: https://issues.apache.org/jira/browse/NIFI-5457 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Percivall >Priority: Major > > Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls > 1.7.0. This can be verified by looking at the DockerFile for the tagged > commit[1], looking at the pushed DockerFile[2], and verifying locally by > running "docker run apache/nifi:1.7.1" and looking at the logs. > > > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] > [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5457) apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0
Joseph Percivall created NIFI-5457: -- Summary: apache/nifi:1.7.1 Dockerhub image uses NiFi 1.7.0 Key: NIFI-5457 URL: https://issues.apache.org/jira/browse/NIFI-5457 Project: Apache NiFi Issue Type: Task Reporter: Joseph Percivall Currently, on DockerHub the tagged image for Apache NiFi 1.7.1 instead pulls 1.7.0. This can be verified by looking at the DockerFile for the tagged commit[1], looking at the pushed DockerFile[2], and verifying locally by running "docker run apache/nifi:1.7.1" and looking at the logs. [1] [https://github.com/apache/nifi/blob/rel/nifi-1.7.1/nifi-docker/dockerhub/Dockerfile#L25] [2] [https://hub.docker.com/r/apache/nifi/~/dockerfile/] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5259) Provenance repo "failed to perform background maintenance procedures" due failing to read schema
Joseph Percivall created NIFI-5259:
--
Summary: Provenance repo "failed to perform background maintenance
procedures" due failing to read schema
Key: NIFI-5259
URL: https://issues.apache.org/jira/browse/NIFI-5259
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.6.0
Environment: Dockerized NiFi v1.6.0, with a link to a Registry
instance, receiving data from MiNiFi java v0.4.0 and NiFi v1.6.0
Reporter: Joseph Percivall
Assignee: Mark Payne
Seeing an odd error (ST below) with the Provenance Repo as a background task
and also when attempting to query it. It's not getting a lot of data and the
issue persists through restarts of the container and also
stop/rm/docker-compose up of the container.
Looking at the code, it's attempting to read the first record in the repo:
final List firstEvents = eventStore.getEvents(0, 1);
Looking through the provenance record itself, it appears the event appears to
just be missing that field altogether.
{quote}2018-06-01 19:32:55,114 ERROR [Provenance Repository Maintenance-1]
o.a.n.p.index.lucene.LuceneEventIndex Failed to perform background maintenance
procedures
java.io.IOException: Invalid Boolean value found when reading 'Repetition' of
field 'Source System FlowFile Identifier'. Expected 0 or 1 but got 145
at
[org.apache.nifi|http://org.apache.nifi/].repository.schema.SchemaRecordReader.readField(SchemaRecordReader.java:107)
at
[org.apache.nifi|http://org.apache.nifi/].repository.schema.SchemaRecordReader.readRecord(SchemaRecordReader.java:72)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.EventIdFirstSchemaRecordReader.readRecord(EventIdFirstSchemaRecordReader.java:138)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.EventIdFirstSchemaRecordReader.nextRecord(EventIdFirstSchemaRecordReader.java:132)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.serialization.CompressableRecordReader.nextRecord(CompressableRecordReader.java:287)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.store.iterator.SequentialRecordReaderEventIterator.nextEvent(SequentialRecordReaderEventIterator.java:73)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.store.iterator.AuthorizingEventIterator.nextEvent(AuthorizingEventIterator.java:47)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:214)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:158)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:148)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.index.lucene.LuceneEventIndex.performMaintenance(LuceneEventIndex.java:650)
at
[org.apache.nifi|http://org.apache.nifi/].provenance.index.lucene.LuceneEventIndex.lambda$initialize$0(LuceneEventIndex.java:156)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5259) Provenance repo "failed to perform background maintenance procedures" due failing to read schema
[
https://issues.apache.org/jira/browse/NIFI-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498600#comment-16498600
]
Joseph Percivall commented on NIFI-5259:
Talking on the Apache NiFi HipChat, [~markap14] said he'd take a look next week.
> Provenance repo "failed to perform background maintenance procedures" due
> failing to read schema
>
>
> Key: NIFI-5259
> URL: https://issues.apache.org/jira/browse/NIFI-5259
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.6.0
> Environment: Dockerized NiFi v1.6.0, with a link to a Registry
> instance, receiving data from MiNiFi java v0.4.0 and NiFi v1.6.0
>Reporter: Joseph Percivall
>Assignee: Mark Payne
>Priority: Major
>
> Seeing an odd error (ST below) with the Provenance Repo as a background task
> and also when attempting to query it. It's not getting a lot of data and the
> issue persists through restarts of the container and also
> stop/rm/docker-compose up of the container.
> Looking at the code, it's attempting to read the first record in the repo:
> final List firstEvents = eventStore.getEvents(0, 1);
> Looking through the provenance record itself, it appears the event appears to
> just be missing that field altogether.
>
> {quote}2018-06-01 19:32:55,114 ERROR [Provenance Repository Maintenance-1]
> o.a.n.p.index.lucene.LuceneEventIndex Failed to perform background
> maintenance procedures
> java.io.IOException: Invalid Boolean value found when reading 'Repetition' of
> field 'Source System FlowFile Identifier'. Expected 0 or 1 but got 145
> at
> [org.apache.nifi|http://org.apache.nifi/].repository.schema.SchemaRecordReader.readField(SchemaRecordReader.java:107)
> at
> [org.apache.nifi|http://org.apache.nifi/].repository.schema.SchemaRecordReader.readRecord(SchemaRecordReader.java:72)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.EventIdFirstSchemaRecordReader.readRecord(EventIdFirstSchemaRecordReader.java:138)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.EventIdFirstSchemaRecordReader.nextRecord(EventIdFirstSchemaRecordReader.java:132)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.serialization.CompressableRecordReader.nextRecord(CompressableRecordReader.java:287)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.store.iterator.SequentialRecordReaderEventIterator.nextEvent(SequentialRecordReaderEventIterator.java:73)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.store.iterator.AuthorizingEventIterator.nextEvent(AuthorizingEventIterator.java:47)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:214)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:158)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.store.PartitionedEventStore.getEvents(PartitionedEventStore.java:148)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.index.lucene.LuceneEventIndex.performMaintenance(LuceneEventIndex.java:650)
> at
> [org.apache.nifi|http://org.apache.nifi/].provenance.index.lucene.LuceneEventIndex.lambda$initialize$0(LuceneEventIndex.java:156)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5158) getStateValue function from NiFi expression language pads extra space in front of the returned value
[ https://issues.apache.org/jira/browse/NIFI-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478056#comment-16478056 ] Joseph Percivall commented on NIFI-5158: FYI, I did some digging last night and couldn't quite figure out the root cause. Will circle back > getStateValue function from NiFi expression language pads extra space in > front of the returned value > > > Key: NIFI-5158 > URL: https://issues.apache.org/jira/browse/NIFI-5158 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Rahul Soni >Priority: Minor > Attachments: getStateValue_Test.xml > > > NiFi 1.6.0 has a bug where getStateValue returns the value with an extra > space padded in front. > Attached is a sample flow, named getStateValue_Test.xml, to recreate the > issue. > PS - The flow work as expected in NiFi 1.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5158) getStateValue function from NiFi expression language pads extra space in front of the returned value
[ https://issues.apache.org/jira/browse/NIFI-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFI-5158: -- Assignee: Joseph Percivall > getStateValue function from NiFi expression language pads extra space in > front of the returned value > > > Key: NIFI-5158 > URL: https://issues.apache.org/jira/browse/NIFI-5158 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Rahul Soni >Assignee: Joseph Percivall >Priority: Minor > Attachments: getStateValue_Test.xml > > > NiFi 1.6.0 has a bug where getStateValue returns the value with an extra > space padded in front. > Attached is a sample flow, named getStateValue_Test.xml, to recreate the > issue. > PS - The flow work as expected in NiFi 1.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5165) PutElasticsearchHttp error handling doesn't properly extract ES6 error reason
[
https://issues.apache.org/jira/browse/NIFI-5165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Percivall updated NIFI-5165:
---
Issue Type: Bug (was: Improvement)
> PutElasticsearchHttp error handling doesn't properly extract ES6 error reason
> -
>
> Key: NIFI-5165
> URL: https://issues.apache.org/jira/browse/NIFI-5165
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.6.0
>Reporter: Joseph Percivall
>Priority: Major
>
> In the event the overall call "succeeds" but a FlowFile within the batch
> fails (leading to the error handling here[1]), the error handling doesn't
> properly pull out the error reason. An example of such error in ES 6:
> {quote}{
> "took": 0,
> "ingest_took": 0,
> "errors": true,
> "items": [
> {
> "index": {
> "_index": "theIndex",
> "_type": "theType",
> "_id": null,
> "status": 400,
> "error": {
> "type": "illegal_argument_exception",
> "reason": "pipeline with id [thePipeline] does not exist"
> }
> }
> }
> ]
> {quote}
> The problem is the extra nesting of "index" which is not accounted for.
>
> [1]
> https://github.com/apache/nifi/blob/4df3eb567d8dff396b0e2380949e971d074dd04b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L367
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5165) PutElasticsearchHttp error handling doesn't properly extract ES6 error reason
Joseph Percivall created NIFI-5165:
--
Summary: PutElasticsearchHttp error handling doesn't properly
extract ES6 error reason
Key: NIFI-5165
URL: https://issues.apache.org/jira/browse/NIFI-5165
Project: Apache NiFi
Issue Type: Improvement
Affects Versions: 1.6.0
Reporter: Joseph Percivall
In the event the overall call "succeeds" but a FlowFile within the batch fails
(leading to the error handling here[1]), the error handling doesn't properly
pull out the error reason. An example of such error in ES 6:
{quote}{
"took": 0,
"ingest_took": 0,
"errors": true,
"items": [
{
"index": {
"_index": "theIndex",
"_type": "theType",
"_id": null,
"status": 400,
"error": {
"type": "illegal_argument_exception",
"reason": "pipeline with id [thePipeline] does not exist"
}
}
}
]
{quote}
The problem is the extra nesting of "index" which is not accounted for.
[1]
https://github.com/apache/nifi/blob/4df3eb567d8dff396b0e2380949e971d074dd04b/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L367
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5162) Registry Client should throttle repeated failure calls
[
https://issues.apache.org/jira/browse/NIFI-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466257#comment-16466257
]
Joseph Percivall commented on NIFI-5162:
Yup, looking again that is what I'm seeing. Was confused as we have 30+
versioned PGs and the logs are flooded with stack traces. Looking at the
timestamp, they are a flood every minute.
Yup, I agree with that approach, a similar case as yielding a processor vs
penalizing a FF. When there is something that would affect all the versioned
PGs, just fail fast.
> Registry Client should throttle repeated failure calls
> --
>
> Key: NIFI-5162
> URL: https://issues.apache.org/jira/browse/NIFI-5162
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Priority: Major
>
> In the event that the controller cannot connect to the Registry instance, it
> will repeatedly send requests as fast as possible that flood the logs with
> errors. Also, none of these errors are displayed to the user in the UI. Below
> is an example:
>
> {quote}{{2018-05-07 13:59:28,295 ERROR [Timer-Driven Process Thread-15]
> o.a.nifi.groups.StandardProcessGroup Failed to synchronize
> StandardProcessGroup[identifier=8f17ccb9-015c-1000-d297-8071b46cf5fe] with
> Flow Registry because could not retrieve version 1 of flow with identifier
> 60cb4fec-393c-46d0-bd9e-466a97f71a35 in bucket
> 3654768f-0762-45c0-9e0f-0fccf04f8402}}
> {{org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving
> flow snapshot: Unknown user with identity 'CN=fake-CN, OU=Hosts, O=Fake Org,
> C=ZZ'. Contact the system administrator.}}
> {{ at
> org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:85)}}
> {{ at
> org.apache.nifi.registry.client.impl.JerseyFlowSnapshotClient.get(JerseyFlowSnapshotClient.java:96)}}
> {{ at
> org.apache.nifi.registry.flow.RestBasedFlowRegistry.getFlowContents(RestBasedFlowRegistry.java:206)}}
> {{ at
> org.apache.nifi.registry.flow.RestBasedFlowRegistry.getFlowContents(RestBasedFlowRegistry.java:220)}}
> {{ at
> org.apache.nifi.groups.StandardProcessGroup.synchronizeWithFlowRegistry(StandardProcessGroup.java:3231)}}
> {{ at
> org.apache.nifi.controller.FlowController$4.run(FlowController.java:786)}}
> {{ at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
> {{ at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)}}
> {{ at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)}}
> {{ at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)}}
> {{ at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
> {{ at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
> {{ at java.lang.Thread.run(Thread.java:748)}}
> {{Caused by: javax.ws.rs.ForbiddenException: HTTP 403 Forbidden}}
> {{ at
> org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1083)}}
> {{ at
> org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:883)}}
> {{ at
> org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:767)}}
> {{ at org.glassfish.jersey.internal.Errors.process(Errors.java:316)}}
> {{ at org.glassfish.jersey.internal.Errors.process(Errors.java:298)}}
> {{ at org.glassfish.jersey.internal.Errors.process(Errors.java:229)}}
> {{ at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)}}
> {{ at
> org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765)}}
> {{ at
> org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:428)}}
> {{ at
> org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:324)}}
> {{ at
> org.apache.nifi.registry.client.impl.JerseyFlowSnapshotClient.lambda$get$1(JerseyFlowSnapshotClient.java:103)}}
> {{ at
> org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:71)}}
> {{ ... 12 common frames omitted}}
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5162) Registry Client should throttle repeated failure calls
Joseph Percivall created NIFI-5162:
--
Summary: Registry Client should throttle repeated failure calls
Key: NIFI-5162
URL: https://issues.apache.org/jira/browse/NIFI-5162
Project: Apache NiFi
Issue Type: Improvement
Reporter: Joseph Percivall
In the event that the controller cannot connect to the Registry instance, it
will repeatedly send requests as fast as possible that flood the logs with
errors. Also, none of these errors are displayed to the user in the UI. Below
is an example:
{quote}{{2018-05-07 13:59:28,295 ERROR [Timer-Driven Process Thread-15]
o.a.nifi.groups.StandardProcessGroup Failed to synchronize
StandardProcessGroup[identifier=8f17ccb9-015c-1000-d297-8071b46cf5fe] with Flow
Registry because could not retrieve version 1 of flow with identifier
60cb4fec-393c-46d0-bd9e-466a97f71a35 in bucket
3654768f-0762-45c0-9e0f-0fccf04f8402}}
{{org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving flow
snapshot: Unknown user with identity 'CN=fake-CN, OU=Hosts, O=Fake Org, C=ZZ'.
Contact the system administrator.}}
{{ at
org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:85)}}
{{ at
org.apache.nifi.registry.client.impl.JerseyFlowSnapshotClient.get(JerseyFlowSnapshotClient.java:96)}}
{{ at
org.apache.nifi.registry.flow.RestBasedFlowRegistry.getFlowContents(RestBasedFlowRegistry.java:206)}}
{{ at
org.apache.nifi.registry.flow.RestBasedFlowRegistry.getFlowContents(RestBasedFlowRegistry.java:220)}}
{{ at
org.apache.nifi.groups.StandardProcessGroup.synchronizeWithFlowRegistry(StandardProcessGroup.java:3231)}}
{{ at org.apache.nifi.controller.FlowController$4.run(FlowController.java:786)}}
{{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
{{ at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)}}
{{ at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)}}
{{ at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)}}
{{ at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
{{Caused by: javax.ws.rs.ForbiddenException: HTTP 403 Forbidden}}
{{ at
org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1083)}}
{{ at
org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:883)}}
{{ at
org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:767)}}
{{ at org.glassfish.jersey.internal.Errors.process(Errors.java:316)}}
{{ at org.glassfish.jersey.internal.Errors.process(Errors.java:298)}}
{{ at org.glassfish.jersey.internal.Errors.process(Errors.java:229)}}
{{ at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)}}
{{ at
org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765)}}
{{ at
org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:428)}}
{{ at
org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:324)}}
{{ at
org.apache.nifi.registry.client.impl.JerseyFlowSnapshotClient.lambda$get$1(JerseyFlowSnapshotClient.java:103)}}
{{ at
org.apache.nifi.registry.client.impl.AbstractJerseyClient.executeAction(AbstractJerseyClient.java:71)}}
{{ ... 12 common frames omitted}}
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5137) Cannot clear state of a controller service
Joseph Percivall created NIFI-5137: -- Summary: Cannot clear state of a controller service Key: NIFI-5137 URL: https://issues.apache.org/jira/browse/NIFI-5137 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.6.0 Reporter: Joseph Percivall To reproduce: # Start with a controller service that is stopped with some state stored # Go to the state manager for that controller service # Attempt to click on the "clear state" text but notice that it is disabled and doesn't do anything Workaround: * Start at step 2 (in the state manager) * Inspect the page using browser web tools * Notice the "class" of the span is "link disabled" * Remove the "disabled" class from the span * It now behaves as expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4795) AllowableValues for AccessPolicySummaryDTO are incorrect
[ https://issues.apache.org/jira/browse/NIFI-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4795: --- Affects Version/s: (was: 1.6.0) > AllowableValues for AccessPolicySummaryDTO are incorrect > > > Key: NIFI-4795 > URL: https://issues.apache.org/jira/browse/NIFI-4795 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Sébastien Bouchex Bellomié >Priority: Minor > Fix For: 1.6.0 > > > org.apache.nifi.authorization.ActionEnum defines lowercase values "read" and > "write", however, org.apache.nifi.web.api.dto.AccessPolicySummaryDTO (action) > defines allowed values as "READ" and "WRITE" (uppercase) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4893) Cannot convert Avro schemas to Record schemas with default value in arrays
[
https://issues.apache.org/jira/browse/NIFI-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Percivall updated NIFI-4893:
---
Affects Version/s: (was: 1.6.0)
> Cannot convert Avro schemas to Record schemas with default value in arrays
> --
>
> Key: NIFI-4893
> URL: https://issues.apache.org/jira/browse/NIFI-4893
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: ALL
>Reporter: Gardella Juan Pablo
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: issue1.zip
>
>
> Given an Avro Schema that has a default array defined, it is not possible to
> be converted to a Nifi Record Schema.
> To reproduce the bug, try to convert the following Avro schema to Record
> Schema:
> {code}
> {
> "type": "record",
> "name": "Foo1",
> "namespace": "foo.namespace",
> "fields": [
> {
> "name": "listOfInt",
> "type": {
> "type": "array",
> "items": "int"
> },
> "doc": "array of ints",
> "default": 0
> }
> ]
> }
> {code}
>
> Using org.apache.nifi.avro.AvroTypeUtil class. Attached a maven project to
> reproduce the issue and also the fix.
> * To reproduce the bug, run "mvn clean test"
> * To test the fix, run "mvn clean test -Ppatch".
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5009) PutParquet processor requires "read filesystem" restricted component permission but should be "write filesystem" permission instead
[ https://issues.apache.org/jira/browse/NIFI-5009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5009: --- Affects Version/s: (was: 1.6.0) > PutParquet processor requires "read filesystem" restricted component > permission but should be "write filesystem" permission instead > --- > > Key: NIFI-5009 > URL: https://issues.apache.org/jira/browse/NIFI-5009 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Reporter: Andrew Lim >Assignee: Matt Gilman >Priority: Minor > Fix For: 1.6.0 > > Attachments: PutParquet_permission.jpg > > > Similar to the other Put*** restricted processors, this is a write > processor, so it should require "write filesystem" permissions. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4958) Travis job will be successful when Maven build fails + add atlas profile
[ https://issues.apache.org/jira/browse/NIFI-4958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4958: --- Affects Version/s: (was: 1.6.0) > Travis job will be successful when Maven build fails + add atlas profile > > > Key: NIFI-4958 > URL: https://issues.apache.org/jira/browse/NIFI-4958 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.6.0 > > > With > [NIFI-4936|https://github.com/apache/nifi/commit/c71409fb5d0a3aef95b05fca9538258d2e2fb907], > the output of the build has been reduced but we loose the output code of the > Maven build command. The profile to build atlas bundle is also missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4959) HandleHttpRequest processor doesn't close/release incomplete message error
[
https://issues.apache.org/jira/browse/NIFI-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Percivall updated NIFI-4959:
---
Affects Version/s: (was: 1.6.0)
> HandleHttpRequest processor doesn't close/release incomplete message error
> --
>
> Key: NIFI-4959
> URL: https://issues.apache.org/jira/browse/NIFI-4959
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0
> Environment: Linux, all versions of nifi-1.X
>Reporter: Wynner
>Priority: Major
> Fix For: 1.6.0
>
>
> I am doing some testing with the HandleHttpRequest processor. My specific
> test, involves sending an incomplete request and closing the connection from
> the sending system. Initially, it throws the error I expect, but it keeps
> throwing the error over and over based on the request expiration configured
> in the StandardHttpContextMap controller service.
> The only way to stop the error message is to stop the processor. In my test,
> I saw one failed request throw an error six times before I stopped the
> processor.
> It doesn't seems to terminate the request on the NiFi side.
> Sample HTTP request
>
> POST/ HTTP/ 1.1
> Host: foo.com
> Content-Type: text/plain
> Content-Length: 130
> say=Hi
>
> I use the telnet command to connect to the system with the processor
> listening, post the message above , close the connection, and then the
> processor starts throws the following error indefinitely
> 2018-03-10 01:36:37,111 ERROR [Timer-Driven Process Thread-6]
> o.a.n.p.standard.HandleHttpRequest
> HandleHttpRequest[id=0d8547f7-0162-1000-9b84-129af2382259]
> HandleHttpRequest[id=0d8547f7-0162-1000-9b84-129af2382259] failed to process
> session due to org.apache.nifi.processor.exception.FlowFileAccessException:
> Failed to import data from
> HttpInputOverHTTP@46e7d12e[c=15,q=0,[0]=null,s=EARLY_EOF] for
> StandardFlowFileRecord[uuid=32bb182d-f619-4b98-b6f8-c1ed50c2736a,claim=,offset=0,name=9714775822613527,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to org.eclipse.jetty.io.EofException: Early EOF: {}
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
> import data from HttpInputOverHTTP@46e7d12e[c=15,q=0,[0]=null,s=EARLY_EOF]
> for
> StandardFlowFileRecord[uuid=32bb182d-f619-4b98-b6f8-c1ed50c2736a,claim=,offset=0,name=9714775822613527,size=0]
> due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to create ContentClaim due to org.eclipse.jetty.io.EofException: Early EOF
> at
> org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2942)
> at
> org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:517)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1123)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.nifi.processor.exception.FlowFileAccessException:
> Unable to create ContentClaim due to org.eclipse.jetty.io.EofException: Early
> EOF
> at
> org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2935)
> ... 13 common frames omitted
> Caused by: org.eclipse.jetty.io.EofException: Early EOF
> at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1104)
> at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1093)
> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35)
> at
>
[jira] [Updated] (NIFI-5033) Cannot update variable referenced in restricted components
[
https://issues.apache.org/jira/browse/NIFI-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Percivall updated NIFI-5033:
---
Affects Version/s: (was: 1.6.0)
> Cannot update variable referenced in restricted components
> --
>
> Key: NIFI-5033
> URL: https://issues.apache.org/jira/browse/NIFI-5033
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.6.0
>
>
> When updating a variable at pg level that references a restricted component
> it will fail. It seems the code is the same for secured and unsecured
> instance and it fails when NiFi is unsecured since the user is unknown.
> It seems it has been introduced by NIFI-4885.
> {noformat}
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread]
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry
> for Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Stopping
> Processors
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread]
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry
> for Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Disabling
> Controller Services
> 2018-03-29 21:10:30,913 INFO [Variable Registry Update Thread]
> o.a.nifi.web.api.ProcessGroupResource In order to update Variable Registry
> for Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50, Applying
> updates to Variable Registry
> 2018-03-29 21:10:30,915 ERROR [Variable Registry Update Thread]
> o.a.nifi.web.api.ProcessGroupResource Failed to update variable registry for
> Process Group with ID 731bbdde-0162-1000-0f00-db6543c34b50
> org.apache.nifi.authorization.AccessDeniedException: Unknown user.
> at
> org.apache.nifi.authorization.resource.RestrictedComponentsAuthorizableFactory$2.checkAuthorization(RestrictedComponentsAuthorizableFactory.java:68)
> at
> org.apache.nifi.controller.ConfiguredComponent.checkAuthorization(ConfiguredComponent.java:129)
> at
> org.apache.nifi.authorization.resource.Authorizable.checkAuthorization(Authorizable.java:183)
> at
> org.apache.nifi.authorization.resource.Authorizable.isAuthorized(Authorizable.java:70)
> at
> org.apache.nifi.web.api.dto.DtoFactory.createPermissionsDto(DtoFactory.java:1798)
> at
> org.apache.nifi.web.api.dto.DtoFactory.createPermissionsDto(DtoFactory.java:1785)
> at
> org.apache.nifi.web.api.dto.DtoFactory.lambda$createAffectedComponentEntities$73(DtoFactory.java:2485)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1540)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.nifi.web.api.dto.DtoFactory.createAffectedComponentEntities(DtoFactory.java:2489)
> at
> org.apache.nifi.web.api.dto.DtoFactory.createVariableRegistryDto(DtoFactory.java:2507)
> at
> org.apache.nifi.web.StandardNiFiServiceFacade.lambda$updateVariableRegistry$36(StandardNiFiServiceFacade.java:950)
> at
> org.apache.nifi.web.StandardNiFiServiceFacade$1.update(StandardNiFiServiceFacade.java:721)
> at
> org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:120)
> at
> org.apache.nifi.web.StandardNiFiServiceFacade.updateComponent(StandardNiFiServiceFacade.java:712)
> at
> org.apache.nifi.web.StandardNiFiServiceFacade.updateVariableRegistry(StandardNiFiServiceFacade.java:947)
> at
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
> at
> org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
> at
>
[jira] [Commented] (NIFI-1927) Improve documentation of property's ability to use Expression Languages
[ https://issues.apache.org/jira/browse/NIFI-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442995#comment-16442995 ] Joseph Percivall commented on NIFI-1927: Actually [~mattyb149] I don't think it covers all the use-cases. It is a great start but not encompassing of this ticket. This can be understood by looking at how property values get evaluated and all the overloaded options. NIFI-4149 covers the basic case of FlowFile or not[1] but does not cover the other all the other case[2]. Specifically, things like the processor passing its own variables (like RouteOnAttribute[3]), exposing stateful variables[4], or any decorators. [1] [https://github.com/apache/nifi/blob/master/nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/StandardPropertyValue.java#L137] [2] [https://github.com/apache/nifi/blob/master/nifi-commons/nifi-expression-language/src/main/java/org/apache/nifi/attribute/expression/language/StandardPropertyValue.java#L153] [3] [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RouteText.java#L129] [4] [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L171] > Improve documentation of property's ability to use Expression Languages > --- > > Key: NIFI-1927 > URL: https://issues.apache.org/jira/browse/NIFI-1927 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Major > > Currently in the documentation for properties there is a simple true/false on > whether or not Expression Language(EL) is supported for that property. This > trivializes the extent of EL and for some hides some functionality. There are > multiple different options when deciding to evaluate EL, the most relevant to > the user are: whether or not a FlowFile is used and if there are any extra > key/value pairs exposed. > The documentation should be explicit so that the user knows what will get > evaluated and what is exposed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5085) InvokeHttp does not close the response in all cases
[ https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5085: --- Status: Patch Available (was: Open) > InvokeHttp does not close the response in all cases > --- > > Key: NIFI-5085 > URL: https://issues.apache.org/jira/browse/NIFI-5085 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.6.0, 1.5.0, 1.4.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), > the response needs to be closed for every instance of the response. > InvokeHttp currently only closes the underlying body stream in the instance > of there existing a body[2][3]. The proper way to do it is how the > HttpNotificationService does, utilizing a try with resources on the > response[4]. > > I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. > I'm still not a 100% sure this is the root cause but one common thing between > those OOM errors is repeated socket timeout exceptions in InvokeHttp (and > even if it's not, it should still be fixed). > > [1] https://github.com/square/okhttp/issues/2311 > [2] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822 > > [3] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894 > > [4] > https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5085) InvokeHttp does not close the response in all cases
[ https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5085: --- Affects Version/s: (was: 0.7.4) (was: 1.3.0) (was: 1.0.1) (was: 1.1.1) (was: 1.2.0) (was: 0.7.1) (was: 1.1.0) (was: 0.6.1) (was: 0.7.0) (was: 0.5.1) (was: 0.4.1) (was: 0.6.0) (was: 0.5.0) (was: 0.4.0) (was: 1.0.0) > InvokeHttp does not close the response in all cases > --- > > Key: NIFI-5085 > URL: https://issues.apache.org/jira/browse/NIFI-5085 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.4.0, 1.5.0, 1.6.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), > the response needs to be closed for every instance of the response. > InvokeHttp currently only closes the underlying body stream in the instance > of there existing a body[2][3]. The proper way to do it is how the > HttpNotificationService does, utilizing a try with resources on the > response[4]. > > I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. > I'm still not a 100% sure this is the root cause but one common thing between > those OOM errors is repeated socket timeout exceptions in InvokeHttp (and > even if it's not, it should still be fixed). > > [1] https://github.com/square/okhttp/issues/2311 > [2] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822 > > [3] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894 > > [4] > https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5085) InvokeHttp does not close the response in all cases
[ https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440082#comment-16440082 ] Joseph Percivall commented on NIFI-5085: Removed the "affectsVersion" for all prior to 1.4.0 as I believe this is a regression hit as part of NIFI-2162 > InvokeHttp does not close the response in all cases > --- > > Key: NIFI-5085 > URL: https://issues.apache.org/jira/browse/NIFI-5085 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.4.0, 1.5.0, 1.6.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), > the response needs to be closed for every instance of the response. > InvokeHttp currently only closes the underlying body stream in the instance > of there existing a body[2][3]. The proper way to do it is how the > HttpNotificationService does, utilizing a try with resources on the > response[4]. > > I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. > I'm still not a 100% sure this is the root cause but one common thing between > those OOM errors is repeated socket timeout exceptions in InvokeHttp (and > even if it's not, it should still be fixed). > > [1] https://github.com/square/okhttp/issues/2311 > [2] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822 > > [3] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894 > > [4] > https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5085) InvokeHttp does not close the response in all cases
[ https://issues.apache.org/jira/browse/NIFI-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440074#comment-16440074 ] Joseph Percivall commented on NIFI-5085: Was curious why I didn't do this in the first place, turns out "closeable" wasn't added to the response until after I refactored it[1]. Not sure when the requirement for always closing it was added though. At the very least, the try with resources should've been added in NIFI-2162 when we updated to OkHttp 3.x. [1] https://github.com/square/okhttp/commit/3699d5c9fd0ad78fc52e3ea317951f9d485f656f > InvokeHttp does not close the response in all cases > --- > > Key: NIFI-5085 > URL: https://issues.apache.org/jira/browse/NIFI-5085 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.0.0, 0.4.0, 0.5.0, 0.6.0, 0.4.1, 0.5.1, 0.7.0, 0.6.1, > 1.1.0, 0.7.1, 1.2.0, 1.1.1, 1.0.1, 1.3.0, 1.4.0, 0.7.4, 1.5.0, 1.6.0 >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Major > > As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), > the response needs to be closed for every instance of the response. > InvokeHttp currently only closes the underlying body stream in the instance > of there existing a body[2][3]. The proper way to do it is how the > HttpNotificationService does, utilizing a try with resources on the > response[4]. > > I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. > I'm still not a 100% sure this is the root cause but one common thing between > those OOM errors is repeated socket timeout exceptions in InvokeHttp (and > even if it's not, it should still be fixed). > > [1] https://github.com/square/okhttp/issues/2311 > [2] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822 > > [3] > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894 > > [4] > https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5086) Many usages of AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch aren't closed properly
Joseph Percivall created NIFI-5086: -- Summary: Many usages of AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch aren't closed properly Key: NIFI-5086 URL: https://issues.apache.org/jira/browse/NIFI-5086 Project: Apache NiFi Issue Type: Bug Reporter: Joseph Percivall Similar to NIFI-5085, all OkHttp responses must be closed[1]. "AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch" returns the actual OkHttp response. All implementing processors should properly close this. Processors that need to be updated: * FetchElasticsearchHttp[2] * PutElasticsearchHttp[3] * PutElasticsearchHttpRecord[4] * QueryElasticsearchHttp[5] * ScrollElasticsearchHtt[6][7] [1] [https://github.com/square/okhttp/issues/2311] [2] [https://github.com/apache/nifi/blob/9736cb9d338b611ff3a096d97fd439cf8b8bbac3/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java#L223 ][3][https://github.com/apache/nifi/blob/e916594b69601bce58e045ba8ae2acf6af66eb46/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L337] [4] [https://github.com/apache/nifi/blob/6b34d3bea9a1fd0923024018d73c3c0b0a807a67/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L400] [5] [https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L287 ][6] [https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L256 ][7][https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L269] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5086) Many usages of AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch aren't closed properly
[ https://issues.apache.org/jira/browse/NIFI-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-5086: --- Description: Similar to NIFI-5085, all OkHttp responses must be closed[1]. "AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch" returns the actual OkHttp response. All implementing processors should properly close this. Processors that need to be updated: * FetchElasticsearchHttp[2] * PutElasticsearchHttp[3] * PutElasticsearchHttpRecord[4] * QueryElasticsearchHttp[5] * ScrollElasticsearchHtt[6][7] [1] [https://github.com/square/okhttp/issues/2311] [2] [[https://github.com/apache/nifi/blob/9736cb9d338b611ff3a096d97fd439cf8b8bbac3/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java#L223] [3][https://github.com/apache/nifi/blob/e916594b69601bce58e045ba8ae2acf6af66eb46/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L337] [4] [https://github.com/apache/nifi/blob/6b34d3bea9a1fd0923024018d73c3c0b0a807a67/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L400] [5] [[https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L287] [6] [[https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L256] [7][https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L269] was: Similar to NIFI-5085, all OkHttp responses must be closed[1]. "AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch" returns the actual OkHttp response. All implementing processors should properly close this. Processors that need to be updated: * FetchElasticsearchHttp[2] * PutElasticsearchHttp[3] * PutElasticsearchHttpRecord[4] * QueryElasticsearchHttp[5] * ScrollElasticsearchHtt[6][7] [1] [https://github.com/square/okhttp/issues/2311] [2] [https://github.com/apache/nifi/blob/9736cb9d338b611ff3a096d97fd439cf8b8bbac3/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearchHttp.java#L223 ][3][https://github.com/apache/nifi/blob/e916594b69601bce58e045ba8ae2acf6af66eb46/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L337] [4] [https://github.com/apache/nifi/blob/6b34d3bea9a1fd0923024018d73c3c0b0a807a67/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L400] [5] [https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L287 ][6] [https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L256 ][7][https://github.com/apache/nifi/blob/6baea8ccffe93e6ea6289cac8970f95e95f797bf/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L269] > Many usages of AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch > aren't closed properly > --- > > Key: NIFI-5086 > URL: https://issues.apache.org/jira/browse/NIFI-5086 > Project: Apache NiFi > Issue Type: Bug >Reporter: Joseph Percivall >Priority: Major > > Similar to NIFI-5085, all OkHttp responses must be closed[1]. > "AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch" returns the > actual OkHttp response. All implementing processors should properly close > this. Processors that need to be updated: > * FetchElasticsearchHttp[2] > * PutElasticsearchHttp[3] > * PutElasticsearchHttpRecord[4] > * QueryElasticsearchHttp[5] > * ScrollElasticsearchHtt[6][7] > [1]
[jira] [Created] (NIFI-5085) InvokeHttp does not close the response in all cases
Joseph Percivall created NIFI-5085: -- Summary: InvokeHttp does not close the response in all cases Key: NIFI-5085 URL: https://issues.apache.org/jira/browse/NIFI-5085 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.6.0, 1.5.0, 0.7.4, 1.4.0, 1.3.0, 1.0.1, 1.1.1, 1.2.0, 0.7.1, 1.1.0, 0.6.1, 0.7.0, 0.5.1, 0.4.1, 0.6.0, 0.5.0, 0.4.0, 1.0.0 Reporter: Joseph Percivall Assignee: Joseph Percivall As stated in this github issue for OkHttp[1] (the library InvokeHttp uses), the response needs to be closed for every instance of the response. InvokeHttp currently only closes the underlying body stream in the instance of there existing a body[2][3]. The proper way to do it is how the HttpNotificationService does, utilizing a try with resources on the response[4]. I ran into this issue when my 1.5.0 instance hit OOM errors multiple times. I'm still not a 100% sure this is the root cause but one common thing between those OOM errors is repeated socket timeout exceptions in InvokeHttp (and even if it's not, it should still be fixed). [1] https://github.com/square/okhttp/issues/2311 [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L822 [3] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L894 [4] https://github.com/apache/nifi/blob/master/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/notification/http/HttpNotificationService.java#L230 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419656#comment-16419656 ] Joseph Percivall commented on NIFI-3576: [~ottobackwards] I agree [~panchicore] and [~wietze], would this path forward solve your use-cases? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419631#comment-16419631 ] Joseph Percivall commented on NIFI-3576: Hmm, the attributes on a no-hits REL FF could be the same as a query-info REL FF. So the main differences would be the name and when it emits a FF. What if we took a best of both worlds and added the query-info relationship and for the property, it is an option of when to emit the FF to that relationship? The options would either be "never", "when there are no hits", or "always" (default to never). > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419599#comment-16419599 ] Joseph Percivall commented on NIFI-3576: There are a couple examples, such as RouteOnAttribute, RouteOnContent, and QueryRecord but the best example for this use case is probably LookUpRecord[1]. Ah looks like it's still an open ticket to add DisplayName to Relationships[2]. Definitely, would rather not break people's flows. Let's just change up the description to better match. [1] [https://github.com/apache/nifi/blob/d7347a2dc3f73e76d03b9b3ef95015dd539e16ac/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupRecord.java#L237] [2] https://issues.apache.org/jira/browse/NIFI-3591 > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419555#comment-16419555 ] Joseph Percivall commented on NIFI-3576: Ah, I see your point about the FlowFile per hit and agree that there's some weirdness with the attributes for an empty "hit" FlowFile. The actual "hit per FF" functionality kinda goes against what is documented so we should change that a bit. With that, I also agree that 3 isn't what we want. So what if we changed the displayName for "success" to "hits" and add a new property to say whether to emit a FF to a 'no hits' relationship. The property would default to not emitting and the relationship would only be available when that prop is true. That way the functionality is fully backwards compatible. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419524#comment-16419524 ] Joseph Percivall commented on NIFI-3576: So to summarize we have a couple of different options: # A new 'original' relationship to route the original FlowFile which has an attribute like 'es.hits.total' - precedent set by InvokeHttp # A 'none found' relationship which routes a FlowFile when there are no hits - precedent set by FetchElasticsearch(5) # Emit a FlowFile to "Success" for but add an attribute 'es.hits.total' - matches the 'Success' relationship documentation which states "All FlowFiles that are read from Elasticsearch are routed to this relationship." I'm leaning more towards option 3. It seems more in line with the original intent of the relationship, and the processor as a whole. It's querying for potential hits, not fetching a single document, so having no results but, a successful query is valid. Also, this means that users won't have an invalid processor on migration (the new relationships wouldn't be routed anywhere). Then we also aren't duplicating logic which could just be in a follow-up RouteOnAttribute processor. Thoughts? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-4325. Resolution: Fixed Fix Version/s: 1.6.0 > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Minor > Fix For: 1.6.0 > > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4325: --- Issue Type: New Feature (was: Improvement) > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Minor > Fix For: 1.6.0 > > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFI-4325: -- Assignee: Mike Thomsen > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4977) PutSyslog should support Expression Language for it's Sender properties
[ https://issues.apache.org/jira/browse/NIFI-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4977: --- Status: Patch Available (was: Open) > PutSyslog should support Expression Language for it's Sender properties > --- > > Key: NIFI-4977 > URL: https://issues.apache.org/jira/browse/NIFI-4977 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Minor > > As part of the continued effort to add access to the variable registry to > processors, the PutSyslog processor should expose expression language to it's > sender properties. These properties control where and how the processor sends > to. > This update should better align PutSyslog with the other PutX processors > which support EL on the sender properties (Slack, TCP, UDP). > These properties are not evaluated per FlowFile and shouldn't be expected to > change between onTriggers (since the sender pool is shared) so it will need > to be documented as such that the -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4977) PutSyslog should support Expression Language for it's Sender properties
Joseph Percivall created NIFI-4977: -- Summary: PutSyslog should support Expression Language for it's Sender properties Key: NIFI-4977 URL: https://issues.apache.org/jira/browse/NIFI-4977 Project: Apache NiFi Issue Type: Improvement Reporter: Joseph Percivall Assignee: Joseph Percivall As part of the continued effort to add access to the variable registry to processors, the PutSyslog processor should expose expression language to it's sender properties. These properties control where and how the processor sends to. This update should better align PutSyslog with the other PutX processors which support EL on the sender properties (Slack, TCP, UDP). These properties are not evaluated per FlowFile and shouldn't be expected to change between onTriggers (since the sender pool is shared) so it will need to be documented as such that the -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394567#comment-16394567 ] Joseph Percivall commented on NIFI-4936: [~joewitt] is there anything we'll need to add to the Migration Guidance for this for those that are utilizing the parent pom for their nar bundles? > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Joseph Witt >Priority: Major > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4956) Cannot import or upgrade versioned flow that includes a CRON_DRIVEN scheduling strategy
[ https://issues.apache.org/jira/browse/NIFI-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393838#comment-16393838 ] Joseph Percivall commented on NIFI-4956: Whelp this is a little embarrassing, lol that appears to be the exact same thing. Should've searched a little harder. I'll double check on master but I'm going to mark this as a duplicate and close it. Thanks [~bende] > Cannot import or upgrade versioned flow that includes a CRON_DRIVEN > scheduling strategy > --- > > Key: NIFI-4956 > URL: https://issues.apache.org/jira/browse/NIFI-4956 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Joseph Percivall >Priority: Major > > To reproduce: > * Create a PG with a process in it with a CRON driven scheduling strategy > * Start version control of the PG > * Attempt create a new PG by "import"ing the versioned flow from above > * Notice that an Error occurs that says "Value '* * * * * ?' is not a valid > Time Duration and the processor exists but has Timer driven scheduling. > * Also, notice that the PG is partially created (if there are other > components they may be created) but is not versioned > This also occurs when upgrading a flow but with a slightly different error > message > "Failed to update flow to new version due to > java.lang.IllegalArgumentException: Value '* * * * *?' is not a valid Time > Duration" > > I checked the flow_storage of the registry instance and it does have > CRON_DRIVEN for the scheduling strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4956) Cannot import or upgrade versioned flow that includes a CRON_DRIVEN scheduling strategy
[ https://issues.apache.org/jira/browse/NIFI-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-4956. Resolution: Duplicate > Cannot import or upgrade versioned flow that includes a CRON_DRIVEN > scheduling strategy > --- > > Key: NIFI-4956 > URL: https://issues.apache.org/jira/browse/NIFI-4956 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Joseph Percivall >Priority: Major > > To reproduce: > * Create a PG with a process in it with a CRON driven scheduling strategy > * Start version control of the PG > * Attempt create a new PG by "import"ing the versioned flow from above > * Notice that an Error occurs that says "Value '* * * * * ?' is not a valid > Time Duration and the processor exists but has Timer driven scheduling. > * Also, notice that the PG is partially created (if there are other > components they may be created) but is not versioned > This also occurs when upgrading a flow but with a slightly different error > message > "Failed to update flow to new version due to > java.lang.IllegalArgumentException: Value '* * * * *?' is not a valid Time > Duration" > > I checked the flow_storage of the registry instance and it does have > CRON_DRIVEN for the scheduling strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4956) Cannot import or upgrade versioned flow that includes a CRON_DRIVEN scheduling strategy
Joseph Percivall created NIFI-4956: -- Summary: Cannot import or upgrade versioned flow that includes a CRON_DRIVEN scheduling strategy Key: NIFI-4956 URL: https://issues.apache.org/jira/browse/NIFI-4956 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.5.0 Reporter: Joseph Percivall To reproduce: * Create a PG with a process in it with a CRON driven scheduling strategy * Start version control of the PG * Attempt create a new PG by "import"ing the versioned flow from above * Notice that an Error occurs that says "Value '* * * * * ?' is not a valid Time Duration and the processor exists but has Timer driven scheduling. * Also, notice that the PG is partially created (if there are other components they may be created) but is not versioned This also occurs when upgrading a flow but with a slightly different error message "Failed to update flow to new version due to java.lang.IllegalArgumentException: Value '* * * * *?' is not a valid Time Duration" I checked the flow_storage of the registry instance and it does have CRON_DRIVEN for the scheduling strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4913) Specifying "run.as" in bootstrap.properties causes environment variables to not be passed to NiFi
[ https://issues.apache.org/jira/browse/NIFI-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4913: --- Labels: beginner (was: ) > Specifying "run.as" in bootstrap.properties causes environment variables to > not be passed to NiFi > - > > Key: NIFI-4913 > URL: https://issues.apache.org/jira/browse/NIFI-4913 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.5.0 >Reporter: Jonathan Peterson >Priority: Minor > Labels: beginner > > I solved this by including "-E" after "sudo" in nifi.sh [1]. > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.5.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh#L312] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4913) Specifying "run.as" in bootstrap.properties causes environment variables to not be passed to NiFi
[ https://issues.apache.org/jira/browse/NIFI-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFI-4913: --- Component/s: (was: Tools and Build) Configuration > Specifying "run.as" in bootstrap.properties causes environment variables to > not be passed to NiFi > - > > Key: NIFI-4913 > URL: https://issues.apache.org/jira/browse/NIFI-4913 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.5.0 >Reporter: Jonathan Peterson >Priority: Minor > > I solved this by including "-E" after "sudo" in nifi.sh [1]. > [1] > [https://github.com/apache/nifi/blob/rel/nifi-1.5.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh#L312] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4874) RPGs to flows that share the same flow and port UUIDs can cause non-deterministic FlowFile transmission
[ https://issues.apache.org/jira/browse/NIFI-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall resolved NIFI-4874. Resolution: Fixed I believe this to be fixed as part of NIFI-3155 > RPGs to flows that share the same flow and port UUIDs can cause > non-deterministic FlowFile transmission > --- > > Key: NIFI-4874 > URL: https://issues.apache.org/jira/browse/NIFI-4874 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Joseph Percivall >Priority: Major > Fix For: 1.5.0 > > > To replicate, set up three different 1.4.0 NiFi instances, one sender, and > two receivers. On one of the senders, create a simple flow consisting of a > single input port with a connection to a LogAttribute processor. Copy this > instance's flow.xml to the other receiving instance. Start the remaining > instances. > On the sender, create a flow with a GenerateFlowFile to a Funnel which has > connections going to two RPGs (one for each receiver). With both RPGs having > disabled transmission generate a single FlowFile (leading to the queues in > front of each RPG with one). > To be explicit, the receiving flows must be started from copies of the same > flow.xml (leading to the same port and flow UUIDs). > Enable and disable transmission on one RPG. Notice the following: > 1: The stats on both increases (NIFI-4571) > 2: Potentially, that the FlowFile was sent to the wrong instance. > Repeat with the other RPG and as needed (since the issue is > non-deterministic). > This also appears to happen shortly after enabling both RPGs > (GenerateFlowFile stopped, enabled both RPGs, start/stop the GenerateFF > relatively quickly to see non-deterministic behavior). > > This appears to have been fixed in 1.5.0 (I can't replicate) and I'm guessing > fixed with NIFI-3155. Logging this issue for explicit visibility of the bug > manifestation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4874) RPGs to flows that share the same flow and port UUIDs can cause non-deterministic FlowFile transmission
Joseph Percivall created NIFI-4874: -- Summary: RPGs to flows that share the same flow and port UUIDs can cause non-deterministic FlowFile transmission Key: NIFI-4874 URL: https://issues.apache.org/jira/browse/NIFI-4874 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.4.0 Reporter: Joseph Percivall Fix For: 1.5.0 To replicate, set up three different 1.4.0 NiFi instances, one sender, and two receivers. On one of the senders, create a simple flow consisting of a single input port with a connection to a LogAttribute processor. Copy this instance's flow.xml to the other receiving instance. Start the remaining instances. On the sender, create a flow with a GenerateFlowFile to a Funnel which has connections going to two RPGs (one for each receiver). With both RPGs having disabled transmission generate a single FlowFile (leading to the queues in front of each RPG with one). To be explicit, the receiving flows must be started from copies of the same flow.xml (leading to the same port and flow UUIDs). Enable and disable transmission on one RPG. Notice the following: 1: The stats on both increases (NIFI-4571) 2: Potentially, that the FlowFile was sent to the wrong instance. Repeat with the other RPG and as needed (since the issue is non-deterministic). This also appears to happen shortly after enabling both RPGs (GenerateFlowFile stopped, enabled both RPGs, start/stop the GenerateFF relatively quickly to see non-deterministic behavior). This appears to have been fixed in 1.5.0 (I can't replicate) and I'm guessing fixed with NIFI-3155. Logging this issue for explicit visibility of the bug manifestation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFI-4871) Data Queuing at Process Group Input Port
[ https://issues.apache.org/jira/browse/NIFI-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361231#comment-16361231 ] Joseph Percivall edited comment on NIFI-4871 at 2/12/18 6:31 PM: - This should probably be handled in the same way as funnels, which has a while loop[1] that transfers all the files in a single onTrigger but in 100 FlowFile batches. [1] [https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java#L365] was (Author: jpercivall): This should probably be handled in the same way was funnels, which has a while loop[1] that transfers all the files in a single onTrigger but in 100 FlowFile batches. [1] https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java#L365 > Data Queuing at Process Group Input Port > > > Key: NIFI-4871 > URL: https://issues.apache.org/jira/browse/NIFI-4871 > Project: Apache NiFi > Issue Type: Bug >Reporter: Brandon DeVries >Priority: Major > > When a flow is under heavy load, we've seen data queue on the way in to a > process group. There is nothing queued *inside* the process group, and no > back pressure... it simply isn't moving files fast enough. As a workaround, > multiple input ports were added... but that's ugly. > The first issue may be the limit of 100 files transferred on any call to > onTrigger()[1]. 100 seems low, ideally it would be nice to fill to the back > pressure point (or some reasonable limit if set to "0"). This may also be > compounded by some scheduling issues we've seen, but not reliably enough to > actually diagnose. But the combination of not transferring enough files on > each run and not running as often as it "should" is definitely causing > issues. The scheduling issue may be complex, but transferring more files > should be pretty straightforward... > > [1] > [https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java#L76] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4871) Data Queuing at Process Group Input Port
[ https://issues.apache.org/jira/browse/NIFI-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361231#comment-16361231 ] Joseph Percivall commented on NIFI-4871: This should probably be handled in the same way was funnels, which has a while loop[1] that transfers all the files in a single onTrigger but in 100 FlowFile batches. [1] https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/controller/StandardFunnel.java#L365 > Data Queuing at Process Group Input Port > > > Key: NIFI-4871 > URL: https://issues.apache.org/jira/browse/NIFI-4871 > Project: Apache NiFi > Issue Type: Bug >Reporter: Brandon DeVries >Priority: Major > > When a flow is under heavy load, we've seen data queue on the way in to a > process group. There is nothing queued *inside* the process group, and no > back pressure... it simply isn't moving files fast enough. As a workaround, > multiple input ports were added... but that's ugly. > The first issue may be the limit of 100 files transferred on any call to > onTrigger()[1]. 100 seems low, ideally it would be nice to fill to the back > pressure point (or some reasonable limit if set to "0"). This may also be > compounded by some scheduling issues we've seen, but not reliably enough to > actually diagnose. But the combination of not transferring enough files on > each run and not running as often as it "should" is definitely causing > issues. The scheduling issue may be complex, but transferring more files > should be pretty straightforward... > > [1] > [https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java#L76] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4837) Thread leak on HandleHTTPRequest processor
[
https://issues.apache.org/jira/browse/NIFI-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Percivall updated NIFI-4837:
---
Description:
When you have multiple HandleHTTPRequest processors trying to listen on the
same port, for every Listen attempt NiFi builds a new thread and never recycles
the old thread which eventually leads to NiFi shutting down when reaching the
OS limit of the number of threads (default is 10.000).
The following error can be seen in nifi-app.log:
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
This has happened before with version 1.2 and probably even with older
versions. but I could also replicate the issue with the latest 1.5 version.
Steps to replicate the issue:
1) build a simple flow with 2 HandleHTTPRequest processors listening on the
same port.
!image-2018-02-02-11-14-51-964.png!
2) Start the processors.
— The second HandleHTTPRequest processor starts logging following as
expected:
2018-02-02 16:18:29,518 ERROR [Timer-Driven Process Thread-3]
o.a.n.p.standard.HandleHttpRequest
HandleHttpRequest[id=af013c62-b26f-1eeb-ae81-8423c70bdc7f] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException: Failed to
initialize the server: {}
org.apache.nifi.processor.exception.ProcessException: Failed to initialize the
server
Caused by: java.net.BindException: Address already in use
...
... 12 common frames omitted
3) Go to the Summary section in NiFi and watch the number of threads going up
to 9959.
!image-2018-02-02-11-16-52-389.png!
With above, I had processors scheduled on primary node only as to not affect
every node.
If you stop the second HandleHTTPRequest processor the threads stop climbing,
but are not released.
After this, NiFi will soon stop.
A restart of NIFi is required to release these threads.
was:
When you have multiple HandleHTTPRequest processors trying to listen on the
same port, for every Listen attempt NiFi builds a new thread and never recycles
the old thread which eventually leads to NiFi shutting down when reaching the
OS limit of the number of threads (default is 10.000).
The following error can be seen in nifi-app.log:
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
This has happened before with HDF 3.0.1 and probably even with older versions.
but I could also replicate the issue with the latest HDF 3.1.
Steps to replicate the issue:
1) build a simple flow with 2 HandleHTTPRequest processors listening on the
same port.
!image-2018-02-02-11-14-51-964.png!
2) Start the processors.
— The second HandleHTTPRequest processor starts logging following as
expected:
2018-02-02 16:18:29,518 ERROR [Timer-Driven Process Thread-3]
o.a.n.p.standard.HandleHttpRequest
HandleHttpRequest[id=af013c62-b26f-1eeb-ae81-8423c70bdc7f] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException: Failed to
initialize the server: {}
org.apache.nifi.processor.exception.ProcessException: Failed to initialize the
server
Caused by: java.net.BindException: Address already in use
...
... 12 common frames omitted
3) Go to the Summary section in NiFi and watch the number of threads going up
to 9959.
!image-2018-02-02-11-16-52-389.png!
With above, I had processors scheduled on primary node only as to not affect
every node.
If you stop the second HandleHTTPRequest processor the threads stop climbing,
but are not released.
After this, NiFi will soon stop.
A restart of NIFi is required to release these threads.
> Thread leak on HandleHTTPRequest processor
> --
>
> Key: NIFI-4837
> URL: https://issues.apache.org/jira/browse/NIFI-4837
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0
> Environment: CENTOS 7
>Reporter: Matthew Clarke
>Priority: Blocker
> Attachments: image-2018-02-02-11-14-51-964.png,
> image-2018-02-02-11-16-52-389.png
>
>
> When you have multiple HandleHTTPRequest processors trying to listen on the
> same port, for every Listen attempt NiFi builds a new thread and never
> recycles the old thread which eventually leads to NiFi shutting down when
> reaching the OS limit of the number of threads (default is 10.000).
> The following error can be seen in nifi-app.log:
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> This has happened before with version 1.2 and probably even with older
[jira] [Created] (NIFI-4817) GetFile is missing a context.yield when using "Polling Interval", causing it to spin
Joseph Percivall created NIFI-4817: -- Summary: GetFile is missing a context.yield when using "Polling Interval", causing it to spin Key: NIFI-4817 URL: https://issues.apache.org/jira/browse/NIFI-4817 Project: Apache NiFi Issue Type: Bug Reporter: Joseph Percivall The "Polling Interval" property is used so that the listing of the directory isn't done on every onTrigger of the processor. The actual check is here[1]. In the event, the fileQueue is empty and it's not yet time to list the directory again, the processor returns here[2]. Since there isn't a context.yield before the return, the processor (assuming the default scheduling period of '0 sec') will immediately attempt to do the same thing. Causing the processor to spin and take up as many concurrent tasks as it's allowed. A "context.yield" should be added before the return so that the processor doesn't just spin between listings. [1] https://github.com/apache/nifi/blob/7f5eabd603bfc326dadc35590bbe69304e8c90fa/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java#L372 [2] https://github.com/apache/nifi/blob/7f5eabd603bfc326dadc35590bbe69304e8c90fa/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java#L406 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-77) Allow a user to see the changes created by the currently loaded version
[ https://issues.apache.org/jira/browse/NIFIREG-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316807#comment-16316807 ] Joseph Percivall commented on NIFIREG-77: - [~bbende] when I originally created the ticket I was thinking of the view within NiFi but given this is the Registry Jira, I agree with you that it should focus on the Registry side and we'd create a ticket for the NiFi side. Partially I didn't want to create a bunch of tickets in the NiFi Jira at the time because the core PR wasn't merged yet. Now that's merged we can go that route though. > Allow a user to see the changes created by the currently loaded version > --- > > Key: NIFIREG-77 > URL: https://issues.apache.org/jira/browse/NIFIREG-77 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.1.0 >Reporter: Joseph Percivall >Priority: Critical > > As a user, I would like to see the changes that are included in a particular > version. More specifically, if I'm on an old version and I upgrade to a > version written by someone else, I have no way to know what changes occurred > during that version upgrade. > A simple solution would be to utilize the same logic which displays the > current differences between local and stored in the registry and use that to > show the differences between the current version N and version N-1. The user > could then change between versions to see the changes that happened as part > of that version. > An even better solution (from a DFM perspective) would be to be able to see > the changes within any version (not just the most recent). That way a DFM > wouldn't have to stop the flow for an extended period of time to view the > changes/differences in different versions but I think that'd be more work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFIREG-77) Allow a user to see the changes created by the currently loaded version
[ https://issues.apache.org/jira/browse/NIFIREG-77?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313361#comment-16313361 ] Joseph Percivall commented on NIFIREG-77: - Hey [~rmoran], Totally agree with the need for a "diff" feature. I see the end goal of the features and UX of versioned flows as very similar to the UX of git. Thinking along the lines of git, the most used features in git are to see the locally uncommitted changes and see what changed in the last X commits (i.e., checking a PR). That said, while they're similar, the mentality of each is quite different and I don't think they should always be treated as such (combining them into one "compare versions" action). The first is checking things that have yet to be saved and can still be changed. The other is verifying changes that have already been saved. So I don't think having a shortcut for "show local changes" and the ability to "compare versions" (including the local one) are mutually exclusive. I understand wanting consistency, but I'm not convinced that changing "show local changes" to only be a part of the "compare versions" would be a better UX. > Allow a user to see the changes created by the currently loaded version > --- > > Key: NIFIREG-77 > URL: https://issues.apache.org/jira/browse/NIFIREG-77 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.1.0 >Reporter: Joseph Percivall >Priority: Critical > > As a user, I would like to see the changes that are included in a particular > version. More specifically, if I'm on an old version and I upgrade to a > version written by someone else, I have no way to know what changes occurred > during that version upgrade. > A simple solution would be to utilize the same logic which displays the > current differences between local and stored in the registry and use that to > show the differences between the current version N and version N-1. The user > could then change between versions to see the changes that happened as part > of that version. > An even better solution (from a DFM perspective) would be to be able to see > the changes within any version (not just the most recent). That way a DFM > wouldn't have to stop the flow for an extended period of time to view the > changes/differences in different versions but I think that'd be more work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (NIFIREG-82) Properly handle when someone goes to registry but without the "/nifi-registry/" path
[ https://issues.apache.org/jira/browse/NIFIREG-82?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall closed NIFIREG-82. --- Resolution: Duplicate > Properly handle when someone goes to registry but without the > "/nifi-registry/" path > > > Key: NIFIREG-82 > URL: https://issues.apache.org/jira/browse/NIFIREG-82 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Trivial > > Currently, Registry goes to the Jetty 404 page. We should have something > similar to NiFi with the page that says "Did you mean: /nifiYou may have > mistyped" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (NIFIREG-89) Add a default URL handler for root '/' instead of returning a 404
[ https://issues.apache.org/jira/browse/NIFIREG-89?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall reassigned NIFIREG-89: --- Assignee: Danny Lane > Add a default URL handler for root '/' instead of returning a 404 > - > > Key: NIFIREG-89 > URL: https://issues.apache.org/jira/browse/NIFIREG-89 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Danny Lane >Assignee: Danny Lane >Priority: Minor > Labels: usability > > Currently when you land on the root or a NiFi Registry deployment you get a > 404 response from Jetty. > It was suggested in the mailing list to add a page similar to the NiFi 'You > may have mistyped...' page. > This is item to track that suggestion and associated work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFIREG-82) Properly handle when someone goes to registry but without the "/nifi-registry/" path
Joseph Percivall created NIFIREG-82: --- Summary: Properly handle when someone goes to registry but without the "/nifi-registry/" path Key: NIFIREG-82 URL: https://issues.apache.org/jira/browse/NIFIREG-82 Project: NiFi Registry Issue Type: Improvement Reporter: Joseph Percivall Priority: Trivial Currently, Registry goes to the Jetty 404 page. We should have something similar to NiFi with the page that says "Did you mean: /nifiYou may have mistyped" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (NIFIREG-77) Allow a user to see the changes created by the currently loaded version
[ https://issues.apache.org/jira/browse/NIFIREG-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Percivall updated NIFIREG-77: Priority: Critical (was: Major) > Allow a user to see the changes created by the currently loaded version > --- > > Key: NIFIREG-77 > URL: https://issues.apache.org/jira/browse/NIFIREG-77 > Project: NiFi Registry > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Critical > > As a user, I would like to see the changes that are included in a particular > version. More specifically, if I'm on an old version and I upgrade to a > version written by someone else, I have no way to know what changes occurred > during that version upgrade. > A simple solution would be to utilize the same logic which displays the > current differences between local and stored in the registry and use that to > show the differences between the current version N and version N-1. The user > could then change between versions to see the changes that happened as part > of that version. > An even better solution (from a DFM perspective) would be to be able to see > the changes within any version (not just the most recent). That way a DFM > wouldn't have to stop the flow for an extended period of time to view the > changes/differences in different versions but I think that'd be more work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFIREG-81) Variable Registry properties that are expected for a versioned Flow should be tracked
Joseph Percivall created NIFIREG-81: --- Summary: Variable Registry properties that are expected for a versioned Flow should be tracked Key: NIFIREG-81 URL: https://issues.apache.org/jira/browse/NIFIREG-81 Project: NiFi Registry Issue Type: Improvement Reporter: Joseph Percivall As a user, pulling down a new versioned flow, I'd like to know which properties are expected in my NiFi's Variable Registry to run the flow. By explicitly calling these out I can more easily configure and understand the flow on deployment. Also, would facilitate maintainability. An easy example of a typical flow that would utilize this is one that relies on an external source which is configured per environment (SSL file paths, Web service URL, etc.). This could also pave the way to set "default" variables within a Registry instance and elsewhere. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFIREG-80) Flows and their versions should be exportable & importable from the UI
Joseph Percivall created NIFIREG-80: --- Summary: Flows and their versions should be exportable & importable from the UI Key: NIFIREG-80 URL: https://issues.apache.org/jira/browse/NIFIREG-80 Project: NiFi Registry Issue Type: Improvement Reporter: Joseph Percivall As a user, I would like to be able to develop a flow in one place using one registry then deploy and continually update that versioned flow to another location (no network connectivity between the two). This could be done by exporting flow and the different versions and allowing them to be uploaded to the UI. This is similar to how you can create patches in git to be applied in other repos. If my understanding is correct, this should be able to be done by using the same REST endpoints that NiFi uses. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
