[jira] [Commented] (NIFI-7505) Add InvokeHTTPRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793510#comment-17793510
]
Ryan Miller commented on NIFI-7505:
---
Continuing from the previous PR [1], I've had a go at implementing the
following changes to the InvokeHTTPRecord Processor:
* Use {{WebClientServiceProvider}} instead of manually instantiating an HTTP
client inside {{{}InvokeHTTPRecord{}}}. This will make the code simpler, reduce
the number of properties and make it easier to write tests.
* Remove the functionality for splitting records by success/failure, and just
have the ability to store the success/failure flag on each record processed. If
we want to split records, we can do that with a separate {{SplitRecord}}
processor.
I've created a new PR [2] with these changes for the purposes of feedback and
additional discussion.
[1] [https://github.com/apache/nifi/pull/7532]
[2]
[https://github.com/apache/nifi/pull/8127|https://github.com/apache/nifi/pull/7532]
> Add InvokeHTTPRecord processor
> --
>
> Key: NIFI-7505
> URL: https://issues.apache.org/jira/browse/NIFI-7505
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Priority: Major
> Labels: Record, http, processor, rest
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Some users have recently requested being able to invoke a specific URL via
> {{InvokeHTTP}} on every record in a flowfile. Currently, the {{InvokeHTTP}}
> processor can only handle one piece of data per flowfile. There are some
> workarounds available for specific patterns with {{LookupRecord}} +
> {{RestLookupService}}, but this is not a complete solution. I propose
> introducing an {{InvokeHTTPRecord}} processor, providing the {{InvokeHTTP}}
> functionality in conjunction with the record processing behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-7505) Add InvokeHTTPRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748486#comment-17748486
]
Peter Spiller commented on NIFI-7505:
-
I've had a go at creating an InvokeHTTPRecord processor, as record-based HTTP
requests are something I've been having issues with for a while. It's based on
the InvokeHTTP processor, but with modifications for record-based operation
(e.g. varying request parameters come from individual records instead of
flowfile attributes):
https://github.com/apache/nifi/pull/7532
It's possible to achieve something like this by using a LookupRecord processor
combined with a RestLookupService, but I've had problems with this approach.
RestLookupService is designed around looking up data, not making requests that
potentially have side-effects. If HTTP requests encounter transient errors,
retrying them is difficult and requires splitting and merging record-based
flowfiles. Semantically, using a LookupService to make requests with
side-effects doesn't feel great.
The PR isn't complete yet (it still needs tests), but I've created it in case
anybody has any feedback on the approach I'm taking.
> Add InvokeHTTPRecord processor
> --
>
> Key: NIFI-7505
> URL: https://issues.apache.org/jira/browse/NIFI-7505
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Priority: Major
> Labels: Record, http, processor, rest
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some users have recently requested being able to invoke a specific URL via
> {{InvokeHTTP}} on every record in a flowfile. Currently, the {{InvokeHTTP}}
> processor can only handle one piece of data per flowfile. There are some
> workarounds available for specific patterns with {{LookupRecord}} +
> {{RestLookupService}}, but this is not a complete solution. I propose
> introducing an {{InvokeHTTPRecord}} processor, providing the {{InvokeHTTP}}
> functionality in conjunction with the record processing behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-7505) Add InvokeHTTPRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128519#comment-17128519
]
Andy LoPresto commented on NIFI-7505:
-
My initial thought was that the request type (HTTP method, etc.) is going to be
the same for each record, so the common use cases are:
# Data enrichment to consistent URL endpoint via {{GET}} per record (i.e. a
_lookup_)
# Batch {{POST}} submission (i.e. each record has some data which needs to be
submitted to a remote endpoint)
# Multiple URL operation (i.e. each record _is_ or _has_ a URL in it, and the
outgoing request URLs will be unique to each record)
I think there should be a configurable property to determine the destination of
the responses. The options would be {{Record - inline}}, {{Record - replace}},
and {{New flowfile}}.
* {{Record - inline}} - puts the HTTP response code & response body in the
record itself. An additional property called {{Record response identifier}}
would accept an optional value to use as the column/field name
* {{Record - replace}} - puts the HTTP response code & response body as the new
record value
* {{New flowfile}} - puts the HTTP response code & response body as the
flowfile content of a new flowfile for each record. This would function similar
to the {{SplitRecord}} or {{PartitionRecord}} processors
Yes, the processor must have a way to reference the record values in order to
craft the URL and request body. I would suggest using the
[RecordPath|https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html]
syntax, as it already exists to extract values from the record structure and is
format agnostic.
> Add InvokeHTTPRecord processor
> --
>
> Key: NIFI-7505
> URL: https://issues.apache.org/jira/browse/NIFI-7505
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Priority: Major
> Labels: Record, http, processor, rest
>
> Some users have recently requested being able to invoke a specific URL via
> {{InvokeHTTP}} on every record in a flowfile. Currently, the {{InvokeHTTP}}
> processor can only handle one piece of data per flowfile. There are some
> workarounds available for specific patterns with {{LookupRecord}} +
> {{RestLookupService}}, but this is not a complete solution. I propose
> introducing an {{InvokeHTTPRecord}} processor, providing the {{InvokeHTTP}}
> functionality in conjunction with the record processing behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7505) Add InvokeHTTPRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128261#comment-17128261
]
Alessandro D'Armiento commented on NIFI-7505:
-
Do you think such a Processor should also emit a different FlowFile for each
record (as each different invocation could succeed or fail independently)
acting in a similar way than SplitRecord?
As per the main behaviour, I guess this processor should be able to add the
record content to the request body (or parameters in case of GET)
What do you think?
> Add InvokeHTTPRecord processor
> --
>
> Key: NIFI-7505
> URL: https://issues.apache.org/jira/browse/NIFI-7505
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Andy LoPresto
>Priority: Major
> Labels: Record, http, processor, rest
>
> Some users have recently requested being able to invoke a specific URL via
> {{InvokeHTTP}} on every record in a flowfile. Currently, the {{InvokeHTTP}}
> processor can only handle one piece of data per flowfile. There are some
> workarounds available for specific patterns with {{LookupRecord}} +
> {{RestLookupService}}, but this is not a complete solution. I propose
> introducing an {{InvokeHTTPRecord}} processor, providing the {{InvokeHTTP}}
> functionality in conjunction with the record processing behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
