[
https://issues.apache.org/jira/browse/NIFI-12544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804950#comment-17804950
]
David Handermann commented on NIFI-12544:
-----------------------------------------
Thanks for the reply, the additional background is helpful. Having some
concrete examples of publicly documented sources is useful for testing any kind
of reference implementation Processor.
Any kind of event handling Processor needs to consider several things, such as
internal event buffering, batching, and record handling.
The Jakarta WS specification includes a [Server-Sent
Events|https://jakartaee.github.io/rest/apidocs/3.0.0/jakarta/ws/rs/sse/package-summary.html]
API for receiving inbound events, so that looks like one possibility for
building a new Processor, using the
[Jersey|https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest3x/sse.html]
implementation. Integrating event handling using the standard NiFi Record
Reader and Writer abstractions should be analogous to other event consuming
Processors.
> Support receiving HTTP SSE (Server Sent Events)
> -----------------------------------------------
>
> Key: NIFI-12544
> URL: https://issues.apache.org/jira/browse/NIFI-12544
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Justin
> Priority: Major
>
> Currently NiFi has no processor which supports long poll type HTTP
> connections, InvokeHTTP expects that every http request will terminate
> shortly and does not forward intermediate results for long-lived connections
> instead hanging until the read timeout is reached and then erroring the
> request. This makes it difficult to connect to any system which utilizes the
> SSE standard to stream events to clients as they happen from NiFi, requiring
> a script processor or external tool to convert the SSE stream into something
> NiFi can handle. Since streaming data processing (streaming events) is
> NiFi's MO, it seems appropriate that there would be a native processor which
> can connect and stream these types of events.
> I also have need of such a processor and would like to upstream the changes,
> hence the ticket before I get too far into an implementation.
> A couple possibilities for implementing these changes:
> * Extend InvokeHTTP to support streaming content when enabled in processor
> options, thus it would forward content as it arrives, possibly broken apart
> via a delimiter/parse config. This would only timing out connections that
> don't make progress in so long (read timeout limits).
> * Add a new processor which is SSE specific which does all of the Proxy,
> Auth, ... work of InvokeHTTP, but outputs SSE events roughly as they arrived
> instead of a single HTTP response per input request. Should this accept
> configuration via FlowFile or be an INPUT_FORBIDDEN processor where
> everything is through Configuration? How do you close a stream
> programmatically if configured via FlowFile?
> * Create a new HTTP processor which has some level of support/awareness of
> content types (application/stream+json, application/x-ngjson,
> text/event-stream, ...) and doesn't time out long connections unless they
> aren't making progress, would output blocks in some predefined size or by a
> delimiter and rely on downstream to decide what to do with the content.
> Ref: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
