[
https://issues.apache.org/jira/browse/NIFI-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953901#comment-14953901
]
Oleg Zhurakousky commented on NIFI-1021:
----------------------------------------
*High level considerations:*
The goal of the event streaming feature is not to represent a reliable source
of events but rather a fast
event distribution mechanism which has little to no impact on the overall
health and/or performance of the NiFi.
It is accomplished via a specialized subscriber with a non-blocking weak
reference bounded event buffer
and a single (from-the-buffer) consuming thread. The buffer is designed in such
way that it will never
block on add(..) call overflowing on itself by overriding events which may not
have been retrieved yet.
For more details please see javadoc for
[WeakReferenceRingBuffer|https://github.com/olegz/nifi/blob/2d3810025f111aa9ccdc9e957e861c12f4727d2a/nifi-api/src/main/java/org/apache/nifi/provenance/WeakReferenceRingBuffer.java]
While the design of the buffer implies that events may be lost in cases of slow
consumer
NiFi still guarantees that those events will end up in the event repository,
and could be
retrieved later. This means that event streaming feature should be used for
specialized use cases only
where event loss is either acceptable or consumer implementation ensures that
events are consumed fast
enough so buffer doesn't overflow.
While the intention of this feature is to provide a real time event streaming
and any type of buffering could be
interpreted as near-real-time, the main purpose of the buffer is to limit event
loss in cases of throughput spikes
which could actually occur in a relatively modest flow, since NiFi
ProcessSession groups events into a collection
before they are sent to the repository implying that 2+ events may arrive
almost instantaneously while significant
amount of time may pass before the next set of events. So, to eliminate the
effects of near-real-time the default buffer size is set to be relatively small
(1024).
> Expose strategy to subscribe to provenance events
> -------------------------------------------------
>
> Key: NIFI-1021
> URL: https://issues.apache.org/jira/browse/NIFI-1021
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
> Affects Versions: 0.3.0
> Reporter: Oleg Zhurakousky
> Assignee: Oleg Zhurakousky
>
> Currently access to the provenance events is exposed thru ReportingTask which
> is scheduled and therefore provides a "near-real-time" access to provenance
> events.
> There are several valid use cases where such events may need to be streamed
> in "real-time" for variety of real time analytics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
