Joseph Witt created NIFI-2418:
---------------------------------
Summary: Provenance should optionally provide back pressure OR
provide best-effort indexing
Key: NIFI-2418
URL: https://issues.apache.org/jira/browse/NIFI-2418
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: Joseph Witt
Today provenance will apply back pressure if the rate of indexing as configured
cannot keep up with the live data flow as configured.
This model makes sense in some cases but is overly prohibitive in many others.
In either case the raw provenance events are available so the tradeoff is the
ability to query them conveniently or not vs ability to keep the live feed
running.
I propose we add a mode, and make it the default, that provenance indexing if
falling behind will effectively be a sampling indexer which will keep skipping
to the most recent items to index. The details of how this might work aren't
clear but the point is that we would intentionally let events fall behind
unindexed. Further a pretty naive approach could be sufficient. We should
flag that this is happening for the user/administrator but otherwise keep
moving.
This means one can still query for events but will only find those that were
indexed (though they'll know it is an incomplete search) and the full rate of
the flow needed will carry on unimpeded by indexing rate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
