Ryan Hendrickson created NIFI-10824:
---------------------------------------
Summary: Yellow indicator to say re-sorting is necessary & add a
warn log if swapping from the Active to Swap Queue takes too long
Key: NIFI-10824
URL: https://issues.apache.org/jira/browse/NIFI-10824
Project: Apache NiFi
Issue Type: Improvement
Reporter: Ryan Hendrickson
Version 1.1.0 introduced visual indication of queue growth relative to back
pressure settings and when back pressure is engaged. .. It goes from Grey to
Red, indicating Back-Pressure has kicked in... It'd be great to add a 3rd
color, like Yellow, indicating that re-sorting will be necessary due to the
back-pressure being larger than the queue size, then if it takes longer than
XXX millis to sort, log something to the bulletin board.
*Background..*
A lot of people don't realize they shouldn't have a queue with a million flow
files in it, especially with a Prioritizor set.
For any Relationship there are 2 underlying queues:
(1) An Active Queue (java.util.PriorityQueue) for the first 20,000
items, defined by the nifi.queue.swap.threshold in nifi.properties.
(2) A Swap Queue (java.util.ArrayList) for the rest of the queue's items.
If the Active Queue is full, every new item, regardless of priority, is placed
on the Swap Queue. No item on the Swap Queue will be re-prioritized until the
entire Active Queue is empty ([Line
284|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java#L284]).
Thus, the PriorityAttributePrioritize only "actively" sorts the first 20,000
items on the relationship. Once the queue is empty, it'll re-sort the rest and
move the top 20,000 over. Then the problem repeats. On our team, no amount of
training can get mitigate NiFi Dataflow users from setting the backpressure to
unlimited or sky-high.
If backpressure is properly set, the time it takes to re-sort the the Swap
Queue should be minimal. In our case, we set the system-wide back-pressure
size default to the same as the swap queue size to prevent any re-sorting.
However, if the backpressure is unset and the queue begins to grow in size, the
re-sorting will take longer and longer. At some threshold, I think it would be
advantageous for NiFi to emit a warn log to the user saying it took over XXX
millis to re-sort the queue and also to show a visual indictor that swapping &
re-sorting will be necessary because the backpressure size is larger than the
swap size.
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java#L271
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
