Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/3131
@patricker thanks for the PR! I had gone down this path before but I backed
out the changes. The changes in this PR will result in obtaining a Write Lock
on the queue of every incoming connection for every running processor in the
graph. This can become quite expensive for a complex flow that is made up of
thousands (or even 10's of thousands) of processors) and result in overall
system performance suffering. This is why we are so care in the FlowFile
Queue's implementation to ensure that isActiveQueueEmpty() never obtains a lock
but instead only references AtomicReference variables.
We should be able to do better, though. For example, when we pull a
FlowFile from the queue, we check if it's penalized. If so, we throw it back
on. Since the queue is ordered, we could do some smart things like looking at
the FlowFile expiration date, then keeping track of the fact that we know all
FlowFiles are penalized until that time is reached - or until a non-penalized
FlowFile is added to the queue.
---
