Arvind Kandpal created NIFI-16540:
-------------------------------------
Summary: ControlRate does not remove expired throttles when using
a Grouping Attribute
Key: NIFI-16540
URL: https://issues.apache.org/jira/browse/NIFI-16540
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Arvind Kandpal
ControlRate keeps one Throttle per value of the Grouping Attribute. The method
clearExpiredThrottles() is supposed to remove throttles that were not used in
the last two time periods, but nothing is ever removed.
The method copies the entries of dataThrottleMap and countThrottleMap into a
new HashSet and then calls iterator.remove() on that copy. Only the temporary
set changes.
The two throttle maps are never updated.
This was introduced in NIFI-10243. Before that change, the iterator removed
entries from the throttle map itself.
Steps to reproduce:
1. Configure ControlRate:
- Rate Control Criteria: flowfile count
- Maximum Rate: 10
- Time Duration: 1 sec
- Grouping Attribute: g
2. Send 1000 FlowFiles, each with a different value of g.
3. Keep sending FlowFiles for more than 2 time periods (for example 50
seconds).
Expected: throttles that are not used any more are removed, and the number of
throttles goes back down.
Actual: the throttle count stays at 1000 and keeps growing (1000 -> 1005 after
50 seconds in the test). Nothing is removed.
Impact: with a high-cardinality Grouping Attribute (for example a user id or
file name), the throttle maps grow without limit until the processor properties
are changed. This leads to growing heap usage.
Code: nifi-standard-processors, ControlRate.java, clearExpiredThrottles()
Existing tests do not cover throttle removal.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
