[
https://issues.apache.org/jira/browse/NIFI-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137079#comment-18137079
]
Anshu Singh commented on NIFI-16540:
------------------------------------
Hi [~arvindk12], Will you be working on this ticket, or should I take it up?
> ControlRate does not remove expired throttles when using a Grouping Attribute
> -----------------------------------------------------------------------------
>
> Key: NIFI-16540
> URL: https://issues.apache.org/jira/browse/NIFI-16540
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Arvind Kandpal
> Priority: Major
>
> ControlRate keeps one Throttle per value of the Grouping Attribute. The
> method clearExpiredThrottles() is supposed to remove throttles that were not
> used in the last two time periods, but nothing is ever removed.
> The method copies the entries of dataThrottleMap and countThrottleMap into a
> new HashSet and then calls iterator.remove() on that copy. Only the temporary
> set changes.
> The two throttle maps are never updated.
> This was introduced in NIFI-10243. Before that change, the iterator removed
> entries from the throttle map itself.
> Steps to reproduce:
> 1. Configure ControlRate:
> - Rate Control Criteria: flowfile count
> - Maximum Rate: 10
> - Time Duration: 1 sec
> - Grouping Attribute: g
> 2. Send 1000 FlowFiles, each with a different value of g.
> 3. Keep sending FlowFiles for more than 2 time periods (for example 50
> seconds).
> Expected: throttles that are not used any more are removed, and the number of
> throttles goes back down.
> Actual: the throttle count stays at 1000 and keeps growing (1000 -> 1005
> after 50 seconds in the test). Nothing is removed.
> Impact: with a high-cardinality Grouping Attribute (for example a user id or
> file name), the throttle maps grow without limit until the processor
> properties are changed. This leads to growing heap usage.
> Code: nifi-standard-processors, ControlRate.java, clearExpiredThrottles()
> Existing tests do not cover throttle removal.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
