[
https://issues.apache.org/jira/browse/NIFI-8340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Turcsanyi updated NIFI-8340:
----------------------------------
Description:
When the content repository reaches its configured threshold (by default, 50%),
it will exert backpressure that prevents any component from writing to it until
archive has occurred. This is designed to prevent a case where the content
repository is written to so fast that cleanup is unable to keep up. This can be
important for use cases where there is extremely high throughput and the nodes
are pretty well tapped out resource-wise.
However, when this happens, it's not at all obvious to the user, as it only
shows an INFO level message in the logs. As a result, users who encounter this
typically think NiFi just suddenly "froze."
To improve the experience here, we should do the following:
* Change default to something less restrictive, such as 75 or 80%. For dev
environments, especially, the content repository is not by itself and it's
common for the drive to be more than 50% full before NiFi is even installed.
* Change the log message from INFO to WARN
* Create a Bulletin at the Controller Level that warns that backpressure is
being applied. Ideally, this could even be created at the component level, but
it's not clear that this is possible without significant refactoring, as the
Content Repository does not know which component is trying to write to it.
The {{nifi.content.repository.archive.backpressure.percentage}} property is not
part of the default nifi.properties currently. Please add it to the default
nifi.properties template and also document the property in the Admin Guide.
was:
When the content repository reaches its configured threshold (by default, 50%),
it will exert backpressure that prevents any component from writing to it until
archive has occurred. This is designed to prevent a case where the content
repository is written to so fast that cleanup is unable to keep up. This can be
important for use cases where there is extremely high throughput and the nodes
are pretty well tapped out resource-wise.
However, when this happens, it's not at all obvious to the user, as it only
shows an INFO level message in the logs. As a result, users who encounter this
typically think NiFi just suddenly "froze."
To improve the experience here, we should do the following:
* Change default to something less restrictive, such as 75 or 80%. For dev
environments, especially, the content repository is not by itself and it's
common for the drive to be more than 50% full before NiFi is even installed.
* Change the log message from INFO to WARN
* Create a Bulletin at the Controller Level that warns that backpressure is
being applied. Ideally, this could even be created at the component level, but
it's not clear that this is possible without significant refactoring, as the
Content Repository does not know which component is trying to write to it.
> If content repository is filling up, create bulletin when exerting
> backpressure
> -------------------------------------------------------------------------------
>
> Key: NIFI-8340
> URL: https://issues.apache.org/jira/browse/NIFI-8340
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Priority: Major
>
> When the content repository reaches its configured threshold (by default,
> 50%), it will exert backpressure that prevents any component from writing to
> it until archive has occurred. This is designed to prevent a case where the
> content repository is written to so fast that cleanup is unable to keep up.
> This can be important for use cases where there is extremely high throughput
> and the nodes are pretty well tapped out resource-wise.
> However, when this happens, it's not at all obvious to the user, as it only
> shows an INFO level message in the logs. As a result, users who encounter
> this typically think NiFi just suddenly "froze."
> To improve the experience here, we should do the following:
> * Change default to something less restrictive, such as 75 or 80%. For dev
> environments, especially, the content repository is not by itself and it's
> common for the drive to be more than 50% full before NiFi is even installed.
> * Change the log message from INFO to WARN
> * Create a Bulletin at the Controller Level that warns that backpressure is
> being applied. Ideally, this could even be created at the component level,
> but it's not clear that this is possible without significant refactoring, as
> the Content Repository does not know which component is trying to write to it.
> The {{nifi.content.repository.archive.backpressure.percentage}} property is
> not part of the default nifi.properties currently. Please add it to the
> default nifi.properties template and also document the property in the Admin
> Guide.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
