[
https://issues.apache.org/jira/browse/NIFI-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199657#comment-15199657
]
Matthew Clarke commented on NIFI-1586:
--------------------------------------
I think that this ticket can be closed out. It does grow to a considerable size
(have seen ~6 GB before zookeeper finally fails).
https://issues.apache.org/jira/browse/NIFI-1626 addresses the large messages
that lead to this failure. The combination of snapCount,
autopurge.pugreInterval, and autopurge.snapRetainCount settings can put some
control on disk utilization. Since those setting do not establish absolutes on
size, we still can't control exactly how much space will be used for state, but
users should be able to play with these settings to achieve a relatively
consistent pattern of disk usage. The only suggestion here is that NiFi add the
snapCount= property to the zookeeper.properties file with a default value of
1000. When not set zookeeper defaults to 100,000 which will of course result
in more disk utilization. By setting it 1000 as a default, we keep the impact
to a minimum and of course users can push that value up to meet their specific
historic state retention needs.
> embedded zookeeper disk utilization grows unbounded
> ---------------------------------------------------
>
> Key: NIFI-1586
> URL: https://issues.apache.org/jira/browse/NIFI-1586
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 0.4.1
> Environment: latest 0.5.1 release
> Reporter: Matthew Clarke
>
> Observed that embedded NiFi zookeeper disk utilization will grow unbounded.
> Zookeeper will occasional create snapshots but at no time will it ever purge
> any of those snapshots it creates. This behavior is documented here:
> https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_administering
> It is the operators responsibility to purge old snapshot files. NiFi needs to
> provide a configuration that will automate this pruge.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
