[jira] [Commented] (NIFI-949) Allow configuration of cleanup resources
[ https://issues.apache.org/jira/browse/NIFI-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748701#comment-15748701 ] Brandon DeVries commented on NIFI-949: -- That's seeming at least possible. We have an 0.7.1 running right now that had ~40 GB in the flow across the cluster, but had 100% consumed the 2 TB of content repo. Restarting one of the boxes and tailing the logs shows all of the expected content repo cleanup, and (although it is still running) the content repo usage is dropping significantly. NIFI-2920 is marked as affecting 0.7.0, 0.6.1, and 0.7.1. Do you know if this issue was really limited to those versions? REgardless, it sounds like we're going to need an 0.8.0 release to pick up that fix... > Allow configuration of cleanup resources > > > Key: NIFI-949 > URL: https://issues.apache.org/jira/browse/NIFI-949 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Brandon DeVries >Priority: Minor > > Allow the allocation of resource to the cleanup of the content and provenance > repositories to be configurable. There are times at which the resources > spent performing cleanup activities can degrade the operation of the flow > sufficiently to cause problems. Sometimes this requires removing the > offending node from the cluster to allow it to clean up without taking on new > data (and thus backing up). Making the cleanup resources configurable could > prevent such a situation by allowing them to be reduced under times of heavy > load to keep the node useable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-949) Allow configuration of cleanup resources
[ https://issues.apache.org/jira/browse/NIFI-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684606#comment-15684606 ] Mark Payne commented on NIFI-949: - [~bicore] thanks for the feedback. I'm actually curious if what you're seeing is NIFI-2925. Whenever FlowFiles are swapped out, they can fail to ever get removed from the content repo. On restart, it will go through and prune them, but this can take quite a long time. > Allow configuration of cleanup resources > > > Key: NIFI-949 > URL: https://issues.apache.org/jira/browse/NIFI-949 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Brandon DeVries >Priority: Minor > > Allow the allocation of resource to the cleanup of the content and provenance > repositories to be configurable. There are times at which the resources > spent performing cleanup activities can degrade the operation of the flow > sufficiently to cause problems. Sometimes this requires removing the > offending node from the cluster to allow it to clean up without taking on new > data (and thus backing up). Making the cleanup resources configurable could > prevent such a situation by allowing them to be reduced under times of heavy > load to keep the node useable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-949) Allow configuration of cleanup resources
[ https://issues.apache.org/jira/browse/NIFI-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678495#comment-15678495 ] Ben Icore commented on NIFI-949: This is still very much an issue even with 0.7.1 if the system has been running under any kind of load for even a few hours, there is a significant amount of repo cleanup that happens when the system is restarted. > Allow configuration of cleanup resources > > > Key: NIFI-949 > URL: https://issues.apache.org/jira/browse/NIFI-949 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Brandon DeVries >Priority: Minor > > Allow the allocation of resource to the cleanup of the content and provenance > repositories to be configurable. There are times at which the resources > spent performing cleanup activities can degrade the operation of the flow > sufficiently to cause problems. Sometimes this requires removing the > offending node from the cluster to allow it to clean up without taking on new > data (and thus backing up). Making the cleanup resources configurable could > prevent such a situation by allowing them to be reduced under times of heavy > load to keep the node useable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-949) Allow configuration of cleanup resources
[ https://issues.apache.org/jira/browse/NIFI-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670879#comment-15670879 ] Brandon DeVries commented on NIFI-949: -- Let me ask around... I'll get back to you. > Allow configuration of cleanup resources > > > Key: NIFI-949 > URL: https://issues.apache.org/jira/browse/NIFI-949 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Brandon DeVries >Priority: Minor > > Allow the allocation of resource to the cleanup of the content and provenance > repositories to be configurable. There are times at which the resources > spent performing cleanup activities can degrade the operation of the flow > sufficiently to cause problems. Sometimes this requires removing the > offending node from the cluster to allow it to clean up without taking on new > data (and thus backing up). Making the cleanup resources configurable could > prevent such a situation by allowing them to be reduced under times of heavy > load to keep the node useable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-949) Allow configuration of cleanup resources
[ https://issues.apache.org/jira/browse/NIFI-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670815#comment-15670815 ] Mark Payne commented on NIFI-949: - [~devriesb] do you still see this as being an issue? Or is it safe to close the ticket now? > Allow configuration of cleanup resources > > > Key: NIFI-949 > URL: https://issues.apache.org/jira/browse/NIFI-949 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Brandon DeVries >Priority: Minor > > Allow the allocation of resource to the cleanup of the content and provenance > repositories to be configurable. There are times at which the resources > spent performing cleanup activities can degrade the operation of the flow > sufficiently to cause problems. Sometimes this requires removing the > offending node from the cluster to allow it to clean up without taking on new > data (and thus backing up). Making the cleanup resources configurable could > prevent such a situation by allowing them to be reduced under times of heavy > load to keep the node useable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
