[
https://issues.apache.org/jira/browse/MINIFI-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107227#comment-16107227
]
marco polo commented on MINIFI-356:
-----------------------------------
Those are good points. In general the memory usage is defined, albeit
non-intuitively, from the configuration. I agree there is room to change this
behavior, but I anticipate that even with the most intelligent backpressure
mechanism we likely can't stop write/read failures.
1) I have been thinking hard about this and the best way to tackle this. My
only comment is that I want to see how people may use the device. Currently you
are are one of those people, so your experiences dictate that perhaps an
alternative may be in order, but I want to balance these ideas with a healthy
dose of skepticism and C2 mindedness.
2) That's a good question. I envision failure meaning the file API returns an
actual error. Excessive I/O wait is a construct that seems far beyond what I
intend here. Hashing output seems reasonable. This is in part why I agree we
may wish to send this thread to the mailing list. Perhaps the larger group as a
whole will have ideas.
3) Interesting point. To your earlier point, we don't want to trigger the OOM
killer, in which case we need to be cognizant of our load. Some of the stuff
I've been doing for MINIFI-339 help to pull in the current RSS and report it
for metrics. That information can be used to watch ourselves. Perhaps we should
have a configured memory limit for the process?
> Create repository failure policy
> --------------------------------
>
> Key: MINIFI-356
> URL: https://issues.apache.org/jira/browse/MINIFI-356
> Project: Apache NiFi MiNiFi
> Issue Type: Improvement
> Components: C++
> Reporter: marco polo
> Assignee: marco polo
>
> Create a failure policy for continuing operations if a repo failure occurs.
> I.e. If writing to disk fails above a threshold ( 100 % for example ), we can
> move to a volatile repo where we can continue operations and report that we
> have a failure.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
