[
https://issues.apache.org/jira/browse/NIFI-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793409#comment-17793409
]
Joe Witt commented on NIFI-12236:
---------------------------------
I went back to quickly review the presumably referenced mailing list thread
titled 'NiFi 2.0 - QuestDB'.
The concerns I noted then remain for me. It would certainly be nice for the
users to have access to persistent and longer duration metric data. That said
I'm not sure our user interface for this is very good when it comes to holding
this information and making interesting visualizations of it for longer
durations. For the type of usage the in memory durations we can reasonably
sustain today seem about right.
What I am seeing more and more is that users of NiFi want this type of
information available in their favorite monitoring or observability tool
whether that is Datadog or Prometheus or something else.
If our goal is a short term but durable store then perhaps we ought to give
them a simple QuestDB based service/process they can run on some node. Then
their NiFi nodes are configured to send/query metrics from that service rather
than it having to live on every node. This also means it would be better
externalized such that maybe they dont even use Quest but instead MySQL or
Postgre or whatever they prefer. In a k8s based deployment I can certainly see
such a model working well.
For users that are looking for more robust data retention and query and
analysis we're better off focusing on getting the data to their preferred tools.
> Improving fault tolerancy of the QuestDB backed metrics repository
> ------------------------------------------------------------------
>
> Key: NIFI-12236
> URL: https://issues.apache.org/jira/browse/NIFI-12236
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Simon Bence
> Assignee: Simon Bence
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Based on the related discussion on the dev email list, the QuestDB handling
> of the metrics repository needs to be improved to have better fault tolerance
> in order to be possible to use as a viable option for default metrics data
> store. This should primarily focus on handling unexpeted database events like
> corrupted database or loss of space on the disk. Any issues should be handled
> with an attempt to keep the database service healthy but in case of that is
> impossible, the priority is to keep NiFi and the core services running, even
> with the price of metrics collection / presentation outage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
