Hi Michael, I did not have time to take a deeper look on this, but I had a fix for cleanup related issues lately: https://github.com/apache/nifi/pull/6374 <https://github.com/apache/nifi/pull/6374>
As for the size of the stored data, it might be perfectly normal. The amount of data primarily depends on the time window you want to monitor* and the amount of components you have in the flow. One of the reasons to add this implementations was actually to do not take up a lot of heap space in case of bigger flows. As for clusters, as far as I remember all the QuestDB instances on all the nodes are working separately, thus you should have the same footprint for a single node setup with the same flow. If you experience a different behaviour, I would happily take a look. As for the 24hours, I will take a look. * Please note that even if you set the time window for 1 day, you might have up to two days of data in the QuestDb as the partitioning is day based and it can drop only full days, after they run out of the monitored time window. Regards, Bence > On 2022. Nov 16., at 9:20, Garland, Michael R <[email protected]> > wrote: > > Further investigation has shown that for the Node Status History, in order to > get it working, I have had to disconnect 4 out of 6 nodes from the cluster in > order to get any node level status history to be displayed without occurring > a parse error. This seems to suggest the QuestDB implementation in NiFi > cannot scale well for multi-clustered deployments with many processors. The > total size of the QuestDB files on each node is 377MB, which is after approx. > 2 days of NiFi running. > > The issue around component level status history only showing a maximum of > 24hrs worth of history still remains. > > Can a bug be raised on this? > > On 2022/11/14 16:24:04 "Garland, Michael R" wrote: > > Hi, > > > > We're trying to make use of the 'EmbeddedQuestDbStatusHistoryRepository' > > implementation of the Status History Repository within NiFi 1.17.0, but > > cannot seem to get this feature working as what is described within the > > NiFi admin guide. > > > > > What we are seeing is that the component Status History viewer within NiFi > > only ever displays 1 days' worth of data, regardless of the configuration > > of the 'nifi.status.repository.questdb.persist.component.days' value > > (leaving the default value of 3) > > > > > A second point, clicking on the 'Node Status History' button from the > > hamburger symbol doesn't load any information on a 6 node cluster that has > > been running for a few days and has ~600MB worth of QuestDB data on each > > node (with the 'nifi.status.repository.questdb.persist.node.days' value > > left at its default of 14 days). After approx. 75 seconds, NiFi displays a > > parse error, with no information in the logs that would explain the cause. > > Has anyone seen this behaviour? > > > > > Many thanks, > > > > Michael > > >
