Re: NiFi 2.0 - QuestDB

2023-08-01 Thread Joe Witt
Hello Here are a couple examples where it seems like we definitely have some work to do for the QuestDB bits. Right now it is blocking full clean builds on Linux machines in my case though works on OSX. See: https://issues.apache.org/jira/browse/NIFI-11896 https://issues.apache.org/jira/browse/N

Re: NiFi 2.0 - QuestDB

2023-07-19 Thread Mark Payne
Sounds good. Thanks Bence. > On Jul 19, 2023, at 11:07 AM, Simon Bence wrote: > > Thanks for the feedback from everyone! > > As I understand the intention is supported and with some preparation > (covering the cases mentioned) it can be done. I will raise some PR in the > foreseeable future t

Re: NiFi 2.0 - QuestDB

2023-07-19 Thread Simon Bence
Thanks for the feedback from everyone! As I understand the intention is supported and with some preparation (covering the cases mentioned) it can be done. I will raise some PR in the foreseeable future to target these questions. Regards, Bence > On 2023. Jul 19., at 16:01, David Handermann >

Re: NiFi 2.0 - QuestDB

2023-07-19 Thread David Handermann
Thanks for the suggestion and additional background Bence, that is very helpful in evaluating the default inclusion approach. I agree with Joe's concern about handling potential corruption. We have recently reduced dependency on the H2 file-backed database driver so that it is now limited to Flow

Re: NiFi 2.0 - QuestDB

2023-07-19 Thread Joe Witt
Bence It no doubt is superior to the current default in terms feature/benefit. We need to just address any fault scenarios. I personally dont care how much space it uses. All good there. I am only focused on fault scenarios and recovery of those. This isnt data we need to protect like content/

Re: NiFi 2.0 - QuestDB

2023-07-19 Thread Simon Bence
Thanks for the quick feedback! Joe: your concerns are relevant, let me provide some details: The database uses some disk space, determined by the number of components and the number of covered days. During adding it I was checking for time usage and however I don’t have the numbers any more, th

Re: NiFi 2.0 - QuestDB

2023-07-19 Thread Joe Witt
Agree functionally How does this handle disk usage? Any manual intervention needed? What if the disk is full where it writes? What if the db somehow becomes corrupted? Id like to ensure this thing is zero ops as much as possible such that in error conditions it resets and gets going again. T

Re: NiFi 2.0 - QuestDB

2023-07-19 Thread Brandon DeVries
I also agree... I've been doing some testing recently, and not having to note relevant stats before a config change and restart is a huge improvement. From: Pierre Villard Sent: Wednesday, July 19, 2023 8:54:13 AM To: dev@nifi.apache.org Subject: Re: NiF

Re: NiFi 2.0 - QuestDB

2023-07-19 Thread Pierre Villard
I do think this provides great value. The possibility to get access to status history of the components and at system level across restart is a great improvement for NiFi troubleshooting. It also gives the ability to store this information for a longer period of time. I'm definitely in favor of mak

NiFi 2.0 - QuestDB

2023-07-19 Thread Simon Bence
Hi Community, I was thinking if it would make sense to set the QuestDB as default for status history backend in 2.0? It is there for a while and I would consider it as a step forward so the new major version might be a good time for the wider audience. It comes with less memory usage for bigger