Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2416
It definitely requires considerable testing. Burgess clearly did testing
during his code review and he had favorable findings. Mark obviously did. I
personally spent multiple weeks in long running protracted tests on a couple of
systems one of which ran uninterrupted at a rate of 120,000 events per second
(so WALI update rate was far higher) for 11 days with zero full gcs, perfectly
stable performance, and in testing before after with a series of hard
stops/restarts all behaviors were favorable. Do you have tests underway with
problematic findings? I'll state that based on my experience with this project
I am very comfortable with this change and my own personally verified
observations and evaluation.
It is critical code so it does help that the author that wrote the previous
implementation also authored this one and that the problems of the previous one
were well stated and the path to write it was clear. That the changes were
significant in nature in terms of total lines/files impacted I dont think
reflects the improved simplicity this brings and as noted it was reusing much
of the core logic of the previous more complicated approach.
The claim of a simple fix being available to close the previous gaps
doesn't appear to be backed with a suggested implementation though it does look
like you received a good response to why that wasn't feasible.
You raised some good points and got a really detailed reply from mark a
month ago after which others such as myself and Burgess stayed active on this
important item. So, in that sense this all seems fine.
Now, having said all of this I do share your view that I would have
preferred to see this be something users could opt-out since the old
implementation even with the known potential issue has been extremely stable
for a very long time and is relied upon heavily. I think using this new
implementation as the default which closes the gap and which handles
automatically reading old repository formatted partitions is correct and
letting users that are on the old one stay on it as they wish is perfectly
valid.
I dont think it is right to revert this change. But I do think it is fair
to reopen the JIRA and move toward a model allowing opt-out and leveraging this
as a default. @markap14 do you agree? @mattyb149 do you?
Thanks
---
