+1 for the validation change. +1 for not calling into user code for GUI refresh.
I understand the logic behind validating whenever we return current state, but that can put a great deal of load on a system unrelated to the actual data flow. For the most part, state changes at discrete points such as configuration, start, onTrigger, etc. When loading the GUI it seems like we should return the last known state, possibly with a GUI option to re-validate the components, to minimize the impact of the user interface side of the sytems on the actual dataflow components. As much as duplicate validation can be eliminated that would help as well. Currently I believe that if Processors A and B validate Service X, the Service X validation will occur twice, contributing to the "exponential" growth Mike mentioned in the ticket. On Tue, Nov 8, 2016 at 12:43 PM, Matt Gilman <[email protected]> wrote: > I also agreed these changes make sense. In addition, another approach we > could consider that has been discussed in the past [1] is to perform > component validation asynchronously. This presents its own challenges but > would also be helpful. We should try to avoid calling into user code in any > web thread. > > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-950 > > On Mon, Nov 7, 2016 at 6:15 PM, Matt Burgess <[email protected]> wrote: > > > Agreed. Also we validate processors on a timer-based strategy in > > FlowController (looks like for snapshotting) and in the web server > > (via ControllerFacade), those seem to happen 6-7 times on that > > interval (which is like 15-20 seconds). Also we validate all > > processors on any change to the canvas (such as moving a processor). > > Besides Mike's suggestion, perhaps we should look at a purely > > event-driven strategy for validating processors if possible? > > > > Regards, > > Matt > > > > On Mon, Nov 7, 2016 at 6:06 PM, Joe Witt <[email protected]> wrote: > > > Makes good sense to me. > > > > > > On Nov 7, 2016 5:39 PM, "Michael Moser" <[email protected]> wrote: > > > > > >> All, > > >> > > >> I would like to propose a fundamental change to processor validation > > based > > >> on observations in https://issues.apache.org/jira/browse/NIFI-2996. I > > >> would > > >> like to validate processors only when they are in the STOPPED state. > > >> > > >> The properties on a processor in the RUNNING state should always be > > valid, > > >> else you should not have been able to start the processor. A processor > > in > > >> the DISABLED statue doesn't show validation results, so it seems a > > waste to > > >> validate its properties. > > >> > > >> The reason I'm proposing this change is because the NiFi UI slows down > > as > > >> you add more processors and controller services to the graph. Beyond > > common > > >> sense expectations that this would be true, it appears that processor > > >> validation is a significant part of the 'cost' on the server when > > >> responding to REST API requests. Some details from my testing are in > > the > > >> JIRA ticket. > > >> > > >> Thoughts? > > >> > > >> Thanks, > > >> -- Mike > > >> > > >
