Hello This is a really important thread I think and thanks for bringing it up.
The reality is that in some cases such as blocked threads there is really very little that I am aware can be done as far as actively killing them. However, we can at least do more than we are and we can further do things like 'sideline' these stuck threads because ultimately we can automatically rollback the affected session, kill off any input/output streams they have access to as far as repository data, etc.. This way even if those threads are sitting there we can move on and do productive things. We could perhaps even have a maximum pool size for orphaned threads and if the limit is reached then notify the user that a restart is required to resolve them. Thanks Joe On Sat, Dec 5, 2015 at 9:35 AM, Aldrin Piri <[email protected]> wrote: > Toivo, Joe, > > Started a draft of this yesterday and glad to see Joe chime in with some > additional input. > > I think now that 0.4.0 is on the cusp of getting wrapped up and put through > its paces, now is a good time to consider something like this that is a big > win for our users and helps iron out some of the rough edges. > > As highlighted in the listed issues and your message, these are certainly > nontrivial. I do think that the mechanism by which stopping and timeouts > are performed should be similar. To this end, the framework could let the > developers expose such properties that help drive the underlying API and > additionally, the UI could provide a kill switch for a task that has a > similar function. This likely introduces some other needs like being able > to differentiate from tasks which could be driven through a viewer that is > along the same lines of the proposed queue FlowFile viewer. > > Given how most things work and likely the general need to cleanup with the > associated things, guaranteeing an immediate stop is something I don't > think we have a good way to get a handle on uniformly, especially on these > processors that are kind of leaving the confines of the framework. > > Short term, I think we could look at developing this functionality into > Execute Process/StreamCommand and see what works to give consistent > timeouts on a per flowfile basis. Long term, we could migrate our lessons > learned from those items to the core. This likely needs to be framework > level as Joe alluded to in avoidance of inconsistency among how each > processor handles similar scenarios. Totally working on a whim, but > something like an @OnTimeout/@OnInterrupt could perhaps fall into the > processor API and potentially in the processor lifecycle depending on how > we are viewing such events. > > Good discussion to get things progressing! > > On Sat, Dec 5, 2015 at 7:38 AM, Joe Gresock <[email protected]> wrote: > >> I like this idea, Toivo. The nifi guys can weigh in on pros/cons of >> incorporating it into the framework, but my team has implemented a >> TimeLimitedCallback and the corresponding Properties that do what you're >> suggesting. It'd be even better if there were hooks in the processor API >> that let you interrupt your code (soft interrupt) or killed the thread >> (hard interrupt). >> >> On Thu, Dec 3, 2015 at 1:08 PM, Toivo Adams <[email protected]> wrote: >> >> > Hi, >> > >> > Sometimes long running processors cannot be easily stopped. >> > See nifi-users, Mon, 23 Nov, ExecuteProcess processor cannot be >> > stopped/started >> > >> > And often it is desirable to be able set running timeout before processor >> > start. >> > https://issues.apache.org/jira/browse/NIFI-528 Add support to specify a >> > timeout on ExecuteStreamCommand, ExecuteScript, and ExecuteProcess >> > >> > Such problems may be in other processors also. For example ExecuteSQL >> does >> > have timeout support but cannot be stopped in the middle of long running >> > query. >> > >> > Both timeouts and stopping can be maybe solved together? >> > There are strong similarities and some code can be reused? >> > And some processor internal redesign is needed for both timeouts and >> stops. >> > >> > More possibly related things: >> > https://issues.apache.org/jira/browse/NIFI-1183 >> > Failed to stop failing processor >> > >> > I looked ExecuteProcess code and it supports some sort >> stopping/cancelling >> > >> > if (!isScheduled()) { >> > getLogger().info("User stopped processor; will terminate >> > process >> > immediately"); >> > longRunningProcess.cancel(true); >> > return; >> > } >> > >> > But this works only at certain time. After processor starts streaming >> > stopping don’t work anymore. >> > >> > Both timeouts and prompt stopping/cancelling can be tricky. >> > And can be implemented in different ways. >> > 1. Will stop sometime in the future, we don't guarantee immediate >> > stop. >> > 2. We guarantee immediate stop, this can be sometimes very >> difficult. >> > >> > And we must accept some limitations. For example JDBC can be forced to >> > timeout/stop. But database server may continue to run query anyway for >> long >> > time after our cancel in client. >> > >> > How to move on? >> > Or we have other higher priority tasks now and it’s not wise to start >> > handling timeouts and stopping? >> > >> > Thanks >> > Toivo >> > >> > >> > >> > >> > -- >> > View this message in context: >> > >> http://apache-nifi-developer-list.39713.n7.nabble.com/Long-running-processors-stopping-and-timeouts-tp5583.html >> > Sent from the Apache NiFi Developer List mailing list archive at >> > Nabble.com. >> > >> >> >> >> -- >> I know what it is to be in need, and I know what it is to have plenty. I >> have learned the secret of being content in any and every situation, >> whether well fed or hungry, whether living in plenty or in want. I can do >> all this through him who gives me strength. *-Philippians 4:12-13* >>
