Can be done by sending "shutdown" message via heartbeat to Stram. Then on stram can shutdown the entire app
Thks Amol On Tue, Jan 17, 2017 at 11:05 PM, Bhupesh Chawda <bhup...@datatorrent.com> wrote: > Yes Ajay, for a graceful shutdown, the data sent out should be processed. > > On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA <ajaygit...@gmail.com> wrote: > > > +1 to idea. > > > > Will this ensure downstream operators to process all data received before > > shutdown is called? > > Also, how do we plan to handle cases where 2 sub-DAGs merge to a single > > operator somewhere downstream, and an operator in one of the sub-DAGs > sends > > ShutdownException. > > > > > > Ajay > > > > On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda < > bhup...@datatorrent.com> > > wrote: > > > > > This JIRA is to stop the DAG in a crude manner, based on an error > > > condition. I think this might also need similar functionality as an > error > > > condition can occur anywhere in the DAG. > > > > > > Perhaps we can modify the same JIRA to include a graceful + ungraceful > > > (kill) shutdown from any operator in the DAG. > > > > > > ~ Bhupesh > > > > > > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi < > tus...@datatorrent.com> > > > wrote: > > > > > > > I think this would be a great addition for batch use cases or use > > > > cases were DAG needs to be shutdown after detecting some > > > > completion/error condition through the operator. We have one Jira > > > > Opened for such functionality > > > > https://issues.apache.org/jira/browse/APEXCORE-503. > > > > > > > > - Tushar. > > > > > > > > > > > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda > > > > <bhup...@datatorrent.com> wrote: > > > > > Hi All, > > > > > > > > > > Currently we can shutdown an Apex app in the following ways: > > > > > 1. Throw ShutdownException() from *all* the input operators > > > > > 2. Use Apex CLI to shutdown an app using the YARN App Id > > > > > > > > > > I think we should have some way of shutting down an application > from > > > > within > > > > > an operator. It is not always true that the trigger for shutdown is > > > sent > > > > by > > > > > the input operator only. Sometimes, an end condition may be > detected > > by > > > > > some operator in the DAG which wants the processing to end. Such a > > > > > shutdown, although triggered from some intermediate operator in the > > > DAG, > > > > > should guarantee graceful shut down of the application. > > > > > > > > > > Thoughts? > > > > > > > > > > ~ Bhupesh > > > > > > > > > >
