Hi Sanjay, The task of the shutdown is to stop the processing from the input operator itself. Once we reach a consistent state (endWindow() for example), we stop the input operator. Thereafter, the next operators in the DAG would only have to process the data which is already emitted by the input operators.
I am not sure of the implementation though; just discussing the need and requirements. ~ Bhupesh On Wed, Jan 18, 2017 at 12:13 PM, Sanjay Pujare <san...@datatorrent.com> wrote: > I have similar question/concerns. When a downstream operator initiates a > shutdown, how is that trigger signaled to and co-ordinated with the input > operators stopping their input reading activity? Or idempotency etc don’t > matter after shutdown? > > > > On 1/17/17, 10:31 PM, "Chinmay Kolhatkar" <chin...@datatorrent.com> wrote: > > +1 for the feature in general. > > Though I have a question about it: Lets say 10th operator in the DAG > initiates the shutdown.. And operator being in quite downstream as > compared > to Input Operator, it might be fairly behind in window it is > processing. > What happens to the data being already processed by upstream operators? > > Is there a possibility where Idempotency might get affected because > termination action is taken in downstream operators? > > -Chinmay. > > > On Wed, Jan 18, 2017 at 11:55 AM, Tushar Gosavi < > tus...@datatorrent.com> > wrote: > > > I think this would be a great addition for batch use cases or use > > cases were DAG needs to be shutdown after detecting some > > completion/error condition through the operator. We have one Jira > > Opened for such functionality > > https://issues.apache.org/jira/browse/APEXCORE-503. > > > > - Tushar. > > > > > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda > > <bhup...@datatorrent.com> wrote: > > > Hi All, > > > > > > Currently we can shutdown an Apex app in the following ways: > > > 1. Throw ShutdownException() from *all* the input operators > > > 2. Use Apex CLI to shutdown an app using the YARN App Id > > > > > > I think we should have some way of shutting down an application > from > > within > > > an operator. It is not always true that the trigger for shutdown > is sent > > by > > > the input operator only. Sometimes, an end condition may be > detected by > > > some operator in the DAG which wants the processing to end. Such a > > > shutdown, although triggered from some intermediate operator in > the DAG, > > > should guarantee graceful shut down of the application. > > > > > > Thoughts? > > > > > > ~ Bhupesh > > > > > >
