Ok. I agree that 2b is the most appropriate. It will require the most code change though. Right now the code does option 1 and test assumes option 2. I'll @Ignore the test for now, and we can make this change in another pull request as I don't want this one to get too big. That means the code will continue with option 1 for now.
-Ryan On Wed, Oct 8, 2014 at 11:49 AM, Steve Blackmon <[email protected]> wrote: > I also prefer 2b. > > The grace period before forceful termination should be configurable > either at compile-time by the pipeline or preferably at run-time via > the type safe config. > > A config fragment like this detected by the runtime and enforced > absolutely would be a good start. The implementation shouldn't be > responsible for parsing args and creating a Map. > > runtime { > shutdown { > stopProviders { > timeAfterStart: "30m" > } > forceful { > timeAfterStopProviders: "5m" > } > } > } > > Steve > Steve Blackmon > [email protected] > > > On Wed, Oct 8, 2014 at 10:55 AM, Robert Douglas > <[email protected]> wrote: > > Hi Ryan, > > > > I think the most important thing is to ensure that anytime "stop()" is > > called, the behavior is deterministic. When that method is called, we > need > > a guarantee that things will completely shut down in a reasonable amount > of > > time. However, I don't like the idea of just abandoning datums that are > > already in the pipeline. If it's possible, I'd prefer to go with an > > approach similar to option 2b. Completely shutdown the providers, and > give > > the Stream some time to attempt to process all currently queued datums. > > However, after a certain period of time, I think that a forceful shutdown > > needs to be executed regardless of the state of the pipeline. > > > > -- Robert Douglas > > > > On Wed, Oct 8, 2014 at 10:31 AM, Ryan Ebanks <[email protected]> > wrote: > > > >> I've been added tests and fixing some issues in the local stream builder > >> class. I need some input on how the stop() method should work so I can > >> write the appropriate tests. > >> > >> Option 1.) Gentle Shutdown > >> Immediately stop the providers from producing more data, but allow for > all > >> data currently in the stream to finish processing. Shutdown time would > be > >> unknown as a result. > >> > >> Option 2.) Forceful Shutdown > >> Immediately stop the providers from producing more data, and stop all > >> processors/writers. Data will be left in the pipeline will be > discarded. > >> shutdown will be immediate > >> > >> Option 2b) Forceful Shutdown after small wait > >> Immediately stop providers from producing more data. Give the > >> processors/writers a small window to try to finish processing (< > 10secs), > >> and if not finished, forcefully shutdown all processors/writers. > >> > >> > >> These options can be changed in the future as more features are added. > For > >> example once datum tracking is implemented, we will probably want a > >> combination of option 1 and 2b. But we need a defined way for it work > now, > >> so tests are testing the correct functionality. > >> > >> -Ryan Ebanks > >> >
