I also prefer 2b.
The grace period before forceful termination should be configurable
either at compile-time by the pipeline or preferably at run-time via
the type safe config.
A config fragment like this detected by the runtime and enforced
absolutely would be a good start. The implementation shouldn't be
responsible for parsing args and creating a Map.
runtime {
shutdown {
stopProviders {
timeAfterStart: "30m"
}
forceful {
timeAfterStopProviders: "5m"
}
}
}
Steve
Steve Blackmon
[email protected]
On Wed, Oct 8, 2014 at 10:55 AM, Robert Douglas
<[email protected]> wrote:
> Hi Ryan,
>
> I think the most important thing is to ensure that anytime "stop()" is
> called, the behavior is deterministic. When that method is called, we need
> a guarantee that things will completely shut down in a reasonable amount of
> time. However, I don't like the idea of just abandoning datums that are
> already in the pipeline. If it's possible, I'd prefer to go with an
> approach similar to option 2b. Completely shutdown the providers, and give
> the Stream some time to attempt to process all currently queued datums.
> However, after a certain period of time, I think that a forceful shutdown
> needs to be executed regardless of the state of the pipeline.
>
> -- Robert Douglas
>
> On Wed, Oct 8, 2014 at 10:31 AM, Ryan Ebanks <[email protected]> wrote:
>
>> I've been added tests and fixing some issues in the local stream builder
>> class. I need some input on how the stop() method should work so I can
>> write the appropriate tests.
>>
>> Option 1.) Gentle Shutdown
>> Immediately stop the providers from producing more data, but allow for all
>> data currently in the stream to finish processing. Shutdown time would be
>> unknown as a result.
>>
>> Option 2.) Forceful Shutdown
>> Immediately stop the providers from producing more data, and stop all
>> processors/writers. Data will be left in the pipeline will be discarded.
>> shutdown will be immediate
>>
>> Option 2b) Forceful Shutdown after small wait
>> Immediately stop providers from producing more data. Give the
>> processors/writers a small window to try to finish processing (< 10secs),
>> and if not finished, forcefully shutdown all processors/writers.
>>
>>
>> These options can be changed in the future as more features are added. For
>> example once datum tracking is implemented, we will probably want a
>> combination of option 1 and 2b. But we need a defined way for it work now,
>> so tests are testing the correct functionality.
>>
>> -Ryan Ebanks
>>
