Hi Chris, Here's the main idea of the SIGTERM shutdown hook approach that we're trying out: https://gist.github.com/zcox/6ec8910bd3f18e36c1a2
We use our own main class to run either the job or the container depending on args, and this main is the ENTRYPOINT of our Docker container, instead of using the standard run-job.sh and run-container.sh scripts. So this is an easy place to add the shutdown hook. Not sure how reusable that would be for others... By "not in a very clean way" I meant that the shutdown hook needs to block jvm shutdown until SamzaContainer has gone through its entire list of shutdown calls, and the only way I could find for the shutdown hook to know this is finished is the shutdownIsComplete flag, set true after SamzaContainer.safeMain() returns, which feels a bit jank, but it seems to work. Thanks, Zach On Tue, Jan 13, 2015 at 2:34 PM, Chris Riccomini < [email protected]> wrote: > Hey Zach, > > Honestly, we just tell YARN to kill everything off. As you said, this > means there is no clean shutdown, but we've lived with thus far. > > Some folks have also used a control message, as you've said, and had their > task call coordinator.shutdown(). Others have used a JMX switch to do the > same. > > I agree that catching a SIGTERM is a little cleaner. Can you open a JIRA > for this? > > Basically, all the solutions you've described are possible. Only the > SIGTERM trap hasn't been implemented. > > > I can sorta make this work, but not in a very clean way. > > > Curious: what's not clean? > > Cheers, > Chris > > On 1/13/15 11:21 AM, "Zach Cox" <[email protected]> wrote: > > >Hi - just wondering what other folks do when a Samza job needs to be > >shutdown purposefully, such as during a deployment. Seems like in this > >case > >you'd want all of the job's Samza containers to cleanly stop processing > >messages, commit offsets/changelog, etc instead of just SIGKILL the jvms > >which will result in out-of-date checkpoints, etc. > > > >I noticed TaskCoordinator has a shutdown method, which the StreamTasks in > >a > >container can use to signal a clean shutdown. Does everyone use this, with > >some kind of shutdown message sent to the StreamTasks using e.g. jmx or a > >kafka topic, as in this thread? > > > http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.html > > > >We're actively developing Samza support for Mesos, so maybe this is > >specific to that world, but would it make sense for a jvm shutdown hook in > >the container to catch a SIGTERM and somehow signal to the container that > >it should shutdown? I can sorta make this work, but not in a very clean > >way. > > > >Just curious what others are doing about graceful shutdown. > > > >Thanks, > >Zach > >
