I enabled glog of mesos driver. Log below for scenario — sequence as follows:
1) shutdown slave 2) monitor agent status in mesos until it shows as “LOST” 3) start slave back up 4) Logs below from executor driver code: https://github.com/mesos/mesos-go/blob/master/api/v0/executor/executor.go#L484-L501 <https://github.com/mesos/mesos-go/blob/master/api/v0/executor/executor.go#L484-L501> I0721 22:20:20.007127 27249 executor.go:377] Executor driver reconnect I0721 22:20:20.007156 27249 executor.go:387] Received reconnect request from slave &SlaveID{Value:*7716ab68-bf95-4fd6-9d07-aa3fd4a65349-S9,XXX_unrecognized:[],} I0721 22:20:20.008659 27249 executor.go:340] Executor driver reregistered I0721 22:20:20.008674 27249 executor.go:351] Re-registered on slave &SlaveID{Value:*7716ab68-bf95-4fd6-9d07-aa3fd4a65349-S9,XXX_unrecognized:[],} I0721 22:20:20.008704 27249 main.go:52] Re-registered Executor on slave I0721 22:20:22.988835 27249 executor.go:509] Executor driver received shutdown I0721 22:20:22.988854 27249 executor.go:516] Executor driver is asked to shutdown I0721 22:20:22.988878 27249 executor.go:602] Stopping the executor driver I0721 22:20:22.988885 27249 messenger.go:295] stopping messenger executor(1)@127.0.0.1:52644.. I0721 22:20:22.988896 27249 http_transporter.go:447] stopping HTTP transport I also put a sleep in Shutdown callback: https://github.com/mesos/mesos-go/blob/master/api/v0/examples/executor/main.go#L110 <https://github.com/mesos/mesos-go/blob/master/api/v0/examples/executor/main.go#L110> However, I expected Shutdown callback getting invoked but don’t see in logs.. it seems likely that executor got terminated abruptly. May be abort or stop is invoked prior to shutdown?. Would you verify behavior from your end for this scenario.. Thanks, Jiamin > On Jul 18, 2017, at 5:42 PM, Zhu Jiamin <jiamin.zhu...@gmail.com> wrote: > > mesos-go executor driver calls driver.stop() after invoking shutdown of > executor > > // this is from mesos-go executor driver > driver.withExecutor(func(e Executor) { e.Shutdown(driver) }) > driver.stop() > > So executor should print log firstly then exit since driver stop happen > after invoking shutdown. > > Will try to enable GLOG and come back to you. > > Many thanks, > Jiamin > > > > > >> On Jul 18, 2017, at 5:24 PM, Vinod Kone <vinodk...@apache.org >> <mailto:vinodk...@apache.org>> wrote: >> >> On Tue, Jul 18, 2017 at 2:11 PM, Zhu Jiamin <jiamin.zhu...@gmail.com >> <mailto:jiamin.zhu...@gmail.com>> wrote: >> >>> 2017/07/18 20:49:19 Stop driver >>> >> >> Am I reading this correctly that mesos-go executor calls `stop()` on the >> driver? Do you know if it calls it when it gets a `shutdown()`? If yes, >> that would explain the behavior. >> >> Looking at the driver code, looks like when it gets a shutdown message from >> the agent the shutdown grace period should be respected. The only way a >> driver would terminate early is if an executor called `stop()` or `abort()` >> in the interim. >> >> Is there any way you could set `GLOG_v=1` in the environment of your >> executor so that driver logs get output? That would make debugging this >> much easier. >
