Interesting this case. +1 2016-11-22 12:27 GMT+08:00 X Brick <[email protected]>:
> Found some issues in the JIRA: > > - MESOS-5368 <https://issues.apache.org/jira/browse/MESOS-5368> > - MESOS-6223 <https://issues.apache.org/jira/browse/MESOS-6223> > - MESOS-3545 <https://issues.apache.org/jira/browse/MESOS-3545> > > Not quite sure the boot_id would fix in next release, but you could backup > the boot_file file ( in your $work_dir/meta/) after slave start. and > restore it with the backup file when restarting, it works well for our > cluster with the persistent volumes. > > 2016-11-09 0:43 GMT+08:00 Hendrik Haddorp <[email protected]>: > >> I have a framework that starts multiple docker containers. The >> configuration (hosts and ports) of my setup need to stay constant. So in a >> first step my framework is claiming resources on the slaves. Once all >> required resources are acquired I start the containers using the docker >> containerizer. When fails I restart it on the same slave with the same >> config. So far I'm tracking the Mesos slave ID and would only restart the >> task if I get an offer for that slave again. As the ID changes now I'm not >> restarting the task anymore. >> >> My assumption was that the slave ID would stay constant so that I could >> for example change the host name and would still recognize the instance or >> start multiple slaves on the same server and easily distinguish them. If >> the slave ID changes I would have expected that all resources connected to >> that would be lost but that doesn't seem to be the case, which is good in >> my case, but rather odd in my opinion. >> >> On 08.11.2016 17:26, Vinod Kone wrote: >> >>> @Hendrik: When maintenance APIs are used, the typical expectation is >>> that the tasks on the machine are stopped (and rescheduled elsewhere in the >>> cluster). That is the reason that the agent gets a new ID. What is the >>> exact problem you are facing? >>> >>> @Justin: This is a known issue that is actively being worked on. >>> https://issues.apache.org/jira/browse/MESOS-5396 >>> >>> >>> On Tue, Nov 8, 2016 at 8:12 AM, Hendrik Haddorp <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Interesting, in one case we also had a reboot but not in the >>> simple restart with a pause test. Losing the ID on restart sounds >>> odd to me. Do you have some further details on that? >>> >>> On 08.11.2016 17:08, Justin Pinkul wrote: >>> >>> >>> Hello, >>> >>> >>> I also hit a very similar problem recently, perhaps it is >>> related. There is special logic inside of the Mesos agent that >>> checks if the machine has rebooted; if it has rebooted it will >>> short circuit the recovery and register with a new agent ID. >>> This is especially problematic with the new >>> --agent_removal_rate_limit and --recovery_agent_removal_limit >>> flags. We hit a power outage and which caused this to happen >>> on every machine in our lab at once, since every agent had a >>> new ID 50% of the ids were considered lost and these safe >>> guards caused our master to kill itself every 15 minutes even >>> after all of the agents were back up and running. Is there any >>> advantage to throwing out the agent ID when rebooting? >>> >>> >>> Thanks, >>> >>> Justin >>> >>> >>> >>> >>> ------------------------------------------------------------ >>> ------------ >>> *From:* Hendrik Haddorp <[email protected] >>> <mailto:[email protected]>> >>> *Sent:* Tuesday, November 8, 2016 12:59 PM >>> *To:* user >>> *Subject:* Slave gets new ID >>> Hi, >>> >>> when we take slaves down for maintenance, as described in >>> http://mesos.apache.org/documentation/latest/maintenance/ >>> <http://mesos.apache.org/documentation/latest/maintenance/>, >>> the slave >>> <http://mesos.apache.org/documentation/latest/maintenance/ >>> <http://mesos.apache.org/documentation/latest/maintenance/>> >>> >>> Apache Mesos - Maintenance Primitives >>> <http://mesos.apache.org/documentation/latest/maintenance/ >>> <http://mesos.apache.org/documentation/latest/maintenance/>> >>> mesos.apache.org <http://mesos.apache.org> >>> Maintenance Primitives. Operators regularly need to perform >>> maintenance tasks on machines that comprise a Mesos cluster. >>> Most Mesos upgrades can be done without ... >>> >>> >>> >>> gets a new ID on start up. Why is that and can it be changed? >>> We are >>> using Mesos 0.28.2. I'm so far only aware of the >>> slave_reregister_timeout. Our restart was within that time >>> frame. When >>> we restart a slave it keeps its ID. However when we wait a few >>> minutes, >>> less then the reregistration timeout, before we restart the >>> slave the ID >>> also changes. >>> >>> regards, >>> Hendrik >>> >>> >>> >>> >> > -- Deshi Xiao Twitter: xds2000 E-mail: xiaods(AT)gmail.com
