Ivan and Tomas- I want to thank you for the quick replies. After I deleted what 'node' from zookeeper, I did the standard "service marathon restart" and marathion did not start. However, I ran marathon manually:
marathon --master zk://10.10.10.51:2181/mesos --zk zk:// 10.10.10.51:2181/marathon and marathon started. Thank you again for the great support! Alex On Wed, Nov 22, 2017 at 11:01 AM, Alex Evonosky <[email protected]> wrote: > Ivan- > > I ran the following: > > zookeepercli -servers=10.10.10.51:2181 -c rm > /marathon/state/migration-in-progress > 2017-11-22 11:00:47 FATAL zk: node does not exist > > > and tried to restart Marathon and same issue. Does this appear to be a > Zookeeper issue? > > Thanks! > > Alex > > > > > On Wed, Nov 22, 2017 at 10:21 AM, Ivan Chernetsky < > [email protected]> wrote: > >> Hi Alex, >> >> If you are sure that the Marathon state in ZK is consistent, you can >> remove the flag using zkCli.sh >> >> For instance, if the ZK connection string for Marathon, you use is >> zk://localhost:2181/marathon, then once connected to ZK using zkCli.sh, >> just execute "rm /marathon/state/migration-in-progress" >> >> This should revolve it. >> >> Please refer to https://github.com/mesosphere/marathon/blob/master/docs/ >> docs/data-migration.md for further details. >> >> Regards, >> Ivan. >> >> On Wed, Nov 22, 2017 at 5:18 PM, Alex Evonosky <[email protected]> >> wrote: >> >>> Tomas- >>> >>> thank you for the reply! I am running marathon 1.5.2, zookeeper: 3.4.8-1 >>> >>> I looked on the referenced gitlab [age, but I really did find the syntax >>> to remove the flag as suggested. Do you happen to know the syntax via the >>> zookeeper cli? >>> >>> Thank you again! >>> >>> >>> On Wed, Nov 22, 2017 at 8:50 AM, Tomas Barton <[email protected]> >>> wrote: >>> >>>> Hi Alex, >>>> >>>> looks like you've restarted Marathon during election. Try to backup >>>> ZooKeeper data and then go to exhibitor / ZooKeeper CLI and remove flag >>>> from Marathon namespace: >>>> >>>> /state/migration-in-progress >>>> >>>> According to https://github.com/mesosphere/marathon/pull/5662 the flag >>>> should be removed upon unsuccessful migration. Which version of Marathon do >>>> you run? >>>> >>>> Tomas >>>> >>>> On 22 November 2017 at 08:21, Alex Evonosky <[email protected]> >>>> wrote: >>>> >>>>> Hello group- >>>>> >>>>> Long time Mesos user and first time post about an issue. I have been >>>>> running mesos 1.4.0 for a while without any issues. The other day, Ubuntu >>>>> upgraded mesos to 1.4.1, which seemed to go ok, however, I reloaded one >>>>> master node to verify it can come back up after the upgrade and now mesos >>>>> and zookeeper appear fine, however, marathon did not recover. Starting >>>>> marathon shows many errors now (attached file). >>>>> >>>>> Could someone let me know what the issue could be from just rebooting >>>>> a server? >>>>> >>>>> >>>>> Thank you! >>>>> >>>>> Alex >>>>> >>>> >>>> >>> >> >
