Hello group-
Long time Mesos user and first time post about an issue. I have been
running mesos 1.4.0 for a while without any issues. The other day, Ubuntu
upgraded mesos to 1.4.1, which seemed to go ok, however, I reloaded one
master node to verify it can come back up after the upgrade and now mesos
and zookeeper appear fine, however, marathon did not recover. Starting
marathon shows many errors now (attached file).
Could someone let me know what the issue could be from just rebooting a
server?
Thank you!
Alex
[INFO] [11/22/2017 02:09:50.438] [main] [SystemMetricsExtension(akka://kamon)]
Starting the Kamon(SystemMetrics) extension
[2017-11-22 02:09:50,455] INFO Sigar library provisioned:
/root/native/libsigar-amd64-linux.so (kamon.sigar.SigarProvisioner:main)
[INFO] [11/22/2017 02:09:50.541] [main] [JMXExtension(akka://kamon)] Starting
the Kamon(JMX) extension
[INFO] [11/22/2017 02:09:50.560] [main] [JMXMetricsExtension(akka://kamon)]
Starting the Kamon(JMXMetrics) extension
registering jmx exporter
[2017-11-22 02:09:50,654] INFO Slf4jLogger started
(akka.event.slf4j.Slf4jLogger:marathon-akka.actor.default-dispatcher-3)
[2017-11-22 02:09:50,705] INFO Starting Marathon
1.5.2/c55acb9a135a8b4b30f8cb1fafa529cfaefc3f8a with --master
zk://10.10.10.51:2181/mesos --zk zk://10.10.10.51:2181/marathon
(mesosphere.marathon.MarathonApp:JMX exporting thread)
[2017-11-22 02:09:50,796] INFO Successfully loaded libmesos: version 1.4.1
(mesosphere.marathon.MarathonApp:JMX exporting thread)
[2017-11-22 02:09:51,835] INFO Starting
(org.apache.curator.framework.imps.CuratorFrameworkImpl:JMX exporting thread)
[2017-11-22 02:09:51,846] INFO Client environment:zookeeper.version=3.4.8--1,
built on 02/06/2016 03:18 GMT (org.apache.zookeeper.ZooKeeper:JMX exporting
thread)
[2017-11-22 02:09:51,847] INFO Client environment:host.name=mesos-master1
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,847] INFO Client environment:java.version=1.8.0_151
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,847] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,847] INFO Client
environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,847] INFO Client
environment:java.class.path=/usr/share/marathon/lib/mesosphere.marathon.marathon-1.5.2.jar:/usr/share/marathon/lib/mesosphere.marathon.plugin-interface-1.5.2.jar:/usr/share/marathon/lib/org.apache.curator.curator-client-2.11.1.jar:/usr/share/marathon/lib/com.thoughtworks.paranamer.paranamer-2.8.jar:/usr/share/marathon/lib/org.slf4j.slf4j-api-1.7.24.jar:/usr/share/marathon/lib/com.google.protobuf.protobuf-java-3.3.0.jar:/usr/share/marathon/lib/net.logstash.logback.logstash-logback-encoder-4.9.jar:/usr/share/marathon/lib/org.mozilla.rhino-1.7.7.jar:/usr/share/marathon/lib/com.fasterxml.jackson.module.jackson-module-scala_2.11-2.7.2.jar:/usr/share/marathon/lib/io.netty.netty-handler-4.0.43.Final.jar:/usr/share/marathon/lib/com.fasterxml.jackson.core.jackson-annotations-2.7.8.jar:/usr/share/marathon/lib/com.google.inject.guice-3.0.jar:/usr/share/marathon/lib/com.getsentry.raven.raven-7.8.6.jar:/usr/share/marathon/lib/io.kamon.kamon-system-metrics_2.11-0.6.7.jar:/usr/share/marathon/lib/org.scala-lang.modules.scala-java8-compat_2.11-0.8.0.jar:/usr/share/marathon/lib/com.fasterxml.jackson.jaxrs.jackson-jaxrs-json-provider-2.7.2.jar:/usr/share/marathon/lib/joda-time.joda-time-2.9.9.jar:/usr/share/marathon/lib/com.typesafe.akka.akka-actor_2.11-2.4.18.jar:/usr/share/marathon/lib/com.sun.jersey.jersey-server-1.18.6.jar:/usr/share/marathon/lib/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.7.8.jar:/usr/share/marathon/lib/org.apache.mesos.mesos-1.4.0.jar:/usr/share/marathon/lib/com.typesafe.netty.netty-reactive-streams-1.0.8.jar:/usr/share/marathon/lib/com.fasterxml.jackson.dataformat.jackson-dataformat-cbor-2.6.6.jar:/usr/share/marathon/lib/io.dropwizard.metrics.metrics-annotation-3.1.2.jar:/usr/share/marathon/lib/org.scala-lang.modules.scala-parser-combinators_2.11-1.0.4.jar:/usr/share/marathon/lib/io.netty.netty-common-4.0.43.Final.jar:/usr/share/marathon/lib/aopalliance.aopalliance-1.0.jar:/usr/share/marathon/lib/net.sf.jopt-simple.jopt-simple-4.6.jar:/usr/share/marathon/lib/com.typesafe.play.play-json_2.11-2.5.14.jar:/usr/share/marathon/lib/io.dropwizard.metrics.metrics-json-3.1.2.jar:/usr/share/marathon/lib/io.kamon.kamon-statsd_2.11-0.6.7.jar:/usr/share/marathon/lib/javax.inject.javax.inject-1.jar:/usr/share/marathon/lib/com.github.fge.json-schema-validator-2.2.6.jar:/usr/share/marathon/lib/org.aspectj.aspectjrt-1.8.10.jar:/usr/share/marathon/lib/com.sksamuel.scapegoat.scalac-scapegoat-plugin_2.11-1.3.0.jar:/usr/share/marathon/lib/org.apache.commons.commons-compress-1.13.jar:/usr/share/marathon/lib/com.typesafe.akka.akka-parsing_2.11-10.0.6.jar:/usr/share/marathon/lib/com.github.fge.uri-template-0.9.jar:/usr/share/marathon/lib/org.asynchttpclient.async-http-client-2.0.25.jar:/usr/share/marathon/lib/org.scala-stm.scala-stm_2.11-0.7.jar:/usr/share/marathon/lib/commons-io.commons-io-2.5.jar:/usr/share/marathon/lib/org.apache.curator.curator-recipes-2.11.1.jar:/usr/share/marathon/lib/commons-beanutils.commons-beanutils-1.9.3.jar:/usr/share/marathon/lib/com.wix.accord-api_2.11-0.5.jar:/usr/share/marathon/lib/io.dropwizard.metrics.metrics-servlets-3.1.2.jar:/usr/share/marathon/lib/org.apache.zookeeper.zookeeper-3.4.8.jar:/usr/share/marathon/lib/com.typesafe.ssl-config-core_2.11-0.2.1.jar:/usr/share/marathon/lib/javax.activation.activation-1.1.jar:/usr/share/marathon/lib/org.asynchttpclient.netty-codec-dns-2.0.25.jar:/usr/share/marathon/lib/org.scala-lang.scala-library-2.11.11.jar:/usr/share/marathon/lib/com.github.fge.json-schema-core-1.2.5.jar:/usr/share/marathon/lib/com.wix.accord-core_2.11-0.5.jar:/usr/share/marathon/lib/org.slf4j.jcl-over-slf4j-1.7.21.jar:/usr/share/marathon/lib/ch.qos.logback.logback-classic-1.2.1.jar:/usr/share/marathon/lib/mesosphere.chaos_2.11-0.8.8.jar:/usr/share/marathon/lib/com.github.spullara.mustache.java.compiler-0.9.0.jar:/usr/share/marathon/lib/org.eclipse.jetty.jetty-io-9.3.6.v20151106.jar:/usr/share/marathon/lib/io.netty.netty-3.7.0.Final.jar:/usr/share/marathon/lib/mesosphere.marathon.ui-1.2.0.jar:/usr/share/marathon/lib/software.amazon.ion.ion-java-1.0.2.jar:/usr/share/marathon/lib/com.typesafe.play.play-iteratees_2.11-2.5.14.jar:/usr/share/marathon/lib/com.typesafe.akka.akka-slf4j_2.11-2.4.18.jar:/usr/share/marathon/lib/org.apache.httpcomponents.httpclient-4.5.2.jar:/usr/share/marathon/lib/com.typesafe.akka.akka-http_2.11-10.0.6.jar:/usr/share/marathon/lib/javax.mail.mailapi-1.4.3.jar:/usr/share/marathon/lib/org.rogach.scallop_2.11-1.0.0.jar:/usr/share/marathon/lib/io.kamon.kamon-jmx_2.11-0.6.7.jar:/usr/share/marathon/lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.7.8.jar:/usr/share/marathon/lib/com.google.code.findbugs.jsr305-3.0.0.jar:/usr/share/marathon/lib/org.apache.httpcomponents.httpcore-4.4.4.jar:/usr/share/marathon/lib/org.joda.joda-convert-1.8.1.jar:/usr/share/marathon/lib/com.github.fge.jackson-coreutils-1.8.jar:/usr/share/marathon/lib/io.dropwizard.metrics.metrics-healthchecks-3.1.2.jar:/usr/share/marathon/lib/org.apache.curator.curator-framework-2.11.1.jar:/usr/share/marathon/lib/org.asynchttpclient.netty-resolver-dns-2.0.25.jar:/usr/share/marathon/lib/org.slf4j.log4j-over-slf4j-1.7.21.jar:/usr/share/marathon/lib/org.aspectj.aspectjweaver-1.8.10.jar:/usr/share/marathon/lib/com.sun.jersey.contribs.jersey-guice-1.18.1.jar:/usr/share/marathon/lib/javax.validation.validation-api-1.1.0.Final.jar:/usr/share/marathon/lib/de.heikoseeberger.akka-http-play-json_2.11-1.10.1.jar:/usr/share/marathon/lib/org.asynchttpclient.async-http-client-netty-utils-2.0.25.jar:/usr/share/marathon/lib/io.dropwizard.metrics.metrics-jersey-3.1.2.jar:/usr/share/marathon/lib/ch.qos.logback.logback-core-1.2.1.jar:/usr/share/marathon/lib/org.eclipse.jetty.jetty-util-9.3.6.v20151106.jar:/usr/share/marathon/lib/org.reactivestreams.reactive-streams-1.0.0.jar:/usr/share/marathon/lib/com.google.guava.guava-19.0.jar:/usr/share/marathon/lib/io.reactivex.rxjava-1.2.4.jar:/usr/share/marathon/lib/com.google.inject.extensions.guice-servlet-3.0.jar:/usr/share/marathon/lib/com.sun.jersey.jersey-servlet-1.18.6.jar:/usr/share/marathon/lib/org.javassist.javassist-3.21.0-GA.jar:/usr/share/marathon/lib/com.fasterxml.uuid.java-uuid-generator-3.1.4.jar:/usr/share/marathon/lib/javax.servlet.javax.servlet-api-3.1.0.jar:/usr/share/marathon/lib/de.heikoseeberger.akka-sse_2.11-2.0.0.jar:/usr/share/marathon/lib/commons-collections.commons-collections-3.2.2.jar:/usr/share/marathon/lib/io.dropwizard.metrics.metrics-core-3.1.2.jar:/usr/share/marathon/lib/io.dropwizard.metrics.metrics-jvm-3.1.2.jar:/usr/share/marathon/lib/org.slf4j.jul-to-slf4j-1.7.21.jar:/usr/share/marathon/lib/commons-codec.commons-codec-1.9.jar:/usr/share/marathon/lib/com.typesafe.play.play-functional_2.11-2.5.14.jar:/usr/share/marathon/lib/io.kamon.sigar-loader-1.6.5-rev002.jar:/usr/share/marathon/lib/com.amazonaws.aws-java-sdk-core-1.11.129.jar:/usr/share/marathon/lib/io.netty.netty-buffer-4.0.43.Final.jar:/usr/share/marathon/lib/org.hibernate.hibernate-validator-5.2.1.Final.jar:/usr/share/marathon/lib/com.fasterxml.jackson.core.jackson-databind-2.7.8.jar:/usr/share/marathon/lib/io.netty.netty-codec-http-4.0.43.Final.jar:/usr/share/marathon/lib/org.scala-lang.modules.scala-async_2.11-0.9.6.jar:/usr/share/marathon/lib/org.javabits.jgrapht.jgrapht-core-0.9.3.jar:/usr/share/marathon/lib/com.typesafe.config-1.3.1.jar:/usr/share/marathon/lib/com.typesafe.scala-logging.scala-logging_2.11-3.5.0.jar:/usr/share/marathon/lib/com.typesafe.akka.akka-http-core_2.11-10.0.6.jar:/usr/share/marathon/lib/org.eclipse.jetty.jetty-http-9.3.6.v20151106.jar:/usr/share/marathon/lib/com.github.fge.msg-simple-1.1.jar:/usr/share/marathon/lib/com.googlecode.libphonenumber.libphonenumber-6.2.jar:/usr/share/marathon/lib/org.hdrhistogram.HdrHistogram-2.1.9.jar:/usr/share/marathon/lib/mesosphere.marathon.api-console-3.0.8-accept.jar:/usr/share/marathon/lib/org.eclipse.jetty.jetty-server-9.3.6.v20151106.jar:/usr/share/marathon/lib/jline.jline-0.9.94.jar:/usr/share/marathon/lib/com.fasterxml.jackson.core.jackson-core-2.8.7.jar:/usr/share/marathon/lib/com.getsentry.raven.raven-logback-7.8.6.jar:/usr/share/marathon/lib/io.reactivex.rxscala_2.11-0.26.5.jar:/usr/share/marathon/lib/com.fasterxml.jackson.module.jackson-module-paranamer-2.7.2.jar:/usr/share/marathon/lib/org.scala-lang.modules.scala-xml_2.11-1.0.5.jar:/usr/share/marathon/lib/net.liftweb.lift-markdown_2.11-2.6.2.jar:/usr/share/marathon/lib/org.scalamacros.resetallattrs_2.11-1.0.0.jar:/usr/share/marathon/lib/org.asynchttpclient.netty-resolver-2.0.25.jar:/usr/share/marathon/lib/com.typesafe.akka.akka-stream_2.11-2.4.18.jar:/usr/share/marathon/lib/com.github.fge.btf-1.2.jar:/usr/share/marathon/lib/com.typesafe.play.play-datacommons_2.11-2.5.14.jar:/usr/share/marathon/lib/org.jvnet.mimepull.mimepull-1.9.3.jar:/usr/share/marathon/lib/com.sun.jersey.jersey-core-1.18.6.jar:/usr/share/marathon/lib/org.jboss.logging.jboss-logging-3.2.1.Final.jar:/usr/share/marathon/lib/org.eclipse.jetty.jetty-servlets-9.3.6.v20151106.jar:/usr/share/marathon/lib/org.eclipse.jetty.jetty-security-9.3.6.v20151106.jar:/usr/share/marathon/lib/com.sun.jersey.contribs.jersey-multipart-1.18.6.jar:/usr/share/marathon/lib/io.kamon.kamon-core_2.11-0.6.7.jar:/usr/share/marathon/lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.7.2.jar:/usr/share/marathon/lib/io.kamon.kamon-datadog_2.11-0.6.7.jar:/usr/share/marathon/lib/com.typesafe.akka.akka-http-xml_2.11-10.0.5.jar:/usr/share/marathon/lib/org.scala-lang.scala-reflect-2.11.11.jar:/usr/share/marathon/lib/io.netty.netty-transport-4.0.43.Final.jar:/usr/share/marathon/lib/com.fasterxml.jackson.jaxrs.jackson-jaxrs-base-2.7.2.jar:/usr/share/marathon/lib/io.dropwizard.metrics.metrics-jetty9-3.1.2.jar:/usr/share/marathon/lib/io.kamon.kamon-scala_2.11-0.6.7.jar:/usr/share/marathon/lib/org.eclipse.jetty.jetty-continuation-9.3.6.v20151106.jar:/usr/share/marathon/lib/com.fasterxml.classmate-1.1.0.jar:/usr/share/marathon/lib/org.eclipse.jetty.jetty-servlet-9.3.6.v20151106.jar:/usr/share/marathon/lib/io.netty.netty-codec-4.0.43.Final.jar:/usr/share/marathon/lib/io.kamon.kamon-autoweave_2.11-0.6.5.jar:/usr/share/marathon/lib/com.lightbend.akka.akka-stream-alpakka-s3_2.11-0.8.jar
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,847] INFO Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,847] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,848] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,848] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,848] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,848] INFO Client environment:os.version=4.10.0-28-generic
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,848] INFO Client environment:user.name=root
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,848] INFO Client environment:user.home=/root
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,848] INFO Client environment:user.dir=/root
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,850] INFO Initiating client connection,
connectString=10.10.10.51:2181 sessionTimeout=10000
watcher=org.apache.curator.ConnectionState@2596d7f4
(org.apache.zookeeper.ZooKeeper:JMX exporting thread)
[2017-11-22 02:09:51,878] INFO Opening socket connection to server
10.10.10.51/10.10.10.51:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn:JMX exporting
thread-SendThread(10.10.10.51:2181))
[2017-11-22 02:09:51,886] INFO Socket connection established to
10.10.10.51/10.10.10.51:2181, initiating session
(org.apache.zookeeper.ClientCnxn:JMX exporting
thread-SendThread(10.10.10.51:2181))
[2017-11-22 02:09:51,936] INFO Session establishment complete on server
10.10.10.51/10.10.10.51:2181, sessionid = 0x15fe27c7c3a000a, negotiated timeout
= 10000 (org.apache.zookeeper.ClientCnxn:JMX exporting
thread-SendThread(10.10.10.51:2181))
[2017-11-22 02:09:51,947] INFO State change: CONNECTED
(org.apache.curator.framework.state.ConnectionStateManager:JMX exporting
thread-EventThread)
[2017-11-22 02:09:52,208] INFO Started TaskTrackerUpdateStepsProcessorImpl
with steps:
* continueOnError(notifyHealthCheckManager)
* continueOnError(notifyRateLimiter)
* continueOnError(notifyLaunchQueue)
* continueOnError(postTaskStatusEvent)
* continueOnError(scaleApp)
(mesosphere.marathon.core.task.tracker.impl.InstanceTrackerUpdateStepProcessorImpl:JMX
exporting thread)
[2017-11-22 02:09:52,306] INFO Calling reviveOffers is enabled. Use
--disable_revive_offers_for_new_apps to disable.
(mesosphere.marathon.core.flow.FlowModule:JMX exporting thread)
[2017-11-22 02:09:52,439] INFO Loading plugins implementing
'mesosphere.marathon.plugin.task.RunSpecTaskProcessor' from these urls: []
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,444] INFO Found 0 plugins.
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,446] INFO Loading plugins implementing
'mesosphere.marathon.plugin.validation.RunSpecValidator' from these urls: []
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,446] INFO Found 0 plugins.
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,621] INFO Loading plugins implementing
'mesosphere.marathon.plugin.auth.Authenticator' from these urls: []
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,621] INFO Found 0 plugins.
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,622] INFO Loading plugins implementing
'mesosphere.marathon.plugin.auth.Authorizer' from these urls: []
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,623] INFO Found 0 plugins.
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,630] INFO Started status update processor
(mesosphere.marathon.core.task.update.impl.TaskStatusUpdateProcessorImpl:JMX
exporting thread)
[2017-11-22 02:09:52,847] INFO Loading plugins implementing
'mesosphere.marathon.plugin.scheduler.SchedulerPlugin' from these urls: []
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,848] INFO Found 0 plugins.
(mesosphere.marathon.core.plugin.impl.PluginManagerImpl:JMX exporting thread)
[2017-11-22 02:09:52,882] INFO All actors suspended:
* Actor[akka://marathon/user/reviveOffersWhenWanted#372962215]
* Actor[akka://marathon/user/offerMatcherStatistics#495250445]
* Actor[akka://marathon/user/instanceTracker#1099627657]
* Actor[akka://marathon/user/launchQueue#632132577]
* Actor[akka://marathon/user/offerMatcherManager#235521723]
* Actor[akka://marathon/user/offerMatcherLaunchTokens#-544757702]
* Actor[akka://marathon/user/deploymentManager#-1914638646]
* Actor[akka://marathon/user/killOverdueStagedTasks#297356721]
* Actor[akka://marathon/user/expungeOverdueLostTasks#27844318]
* Actor[akka://marathon/user/taskKillServiceActor#175768796]
* Actor[akka://marathon/user/rateLimiter#-1542909685]
* Actor[akka://marathon/user/offersWantedForReconciliation#-664527739]
(mesosphere.marathon.core.leadership.impl.LeadershipCoordinatorActor:marathon-akka.actor.default-dispatcher-2)
[2017-11-22 02:09:52,950] INFO Logging initialized @5429ms
(org.eclipse.jetty.util.log:JMX exporting thread)
[2017-11-22 02:09:53,008] INFO Adding HTTP support.
(mesosphere.chaos.http.HttpModule:JMX exporting thread)
[2017-11-22 02:09:53,009] INFO No HTTPS support configured.
(mesosphere.chaos.http.HttpModule:JMX exporting thread)
[2017-11-22 02:09:53,097] INFO Starting up
(mesosphere.marathon.MarathonSchedulerService:MarathonSchedulerService)
[2017-11-22 02:09:53,099] INFO Beginning run
(mesosphere.marathon.MarathonSchedulerService:MarathonSchedulerService)
[2017-11-22 02:09:53,099] INFO MarathonSchedulerService [RUNNING] offered
leadership
(mesosphere.marathon.core.election.impl.CuratorElectionService:MarathonSchedulerService)
[2017-11-22 02:09:53,099] INFO Going to acquire leadership
(mesosphere.marathon.core.election.impl.CuratorElectionService:MarathonSchedulerService)
[2017-11-22 02:09:53,098] INFO jetty-9.3.6.v20151106
(org.eclipse.jetty.server.Server:$anon$1 STARTING)
[2017-11-22 02:09:53,114] INFO Will do leader election through
10.10.10.51:2181
(mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
[2017-11-22 02:09:53,120] INFO Starting
(org.apache.curator.framework.imps.CuratorFrameworkImpl:pool-3-thread-1)
[2017-11-22 02:09:53,121] INFO Initiating client connection,
connectString=10.10.10.51:2181 sessionTimeout=10000
watcher=org.apache.curator.ConnectionState@7e7e09b0
(org.apache.zookeeper.ZooKeeper:pool-3-thread-1)
[2017-11-22 02:09:53,124] INFO Opening socket connection to server
10.10.10.51/10.10.10.51:2181. Will not attempt to authenticate using SASL
(unknown error)
(org.apache.zookeeper.ClientCnxn:pool-3-thread-1-SendThread(10.10.10.51:2181))
[2017-11-22 02:09:53,125] INFO Socket connection established to
10.10.10.51/10.10.10.51:2181, initiating session
(org.apache.zookeeper.ClientCnxn:pool-3-thread-1-SendThread(10.10.10.51:2181))
[2017-11-22 02:09:53,184] INFO Now standing by. Closing existing handles and
rejecting new.
(mesosphere.marathon.core.event.impl.stream.HttpEventStreamActor:marathon-akka.actor.default-dispatcher-2)
[2017-11-22 02:09:53,187] INFO Session establishment complete on server
10.10.10.51/10.10.10.51:2181, sessionid = 0x15fe27c7c3a000b, negotiated timeout
= 10000
(org.apache.zookeeper.ClientCnxn:pool-3-thread-1-SendThread(10.10.10.51:2181))
[2017-11-22 02:09:53,187] INFO State change: CONNECTED
(org.apache.curator.framework.state.ConnectionStateManager:pool-3-thread-1-EventThread)
[2017-11-22 02:09:53,251] INFO Leader elected
(mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
[2017-11-22 02:09:53,252] INFO Starting MarathonSchedulerService [RUNNING]'s
leadership
(mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
[2017-11-22 02:09:53,252] INFO As new leader running the driver
(mesosphere.marathon.MarathonSchedulerService:pool-3-thread-1)
[2017-11-22 02:09:53,391] INFO Registering
com.codahale.metrics.jersey.InstrumentedResourceMethodDispatchAdapter as a
provider class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,398] INFO Registering
mesosphere.marathon.api.MarathonExceptionMapper as a provider class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,399] INFO Registering
mesosphere.marathon.api.SystemResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,399] INFO Registering
mesosphere.marathon.api.v2.AppsResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,399] INFO Registering
mesosphere.marathon.api.v2.PodsResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,400] INFO Registering
mesosphere.marathon.api.v2.TasksResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,400] INFO Registering
mesosphere.marathon.api.v2.QueueResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,400] INFO Registering
mesosphere.marathon.api.v2.GroupsResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,401] INFO Registering
mesosphere.marathon.api.v2.InfoResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,401] INFO Registering
mesosphere.marathon.api.v2.LeaderResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,401] INFO Registering
mesosphere.marathon.api.v2.DeploymentsResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,401] INFO Registering
mesosphere.marathon.api.v2.SchemaResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,401] INFO Registering
mesosphere.marathon.api.v2.PluginsResource as a root resource class
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,414] INFO Initiating Jersey application, version 'Jersey:
1.18.6 10/26/2016 11:30 AM'
(com.sun.jersey.server.impl.application.WebApplicationImpl:$anon$1 STARTING)
[2017-11-22 02:09:53,583] INFO Binding
com.codahale.metrics.jersey.InstrumentedResourceMethodDispatchAdapter to
GuiceManagedComponentProvider with the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:53,632] INFO Binding
mesosphere.marathon.api.MarathonExceptionMapper to
GuiceManagedComponentProvider with the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,577] ERROR Fatal error while starting leadership of
Some(MarathonSchedulerService [RUNNING]). Exiting now
(mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
mesosphere.marathon.StoreCommandFailedException: Migration is already in
progress; /migration-in-progress node already exists
at
mesosphere.marathon.core.storage.store.impl.zk.ZkPersistenceStore$stateMachine$macro$562$1.apply(ZkPersistenceStore.scala:364)
at
mesosphere.marathon.core.storage.store.impl.zk.ZkPersistenceStore$stateMachine$macro$562$1.apply(ZkPersistenceStore.scala:359)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at
mesosphere.marathon.core.async.ContextPropagatingExecutionContext$$anon$1$$anon$2$$anonfun$run$1.apply$mcV$sp(ExecutionContexts.scala:20)
at
mesosphere.marathon.core.async.ContextPropagatingExecutionContext$$anon$1$$anon$2$$anonfun$run$1.apply(ExecutionContexts.scala:20)
at
mesosphere.marathon.core.async.ContextPropagatingExecutionContext$$anon$1$$anon$2$$anonfun$run$1.apply(ExecutionContexts.scala:20)
at
mesosphere.marathon.core.async.package$.propagateContext(package.scala:15)
at
mesosphere.marathon.core.async.ContextPropagatingExecutionContext$$anon$1$$anon$2.run(ExecutionContexts.scala:20)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException:
KeeperErrorCode = NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at
mesosphere.marathon.core.storage.store.impl.zk.ZkFuture.processResult(ZkFuture.scala:30)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:749)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:522)
at
org.apache.curator.framework.imps.CreateBuilderImpl.sendBackgroundResponse(CreateBuilderImpl.java:580)
at
org.apache.curator.framework.imps.CreateBuilderImpl.access$900(CreateBuilderImpl.java:44)
at
org.apache.curator.framework.imps.CreateBuilderImpl$6.processResult(CreateBuilderImpl.java:534)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:613)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[2017-11-22 02:09:54,577] INFO Stopping the election service
(mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
[2017-11-22 02:09:54,590] INFO backgroundOperationsLoop exiting
(org.apache.curator.framework.imps.CuratorFrameworkImpl:Curator-Framework-0)
[2017-11-22 02:09:54,599] INFO Binding mesosphere.marathon.api.SystemResource
to GuiceManagedComponentProvider with the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,631] INFO Binding mesosphere.marathon.api.v2.AppsResource
to GuiceManagedComponentProvider with the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,647] INFO Binding mesosphere.marathon.api.v2.PodsResource
to GuiceManagedComponentProvider with the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,655] INFO Binding
mesosphere.marathon.api.v2.TasksResource to GuiceManagedComponentProvider with
the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,657] INFO Binding
mesosphere.marathon.api.v2.QueueResource to GuiceManagedComponentProvider with
the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,663] INFO Binding
mesosphere.marathon.api.v2.GroupsResource to GuiceManagedComponentProvider with
the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,665] INFO Session: 0x15fe27c7c3a000b closed
(org.apache.zookeeper.ZooKeeper:pool-3-thread-1)
[2017-11-22 02:09:54,665] INFO Stopping MarathonSchedulerService [RUNNING]'s
leadership
(mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
[2017-11-22 02:09:54,665] INFO Lost leadership
(mesosphere.marathon.MarathonSchedulerService:pool-3-thread-1)
[2017-11-22 02:09:54,666] INFO EventThread shut down for session:
0x15fe27c7c3a000b (org.apache.zookeeper.ClientCnxn:pool-3-thread-1-EventThread)
[2017-11-22 02:09:54,667] INFO Binding mesosphere.marathon.api.v2.InfoResource
to GuiceManagedComponentProvider with the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,670] INFO Binding
mesosphere.marathon.api.v2.LeaderResource to GuiceManagedComponentProvider with
the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,673] INFO Stopped MarathonSchedulerService [RUNNING]'s
leadership
(mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
[2017-11-22 02:09:54,673] INFO Terminating due to leadership abdication or
failure
(mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
[2017-11-22 02:09:54,673] INFO Now standing by. Closing existing handles and
rejecting new.
(mesosphere.marathon.core.event.impl.stream.HttpEventStreamActor:marathon-akka.actor.default-dispatcher-4)
[2017-11-22 02:09:54,674] INFO Binding
mesosphere.marathon.api.v2.DeploymentsResource to GuiceManagedComponentProvider
with the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,681] INFO Binding
mesosphere.marathon.api.v2.SchemaResource to GuiceManagedComponentProvider with
the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
[2017-11-22 02:09:54,686] INFO Binding
mesosphere.marathon.api.v2.PluginsResource to GuiceManagedComponentProvider
with the scope "Singleton"
(com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory:$anon$1
STARTING)
