[
https://issues.apache.org/jira/browse/MESOS-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Chen closed MESOS-2252.
-------------------------------
Resolution: Fixed
> Docker containers fail to start with "future discarded" error
> -------------------------------------------------------------
>
> Key: MESOS-2252
> URL: https://issues.apache.org/jira/browse/MESOS-2252
> Project: Mesos
> Issue Type: Bug
> Components: docker, slave
> Affects Versions: 0.21.0
> Environment: Mesos slaves in containers, image
> mesosphere/mesos-slave:0.21.0-1.0.ubuntu1404 on docker hub. Docker 1.4.1,
> marathon 0.8.0-SNAPSHOT
> Reporter: Ian Babrou
> Labels: docker, executors, slave
>
> I tried to launch my dockerized app with 50 tasks on marathon and all tasks
> failed to run. Usually app works just fine.
> Backstory:
> https://github.com/mesosphere/marathon/issues/1083#issuecomment-71196704
> Marathon logs:
> [2015-01-23 13:22:30,163] INFO Starting app /topface/prod-test/app
> (mesosphere.marathon.SchedulerActions:363)
> [2015-01-23 13:22:30,165] INFO Already running 0 instances of
> /topface/prod-test/app. Not scaling.
> (mesosphere.marathon.SchedulerActions:512)
> [2015-01-23 13:22:35,339] INFO Received status update for task
> topface_prod-test_app.e2baeae4-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> [2015-01-23 13:22:35,367] INFO Task
> topface_prod-test_app.e2baeae4-a302-11e4-bea0-56847afe9799 expunged and
> removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)
> [2015-01-23 13:22:35,368] INFO Sending event notification.
> (mesosphere.marathon.MarathonScheduler:262)
> [2015-01-23 13:22:35,369] INFO Task launch delay for [/topface/prod-test/app]
> is now [999483319 nanoseconds] (mesosphere.util.RateLimiter:35)
> [2015-01-23 13:22:45,345] INFO Received status update for task
> topface_prod-test_app.e8f945c2-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> [2015-01-23 13:22:45,359] INFO Task
> topface_prod-test_app.e8f945c2-a302-11e4-bea0-56847afe9799 expunged and
> removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)
> [2015-01-23 13:22:45,360] INFO Sending event notification.
> (mesosphere.marathon.MarathonScheduler:262)
> [2015-01-23 13:22:45,360] INFO Task launch delay for [/topface/prod-test/app]
> is now [999838313 nanoseconds] (mesosphere.util.RateLimiter:35)
> [2015-01-23 13:23:31,942] INFO Received status update for task
> topface_prod-test_app.e2bcbfae-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> [2015-01-23 13:23:31,946] INFO Task launch delay for [/topface/prod-test/app]
> is now [1149948119 nanoseconds] (mesosphere.util.RateLimiter:35)
> [2015-01-23 13:23:31,946] INFO Task
> topface_prod-test_app.e2bcbfae-a302-11e4-bea0-56847afe9799 expunged and
> removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)
> [2015-01-23 13:23:31,946] INFO Sending event notification.
> (mesosphere.marathon.MarathonScheduler:262)
> [2015-01-23 13:23:31,955] INFO Received status update for task
> topface_prod-test_app.e2c460d9-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> [2015-01-23 13:23:31,957] INFO Task launch delay for [/topface/prod-test/app]
> is now [1321950877 nanoseconds] (mesosphere.util.RateLimiter:35)
> [2015-01-23 13:23:31,958] INFO Task
> topface_prod-test_app.e2c460d9-a302-11e4-bea0-56847afe9799 expunged and
> removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)
> [2015-01-23 13:23:31,958] INFO Sending event notification.
> (mesosphere.marathon.MarathonScheduler:262)
> [2015-01-23 13:23:31,958] INFO Received status update for task
> topface_prod-test_app.e2bb3906-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> [2015-01-23 13:23:31,960] INFO Task launch delay for [/topface/prod-test/app]
> is now [1519954162 nanoseconds] (mesosphere.util.RateLimiter:35)
> [2015-01-23 13:23:31,960] INFO Task
> topface_prod-test_app.e2bb3906-a302-11e4-bea0-56847afe9799 expunged and
> removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)
> [2015-01-23 13:23:31,961] INFO Sending event notification.
> (mesosphere.marathon.MarathonScheduler:262)
> [2015-01-23 13:23:31,961] INFO Received status update for task
> topface_prod-test_app.e2c30146-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> [2015-01-23 13:23:31,963] INFO Task launch delay for [/topface/prod-test/app]
> is now [1746973326 nanoseconds] (mesosphere.util.RateLimiter:35)
> [2015-01-23 13:23:31,970] INFO Task
> topface_prod-test_app.e2c30146-a302-11e4-bea0-56847afe9799 expunged and
> removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)
> [2015-01-23 13:23:31,970] INFO Sending event notification.
> (mesosphere.marathon.MarathonScheduler:262)
> [2015-01-23 13:23:31,970] INFO Received status update for task
> topface_prod-test_app.e2ba9cc2-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> [2015-01-23 13:23:31,973] INFO Task
> topface_prod-test_app.e2ba9cc2-a302-11e4-bea0-56847afe9799 expunged and
> removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)
> [2015-01-23 13:23:31,973] INFO Task launch delay for [/topface/prod-test/app]
> is now [2008991202 nanoseconds] (mesosphere.util.RateLimiter:35)
> [2015-01-23 13:23:31,973] INFO Sending event notification.
> (mesosphere.marathon.MarathonScheduler:262)
> [2015-01-23 13:23:31,973] INFO Received status update for task
> topface_prod-test_app.e2bc4a7c-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> [2015-01-23 13:23:31,975] INFO Task launch delay for [/topface/prod-test/app]
> is now [2309993195 nanoseconds] (mesosphere.util.RateLimiter:35)
> [2015-01-23 13:23:31,976] INFO Task
> topface_prod-test_app.e2bc4a7c-a302-11e4-bea0-56847afe9799 expunged and
> removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)
> [2015-01-23 13:23:31,976] INFO Sending event notification.
> (mesosphere.marathon.MarathonScheduler:262)
> [2015-01-23 13:23:31,976] INFO Received status update for task
> topface_prod-test_app.e2bb11f5-a302-11e4-bea0-56847afe9799: TASK_FAILED
> (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)
> First task failed to start because of network setup (docker registry was
> unavailable). Second task ended up on the same host and failed as well:
> E0123 13:22:35.287389 13 slave.cpp:2787] Container
> '0a1225ce-98bd-4f83-a417-b7cf72bb90e8' for executor
> 'topface_prod-test_app.e2baeae4-a302-11e4-bea0-56847afe9799' of framework
> '20141003-172543-3892422848-5050-1-0000' failed to start: Failed to 'docker
> pull docker.core.tf/topface-prod-app:20150123019': exit status = exited with
> status 1 stderr = time="2015-01-23T13:22:35Z" level="fatal" msg="Error:
> Invalid registry endpoint https://docker.core.tf/v1/: Get
> https://docker.core.tf/v1/_ping: dial tcp 10.5.1.194:443: connection timed
> out. If this private registry supports only HTTP or HTTPS with an unknown CA
> certificate, please add `--insecure-registry docker.core.tf` to the daemon's
> arguments. In the case of HTTPS, if you have access to the registry's CA
> certificate, no need for the flag; simply place the CA certificate at
> /etc/docker/certs.d/docker.core.tf/ca.crt"
> E0123 13:22:35.303208 13 slave.cpp:2882] Termination of executor
> 'topface_prod-test_app.e2baeae4-a302-11e4-bea0-56847afe9799' of framework
> '20141003-172543-3892422848-5050-1-0000' failed: Unknown container:
> 0a1225ce-98bd-4f83-a417-b7cf72bb90e8
> E0123 13:22:35.303503 6 slave.cpp:3134] Failed to unmonitor container for
> executor topface_prod-test_app.e2baeae4-a302-11e4-bea0-56847afe9799 of
> framework 20141003-172543-3892422848-5050-1-0000: Not monitored
> W0123 13:22:35.304908 11 docker.cpp:1184] Ignoring updating unknown
> container: 0a1225ce-98bd-4f83-a417-b7cf72bb90e8
> E0123 13:22:45.330379 12 slave.cpp:2787] Container
> '60a2fe62-4d64-4594-b1be-7e5795d6323c' for executor
> 'topface_prod-test_app.e8f945c2-a302-11e4-bea0-56847afe9799' of framework
> '20141003-172543-3892422848-5050-1-0000' failed to start: Failed to 'docker
> pull docker.core.tf/topface-prod-app:20150123019': exit status = exited with
> status 1 stderr = time="2015-01-23T13:22:45Z" level="fatal" msg="Error:
> Invalid registry endpoint https://docker.core.tf/v1/: Get
> https://docker.core.tf/v1/_ping: dial tcp 10.5.1.194:443: connection timed
> out. If this private registry supports only HTTP or HTTPS with an unknown CA
> certificate, please add `--insecure-registry docker.core.tf` to the daemon's
> arguments. In the case of HTTPS, if you have access to the registry's CA
> certificate, no need for the flag; simply place the CA certificate at
> /etc/docker/certs.d/docker.core.tf/ca.crt"
> E0123 13:22:45.330746 12 slave.cpp:2882] Termination of executor
> 'topface_prod-test_app.e8f945c2-a302-11e4-bea0-56847afe9799' of framework
> '20141003-172543-3892422848-5050-1-0000' failed: Unknown container:
> 60a2fe62-4d64-4594-b1be-7e5795d6323c
> E0123 13:22:45.340802 9 slave.cpp:3134] Failed to unmonitor container for
> executor topface_prod-test_app.e8f945c2-a302-11e4-bea0-56847afe9799 of
> framework 20141003-172543-3892422848-5050-1-0000: Not monitored
> W0123 13:22:45.342725 11 docker.cpp:1184] Ignoring updating unknown
> container: 60a2fe62-4d64-4594-b1be-7e5795d6323c
> Third task failed because of future discarded error:
> E0123 13:23:31.906733 12 slave.cpp:2787] Container
> 'bd0337a2-41f4-4308-85a9-68a3ff0475e6' for executor
> 'topface_prod-test_app.e2bcbfae-a302-11e4-bea0-56847afe9799' of framework
> '20141003-172543-3892422848-5050-1-0000' failed to start: future discarded
> E0123 13:23:31.907039 12 slave.cpp:2882] Termination of executor
> 'topface_prod-test_app.e2bcbfae-a302-11e4-bea0-56847afe9799' of framework
> '20141003-172543-3892422848-5050-1-0000' failed: Unknown container:
> bd0337a2-41f4-4308-85a9-68a3ff0475e6
> E0123 13:23:31.907260 7 slave.cpp:3134] Failed to unmonitor container for
> executor topface_prod-test_app.e2bcbfae-a302-11e4-bea0-56847afe9799 of
> framework 20141003-172543-3892422848-5050-1-0000: Not monitored
> Fourth task failed because of future discarded error too:
> E0123 13:23:31.932677 8 slave.cpp:2787] Container
> '782c163a-9238-4f3b-b9fd-dcc50579322a' for executor
> 'topface_prod-test_app.e2c460d9-a302-11e4-bea0-56847afe9799' of framework
> '20141003-172543-3892422848-5050-1-0000' failed to start: future discarded
> E0123 13:23:31.933078 8 slave.cpp:2882] Termination of executor
> 'topface_prod-test_app.e2c460d9-a302-11e4-bea0-56847afe9799' of framework
> '20141003-172543-3892422848-5050-1-0000' failed: Unknown container:
> 782c163a-9238-4f3b-b9fd-dcc50579322a
> E0123 13:23:31.967974 6 slave.cpp:3134] Failed to unmonitor container for
> executor topface_prod-test_app.e2c460d9-a302-11e4-bea0-56847afe9799 of
> framework 20141003-172543-3892422848-5050-1-0000: Not monitored
> I think this "future discarded" thing should be fixed. Ideally more
> understandable error message should be introduced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
