Maybe you could post your executor stdout/stderr so that we could know whether health check running not.
On Wed, Oct 7, 2015 at 2:15 PM, haosdent <[email protected]> wrote: > marathon also use mesos health check. When I use health check, I could saw > the log like this in executor stdout. > > ``` > Registered docker executor on xxxxx > Starting task test-health-check.822a5fd2-6cba-11e5-b5ce-0a0027000000 > Launching health check process: > /home/haosdent/mesos/build/src/.libs/mesos-health-check --executor=xxxx > Health check process launched at pid: 9895 > Received task health update, healthy: true > ``` > > On Wed, Oct 7, 2015 at 12:51 PM, Jay Taylor <[email protected]> wrote: > >> I am using my own framework, and the full task info I'm using is posted >> earlier in this thread. Do you happen to know if Marathon uses Mesos's >> health checks for its health check system? >> >> >> >> On Oct 6, 2015, at 9:01 PM, haosdent <[email protected]> wrote: >> >> Yes, launch the health task through its definition in taskinfo. Do you >> launch your task through Marathon? I could test it in my side. >> >> On Wed, Oct 7, 2015 at 11:56 AM, Jay Taylor <[email protected]> wrote: >> >>> Precisely, and there are none of those statements. Are you or others >>> confident health-checks are part of the code path when defined via task >>> info for docker container tasks? Going through the code, I wasn't able to >>> find the linkage for anything other than health-checks triggered through a >>> custom executor. >>> >>> With that being said it is a pretty good sized code base and I'm not >>> very familiar with it, so my analysis this far has by no means been >>> exhaustive. >>> >>> >>> >>> On Oct 6, 2015, at 8:41 PM, haosdent <[email protected]> wrote: >>> >>> When health check launch, it would have a log like this in your executor >>> stdout >>> ``` >>> Health check process launched at pid xxx >>> ``` >>> >>> On Wed, Oct 7, 2015 at 11:37 AM, Jay Taylor <[email protected]> wrote: >>> >>>> I'm happy to try this, however wouldn't there be output in the logs >>>> with the string "health" or "Health" if the health-check were active? None >>>> of my master or slave logs contain the string.. >>>> >>>> >>>> >>>> On Oct 6, 2015, at 7:45 PM, haosdent <[email protected]> wrote: >>>> >>>> Could you use "exit 1" instead of "sleep 5" to see whether could see >>>> unhealthy status in your task stdout/stderr. >>>> >>>> On Wed, Oct 7, 2015 at 10:38 AM, Jay Taylor <[email protected]> >>>> wrote: >>>> >>>>> My current version is 0.24.1. >>>>> >>>>> On Tue, Oct 6, 2015 at 7:30 PM, haosdent <[email protected]> wrote: >>>>> >>>>>> yes, adam also help commit it to 0.23.1 and 0.24.1 >>>>>> https://github.com/apache/mesos/commit/8c0ed92de3925d4312429bfba01b9b1ccbcbbef0 >>>>>> >>>>>> https://github.com/apache/mesos/commit/09e367cd69aa39c156c9326d44f4a7b829ba3db7 >>>>>> Are you use one of this version? >>>>>> >>>>>> On Wed, Oct 7, 2015 at 10:26 AM, haosdent <[email protected]> wrote: >>>>>> >>>>>>> I remember 0.23.1 and 0.24.1 contains this backport, let me double >>>>>>> check. >>>>>>> >>>>>>> On Wed, Oct 7, 2015 at 10:01 AM, Jay Taylor <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Oops- Now I see you already said it's in master. I'll look there :) >>>>>>>> >>>>>>>> Thanks again! >>>>>>>> >>>>>>>> On Tue, Oct 6, 2015 at 6:59 PM, Jay Taylor <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Great, thanks for the quick reply Tim! >>>>>>>>> >>>>>>>>> Do you know if there is a branch I can checkout to test it out? >>>>>>>>> >>>>>>>>> On Tue, Oct 6, 2015 at 6:54 PM, Timothy Chen <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Jay, >>>>>>>>>> >>>>>>>>>> We just added health check support for docker tasks that's in >>>>>>>>>> master but not yet released. It will run docker exec with the >>>>>>>>>> command you >>>>>>>>>> provided as health checks. >>>>>>>>>> >>>>>>>>>> It should be in the next release. >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> Tim >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Oct 6, 2015, at 6:49 PM, Jay Taylor <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Does Mesos support health checks for docker image tasks? Mesos >>>>>>>>>> seems to be ignoring the TaskInfo.HealthCheck field for me. >>>>>>>>>> >>>>>>>>>> Example TaskInfo JSON received back from Mesos: >>>>>>>>>> >>>>>>>>>> { >>>>>>>>>>> >>>>>>>>>>> "name":"hello-app.web.v3", >>>>>>>>>>> >>>>>>>>>>> "task_id":{ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> "value":"hello-app_web-v3.fc05a1a5-1e06-4e61-9879-be0d97cd3eec" >>>>>>>>>>> >>>>>>>>>>> }, >>>>>>>>>>> >>>>>>>>>>> "slave_id":{ >>>>>>>>>>> >>>>>>>>>>> "value":"20150924-210922-1608624320-5050-1792-S1" >>>>>>>>>>> >>>>>>>>>>> }, >>>>>>>>>>> >>>>>>>>>>> "resources":[ >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> >>>>>>>>>>> "name":"cpus", >>>>>>>>>>> >>>>>>>>>>> "type":0, >>>>>>>>>>> >>>>>>>>>>> "scalar":{ >>>>>>>>>>> >>>>>>>>>>> "value":0.1 >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> }, >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> >>>>>>>>>>> "name":"mem", >>>>>>>>>>> >>>>>>>>>>> "type":0, >>>>>>>>>>> >>>>>>>>>>> "scalar":{ >>>>>>>>>>> >>>>>>>>>>> "value":256 >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> }, >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> >>>>>>>>>>> "name":"ports", >>>>>>>>>>> >>>>>>>>>>> "type":1, >>>>>>>>>>> >>>>>>>>>>> "ranges":{ >>>>>>>>>>> >>>>>>>>>>> "range":[ >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> >>>>>>>>>>> "begin":31002, >>>>>>>>>>> >>>>>>>>>>> "end":31002 >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> ] >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> ], >>>>>>>>>>> >>>>>>>>>>> "command":{ >>>>>>>>>>> >>>>>>>>>>> "container":{ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> "image":"docker-services1a:5000/test/app-81-1-hello-app-103" >>>>>>>>>>> >>>>>>>>>>> }, >>>>>>>>>>> >>>>>>>>>>> "shell":false >>>>>>>>>>> >>>>>>>>>>> }, >>>>>>>>>>> >>>>>>>>>>> "container":{ >>>>>>>>>>> >>>>>>>>>>> "type":1, >>>>>>>>>>> >>>>>>>>>>> "docker":{ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> "image":"docker-services1a:5000/gig1/app-81-1-hello-app-103", >>>>>>>>>>> >>>>>>>>>>> "network":2, >>>>>>>>>>> >>>>>>>>>>> "port_mappings":[ >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> >>>>>>>>>>> "host_port":31002, >>>>>>>>>>> >>>>>>>>>>> "container_port":8000, >>>>>>>>>>> >>>>>>>>>>> "protocol":"tcp" >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> ], >>>>>>>>>>> >>>>>>>>>>> "privileged":false, >>>>>>>>>>> >>>>>>>>>>> "parameters":[], >>>>>>>>>>> >>>>>>>>>>> "force_pull_image":false >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> }, >>>>>>>>>>> >>>>>>>>>>> "health_check":{ >>>>>>>>>>> >>>>>>>>>>> "delay_seconds":5, >>>>>>>>>>> >>>>>>>>>>> "interval_seconds":10, >>>>>>>>>>> >>>>>>>>>>> "timeout_seconds":10, >>>>>>>>>>> >>>>>>>>>>> "consecutive_failures":3, >>>>>>>>>>> >>>>>>>>>>> "grace_period_seconds":0, >>>>>>>>>>> >>>>>>>>>>> "command":{ >>>>>>>>>>> >>>>>>>>>>> "shell":true, >>>>>>>>>>> >>>>>>>>>>> "value":"sleep 5", >>>>>>>>>>> >>>>>>>>>>> "user":"root" >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> I have searched all machines and containers to see if they ever >>>>>>>>>> run the command (in this case `sleep 5`), but have not found any >>>>>>>>>> indication >>>>>>>>>> that it is being executed. >>>>>>>>>> >>>>>>>>>> In the mesos src code the health-checks are invoked from >>>>>>>>>> src/launcher/executor.cpp CommandExecutorProcess::launchTask. Does >>>>>>>>>> this >>>>>>>>>> mean that health-checks are only supported for custom executors and >>>>>>>>>> not for >>>>>>>>>> docker tasks? >>>>>>>>>> >>>>>>>>>> What I am trying to accomplish is to have the 0/non-zero >>>>>>>>>> exit-status of a health-check command translate to task health. >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> Jay >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, >>>>>>> Haosdent Huang >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Haosdent Huang >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Haosdent Huang >>>> >>>> >>> >>> >>> -- >>> Best Regards, >>> Haosdent Huang >>> >>> >> >> >> -- >> Best Regards, >> Haosdent Huang >> >> > > > -- > Best Regards, > Haosdent Huang > -- Best Regards, Haosdent Huang
