Are those the stdout logs of the Agent? Because I don't see the --launcher-dir set, however, if I look into one that is running off the same 0.24.1 package, this is what I see:
I1012 14:56:36.933856 1704 slave.cpp:191] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --attributes="rack:r2d2;pod:demo,dev" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="docker,mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --initialize_driver_logging="true" --ip="192.168.33.11" --isolation="cgroups/cpu,cgroups/mem" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/local/mesos/logs/agent" --logbufsecs="0" --logging_level="INFO" --master="zk://192.168.33.1:2181/mesos/vagrant" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="ports:[9000-10000];ephemeral_ports:[32768-57344]" --revocable_cpu_low_priority="true" --sandbox_directory="/var/local/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/var/local/mesos/agent" (this is run off the Vagrantfile at [0] in case you want to reproduce). That agent is not run via the init command, though, I execute it manually via the `run-agent.sh` in the same directory. I don't really think this matters, but I assume you also restarted the agent after making the config changes? (and, for your own sanity - you can double check the version by looking at the very head of the logs). -- *Marco Massenzio* Distributed Systems Engineer http://codetrips.com On Mon, Oct 12, 2015 at 10:50 PM, Jay Taylor <[email protected]> wrote: > Hi Haosdent and Mesos friends, > > I've rebuilt the cluster from scratch and installed mesos 0.24.1 from the > mesosphere apt repo: > > $ dpkg -l | grep mesos > ii mesos 0.24.1-0.2.35.ubuntu1404 > amd64 Cluster resource manager with efficient resource isolation > > Then added the `launcher_dir' flag to /etc/mesos-slave/launcher_dir on the > slaves: > > mesos-worker1a:~$ cat /etc/mesos-slave/launcher_dir > /usr/libexec/mesos > > And yet the task health-checks are still being launched from the sandbox > directory like before! > > I've also tested setting the MESOS_LAUNCHER_DIR env var and get the > identical result (just as before on the cluster where many versions of > mesos had been installed): > > STDOUT: > > --container="mesos-20151012-184440-1625401536-5050-23953-S0.62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb" >> --docker="docker" --help="false" --initialize_driver_logging="true" >> --logbufsecs="0" --logging_level="INFO" >> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >> --sandbox_directory="/tmp/mesos/slaves/20151012-184440-1625401536-5050-23953-S0/frameworks/20151012-184440-1625401536-5050-23953-0000/executors/hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91/runs/62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb" >> --stop_timeout="0ns" >> --container="mesos-20151012-184440-1625401536-5050-23953-S0.62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb" >> --docker="docker" --help="false" --initialize_driver_logging="true" >> --logbufsecs="0" --logging_level="INFO" >> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >> --sandbox_directory="/tmp/mesos/slaves/20151012-184440-1625401536-5050-23953-S0/frameworks/20151012-184440-1625401536-5050-23953-0000/executors/hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91/runs/62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb" >> --stop_timeout="0ns" >> Registered docker executor on mesos-worker1a >> Starting task hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91 >> Launching health check process: >> /tmp/mesos/slaves/20151012-184440-1625401536-5050-23953-S0/frameworks/20151012-184440-1625401536-5050-23953-0000/executors/hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91/runs/62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb/mesos-health-check >> --executor=(1)@192.168.225.58:48912 >> --health_check_json={"command":{"shell":true,"value":"docker exec >> mesos-20151012-184440-1625401536-5050-23953-S0.62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb >> sh -c \" curl --silent --show-error --fail --tcp-nodelay --head -X GET >> --user-agent flux-capacitor-health-checker --max-time 1 http:\/\/ >> 127.0.0.1:8000 >> \""},"consecutive_failures":6,"delay_seconds":15,"grace_period_seconds":10,"interval_seconds":1,"timeout_seconds":1} >> --task_id=hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91 >> Health check process launched at pid: 11253 > > > > STDERR: > > --container="mesos-20151012-184440-1625401536-5050-23953-S0.62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb" >> --docker="docker" --help="false" --initialize_driver_logging="true" >> --logbufsecs="0" --logging_level="INFO" >> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >> --sandbox_directory="/tmp/mesos/slaves/20151012-184440-1625401536-5050-23953-S0/frameworks/20151012-184440-1625401536-5050-23953-0000/executors/hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91/runs/62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb" >> --stop_timeout="0ns" >> --container="mesos-20151012-184440-1625401536-5050-23953-S0.62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb" >> --docker="docker" --help="false" --initialize_driver_logging="true" >> --logbufsecs="0" --logging_level="INFO" >> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >> --sandbox_directory="/tmp/mesos/slaves/20151012-184440-1625401536-5050-23953-S0/frameworks/20151012-184440-1625401536-5050-23953-0000/executors/hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91/runs/62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb" >> --stop_timeout="0ns" >> Registered docker executor on mesos-worker1a >> Starting task hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91 >> *Launching health check process: >> /tmp/mesos/slaves/20151012-184440-1625401536-5050-23953-S0/frameworks/20151012-184440-1625401536-5050-23953-0000/executors/hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91/runs/62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb/mesos-health-check* >> --executor=(1)@192.168.225.58:48912 >> --health_check_json={"command":{"shell":true,"value":"docker exec >> mesos-20151012-184440-1625401536-5050-23953-S0.62d43b8f-6cd1-4c53-9ac8-84dbfc45bbcb >> sh -c \" curl --silent --show-error --fail --tcp-nodelay --head -X GET >> --user-agent flux-capacitor-health-checker --max-time 1 http:\/\/ >> 127.0.0.1:8000 >> \""},"consecutive_failures":6,"delay_seconds":15,"grace_period_seconds":10,"interval_seconds":1,"timeout_seconds":1} >> --task_id=hello-app_web-v3.33597b73-1943-41b4-a308-76132eebcc91 >> Health check process launched at pid: 11253 > > > Any ideas on where to go from here? Is there any additional information I > can provide? > > Thanks as always, > Jay > > > On Thu, Oct 8, 2015 at 9:23 PM, haosdent <[email protected]> wrote: > >> For flag sent to the executor from containerizer, the flag would >> stringify and become a command line parameter when launch executor. >> >> You could see this in >> https://github.com/apache/mesos/blob/master/src/slave/containerizer/docker.cpp#L279-L288 >> >> But for launcher_dir, the executor get it from `argv[0]`, as you >> mentioned above. >> ``` >> string path = >> envPath.isSome() ? envPath.get() >> : os::realpath(Path(argv[0]).dirname()).get(); >> >> ``` >> So I want to figure out why your argv[0] would become sandbox dir, not >> "/usr/libexec/mesos". >> >> On Fri, Oct 9, 2015 at 12:03 PM, Jay Taylor <[email protected]> wrote: >> >>> I see. And then how are the flags sent to the executor? >>> >>> >>> >>> On Oct 8, 2015, at 8:56 PM, haosdent <[email protected]> wrote: >>> >>> Yes. The related code is located in >>> https://github.com/apache/mesos/blob/master/src/slave/main.cpp#L123 >>> >>> In fact, environment variables starts with MESOS_ would load as flags >>> variables. >>> >>> https://github.com/apache/mesos/blob/master/3rdparty/libprocess/3rdparty/stout/include/stout/flags/flags.hpp#L52 >>> >>> On Fri, Oct 9, 2015 at 11:33 AM, Jay Taylor <[email protected]> wrote: >>> >>>> One question for you haosdent- >>>> >>>> You mentioned that the flags.launcher_dir should propagate to the >>>> docker executor all the way up the chain. Can you show me where this logic >>>> is in the codebase? I didn't see where that was happening and would like >>>> to understand the mechanism. >>>> >>>> Thanks! >>>> Jay >>>> >>>> >>>> >>>> On Oct 8, 2015, at 8:29 PM, Jay Taylor <[email protected]> wrote: >>>> >>>> Maybe tomorrow I will build a fresh cluster from scratch to see if the >>>> broken behavior experienced today still persists. >>>> >>>> On Oct 8, 2015, at 7:52 PM, haosdent <[email protected]> wrote: >>>> >>>> As far as I know, MESOS_LAUNCHER_DIR is works by set flags.launcher_dir >>>> which would find mesos-docker-executor and mesos-health-check under this >>>> dir. Although the env is not propagated, but MESOS_LAUNCHER_DIR still >>>> works because flags.launcher_dir is get from it. >>>> >>>> For example, because I >>>> ``` >>>> export MESOS_LAUNCHER_DIR=/tmp >>>> ``` >>>> before start mesos-slave. So when I launch slave, I could find this log >>>> in slave log >>>> ``` >>>> I1009 10:27:26.594599 1416 slave.cpp:203] Flags at startup: >>>> xxxxx --launcher_dir="/tmp" >>>> ``` >>>> >>>> And from your log, I not sure why your MESOS_LAUNCHER_DIR become >>>> sandbox dir. Is it because MESOS_LAUNCHER_DIR is overrided in your other >>>> scripts? >>>> >>>> >>>> On Fri, Oct 9, 2015 at 1:56 AM, Jay Taylor <[email protected]> wrote: >>>> >>>>> I haven't ever changed MESOS_LAUNCHER_DIR/--launcher_dir before. >>>>> >>>>> I just tried setting both the env var and flag on the slaves, and have >>>>> determined that the env var is not present when it is being checked >>>>> src/docker/executor.cpp @ line 573: >>>>> >>>>> const Option<string> envPath = os::getenv("MESOS_LAUNCHER_DIR"); >>>>>> string path = >>>>>> envPath.isSome() ? envPath.get() >>>>>> : os::realpath(Path(argv[0]).dirname()).get(); >>>>>> cout << "MESOS_LAUNCHER_DIR: envpath.isSome()->" << >>>>>> (envPath.isSome() ? "yes" : "no") << endl; >>>>>> cout << "MESOS_LAUNCHER_DIR: path='" << path << "'" << endl; >>>>> >>>>> >>>>> Exported MESOS_LAUNCHER_DIR env var (and verified it is correctly >>>>> propagated along up to the point of mesos-slave launch): >>>>> >>>>> $ cat /etc/default/mesos-slave >>>>>> export >>>>>> MESOS_MASTER="zk://mesos-primary1a:2181,mesos-primary2a:2181,mesos-primary3a:2181/mesos" >>>>>> export MESOS_CONTAINERIZERS="mesos,docker" >>>>>> export MESOS_EXECUTOR_REGISTRATION_TIMEOUT="5mins" >>>>>> export MESOS_PORT="5050" >>>>>> export MESOS_LAUNCHER_DIR="/usr/libexec/mesos" >>>>> >>>>> >>>>> TASK OUTPUT: >>>>> >>>>> >>>>>> *MESOS_LAUNCHER_DIR: envpath.isSome()->no**MESOS_LAUNCHER_DIR: >>>>>> path='/tmp/mesos/slaves/61373c0e-7349-4173-ab8d-9d7b260e8a30-S1/frameworks/20150924-210922-1608624320-5050-1792-0020/executors/hello-app_web-v3.22f9c7e4-2109-48a9-998e-e116141ec253/runs/41f8eed6-ec6c-4e6f-b1aa-0a2817a600ad'* >>>>>> Registered docker executor on mesos-worker2a >>>>>> Starting task hello-app_web-v3.22f9c7e4-2109-48a9-998e-e116141ec253 >>>>>> Launching health check process: >>>>>> /tmp/mesos/slaves/61373c0e-7349-4173-ab8d-9d7b260e8a30-S1/frameworks/20150924-210922-1608624320-5050-1792-0020/executors/hello-app_web-v3.22f9c7e4-2109-48a9-998e-e116141ec253/runs/41f8eed6-ec6c-4e6f-b1aa-0a2817a600ad/mesos-health-check >>>>>> --executor=(1)@192.168.225.59:44523 >>>>>> --health_check_json={"command":{"shell":true,"value":"docker exec >>>>>> mesos-61373c0e-7349-4173-ab8d-9d7b260e8a30-S1.41f8eed6-ec6c-4e6f-b1aa-0a2817a600ad >>>>>> sh -c \" \/bin\/bash >>>>>> \""},"consecutive_failures":3,"delay_seconds":5.0,"grace_period_seconds":10.0,"interval_seconds":10.0,"timeout_seconds":10.0} >>>>>> --task_id=hello-app_web-v3.22f9c7e4-2109-48a9-998e-e116141ec253 >>>>>> Health check process launched at pid: 2519 >>>>> >>>>> >>>>> The env var is not propagated when the docker executor is launched >>>>> in src/slave/containerizer/docker.cpp around line 903: >>>>> >>>>> vector<string> argv; >>>>>> argv.push_back("mesos-docker-executor"); >>>>>> // Construct the mesos-docker-executor using the "name" we gave the >>>>>> // container (to distinguish it from Docker containers not created >>>>>> // by Mesos). >>>>>> Try<Subprocess> s = subprocess( >>>>>> path::join(flags.launcher_dir, "mesos-docker-executor"), >>>>>> argv, >>>>>> Subprocess::PIPE(), >>>>>> Subprocess::PATH(path::join(container->directory, "stdout")), >>>>>> Subprocess::PATH(path::join(container->directory, "stderr")), >>>>>> dockerFlags(flags, container->name(), container->directory), >>>>>> environment, >>>>>> lambda::bind(&setup, container->directory)); >>>>> >>>>> >>>>> A little ways above we can see the environment is setup w/ the >>>>> container tasks defined env vars. >>>>> >>>>> See src/slave/containerizer/docker.cpp around line 871: >>>>> >>>>> // Include any enviroment variables from ExecutorInfo. >>>>>> foreach (const Environment::Variable& variable, >>>>>> container->executor.command().environment().variables()) { >>>>>> environment[variable.name()] = variable.value(); >>>>>> } >>>>> >>>>> >>>>> Should I file a JIRA for this? Have I overlooked anything? >>>>> >>>>> >>>>> On Wed, Oct 7, 2015 at 8:11 PM, haosdent <[email protected]> wrote: >>>>> >>>>>> >Not sure what was going on with health-checks in 0.24.0. >>>>>> 0.24.1 should be works. >>>>>> >>>>>> >Do any of you know which host the path >>>>>> "/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check" >>>>>> should exist on? It definitely doesn't exist on the slave, hence >>>>>> execution >>>>>> failing. >>>>>> >>>>>> Does you set MESOS_LAUNCHER_DIR/--launcher_dir incorrectly before? We >>>>>> got mesos-health-check from MESOS_LAUNCHER_DIR/--launcher_id or use the >>>>>> same dir of mesos-docker-executor. >>>>>> >>>>>> On Thu, Oct 8, 2015 at 10:46 AM, Jay Taylor <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Maybe I spoke too soon. >>>>>>> >>>>>>> Now the checks are attempting to run, however the STDERR is not >>>>>>> looking good. I've added some debugging to the error message output to >>>>>>> show the path, argv, and envp variables: >>>>>>> >>>>>>> STDOUT: >>>>>>> >>>>>>> --container="mesos-16b49e90-6852-4c91-8e70-d89c54f25668-S1.73dbfe88-1dbb-4f61-9a52-c365558cdbfc" >>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>>>>>> --initialize_driver_logging="true" --logbufsecs="0" >>>>>>>> --logging_level="INFO" >>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>>>> --sandbox_directory="/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc" >>>>>>>> --stop_timeout="0ns" >>>>>>>> --container="mesos-16b49e90-6852-4c91-8e70-d89c54f25668-S1.73dbfe88-1dbb-4f61-9a52-c365558cdbfc" >>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>>>>>> --initialize_driver_logging="true" --logbufsecs="0" >>>>>>>> --logging_level="INFO" >>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>>>> --sandbox_directory="/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc" >>>>>>>> --stop_timeout="0ns" >>>>>>>> Registered docker executor on mesos-worker2a >>>>>>>> Starting task >>>>>>>> app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0 >>>>>>>> Launching health check process: >>>>>>>> /tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check >>>>>>>> --executor=(1)@192.168.225.59:43917 >>>>>>>> --health_check_json={"command":{"shell":true,"value":"docker exec >>>>>>>> mesos-16b49e90-6852-4c91-8e70-d89c54f25668-S1.73dbfe88-1dbb-4f61-9a52-c365558cdbfc >>>>>>>> sh -c \" exit 1 >>>>>>>> \""},"consecutive_failures":3,"delay_seconds":0.0,"grace_period_seconds":10.0,"interval_seconds":10.0,"timeout_seconds":10.0} >>>>>>>> --task_id=app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0 >>>>>>>> Health check process launched at pid: 3012 >>>>>>> >>>>>>> >>>>>>> STDERR: >>>>>>> >>>>>>> I1008 02:17:28.870434 2770 exec.cpp:134] Version: 0.26.0 >>>>>>>> I1008 02:17:28.871860 2778 exec.cpp:208] Executor registered on >>>>>>>> slave 16b49e90-6852-4c91-8e70-d89c54f25668-S1 >>>>>>>> WARNING: Your kernel does not support swap limit capabilities, >>>>>>>> memory limited without swap. >>>>>>>> ABORT: (src/subprocess.cpp:180): Failed to os::execvpe in childMain >>>>>>>> (path.c_str()='/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check', >>>>>>>> argv='/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check', >>>>>>>> envp=''): No such file or directory*** Aborted at 1444270649 (unix >>>>>>>> time) >>>>>>>> try "date -d @1444270649" if you are using GNU date *** >>>>>>>> PC: @ 0x7f4a37ec6cc9 (unknown) >>>>>>>> *** SIGABRT (@0xbc4) received by PID 3012 (TID 0x7f4a2f9f6700) from >>>>>>>> PID 3012; stack trace: *** >>>>>>>> @ 0x7f4a38265340 (unknown) >>>>>>>> @ 0x7f4a37ec6cc9 (unknown) >>>>>>>> @ 0x7f4a37eca0d8 (unknown) >>>>>>>> @ 0x4191e2 _Abort() >>>>>>>> @ 0x41921c _Abort() >>>>>>>> @ 0x7f4a39dc2768 process::childMain() >>>>>>>> @ 0x7f4a39dc4f59 std::_Function_handler<>::_M_invoke() >>>>>>>> @ 0x7f4a39dc24fc process::defaultClone() >>>>>>>> @ 0x7f4a39dc34fb process::subprocess() >>>>>>>> @ 0x43cc9c >>>>>>>> mesos::internal::docker::DockerExecutorProcess::launchHealthCheck() >>>>>>>> @ 0x7f4a39d924f4 process::ProcessManager::resume() >>>>>>>> @ 0x7f4a39d92827 >>>>>>>> _ZNSt6thread5_ImplISt12_Bind_simpleIFSt5_BindIFZN7process14ProcessManager12init_threadsEvEUlRKSt11atomic_boolE_St17reference_wrapperIS6_EEEvEEE6_M_runEv >>>>>>>> @ 0x7f4a38a47e40 (unknown) >>>>>>>> @ 0x7f4a3825d182 start_thread >>>>>>>> @ 0x7f4a37f8a47d (unknown) >>>>>>> >>>>>>> >>>>>>> Do any of you know which host the path >>>>>>> "/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check" >>>>>>> should exist on? It definitely doesn't exist on the slave, hence >>>>>>> execution failing. >>>>>>> >>>>>>> This is with current master, git hash >>>>>>> 5058fac1083dc91bca54d33c26c810c17ad95dd1. >>>>>>> >>>>>>> commit 5058fac1083dc91bca54d33c26c810c17ad95dd1 >>>>>>>> Author: Anand Mazumdar <[email protected]> >>>>>>>> Date: Tue Oct 6 17:37:41 2015 -0700 >>>>>>> >>>>>>> >>>>>>> -Jay >>>>>>> >>>>>>> On Wed, Oct 7, 2015 at 5:23 PM, Jay Taylor <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Update: >>>>>>>> >>>>>>>> I used https://github.com/deric/mesos-deb-packaging to compile and >>>>>>>> package the latest master (0.26.x) and deployed it to the cluster, and >>>>>>>> now >>>>>>>> health checks are working as advertised in both Marathon and my own >>>>>>>> framework! Not sure what was going on with health-checks in 0.24.0.. >>>>>>>> >>>>>>>> Anyways, thanks again for your help Haosdent! >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Jay >>>>>>>> >>>>>>>> On Wed, Oct 7, 2015 at 12:53 PM, Jay Taylor <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Haosdent, >>>>>>>>> >>>>>>>>> Can you share your Marathon POST request that results in Mesos >>>>>>>>> executing the health checks? >>>>>>>>> >>>>>>>>> Since we can reference the Marathon framework, I've been doing >>>>>>>>> some digging around. >>>>>>>>> >>>>>>>>> Here are the details of my setup and findings: >>>>>>>>> >>>>>>>>> I put a few small hacks in Marathon: >>>>>>>>> >>>>>>>>> (1) Added com.googlecode.protobuf.format to Marathon's dependencies >>>>>>>>> >>>>>>>>> (2) Edited the following files so TaskInfo is dumped as JSON to >>>>>>>>> /tmp/X in both the TaskFactory as well an right before the task is >>>>>>>>> sent to >>>>>>>>> Mesos via driver.launchTasks: >>>>>>>>> >>>>>>>>> src/main/scala/mesosphere/marathon/tasks/DefaultTaskFactory.scala: >>>>>>>>> >>>>>>>>> $ git diff >>>>>>>>>> src/main/scala/mesosphere/marathon/tasks/DefaultTaskFactory.scala >>>>>>>>>> @@ -25,6 +25,12 @@ class DefaultTaskFactory @Inject() ( >>>>>>>>>> >>>>>>>>>> new TaskBuilder(app, taskIdUtil.newTaskId, >>>>>>>>>> config).buildIfMatches(offer, runningTasks).map { >>>>>>>>>> case (taskInfo, ports) => >>>>>>>>>> + import com.googlecode.protobuf.format.JsonFormat >>>>>>>>>> + import java.io._ >>>>>>>>>> + val bw = new BufferedWriter(new FileWriter(new >>>>>>>>>> File("/tmp/taskjson1-" + taskInfo.getTaskId.getValue))) >>>>>>>>>> + bw.write(JsonFormat.printToString(taskInfo)) >>>>>>>>>> + bw.write("\n") >>>>>>>>>> + bw.close() >>>>>>>>>> CreatedTask( >>>>>>>>>> taskInfo, >>>>>>>>>> MarathonTasks.makeTask( >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> src/main/scala/mesosphere/marathon/core/launcher/impl/TaskLauncherImpl.scala: >>>>>>>>> >>>>>>>>> $ git diff >>>>>>>>>> src/main/scala/mesosphere/marathon/core/launcher/impl/TaskLauncherImpl.scala >>>>>>>>>> @@ -24,6 +24,16 @@ private[launcher] class TaskLauncherImpl( >>>>>>>>>> override def launchTasks(offerID: OfferID, taskInfos: >>>>>>>>>> Seq[TaskInfo]): Boolean = { >>>>>>>>>> val launched = withDriver(s"launchTasks($offerID)") { driver >>>>>>>>>> => >>>>>>>>>> import scala.collection.JavaConverters._ >>>>>>>>>> + var i = 0 >>>>>>>>>> + for (i <- 0 to taskInfos.length - 1) { >>>>>>>>>> + import com.googlecode.protobuf.format.JsonFormat >>>>>>>>>> + import java.io._ >>>>>>>>>> + val file = new File("/tmp/taskJson2-" + i.toString() + >>>>>>>>>> "-" + taskInfos(i).getTaskId.getValue) >>>>>>>>>> + val bw = new BufferedWriter(new FileWriter(file)) >>>>>>>>>> + bw.write(JsonFormat.printToString(taskInfos(i))) >>>>>>>>>> + bw.write("\n") >>>>>>>>>> + bw.close() >>>>>>>>>> + } >>>>>>>>>> driver.launchTasks(Collections.singleton(offerID), >>>>>>>>>> taskInfos.asJava) >>>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> Then I built and deployed the hacked Marathon and restarted the >>>>>>>>> marathon service. >>>>>>>>> >>>>>>>>> Next I created the app via the Marathon API ("hello app" is a >>>>>>>>> container with a simple hello-world ruby app running on >>>>>>>>> 0.0.0.0:8000) >>>>>>>>> >>>>>>>>> curl http://mesos-primary1a:8080/v2/groups -XPOST >>>>>>>>>> -H'Content-Type: application/json' -d' >>>>>>>>>> { >>>>>>>>>> "id": "/app-81-1-hello-app", >>>>>>>>>> "apps": [ >>>>>>>>>> { >>>>>>>>>> "id": "/app-81-1-hello-app/web-v11", >>>>>>>>>> "container": { >>>>>>>>>> "type": "DOCKER", >>>>>>>>>> "docker": { >>>>>>>>>> "image": >>>>>>>>>> "docker-services1a:5000/gig1/app-81-1-hello-app-1444240966", >>>>>>>>>> "network": "BRIDGE", >>>>>>>>>> "portMappings": [ >>>>>>>>>> { >>>>>>>>>> "containerPort": 8000, >>>>>>>>>> "hostPort": 0, >>>>>>>>>> "protocol": "tcp" >>>>>>>>>> } >>>>>>>>>> ] >>>>>>>>>> } >>>>>>>>>> }, >>>>>>>>>> "env": { >>>>>>>>>> >>>>>>>>>> }, >>>>>>>>>> "healthChecks": [ >>>>>>>>>> { >>>>>>>>>> "protocol": "COMMAND", >>>>>>>>>> "command": {"value": "exit 1"}, >>>>>>>>>> "gracePeriodSeconds": 10, >>>>>>>>>> "intervalSeconds": 10, >>>>>>>>>> "timeoutSeconds": 10, >>>>>>>>>> "maxConsecutiveFailures": 3 >>>>>>>>>> } >>>>>>>>>> ], >>>>>>>>>> "instances": 1, >>>>>>>>>> "cpus": 1, >>>>>>>>>> "mem": 512 >>>>>>>>>> } >>>>>>>>>> ] >>>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> $ ls /tmp/ >>>>>>>>>> >>>>>>>>>> taskjson1-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>>>>> >>>>>>>>>> taskJson2-0-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>>>> >>>>>>>>> >>>>>>>>> Do they match? >>>>>>>>> >>>>>>>>> $ md5sum /tmp/task* >>>>>>>>>> 1b5115997e78e2611654059249d99578 >>>>>>>>>> >>>>>>>>>> /tmp/taskjson1-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>>>>> 1b5115997e78e2611654059249d99578 >>>>>>>>>> >>>>>>>>>> /tmp/taskJson2-0-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>>>> >>>>>>>>> >>>>>>>>> Yes, so I am confident this is the information being sent across >>>>>>>>> the wire to Mesos. >>>>>>>>> >>>>>>>>> Do they contain any health-check information? >>>>>>>>> >>>>>>>>> $ cat >>>>>>>>>> /tmp/taskjson1-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>>>>> { >>>>>>>>>> "name":"web-v11.app-81-1-hello-app", >>>>>>>>>> "task_id":{ >>>>>>>>>> >>>>>>>>>> "value":"app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0" >>>>>>>>>> }, >>>>>>>>>> "slave_id":{ >>>>>>>>>> "value":"20150924-210922-1608624320-5050-1792-S1" >>>>>>>>>> }, >>>>>>>>>> "resources":[ >>>>>>>>>> { >>>>>>>>>> "name":"cpus", >>>>>>>>>> "type":"SCALAR", >>>>>>>>>> "scalar":{ >>>>>>>>>> "value":1.0 >>>>>>>>>> }, >>>>>>>>>> "role":"*" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"mem", >>>>>>>>>> "type":"SCALAR", >>>>>>>>>> "scalar":{ >>>>>>>>>> "value":512.0 >>>>>>>>>> }, >>>>>>>>>> "role":"*" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"ports", >>>>>>>>>> "type":"RANGES", >>>>>>>>>> "ranges":{ >>>>>>>>>> "range":[ >>>>>>>>>> { >>>>>>>>>> "begin":31641, >>>>>>>>>> "end":31641 >>>>>>>>>> } >>>>>>>>>> ] >>>>>>>>>> }, >>>>>>>>>> "role":"*" >>>>>>>>>> } >>>>>>>>>> ], >>>>>>>>>> "command":{ >>>>>>>>>> "environment":{ >>>>>>>>>> "variables":[ >>>>>>>>>> { >>>>>>>>>> "name":"PORT_8000", >>>>>>>>>> "value":"31641" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"MARATHON_APP_VERSION", >>>>>>>>>> "value":"2015-10-07T19:35:08.386Z" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"HOST", >>>>>>>>>> "value":"mesos-worker1a" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"MARATHON_APP_DOCKER_IMAGE", >>>>>>>>>> >>>>>>>>>> "value":"docker-services1a:5000/gig1/app-81-1-hello-app-1444240966" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"MESOS_TASK_ID", >>>>>>>>>> >>>>>>>>>> "value":"app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"PORT", >>>>>>>>>> "value":"31641" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"PORTS", >>>>>>>>>> "value":"31641" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"MARATHON_APP_ID", >>>>>>>>>> "value":"/app-81-1-hello-app/web-v11" >>>>>>>>>> }, >>>>>>>>>> { >>>>>>>>>> "name":"PORT0", >>>>>>>>>> "value":"31641" >>>>>>>>>> } >>>>>>>>>> ] >>>>>>>>>> }, >>>>>>>>>> "shell":false >>>>>>>>>> }, >>>>>>>>>> "container":{ >>>>>>>>>> "type":"DOCKER", >>>>>>>>>> "docker":{ >>>>>>>>>> >>>>>>>>>> "image":"docker-services1a:5000/gig1/app-81-1-hello-app-1444240966", >>>>>>>>>> "network":"BRIDGE", >>>>>>>>>> "port_mappings":[ >>>>>>>>>> { >>>>>>>>>> "host_port":31641, >>>>>>>>>> "container_port":8000, >>>>>>>>>> "protocol":"tcp" >>>>>>>>>> } >>>>>>>>>> ], >>>>>>>>>> "privileged":false, >>>>>>>>>> "force_pull_image":false >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> No, I don't see anything about any health check. >>>>>>>>> >>>>>>>>> Mesos STDOUT for the launched task: >>>>>>>>> >>>>>>>>> --container="mesos-20150924-210922-1608624320-5050-1792-S1.14335f1f-3774-4862-a55b-e9c76cd0f2da" >>>>>>>>>> --docker="docker" --help="false" --initialize_driver_logging="true" >>>>>>>>>> --logbufsecs="0" --logging_level="INFO" >>>>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>>>>>> --sandbox_directory="/tmp/mesos/slaves/20150924-210922-1608624320-5050-1792-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0/runs/14335f1f-3774-4862-a55b-e9c76cd0f2da" >>>>>>>>>> --stop_timeout="0ns" >>>>>>>>>> --container="mesos-20150924-210922-1608624320-5050-1792-S1.14335f1f-3774-4862-a55b-e9c76cd0f2da" >>>>>>>>>> --docker="docker" --help="false" --initialize_driver_logging="true" >>>>>>>>>> --logbufsecs="0" --logging_level="INFO" >>>>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>>>>>> --sandbox_directory="/tmp/mesos/slaves/20150924-210922-1608624320-5050-1792-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0/runs/14335f1f-3774-4862-a55b-e9c76cd0f2da" >>>>>>>>>> --stop_timeout="0ns" >>>>>>>>>> Registered docker executor on mesos-worker1a >>>>>>>>>> Starting task >>>>>>>>>> app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>>>> >>>>>>>>> >>>>>>>>> And STDERR: >>>>>>>>> >>>>>>>>> I1007 19:35:08.790743 4612 exec.cpp:134] Version: 0.24.0 >>>>>>>>>> I1007 19:35:08.793416 4619 exec.cpp:208] Executor registered on >>>>>>>>>> slave 20150924-210922-1608624320-5050-1792-S1 >>>>>>>>>> WARNING: Your kernel does not support swap limit capabilities, >>>>>>>>>> memory limited without swap. >>>>>>>>> >>>>>>>>> >>>>>>>>> Again, nothing about any health checks. >>>>>>>>> >>>>>>>>> Any ideas of other things to try or what I could be missing? >>>>>>>>> Can't say either way about the Mesos health-check system working or >>>>>>>>> not if >>>>>>>>> Marathon won't put the health-check into the task it sends to Mesos. >>>>>>>>> >>>>>>>>> Thanks for all your help! >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Jay >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>> On Tue, Oct 6, 2015 at 11:24 PM, haosdent <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Maybe you could post your executor stdout/stderr so that we could >>>>>>>>>> know whether health check running not. >>>>>>>>>> >>>>>>>>>> On Wed, Oct 7, 2015 at 2:15 PM, haosdent <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> marathon also use mesos health check. When I use health check, I >>>>>>>>>>> could saw the log like this in executor stdout. >>>>>>>>>>> >>>>>>>>>>> ``` >>>>>>>>>>> Registered docker executor on xxxxx >>>>>>>>>>> Starting task >>>>>>>>>>> test-health-check.822a5fd2-6cba-11e5-b5ce-0a0027000000 >>>>>>>>>>> Launching health check process: >>>>>>>>>>> /home/haosdent/mesos/build/src/.libs/mesos-health-check >>>>>>>>>>> --executor=xxxx >>>>>>>>>>> Health check process launched at pid: 9895 >>>>>>>>>>> Received task health update, healthy: true >>>>>>>>>>> ``` >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 7, 2015 at 12:51 PM, Jay Taylor <[email protected] >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> I am using my own framework, and the full task info I'm using >>>>>>>>>>>> is posted earlier in this thread. Do you happen to know if >>>>>>>>>>>> Marathon uses >>>>>>>>>>>> Mesos's health checks for its health check system? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Oct 6, 2015, at 9:01 PM, haosdent <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Yes, launch the health task through its definition in taskinfo. >>>>>>>>>>>> Do you launch your task through Marathon? I could test it in my >>>>>>>>>>>> side. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Oct 7, 2015 at 11:56 AM, Jay Taylor < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Precisely, and there are none of those statements. Are you or >>>>>>>>>>>>> others confident health-checks are part of the code path when >>>>>>>>>>>>> defined via >>>>>>>>>>>>> task info for docker container tasks? Going through the code, I >>>>>>>>>>>>> wasn't >>>>>>>>>>>>> able to find the linkage for anything other than health-checks >>>>>>>>>>>>> triggered >>>>>>>>>>>>> through a custom executor. >>>>>>>>>>>>> >>>>>>>>>>>>> With that being said it is a pretty good sized code base and >>>>>>>>>>>>> I'm not very familiar with it, so my analysis this far has by no >>>>>>>>>>>>> means been >>>>>>>>>>>>> exhaustive. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Oct 6, 2015, at 8:41 PM, haosdent <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> When health check launch, it would have a log like this in >>>>>>>>>>>>> your executor stdout >>>>>>>>>>>>> ``` >>>>>>>>>>>>> Health check process launched at pid xxx >>>>>>>>>>>>> ``` >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Oct 7, 2015 at 11:37 AM, Jay Taylor < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I'm happy to try this, however wouldn't there be output in >>>>>>>>>>>>>> the logs with the string "health" or "Health" if the >>>>>>>>>>>>>> health-check were >>>>>>>>>>>>>> active? None of my master or slave logs contain the string.. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Oct 6, 2015, at 7:45 PM, haosdent <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Could you use "exit 1" instead of "sleep 5" to see whether >>>>>>>>>>>>>> could see unhealthy status in your task stdout/stderr. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 10:38 AM, Jay Taylor < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> My current version is 0.24.1. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 7:30 PM, haosdent <[email protected] >>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> yes, adam also help commit it to 0.23.1 and 0.24.1 >>>>>>>>>>>>>>>> https://github.com/apache/mesos/commit/8c0ed92de3925d4312429bfba01b9b1ccbcbbef0 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://github.com/apache/mesos/commit/09e367cd69aa39c156c9326d44f4a7b829ba3db7 >>>>>>>>>>>>>>>> Are you use one of this version? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 10:26 AM, haosdent < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I remember 0.23.1 and 0.24.1 contains this backport, let >>>>>>>>>>>>>>>>> me double check. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 10:01 AM, Jay Taylor < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Oops- Now I see you already said it's in master. I'll >>>>>>>>>>>>>>>>>> look there :) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks again! >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 6:59 PM, Jay Taylor < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Great, thanks for the quick reply Tim! >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Do you know if there is a branch I can checkout to test >>>>>>>>>>>>>>>>>>> it out? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 6:54 PM, Timothy Chen < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Jay, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> We just added health check support for docker tasks >>>>>>>>>>>>>>>>>>>> that's in master but not yet released. It will run docker >>>>>>>>>>>>>>>>>>>> exec with the >>>>>>>>>>>>>>>>>>>> command you provided as health checks. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> It should be in the next release. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Tim >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Oct 6, 2015, at 6:49 PM, Jay Taylor < >>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Does Mesos support health checks for docker image >>>>>>>>>>>>>>>>>>>> tasks? Mesos seems to be ignoring the >>>>>>>>>>>>>>>>>>>> TaskInfo.HealthCheck field for me. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Example TaskInfo JSON received back from Mesos: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "name":"hello-app.web.v3", >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "task_id":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> "value":"hello-app_web-v3.fc05a1a5-1e06-4e61-9879-be0d97cd3eec" >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "slave_id":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "value":"20150924-210922-1608624320-5050-1792-S1" >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "resources":[ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "name":"cpus", >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "type":0, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "scalar":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "value":0.1 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "name":"mem", >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "type":0, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "scalar":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "value":256 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "name":"ports", >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "type":1, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "ranges":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "range":[ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "begin":31002, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "end":31002 >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> ] >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> ], >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "command":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "container":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> "image":"docker-services1a:5000/test/app-81-1-hello-app-103" >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "shell":false >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "container":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "type":1, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "docker":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> "image":"docker-services1a:5000/gig1/app-81-1-hello-app-103", >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "network":2, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "port_mappings":[ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "host_port":31002, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "container_port":8000, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "protocol":"tcp" >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> ], >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "privileged":false, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "parameters":[], >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "force_pull_image":false >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "health_check":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "delay_seconds":5, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "interval_seconds":10, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "timeout_seconds":10, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "consecutive_failures":3, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "grace_period_seconds":0, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "command":{ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "shell":true, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "value":"sleep 5", >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> "user":"root" >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I have searched all machines and containers to see if >>>>>>>>>>>>>>>>>>>> they ever run the command (in this case `sleep 5`), but >>>>>>>>>>>>>>>>>>>> have not found any >>>>>>>>>>>>>>>>>>>> indication that it is being executed. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> In the mesos src code the health-checks are invoked >>>>>>>>>>>>>>>>>>>> from src/launcher/executor.cpp >>>>>>>>>>>>>>>>>>>> CommandExecutorProcess::launchTask. Does >>>>>>>>>>>>>>>>>>>> this mean that health-checks are only supported for custom >>>>>>>>>>>>>>>>>>>> executors and >>>>>>>>>>>>>>>>>>>> not for docker tasks? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> What I am trying to accomplish is to have the >>>>>>>>>>>>>>>>>>>> 0/non-zero exit-status of a health-check command translate >>>>>>>>>>>>>>>>>>>> to task health. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>>>>> Jay >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>> Haosdent Huang >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>> Haosdent Huang >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Haosdent Huang >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>> Haosdent Huang >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> Haosdent Huang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Best Regards, >>>>>>>>>>> Haosdent Huang >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best Regards, >>>>>>>>>> Haosdent Huang >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Haosdent Huang >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Haosdent Huang >>>> >>>> >>> >>> >>> -- >>> Best Regards, >>> Haosdent Huang >>> >>> >> >> >> -- >> Best Regards, >> Haosdent Huang >> > >
