Paul, This would be relevant for any system which is automatically deleting files in /tmp. It looks like in Ubuntu, the default behavior is for /tmp to be completely nuked at boot time. Was the agent node rebooted prior to this problem?
On Tue, Mar 29, 2016 at 2:29 PM, Paul Bell <[email protected]> wrote: > Hi Greg, > > Thanks very much for your quick reply. > > I simply forgot to mention platform. It's Ubuntu 14.04 LTS and it's not > systemd. I will look at the link you provide. > > Is there any chance that it might apply to non-systemd platforms? > > Cordially, > > Paul > > On Tue, Mar 29, 2016 at 5:18 PM, Greg Mann <[email protected]> wrote: > >> Hi Paul, >> Noticing the logging output, "Failed to find resources file >> '/tmp/mesos/meta/resources/resources.info'", I wonder if your trouble >> may be related to the location of your agent's work_dir. See this ticket: >> https://issues.apache.org/jira/browse/MESOS-4541 >> >> Some users have reported issues resulting from the systemd-tmpfiles >> service garbage collecting files in /tmp, perhaps this is related? What >> platform is your agent running on? >> >> You could try specifying a different agent work directory outside of >> /tmp/ via the `--work_dir` command-line flag. >> >> Cheers, >> Greg >> >> >> On Tue, Mar 29, 2016 at 2:08 PM, Paul Bell <[email protected]> wrote: >> >>> Hi, >>> >>> I am hoping someone can shed some light on this. >>> >>> An agent node failed to start, that is, when I did "service mesos-slave >>> start" the service came up briefly & then stopped. Before stopping it >>> produced the log shown below. The last thing it wrote is "Trying to create >>> path '/mesos' in Zookeeper". >>> >>> This mention of the mesos znode prompted me to go for a clean slate by >>> removing the mesos znode from Zookeeper. >>> >>> After doing this, the mesos-slave service started perfectly. >>> >>> What might be happening here, and also what's the right way to >>> trouble-shoot such a problem? Mesos is version 0.23.0. >>> >>> Thanks for your help. >>> >>> -Paul >>> >>> >>> Log file created at: 2016/03/29 14:19:39 >>> Running on machine: 71.100.202.193 >>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg >>> I0329 14:19:39.512249 5870 logging.cpp:172] INFO level logging started! >>> I0329 14:19:39.512564 5870 main.cpp:162] Build: 2015-07-24 10:05:39 by >>> root >>> I0329 14:19:39.512588 5870 main.cpp:164] Version: 0.23.0 >>> I0329 14:19:39.512600 5870 main.cpp:167] Git tag: 0.23.0 >>> I0329 14:19:39.512612 5870 main.cpp:171] Git SHA: >>> 4ce5475346a0abb7ef4b7ffc9836c5836d7c7a66 >>> I0329 14:19:39.615172 5870 containerizer.cpp:111] Using isolation: >>> posix/cpu,posix/mem >>> I0329 14:19:39.615697 5870 main.cpp:249] Starting Mesos slave >>> I0329 14:19:39.616267 5870 slave.cpp:190] Slave started on 1)@ >>> 71.100.202.193:5051 >>> I0329 14:19:39.616286 5870 slave.cpp:191] Flags at startup: >>> --attributes="hostType:shard1" --authenticatee="crammd5" >>> --cgroups_cpu_enable_pids_and_tids_count="false" >>> --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" >>> --cgroups_limit_swap="false" --cgroups_root="mesos" >>> --container_disk_watch_interval="15secs" --containerizers="docker,mesos" >>> --default_role="*" --disk_watch_interval="1mins" >>> --docker="/usr/local/ecxmcc/weaveShim" --docker_kill_orphans="true" >>> --docker_remove_delay="6hrs" >>> --docker_sandbox_directory="/mnt/mesos/sandbox" >>> --docker_socket="/var/run/docker.sock" --docker_stop_timeout="15secs" >>> --enforce_container_disk_quota="false" >>> --executor_registration_timeout="5mins" >>> --executor_shutdown_grace_period="5secs" >>> --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" >>> --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" >>> --hadoop_home="" --help="false" --hostname="71.100.202.193" >>> --initialize_driver_logging="true" --ip="71.100.202.193" >>> --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" >>> --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" >>> --master="zk://71.100.202.191:2181/mesos" >>> --oversubscribed_resources_interval="15secs" --perf_duration="10secs" >>> --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" >>> --quiet="false" --recover="reconnect" --recovery_timeout="15mins" >>> --registration_backoff_factor="1secs" >>> --resource_monitoring_interval="1secs" --revocable_cpu_low_priority="true" >>> --strict="true" --switch_user="true" --version="false" >>> --work_dir="/tmp/mesos" >>> I0329 14:19:39.616835 5870 slave.cpp:354] Slave resources: cpus(*):4; >>> mem(*):23089; disk(*):122517; ports(*):[31000-32000] >>> I0329 14:19:39.617032 5870 slave.cpp:384] Slave hostname: 71.100.202.193 >>> I0329 14:19:39.617046 5870 slave.cpp:389] Slave checkpoint: true >>> I0329 14:19:39.618841 5894 state.cpp:36] Recovering state from >>> '/tmp/mesos/meta' >>> I0329 14:19:39.618872 5894 state.cpp:672] Failed to find resources file >>> '/tmp/mesos/meta/resources/resources.info' >>> I0329 14:19:39.619730 5898 group.cpp:313] Group process (group(1)@ >>> 71.100.202.193:5051) connected to ZooKeeper >>> I0329 14:19:39.619760 5898 group.cpp:787] Syncing group operations: >>> queue size (joins, cancels, datas) = (0, 0, 0) >>> I0329 14:19:39.619773 5898 group.cpp:385] Trying to create path >>> '/mesos' in ZooKeeper >>> >>> >> >
