The work_dir was set to /tmp/mesos by default, I've deleted it and tried to start the slave again. The dir is not being recreated now, just a continual service failure.
On Thu, Aug 6, 2015 at 2:59 PM, craig w <[email protected]> wrote: > Have you tried clearing out the data in the slave's work_dir? for example > if work dir is /var/mesos, rm -rf /var/mesos/* then start the slave? > > On Thu, Aug 6, 2015 at 6:55 AM, Stephen Knight <[email protected]> wrote: > >> My system doesn't support cat with systemctl for some reason but here is >> the contents of /usr/lib/systemd/system/mesos-slave.service >> >> [Unit] >> >> Description=Mesos Slave >> >> After=network.target >> >> Wants=network.target >> >> >> [Service] >> >> ExecStart=/usr/bin/mesos-init-wrapper slave >> >> KillMode=process >> >> Restart=always >> >> RestartSec=20 >> >> LimitNOFILE=16384 >> >> CPUAccounting=true >> >> MemoryAccounting=true >> >> >> [Install] >> >> WantedBy=multi-user.target >> >> >> What are the required flags to start it manually? >> >> On Thu, Aug 6, 2015 at 2:51 PM, haosdent <[email protected]> wrote: >> >>> Or you could try "systemctl cat mesos-slave.service" and show us the >>> file content. >>> >>> On Thu, Aug 6, 2015 at 6:49 PM, haosdent <[email protected]> wrote: >>> >>>> From this message, I think "systemctl status mesos-slave.service -l" >>>> run mesos-slave with uncorrect flags. And the status out of it is the help >>>> message of slave. Could you try to start mesos-slave in manual way? Not >>>> through systemctl. >>>> >>>> On Thu, Aug 6, 2015 at 6:41 PM, Stephen Knight <[email protected]> >>>> wrote: >>>> >>>>> systemctl gives me the following output on CentOS: The command to >>>>> start I ran was "systemctl start mesos-slave.service" >>>>> >>>>> [root@ip-172-31-35-167 mesos]# systemctl status mesos-slave.service -l >>>>> >>>>> mesos-slave.service - Mesos Slave >>>>> >>>>> Loaded: loaded (/usr/lib/systemd/system/mesos-slave.service; >>>>> enabled) >>>>> >>>>> Drop-In: /etc/systemd/system/mesos-slave.service.d >>>>> >>>>> └─mesos-slave-containerizers.conf >>>>> >>>>> Active: activating (auto-restart) (Result: exit-code) since Thu >>>>> 2015-08-06 10:38:08 UTC; 2s ago >>>>> >>>>> Process: 1472 ExecStart=/usr/bin/mesos-init-wrapper slave *(code=exited, >>>>> status=1/FAILURE)* >>>>> >>>>> Main PID: 1472 (code=exited, status=1/FAILURE) >>>>> >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *If >>>>> strict=false, any expected errors (e.g., slave cannot recover* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>>> *information >>>>> about an executor, because the slave died right before* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *the >>>>> executor registered.) during recovery are ignored and as much* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *state >>>>> as possible is recovered.* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>>> *(default: >>>>> true)* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>>> *--[no-]switch_user >>>>> Whether to run tasks as the user who* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>>> *submitted >>>>> them rather than the user running* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *the >>>>> slave (requires setuid permission) (default: true)* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>>> *--[no-]version >>>>> Show version and exit. (default: false)* >>>>> >>>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>>> *--work_dir=VALUE >>>>> Directory path to place framework work >>>>> directories* >>>>> >>>>> >>>>> >>>>> I've also run strace against it, nothing sticks out: >>>>> >>>>> >>>>> strace systemctl start mesos-slave.service >>>>> >>>>> execve("/bin/systemctl", ["systemctl", "start", >>>>> "mesos-slave.service"], [/* 18 vars */]) = 0 >>>>> >>>>> brk(0) = 0x7f5c2af9f000 >>>>> >>>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>>> 0) = 0x7f5c2a5c6000 >>>>> >>>>> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or >>>>> directory) >>>>> >>>>> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0644, st_size=20940, ...}) = 0 >>>>> >>>>> mmap(NULL, 20940, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5c2a5c0000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libsystemd-daemon.so.0", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\r\0\0\0\0\0\0"..., >>>>> 832) >>>>> = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=15216, ...}) = 0 >>>>> >>>>> mmap(NULL, 2109448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c2a1a2000 >>>>> >>>>> mprotect(0x7f5c2a1a4000, 2097152, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c2a3a4000, 4096, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f5c2a3a4000 >>>>> >>>>> mmap(0x7f5c2a3a5000, 8, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c2a3a5000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libdbus-1.so.3", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@x\0\0\0\0\0\0"..., >>>>> 832) = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=304536, ...}) = 0 >>>>> >>>>> mmap(NULL, 2390496, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c29f5a000 >>>>> >>>>> mprotect(0x7f5c29fa0000, 2097152, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c2a1a0000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x46000) = 0x7f5c2a1a0000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\"\0\0\0\0\0\0"..., >>>>> 832) >>>>> = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=44088, ...}) = 0 >>>>> >>>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>>> 0) = 0x7f5c2a5bf000 >>>>> >>>>> mmap(NULL, 2128952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c29d52000 >>>>> >>>>> mprotect(0x7f5c29d59000, 2093056, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c29f58000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f5c29f58000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240d\0\0\0\0\0\0"..., 832) >>>>> = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=147120, ...}) = 0 >>>>> >>>>> mmap(NULL, 2246784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c29b2d000 >>>>> >>>>> mprotect(0x7f5c29b4e000, 2097152, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c29d4e000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f5c29d4e000 >>>>> >>>>> mmap(0x7f5c29d50000, 6272, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c29d50000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000/\0\0\0\0\0\0"..., 832) >>>>> = >>>>> 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=153184, ...}) = 0 >>>>> >>>>> mmap(NULL, 2245240, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c29908000 >>>>> >>>>> mprotect(0x7f5c2992c000, 2093056, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c29b2b000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f5c29b2b000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libgcrypt.so.11", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0u\0\0\0\0\0\0"..., 832) = >>>>> 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=534488, ...}) = 0 >>>>> >>>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>>> 0) = 0x7f5c2a5be000 >>>>> >>>>> mmap(NULL, 2621456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c29687000 >>>>> >>>>> mprotect(0x7f5c29703000, 2097152, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c29903000, 16384, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7c000) = 0x7f5c29903000 >>>>> >>>>> mmap(0x7f5c29907000, 16, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c29907000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360*\0\0\0\0\0\0"..., 832) >>>>> = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=88720, ...}) = 0 >>>>> >>>>> mmap(NULL, 2184192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c29471000 >>>>> >>>>> mprotect(0x7f5c29486000, 2093056, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c29685000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7f5c29685000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\34\2\0\0\0\0\0"..., 832) >>>>> = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=2107760, ...}) = 0 >>>>> >>>>> mmap(NULL, 3932736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c290b0000 >>>>> >>>>> mprotect(0x7f5c29266000, 2097152, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c29466000, 24576, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f5c29466000 >>>>> >>>>> mmap(0x7f5c2946c000, 16960, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c2946c000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., >>>>> 832) = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=19512, ...}) = 0 >>>>> >>>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>>> 0) = 0x7f5c2a5bd000 >>>>> >>>>> mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c28eac000 >>>>> >>>>> mprotect(0x7f5c28eaf000, 2093056, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c290ae000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f5c290ae000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240l\0\0\0\0\0\0"..., 832) >>>>> = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=141616, ...}) = 0 >>>>> >>>>> mmap(NULL, 2208864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c28c90000 >>>>> >>>>> mprotect(0x7f5c28ca6000, 2097152, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c28ea6000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f5c28ea6000 >>>>> >>>>> mmap(0x7f5c28ea8000, 13408, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c28ea8000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., >>>>> 832) = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=398272, ...}) = 0 >>>>> >>>>> mmap(NULL, 2490888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c28a2f000 >>>>> >>>>> mprotect(0x7f5c28a8e000, 2097152, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c28c8e000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5f000) = 0x7f5c28c8e000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> open("/lib64/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> read(3, >>>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\n\0\0\0\0\0\0"..., 832) >>>>> = 832 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0755, st_size=19384, ...}) = 0 >>>>> >>>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>>> 0) = 0x7f5c2a5bc000 >>>>> >>>>> mmap(NULL, 2113656, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>>> 0) = 0x7f5c2882a000 >>>>> >>>>> mprotect(0x7f5c2882e000, 2093056, PROT_NONE) = 0 >>>>> >>>>> mmap(0x7f5c28a2d000, 8192, PROT_READ|PROT_WRITE, >>>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f5c28a2d000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>>> 0) = 0x7f5c2a5bb000 >>>>> >>>>> mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>>> 0) = 0x7f5c2a5b9000 >>>>> >>>>> arch_prctl(ARCH_SET_FS, 0x7f5c2a5b9880) = 0 >>>>> >>>>> mprotect(0x7f5c29466000, 16384, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c28a2d000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c28ea6000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c28c8e000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c290ae000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c29685000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c29903000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c29b2b000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c29d4e000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c29f58000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c2a1a0000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c2a3a4000, 4096, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c2a81d000, 8192, PROT_READ) = 0 >>>>> >>>>> mprotect(0x7f5c2a5c7000, 4096, PROT_READ) = 0 >>>>> >>>>> munmap(0x7f5c2a5c0000, 20940) = 0 >>>>> >>>>> set_tid_address(0x7f5c2a5b9b50) = 1545 >>>>> >>>>> set_robust_list(0x7f5c2a5b9b60, 24) = 0 >>>>> >>>>> rt_sigaction(SIGRTMIN, {0x7f5c28c96780, [], SA_RESTORER|SA_SIGINFO, >>>>> 0x7f5c28c9f130}, NULL, 8) = 0 >>>>> >>>>> rt_sigaction(SIGRT_1, {0x7f5c28c96810, [], >>>>> SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f5c28c9f130}, NULL, 8) = 0 >>>>> >>>>> rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 >>>>> >>>>> getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, >>>>> rlim_max=RLIM64_INFINITY}) = 0 >>>>> >>>>> brk(0) = 0x7f5c2af9f000 >>>>> >>>>> brk(0x7f5c2afc0000) = 0x7f5c2afc0000 >>>>> >>>>> access("/etc/system-fips", F_OK) = -1 ENOENT (No such file or >>>>> directory) >>>>> >>>>> statfs("/sys/fs/selinux", {f_type=0xf97cff8c, f_bsize=4096, >>>>> f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, >>>>> f_namelen=255, f_frsize=4096}) = 0 >>>>> >>>>> statfs("/sys/fs/selinux", {f_type=0xf97cff8c, f_bsize=4096, >>>>> f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, >>>>> f_namelen=255, f_frsize=4096}) = 0 >>>>> >>>>> stat("/sys/fs/selinux", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 >>>>> >>>>> open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 >>>>> >>>>> fstat(3, {st_mode=S_IFREG|0644, st_size=106065056, ...}) = 0 >>>>> >>>>> mmap(NULL, 106065056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5c22303000 >>>>> >>>>> close(3) = 0 >>>>> >>>>> ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or >>>>> TCGETS, {B9600 opost isig icanon echo ...}) = 0 >>>>> >>>>> stat("/proc/1/root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>>> >>>>> stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>>> >>>>> stat("/proc/1/root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>>> >>>>> stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>>> >>>>> lstat("/run/systemd/system/", {st_mode=S_IFDIR|0755, st_size=80, ...}) >>>>> = 0 >>>>> >>>>> geteuid() = 0 >>>>> >>>>> socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3 >>>>> >>>>> connect(3, {sa_family=AF_LOCAL, sun_path="/run/systemd/private"}, 22) >>>>> = 0 >>>>> >>>>> fcntl(3, F_GETFL) = 0x2 (flags O_RDWR) >>>>> >>>>> fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0 >>>>> >>>>> geteuid() = 0 >>>>> >>>>> getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0 >>>>> >>>>> getsockopt(3, SOL_SOCKET, SO_PEERCRED, {pid=1, uid=0, gid=0}, [12]) = 0 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 716107629}) = 0 >>>>> >>>>> poll([{fd=3, events=POLLOUT}], 1, 90000) = 1 ([{fd=3, >>>>> revents=POLLOUT}]) >>>>> >>>>> sendto(3, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1 >>>>> >>>>> sendto(3, "AUTH EXTERNAL 30\r\n", 18, MSG_NOSIGNAL, NULL, 0) = 18 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 716225119}) = 0 >>>>> >>>>> poll([{fd=3, events=POLLIN}], 1, 90000) = 1 ([{fd=3, revents=POLLIN}]) >>>>> >>>>> read(3, "OK c2d5f933b89e938a29b77c0355c33"..., 2048) = 37 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 716481724}) = 0 >>>>> >>>>> poll([{fd=3, events=POLLOUT}], 1, 90000) = 1 ([{fd=3, >>>>> revents=POLLOUT}]) >>>>> >>>>> sendto(3, "NEGOTIATE_UNIX_FD\r\n", 19, MSG_NOSIGNAL, NULL, 0) = 19 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 716552147}) = 0 >>>>> >>>>> poll([{fd=3, events=POLLIN}], 1, 90000) = 1 ([{fd=3, revents=POLLIN}]) >>>>> >>>>> read(3, "AGREE_UNIX_FD\r\n", 2048) = 15 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 716708588}) = 0 >>>>> >>>>> poll([{fd=3, events=POLLOUT}], 1, 90000) = 1 ([{fd=3, >>>>> revents=POLLOUT}]) >>>>> >>>>> sendto(3, "BEGIN\r\n", 7, MSG_NOSIGNAL, NULL, 0) = 7 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 716773800}) = 0 >>>>> >>>>> stat("/proc/1/root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>>> >>>>> stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>>> >>>>> ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or >>>>> TCGETS, {B9600 opost isig icanon echo ...}) = 0 >>>>> >>>>> rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0 >>>>> >>>>> clone(child_stack=0, >>>>> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, >>>>> child_tidptr=0x7f5c2a5b9b50) = 1546 >>>>> >>>>> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 >>>>> >>>>> gettid() = 1545 >>>>> >>>>> sendmsg(3, {msg_name(0)=NULL, >>>>> msg_iov(2)=[{"l\1\0\1$\0\0\0\1\0\0\0\240\0\0\0\1\1o\0\31\0\0\0/org/fre"..., >>>>> 176}, {"\23\0\0\0mesos-slave.service\0\7\0\0\0repl"..., 36}], >>>>> msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 212 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 717164627}) = 0 >>>>> >>>>> poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}]) >>>>> >>>>> recvmsg(3, {msg_name(0)=NULL, >>>>> msg_iov(1)=[{"l\2\1\1'\0\0\0\1\0\0\0\17\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1o\0\0"..., >>>>> 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = >>>>> 1143 >>>>> >>>>> recvmsg(3, 0x7fff05789c50, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource >>>>> temporarily unavailable) >>>>> >>>>> sendmsg(3, {msg_name(0)=NULL, >>>>> msg_iov(2)=[{"l\1\0\1\30\0\0\0\2\0\0\0\227\0\0\0\1\1o\0\31\0\0\0/org/fre"..., >>>>> 168}, {"\23\0\0\0mesos-slave.service\0", 24}], msg_controllen=0, >>>>> msg_flags=0}, MSG_NOSIGNAL) = 192 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 719385633}) = 0 >>>>> >>>>> poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}]) >>>>> >>>>> recvmsg(3, {msg_name(0)=NULL, >>>>> msg_iov(1)=[{"l\2\1\1;\0\0\0\2\0\0\0\17\0\0\0\5\1u\0\2\0\0\0\10\1g\0\1o\0\0"..., >>>>> 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = >>>>> 91 >>>>> >>>>> recvmsg(3, 0x7fff05789c50, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource >>>>> temporarily unavailable) >>>>> >>>>> sendmsg(3, {msg_name(0)=NULL, >>>>> msg_iov(2)=[{"l\1\0\0019\0\0\0\3\0\0\0\250\0\0\0\1\1o\0006\0\0\0/org/fre"..., >>>>> 184}, {"\35\0\0\0org.freedesktop.systemd1.Uni"..., 57}], msg_controllen=0, >>>>> msg_flags=0}, MSG_NOSIGNAL) = 241 >>>>> >>>>> clock_gettime(CLOCK_MONOTONIC, {774, 719741008}) = 0 >>>>> >>>>> poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}]) >>>>> >>>>> recvmsg(3, {msg_name(0)=NULL, >>>>> msg_iov(1)=[{"l\2\1\1\10\0\0\0\3\0\0\0\17\0\0\0\5\1u\0\3\0\0\0\10\1g\0\1v\0\0"..., >>>>> 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = >>>>> 40 >>>>> >>>>> recvmsg(3, 0x7fff05789c50, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource >>>>> temporarily unavailable) >>>>> >>>>> poll([{fd=3, events=POLLIN}], 1, 4294967295) = 1 ([{fd=3, >>>>> revents=POLLIN}]) >>>>> >>>>> recvmsg(3, {msg_name(0)=NULL, >>>>> msg_iov(1)=[{"l\4\1\1M\0\0\0\206\1\0\0z\0\0\0\1\1o\0\31\0\0\0/org/fre"..., >>>>> 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = >>>>> 1089 >>>>> >>>>> recvmsg(3, 0x7fff0578a030, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource >>>>> temporarily unavailable) >>>>> >>>>> close(3) = 0 >>>>> >>>>> kill(1546, SIGTERM) = 0 >>>>> >>>>> kill(1546, SIGCONT) = 0 >>>>> >>>>> waitid(P_PID, 1546, {si_signo=SIGCHLD, si_code=CLD_EXITED, >>>>> si_pid=1546, si_status=0, si_utime=0, si_stime=0}, WEXITED, NULL) = 0 >>>>> >>>>> --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1546, >>>>> si_status=0, si_utime=0, si_stime=0} --- >>>>> >>>>> exit_group(0) = ? >>>>> >>>>> +++ exited with 0 +++ >>>>> >>>>> >>>>> On Thu, Aug 6, 2015 at 2:36 PM, haosdent <[email protected]> wrote: >>>>> >>>>>> You use service mesos-slave to start slave failed so that it could >>>>>> not connect to master, right? Does it have any log about this? >>>>>> >>>>>> On Thu, Aug 6, 2015 at 6:30 PM, Stephen Knight <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Yes, I have tried on both environments (one CentOS 7 and one Ubuntu) >>>>>>> and they behave the exact same way. I probably should clarify though >>>>>>> that >>>>>>> after a restart the agent fails to start on the slave, which is why it >>>>>>> never reaches the master. >>>>>>> >>>>>>> I can however run a smoke test and get it to connect from a slave to >>>>>>> the master by passing a job, but the slave agent won't start. The guide >>>>>>> I >>>>>>> followed is at: >>>>>>> http://open.mesosphere.com/getting-started/datacenter/install/ >>>>>>> >>>>>>> The environment is by no means live, I can share details with you if >>>>>>> you can help me to that degree. >>>>>>> >>>>>>> On Thu, Aug 6, 2015 at 2:23 PM, haosdent <[email protected]> wrote: >>>>>>> >>>>>>>> Hi, @Stephen From your slave log, could not see the restart log >>>>>>>> about slave. Are you sure you restart slave after reboot? >>>>>>>> >>>>>>>> On Thu, Aug 6, 2015 at 5:54 PM, Stephen Knight <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Klaus, I have attached all from a master and a slave. >>>>>>>>> >>>>>>>>> I've replicated the problem over and over again, not sure what to >>>>>>>>> make of it. First registration is fine but then if I reboot the >>>>>>>>> service for >>>>>>>>> mesos-slave (process restart of full server restart) it never connects >>>>>>>>> again. >>>>>>>>> >>>>>>>>> The VM's are in the same VPC on AWS with an open security group >>>>>>>>> between them. >>>>>>>>> >>>>>>>>> On Thu, Aug 6, 2015 at 12:41 PM, Klaus Ma <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Stephen, >>>>>>>>>> >>>>>>>>>> Would you share the log of master & slave? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Klaus >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2015年08月06日 16:07, Stephen Knight wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I was wondering if anyone can help me. I have a test setup, 1 >>>>>>>>>> master/zookeeper and 2 slaves on Ubuntu 14.04. >>>>>>>>>> >>>>>>>>>> When I initialize the slaves the first time it all works and they >>>>>>>>>> register with the master (I can see it on x.x.x.x:5050) but when I >>>>>>>>>> reboot >>>>>>>>>> those slaves for any reason, they never re-register. Am I missing >>>>>>>>>> something? >>>>>>>>>> >>>>>>>>>> Thx >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> --- >>>>>>>>>> Stephen Knight >>>>>>>>>> Infrastructure Consultant >>>>>>>>>> >>>>>>>>>> Pivotal Services @ EMC >>>>>>>>>> +971 (0)56 538 2071 >>>>>>>>>> >>>>>>>>>> [email protected] >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>> Pivotal.io >>>>>>>>>> >>>>>>>>>> Notice of Confidentiality - This email message is for the sole >>>>>>>>>> use of the intended recipient and may contain confidential and >>>>>>>>>> privileged >>>>>>>>>> information. Any unauthorized review, use, disclosure or >>>>>>>>>> distribution is >>>>>>>>>> prohibited. If you are not the intended recipient, please contact the >>>>>>>>>> sender by reply email and destroy all copies of the original message. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best Regards, >>>>>>>> Haosdent Huang >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Haosdent Huang >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Haosdent Huang >>>> >>> >>> >>> >>> -- >>> Best Regards, >>> Haosdent Huang >>> >> >> > > > -- > > https://github.com/mindscratch > https://www.google.com/+CraigWickesser > https://twitter.com/mind_scratch > https://twitter.com/craig_links > >
