Have you tried clearing out the data in the slave's work_dir? for example if work dir is /var/mesos, rm -rf /var/mesos/* then start the slave?
On Thu, Aug 6, 2015 at 6:55 AM, Stephen Knight <[email protected]> wrote: > My system doesn't support cat with systemctl for some reason but here is > the contents of /usr/lib/systemd/system/mesos-slave.service > > [Unit] > > Description=Mesos Slave > > After=network.target > > Wants=network.target > > > [Service] > > ExecStart=/usr/bin/mesos-init-wrapper slave > > KillMode=process > > Restart=always > > RestartSec=20 > > LimitNOFILE=16384 > > CPUAccounting=true > > MemoryAccounting=true > > > [Install] > > WantedBy=multi-user.target > > > What are the required flags to start it manually? > > On Thu, Aug 6, 2015 at 2:51 PM, haosdent <[email protected]> wrote: > >> Or you could try "systemctl cat mesos-slave.service" and show us the >> file content. >> >> On Thu, Aug 6, 2015 at 6:49 PM, haosdent <[email protected]> wrote: >> >>> From this message, I think "systemctl status mesos-slave.service -l" run >>> mesos-slave with uncorrect flags. And the status out of it is the help >>> message of slave. Could you try to start mesos-slave in manual way? Not >>> through systemctl. >>> >>> On Thu, Aug 6, 2015 at 6:41 PM, Stephen Knight <[email protected]> >>> wrote: >>> >>>> systemctl gives me the following output on CentOS: The command to start >>>> I ran was "systemctl start mesos-slave.service" >>>> >>>> [root@ip-172-31-35-167 mesos]# systemctl status mesos-slave.service -l >>>> >>>> mesos-slave.service - Mesos Slave >>>> >>>> Loaded: loaded (/usr/lib/systemd/system/mesos-slave.service; enabled) >>>> >>>> Drop-In: /etc/systemd/system/mesos-slave.service.d >>>> >>>> └─mesos-slave-containerizers.conf >>>> >>>> Active: activating (auto-restart) (Result: exit-code) since Thu >>>> 2015-08-06 10:38:08 UTC; 2s ago >>>> >>>> Process: 1472 ExecStart=/usr/bin/mesos-init-wrapper slave *(code=exited, >>>> status=1/FAILURE)* >>>> >>>> Main PID: 1472 (code=exited, status=1/FAILURE) >>>> >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *If >>>> strict=false, any expected errors (e.g., slave cannot recover* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>> *information >>>> about an executor, because the slave died right before* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *the >>>> executor registered.) during recovery are ignored and as much* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *state >>>> as possible is recovered.* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *(default: >>>> true)* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>> *--[no-]switch_user >>>> Whether to run tasks as the user who* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *submitted >>>> them rather than the user running* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: *the >>>> slave (requires setuid permission) (default: true)* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>> *--[no-]version >>>> Show version and exit. (default: false)* >>>> >>>> Aug 06 10:38:08 ip-172-31-35-167.ec2.internal mesos-slave[1483]: >>>> *--work_dir=VALUE >>>> Directory path to place framework work >>>> directories* >>>> >>>> >>>> >>>> I've also run strace against it, nothing sticks out: >>>> >>>> >>>> strace systemctl start mesos-slave.service >>>> >>>> execve("/bin/systemctl", ["systemctl", "start", "mesos-slave.service"], >>>> [/* 18 vars */]) = 0 >>>> >>>> brk(0) = 0x7f5c2af9f000 >>>> >>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>> 0) = 0x7f5c2a5c6000 >>>> >>>> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or >>>> directory) >>>> >>>> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> fstat(3, {st_mode=S_IFREG|0644, st_size=20940, ...}) = 0 >>>> >>>> mmap(NULL, 20940, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5c2a5c0000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libsystemd-daemon.so.0", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\r\0\0\0\0\0\0"..., 832) >>>> = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=15216, ...}) = 0 >>>> >>>> mmap(NULL, 2109448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c2a1a2000 >>>> >>>> mprotect(0x7f5c2a1a4000, 2097152, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c2a3a4000, 4096, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f5c2a3a4000 >>>> >>>> mmap(0x7f5c2a3a5000, 8, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c2a3a5000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libdbus-1.so.3", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@x\0\0\0\0\0\0"..., >>>> 832) = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=304536, ...}) = 0 >>>> >>>> mmap(NULL, 2390496, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c29f5a000 >>>> >>>> mprotect(0x7f5c29fa0000, 2097152, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c2a1a0000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x46000) = 0x7f5c2a1a0000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\"\0\0\0\0\0\0"..., 832) >>>> = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=44088, ...}) = 0 >>>> >>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>> 0) = 0x7f5c2a5bf000 >>>> >>>> mmap(NULL, 2128952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c29d52000 >>>> >>>> mprotect(0x7f5c29d59000, 2093056, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c29f58000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f5c29f58000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240d\0\0\0\0\0\0"..., 832) >>>> = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=147120, ...}) = 0 >>>> >>>> mmap(NULL, 2246784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c29b2d000 >>>> >>>> mprotect(0x7f5c29b4e000, 2097152, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c29d4e000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f5c29d4e000 >>>> >>>> mmap(0x7f5c29d50000, 6272, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c29d50000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000/\0\0\0\0\0\0"..., 832) = >>>> 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=153184, ...}) = 0 >>>> >>>> mmap(NULL, 2245240, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c29908000 >>>> >>>> mprotect(0x7f5c2992c000, 2093056, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c29b2b000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f5c29b2b000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libgcrypt.so.11", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0u\0\0\0\0\0\0"..., 832) = >>>> 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=534488, ...}) = 0 >>>> >>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>> 0) = 0x7f5c2a5be000 >>>> >>>> mmap(NULL, 2621456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c29687000 >>>> >>>> mprotect(0x7f5c29703000, 2097152, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c29903000, 16384, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7c000) = 0x7f5c29903000 >>>> >>>> mmap(0x7f5c29907000, 16, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c29907000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360*\0\0\0\0\0\0"..., 832) >>>> = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=88720, ...}) = 0 >>>> >>>> mmap(NULL, 2184192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c29471000 >>>> >>>> mprotect(0x7f5c29486000, 2093056, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c29685000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7f5c29685000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\34\2\0\0\0\0\0"..., 832) >>>> = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=2107760, ...}) = 0 >>>> >>>> mmap(NULL, 3932736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c290b0000 >>>> >>>> mprotect(0x7f5c29266000, 2097152, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c29466000, 24576, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f5c29466000 >>>> >>>> mmap(0x7f5c2946c000, 16960, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c2946c000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., >>>> 832) = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=19512, ...}) = 0 >>>> >>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>> 0) = 0x7f5c2a5bd000 >>>> >>>> mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c28eac000 >>>> >>>> mprotect(0x7f5c28eaf000, 2093056, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c290ae000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f5c290ae000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240l\0\0\0\0\0\0"..., 832) >>>> = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=141616, ...}) = 0 >>>> >>>> mmap(NULL, 2208864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c28c90000 >>>> >>>> mprotect(0x7f5c28ca6000, 2097152, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c28ea6000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f5c28ea6000 >>>> >>>> mmap(0x7f5c28ea8000, 13408, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c28ea8000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., >>>> 832) = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=398272, ...}) = 0 >>>> >>>> mmap(NULL, 2490888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c28a2f000 >>>> >>>> mprotect(0x7f5c28a8e000, 2097152, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c28c8e000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5f000) = 0x7f5c28c8e000 >>>> >>>> close(3) = 0 >>>> >>>> open("/lib64/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> read(3, >>>> "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\n\0\0\0\0\0\0"..., 832) >>>> = 832 >>>> >>>> fstat(3, {st_mode=S_IFREG|0755, st_size=19384, ...}) = 0 >>>> >>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>> 0) = 0x7f5c2a5bc000 >>>> >>>> mmap(NULL, 2113656, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, >>>> 0) = 0x7f5c2882a000 >>>> >>>> mprotect(0x7f5c2882e000, 2093056, PROT_NONE) = 0 >>>> >>>> mmap(0x7f5c28a2d000, 8192, PROT_READ|PROT_WRITE, >>>> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f5c28a2d000 >>>> >>>> close(3) = 0 >>>> >>>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>> 0) = 0x7f5c2a5bb000 >>>> >>>> mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, >>>> 0) = 0x7f5c2a5b9000 >>>> >>>> arch_prctl(ARCH_SET_FS, 0x7f5c2a5b9880) = 0 >>>> >>>> mprotect(0x7f5c29466000, 16384, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c28a2d000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c28ea6000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c28c8e000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c290ae000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c29685000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c29903000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c29b2b000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c29d4e000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c29f58000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c2a1a0000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c2a3a4000, 4096, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c2a81d000, 8192, PROT_READ) = 0 >>>> >>>> mprotect(0x7f5c2a5c7000, 4096, PROT_READ) = 0 >>>> >>>> munmap(0x7f5c2a5c0000, 20940) = 0 >>>> >>>> set_tid_address(0x7f5c2a5b9b50) = 1545 >>>> >>>> set_robust_list(0x7f5c2a5b9b60, 24) = 0 >>>> >>>> rt_sigaction(SIGRTMIN, {0x7f5c28c96780, [], SA_RESTORER|SA_SIGINFO, >>>> 0x7f5c28c9f130}, NULL, 8) = 0 >>>> >>>> rt_sigaction(SIGRT_1, {0x7f5c28c96810, [], >>>> SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f5c28c9f130}, NULL, 8) = 0 >>>> >>>> rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 >>>> >>>> getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) >>>> = 0 >>>> >>>> brk(0) = 0x7f5c2af9f000 >>>> >>>> brk(0x7f5c2afc0000) = 0x7f5c2afc0000 >>>> >>>> access("/etc/system-fips", F_OK) = -1 ENOENT (No such file or >>>> directory) >>>> >>>> statfs("/sys/fs/selinux", {f_type=0xf97cff8c, f_bsize=4096, f_blocks=0, >>>> f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, >>>> f_frsize=4096}) = 0 >>>> >>>> statfs("/sys/fs/selinux", {f_type=0xf97cff8c, f_bsize=4096, f_blocks=0, >>>> f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, >>>> f_frsize=4096}) = 0 >>>> >>>> stat("/sys/fs/selinux", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 >>>> >>>> open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 >>>> >>>> fstat(3, {st_mode=S_IFREG|0644, st_size=106065056, ...}) = 0 >>>> >>>> mmap(NULL, 106065056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5c22303000 >>>> >>>> close(3) = 0 >>>> >>>> ioctl(1, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or >>>> TCGETS, {B9600 opost isig icanon echo ...}) = 0 >>>> >>>> stat("/proc/1/root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>> >>>> stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>> >>>> stat("/proc/1/root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>> >>>> stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>> >>>> lstat("/run/systemd/system/", {st_mode=S_IFDIR|0755, st_size=80, ...}) >>>> = 0 >>>> >>>> geteuid() = 0 >>>> >>>> socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3 >>>> >>>> connect(3, {sa_family=AF_LOCAL, sun_path="/run/systemd/private"}, 22) = >>>> 0 >>>> >>>> fcntl(3, F_GETFL) = 0x2 (flags O_RDWR) >>>> >>>> fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0 >>>> >>>> geteuid() = 0 >>>> >>>> getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0 >>>> >>>> getsockopt(3, SOL_SOCKET, SO_PEERCRED, {pid=1, uid=0, gid=0}, [12]) = 0 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 716107629}) = 0 >>>> >>>> poll([{fd=3, events=POLLOUT}], 1, 90000) = 1 ([{fd=3, revents=POLLOUT}]) >>>> >>>> sendto(3, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1 >>>> >>>> sendto(3, "AUTH EXTERNAL 30\r\n", 18, MSG_NOSIGNAL, NULL, 0) = 18 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 716225119}) = 0 >>>> >>>> poll([{fd=3, events=POLLIN}], 1, 90000) = 1 ([{fd=3, revents=POLLIN}]) >>>> >>>> read(3, "OK c2d5f933b89e938a29b77c0355c33"..., 2048) = 37 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 716481724}) = 0 >>>> >>>> poll([{fd=3, events=POLLOUT}], 1, 90000) = 1 ([{fd=3, revents=POLLOUT}]) >>>> >>>> sendto(3, "NEGOTIATE_UNIX_FD\r\n", 19, MSG_NOSIGNAL, NULL, 0) = 19 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 716552147}) = 0 >>>> >>>> poll([{fd=3, events=POLLIN}], 1, 90000) = 1 ([{fd=3, revents=POLLIN}]) >>>> >>>> read(3, "AGREE_UNIX_FD\r\n", 2048) = 15 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 716708588}) = 0 >>>> >>>> poll([{fd=3, events=POLLOUT}], 1, 90000) = 1 ([{fd=3, revents=POLLOUT}]) >>>> >>>> sendto(3, "BEGIN\r\n", 7, MSG_NOSIGNAL, NULL, 0) = 7 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 716773800}) = 0 >>>> >>>> stat("/proc/1/root", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>> >>>> stat("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 >>>> >>>> ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or >>>> TCGETS, {B9600 opost isig icanon echo ...}) = 0 >>>> >>>> rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0 >>>> >>>> clone(child_stack=0, >>>> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, >>>> child_tidptr=0x7f5c2a5b9b50) = 1546 >>>> >>>> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 >>>> >>>> gettid() = 1545 >>>> >>>> sendmsg(3, {msg_name(0)=NULL, >>>> msg_iov(2)=[{"l\1\0\1$\0\0\0\1\0\0\0\240\0\0\0\1\1o\0\31\0\0\0/org/fre"..., >>>> 176}, {"\23\0\0\0mesos-slave.service\0\7\0\0\0repl"..., 36}], >>>> msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 212 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 717164627}) = 0 >>>> >>>> poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}]) >>>> >>>> recvmsg(3, {msg_name(0)=NULL, >>>> msg_iov(1)=[{"l\2\1\1'\0\0\0\1\0\0\0\17\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1o\0\0"..., >>>> 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = >>>> 1143 >>>> >>>> recvmsg(3, 0x7fff05789c50, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> >>>> sendmsg(3, {msg_name(0)=NULL, >>>> msg_iov(2)=[{"l\1\0\1\30\0\0\0\2\0\0\0\227\0\0\0\1\1o\0\31\0\0\0/org/fre"..., >>>> 168}, {"\23\0\0\0mesos-slave.service\0", 24}], msg_controllen=0, >>>> msg_flags=0}, MSG_NOSIGNAL) = 192 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 719385633}) = 0 >>>> >>>> poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}]) >>>> >>>> recvmsg(3, {msg_name(0)=NULL, >>>> msg_iov(1)=[{"l\2\1\1;\0\0\0\2\0\0\0\17\0\0\0\5\1u\0\2\0\0\0\10\1g\0\1o\0\0"..., >>>> 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = >>>> 91 >>>> >>>> recvmsg(3, 0x7fff05789c50, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> >>>> sendmsg(3, {msg_name(0)=NULL, >>>> msg_iov(2)=[{"l\1\0\0019\0\0\0\3\0\0\0\250\0\0\0\1\1o\0006\0\0\0/org/fre"..., >>>> 184}, {"\35\0\0\0org.freedesktop.systemd1.Uni"..., 57}], msg_controllen=0, >>>> msg_flags=0}, MSG_NOSIGNAL) = 241 >>>> >>>> clock_gettime(CLOCK_MONOTONIC, {774, 719741008}) = 0 >>>> >>>> poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}]) >>>> >>>> recvmsg(3, {msg_name(0)=NULL, >>>> msg_iov(1)=[{"l\2\1\1\10\0\0\0\3\0\0\0\17\0\0\0\5\1u\0\3\0\0\0\10\1g\0\1v\0\0"..., >>>> 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = >>>> 40 >>>> >>>> recvmsg(3, 0x7fff05789c50, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> >>>> poll([{fd=3, events=POLLIN}], 1, 4294967295) = 1 ([{fd=3, >>>> revents=POLLIN}]) >>>> >>>> recvmsg(3, {msg_name(0)=NULL, >>>> msg_iov(1)=[{"l\4\1\1M\0\0\0\206\1\0\0z\0\0\0\1\1o\0\31\0\0\0/org/fre"..., >>>> 2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = >>>> 1089 >>>> >>>> recvmsg(3, 0x7fff0578a030, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> >>>> close(3) = 0 >>>> >>>> kill(1546, SIGTERM) = 0 >>>> >>>> kill(1546, SIGCONT) = 0 >>>> >>>> waitid(P_PID, 1546, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1546, >>>> si_status=0, si_utime=0, si_stime=0}, WEXITED, NULL) = 0 >>>> >>>> --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1546, >>>> si_status=0, si_utime=0, si_stime=0} --- >>>> >>>> exit_group(0) = ? >>>> >>>> +++ exited with 0 +++ >>>> >>>> >>>> On Thu, Aug 6, 2015 at 2:36 PM, haosdent <[email protected]> wrote: >>>> >>>>> You use service mesos-slave to start slave failed so that it could not >>>>> connect to master, right? Does it have any log about this? >>>>> >>>>> On Thu, Aug 6, 2015 at 6:30 PM, Stephen Knight <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Yes, I have tried on both environments (one CentOS 7 and one Ubuntu) >>>>>> and they behave the exact same way. I probably should clarify though that >>>>>> after a restart the agent fails to start on the slave, which is why it >>>>>> never reaches the master. >>>>>> >>>>>> I can however run a smoke test and get it to connect from a slave to >>>>>> the master by passing a job, but the slave agent won't start. The guide I >>>>>> followed is at: >>>>>> http://open.mesosphere.com/getting-started/datacenter/install/ >>>>>> >>>>>> The environment is by no means live, I can share details with you if >>>>>> you can help me to that degree. >>>>>> >>>>>> On Thu, Aug 6, 2015 at 2:23 PM, haosdent <[email protected]> wrote: >>>>>> >>>>>>> Hi, @Stephen From your slave log, could not see the restart log >>>>>>> about slave. Are you sure you restart slave after reboot? >>>>>>> >>>>>>> On Thu, Aug 6, 2015 at 5:54 PM, Stephen Knight <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Klaus, I have attached all from a master and a slave. >>>>>>>> >>>>>>>> I've replicated the problem over and over again, not sure what to >>>>>>>> make of it. First registration is fine but then if I reboot the >>>>>>>> service for >>>>>>>> mesos-slave (process restart of full server restart) it never connects >>>>>>>> again. >>>>>>>> >>>>>>>> The VM's are in the same VPC on AWS with an open security group >>>>>>>> between them. >>>>>>>> >>>>>>>> On Thu, Aug 6, 2015 at 12:41 PM, Klaus Ma <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Stephen, >>>>>>>>> >>>>>>>>> Would you share the log of master & slave? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Klaus >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2015年08月06日 16:07, Stephen Knight wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I was wondering if anyone can help me. I have a test setup, 1 >>>>>>>>> master/zookeeper and 2 slaves on Ubuntu 14.04. >>>>>>>>> >>>>>>>>> When I initialize the slaves the first time it all works and they >>>>>>>>> register with the master (I can see it on x.x.x.x:5050) but when I >>>>>>>>> reboot >>>>>>>>> those slaves for any reason, they never re-register. Am I missing >>>>>>>>> something? >>>>>>>>> >>>>>>>>> Thx >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> --- >>>>>>>>> Stephen Knight >>>>>>>>> Infrastructure Consultant >>>>>>>>> >>>>>>>>> Pivotal Services @ EMC >>>>>>>>> +971 (0)56 538 2071 >>>>>>>>> >>>>>>>>> [email protected] >>>>>>>>> [email protected] >>>>>>>>> >>>>>>>>> Pivotal.io >>>>>>>>> >>>>>>>>> Notice of Confidentiality - This email message is for the sole use >>>>>>>>> of the intended recipient and may contain confidential and privileged >>>>>>>>> information. Any unauthorized review, use, disclosure or distribution >>>>>>>>> is >>>>>>>>> prohibited. If you are not the intended recipient, please contact the >>>>>>>>> sender by reply email and destroy all copies of the original message. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, >>>>>>> Haosdent Huang >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, >>>>> Haosdent Huang >>>>> >>>> >>>> >>> >>> >>> -- >>> Best Regards, >>> Haosdent Huang >>> >> >> >> >> -- >> Best Regards, >> Haosdent Huang >> > > -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
