> On March 13, 2013, 8:03 p.m., Thomas Marshall wrote: > > After applying this patch, I get the following output: > > > > > make check > > ... > > [ RUN ] SlaveRecoveryTest/0.ReconnectExecutor > > Checkpointing SlaveInfo to > > '/tmp/SlaveRecoveryTest_0_ReconnectExecutor_7r5qCw/meta/slaves/201303131255-1883310602-49297-1281-0/slave.info' > > Checkpointing FrameworkInfo to > > '/tmp/SlaveRecoveryTest_0_ReconnectExecutor_7r5qCw/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/framework.info' > > Checkpointing 'scheduler(14)@10.10.65.112:49297' to > > /tmp/SlaveRecoveryTest_0_ReconnectExecutor_7r5qCw/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/framework.pid > > Checkpointing ExecutorInfo to > > '/tmp/SlaveRecoveryTest_0_ReconnectExecutor_7r5qCw/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/88f6493c-9dd2-46b3-86b7-1f89d6b57ad3/executor.info' > > Checkpointing Task to > > '/tmp/SlaveRecoveryTest_0_ReconnectExecutor_7r5qCw/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/88f6493c-9dd2-46b3-86b7-1f89d6b57ad3/runs/96eb66d8-612e-47d2-8fec-7f0c980c0ced/tasks/88f6493c-9dd2-46b3-86b7-1f89d6b57ad3/task.info' > > Checkpointing forked pid 1334 > > Checkpointing '1334' to > > /tmp/SlaveRecoveryTest_0_ReconnectExecutor_7r5qCw/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/88f6493c-9dd2-46b3-86b7-1f89d6b57ad3/runs/96eb66d8-612e-47d2-8fec-7f0c980c0ced/pids/forked.pid > > Fetching resources into > > /tmp/SlaveRecoveryTest_0_ReconnectExecutor_7r5qCw/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/88f6493c-9dd2-46b3-86b7-1f89d6b57ad3/runs/96eb66d8-612e-47d2-8fec-7f0c980c0ced > > Checkpointing 'executor(1)@10.10.65.112:49325' to > > /tmp/SlaveRecoveryTest_0_ReconnectExecutor_7r5qCw/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/88f6493c-9dd2-46b3-86b7-1f89d6b57ad3/runs/96eb66d8-612e-47d2-8fec-7f0c980c0ced/pids/libprocess.pid > > Registered executor on thomass-macbook-pro-4.local > > Starting task 88f6493c-9dd2-46b3-86b7-1f89d6b57ad3 > > sh -c 'sleep 1000' > > Re-registered executor on thomass-macbook-pro-4.local > > Waited on process 1350, returned status 15 > > [ OK ] SlaveRecoveryTest/0.ReconnectExecutor (138 ms) > > [ RUN ] SlaveRecoveryTest/0.RecoverUnregisteredExecutor > > Checkpointing SlaveInfo to > > '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_CuAC1i/meta/slaves/201303131255-1883310602-49297-1281-0/slave.info' > > Checkpointing FrameworkInfo to > > '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_CuAC1i/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/framework.info' > > Checkpointing 'scheduler(15)@10.10.65.112:49297' to > > /tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_CuAC1i/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/framework.pid > > Checkpointing ExecutorInfo to > > '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_CuAC1i/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/8fe83fa2-e257-4a2e-8436-50bd6704980e/executor.info' > > Checkpointing Task to > > '/tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_CuAC1i/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/8fe83fa2-e257-4a2e-8436-50bd6704980e/runs/f1ec30f0-f9ff-4868-b6f4-ef0ca3a43522/tasks/8fe83fa2-e257-4a2e-8436-50bd6704980e/task.info' > > Checkpointing forked pid 1351 > > Checkpointing '1351' to > > /tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_CuAC1i/meta/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/8fe83fa2-e257-4a2e-8436-50bd6704980e/runs/f1ec30f0-f9ff-4868-b6f4-ef0ca3a43522/pids/forked.pid > > Fetching resources into > > /tmp/SlaveRecoveryTest_0_RecoverUnregisteredExecutor_CuAC1i/slaves/201303131255-1883310602-49297-1281-0/frameworks/201303131255-1883310602-49297-1281-0000/executors/8fe83fa2-e257-4a2e-8436-50bd6704980e/runs/f1ec30f0-f9ff-4868-b6f4-ef0ca3a43522 > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > ../../src/tests/slave_recovery_tests.cpp:634: Failure > > Failed > > Waited too long for 'statusUpdateCall' > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > dyld: DYLD_ environment variables being ignored because main executable > > (/bin/ps) is setuid or setgid > > ... > > > > > > The "dyld: DYLD_ environment variables being ignored because main > > executable (/bin/ps) is setuid or setgid" also occurs if, for example, if I > > run mesos-master.sh, mesos-slave.sh, and test-framework, the slave will > > output this error after the framework exits, and if I don't kill it almost > > immediately it crashes my entire machine. Again, running this on OSX > > Mountain Lion. > > Thomas Marshall wrote: > Additionally, updating to this patch doesn't seem to have any effect on > the original problem when running on Ubuntu. > > Vinod Kone wrote: > Looks like, one of the bugs here is the fact that 'ps' on MacOSX 10.8 > reports session id as "0" for every process on the system! This manifests as > isolation module trying to kill all the processes (via "killtree.sh") when it > tries to kill an executor! > > I have tested with a fix in killtree.sh, to guard against session ids > being "0". This unbreaks the test. Still digging on the reason behind the > 'dyld' warnings above!
OK. The DYLD warnings are apparently because the killtree script is being called with DYLD_LIBRARY_PATH being set by the mesos slave. This doesn't play nice when running setuid programs like /bin/ps. I will send a fix out. Here is the env the killtree.sh was operating under. TERM_PROGRAM=Apple_Terminal SHELL=/bin/zsh TERM=xterm-256color MESOS_WEBUI_DIR=/Users/vinod/workspace/apache/mesos/build/../src/webui TMPDIR=/var/folders/9w/j_qrfhv52g7fbnnxgx6qxr4r0000gq/T/ Apple_PubSub_Socket_Render=/tmp/launch-RWxoYW/Render TERM_PROGRAM_VERSION=309 TERM_SESSION_ID=72203344-4A3F-44EF-BE01-86CCF7832991 DUALCASE=1 MESOS_LAUNCHER_DIR=/Users/vinod/workspace/apache/mesos/build/src USER=vinod COMMAND_MODE=unix2003 SSH_AUTH_SOCK=/tmp/launch-e4WAKU/Listeners Apple_Ubiquity_Message=/tmp/launch-unwDEU/Apple_Ubiquity_Message __CF_USER_TEXT_ENCODING=0x1F7:0:0 PAGER=less LSCOLORS=Gxfxcxdxbxegedabagacad GLOG_v=1 PWD=/Users/vinod/workspace/apache/mesos/build LANG=en_US.UTF-8 BIN_SH=xpg4 HOME=/Users/vinod SHLVL=3 MESOS_KILLTREE=/Users/vinod/workspace/apache/mesos/build/../src/scripts/killtree.sh GREP_OPTIONS=--color=auto DYLD_LIBRARY_PATH=/Users/vinod/workspace/apache/mesos/build/src/.libs LOGNAME=vinod LESS= -R LC_CTYPE=en_US.UTF-8 LESSOPEN=| /usr/local/Cellar/source-highlight/3.1.7/bin/src-hilite-lesspipe.sh %s DISPLAY=/tmp/launch-l8T1bm/org.macosforge.xquartz:0 MESOS_SOURCE_DIR=/Users/vinod/workspace/apache/mesos - Vinod ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9907/#review17822 ----------------------------------------------------------- On March 13, 2013, 7:13 p.m., Vinod Kone wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/9907/ > ----------------------------------------------------------- > > (Updated March 13, 2013, 7:13 p.m.) > > > Review request for mesos, Benjamin Hindman, Thomas Marshall, and Ben Mahler. > > > Description > ------- > > See summary. > > > Diffs > ----- > > src/tests/slave_recovery_tests.cpp 2a2fa281bba8e00e93a770828f5a93aff5539cf4 > > Diff: https://reviews.apache.org/r/9907/diff/ > > > Testing > ------- > > make check > > > Thanks, > > Vinod Kone > >
