Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
> On March 22, 2016, 1:02 a.m., Joshua Cohen wrote: > > Sorry for the delay on this. After you filed the pull request, I > > investigated a bit what will be required once Mesos 0.30.0 lands: > > https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes > > beyond the failure to find `sys.executable` when $PATH is not set. As even > > after switching back to chmod+x on the runner, the task failed further down > > the stack. > > > > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which > > should allow `sys.executable` to continue working and will allow any tasks > > users have running which have come to rely on Thermos setting it for them > > to behave as expected. The problem is, I haven't had time to figure out > > what we should set $PATH to yet ;) (anyone have any thoughts?). > > > > I know this is probably more info than you bargained for when you opened > > what seemed like a simple pull request. I'm not opposed to accepting this > > patch (with a TODO to restore `sys.executable` when we figure out what to > > do about setting $PATH) if it unblocks your use case, but can you confirm > > that you're actually able to run the Mesos agent with > > `--executor_environment_variables='{}'` and still launch tasks? > > Pierre Cheynier wrote: > Hi, > > Sorry for my delay on opening this review on apache.org. > Actually, I started to use Aurora and faced this issue cause my setup use > `--executor_environment_variables`. > I first investigated in the Python default setup on CentOS, the pants/pex > build system and then the differences between the vagrant box provided in the > repo and mine before discovering that this is the origin of the issue. > > I admit this is a simple fix, I just tried to understand the reason of > using this mechanism and found that it was changed 2 years ago. > In 0.27.2 it works well by using this patch (I'm now able to run > something). > > Joshua Cohen wrote: > So he's the behavior I see when I run the Mesos agent with > `--executor_environment_variables='{}'`. Without this patch, as you note, > tasks fail to start with a permission denied error (due to `sys.executable` > being unset). After applying this patch, tasks start up, but processes fail > to fork: > https://www.dropbox.com/s/7zzy574xdy02ukq/Screen%20Shot%202016-03-24%20at%203.49.24%20PM.png?dl=0. > > Are you by chance running Docker tasks? > > Pierre Cheynier wrote: > I'm running tasks on the Mesos containerizer for now. > Your problem is not purely a side-effect of not passing any environment > variables ? (your tasks themselves maybe need PATH or LD_LIBRARY_PATH ?) > > Joshua Cohen wrote: > It's not the task that requires PATH to be set, it's the thermos runner > itself: > https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/core/process.py#L397-L402 > > This is why I'm confused as to how this works for you unless PATH is > somehow being set elsewhere (and if PATH is set elsewhere, then > `sys.executable` should work as well). Using the Mesos Containerizer, it probably depends on the shell used and the need for the tasks itself to get a specific environment. But I'm really surprised, cause even when I'm using your vagrant setup (patch cherry-picked on top of the master branch + vagrant up), I'm able to launch your two sample tasks (hello_world + cron_hello_world). I can provide screenshots if needed. The portion of code you pointed me to is supposed to enrich the environment of the subshell. I presume that if the child absolutely need the PATH/LD_LIBRARY_PATH/whatever, the operator will have to add it in --executor_environment_variable OR the user will add it by itself. This is what it's supposed to happen when Mesos 0.30 will be released, right ? - Pierre --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/#review124697 --- On March 22, 2016, 9:50 a.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45104/ > --- > > (Updated March 22, 2016, 9:50 a.m.) > > > Review request for Aurora, Joshua Cohen and Bill Farner. > > > Repository: aurora > > > Description > --- > > When using `--executor_environment_variables` without explicitely > passing LD_LIBRARY_PATH, `sys.executable` returns an empty string > resulting in a '[Errno 13] Permission denied' error for every launched > task. > > Moreover, it seems that this feature is coming in 0.30: "Executors no > longer inherit environment variables from the agent". > > This patch partially revert back 07ce21d where chmod_x method was > removed in favor
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
> On March 22, 2016, 1:02 a.m., Joshua Cohen wrote: > > Sorry for the delay on this. After you filed the pull request, I > > investigated a bit what will be required once Mesos 0.30.0 lands: > > https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes > > beyond the failure to find `sys.executable` when $PATH is not set. As even > > after switching back to chmod+x on the runner, the task failed further down > > the stack. > > > > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which > > should allow `sys.executable` to continue working and will allow any tasks > > users have running which have come to rely on Thermos setting it for them > > to behave as expected. The problem is, I haven't had time to figure out > > what we should set $PATH to yet ;) (anyone have any thoughts?). > > > > I know this is probably more info than you bargained for when you opened > > what seemed like a simple pull request. I'm not opposed to accepting this > > patch (with a TODO to restore `sys.executable` when we figure out what to > > do about setting $PATH) if it unblocks your use case, but can you confirm > > that you're actually able to run the Mesos agent with > > `--executor_environment_variables='{}'` and still launch tasks? > > Pierre Cheynier wrote: > Hi, > > Sorry for my delay on opening this review on apache.org. > Actually, I started to use Aurora and faced this issue cause my setup use > `--executor_environment_variables`. > I first investigated in the Python default setup on CentOS, the pants/pex > build system and then the differences between the vagrant box provided in the > repo and mine before discovering that this is the origin of the issue. > > I admit this is a simple fix, I just tried to understand the reason of > using this mechanism and found that it was changed 2 years ago. > In 0.27.2 it works well by using this patch (I'm now able to run > something). > > Joshua Cohen wrote: > So he's the behavior I see when I run the Mesos agent with > `--executor_environment_variables='{}'`. Without this patch, as you note, > tasks fail to start with a permission denied error (due to `sys.executable` > being unset). After applying this patch, tasks start up, but processes fail > to fork: > https://www.dropbox.com/s/7zzy574xdy02ukq/Screen%20Shot%202016-03-24%20at%203.49.24%20PM.png?dl=0. > > Are you by chance running Docker tasks? > > Pierre Cheynier wrote: > I'm running tasks on the Mesos containerizer for now. > Your problem is not purely a side-effect of not passing any environment > variables ? (your tasks themselves maybe need PATH or LD_LIBRARY_PATH ?) It's not the task that requires PATH to be set, it's the thermos runner itself: https://github.com/apache/aurora/blob/master/src/main/python/apache/thermos/core/process.py#L397-L402 This is why I'm confused as to how this works for you unless PATH is somehow being set elsewhere (and if PATH is set elsewhere, then `sys.executable` should work as well). - Joshua --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/#review124697 --- On March 22, 2016, 9:50 a.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45104/ > --- > > (Updated March 22, 2016, 9:50 a.m.) > > > Review request for Aurora, Joshua Cohen and Bill Farner. > > > Repository: aurora > > > Description > --- > > When using `--executor_environment_variables` without explicitely > passing LD_LIBRARY_PATH, `sys.executable` returns an empty string > resulting in a '[Errno 13] Permission denied' error for every launched > task. > > Moreover, it seems that this feature is coming in 0.30: "Executors no > longer inherit environment variables from the agent". > > This patch partially revert back 07ce21d where chmod_x method was > removed in favor of using sys.executable. > > > Diffs > - > > src/main/python/apache/aurora/executor/thermos_task_runner.py > 3896e3841562600379705dbf78a6f62728246348 > > Diff: https://reviews.apache.org/r/45104/diff/ > > > Testing > --- > > Make Aurora executor run on CentOS7, while running mesos agent with > `--executor_environment_variables` option and no excplicit $PATH set. > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
> On mars 22, 2016, 1:02 matin, Joshua Cohen wrote: > > Sorry for the delay on this. After you filed the pull request, I > > investigated a bit what will be required once Mesos 0.30.0 lands: > > https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes > > beyond the failure to find `sys.executable` when $PATH is not set. As even > > after switching back to chmod+x on the runner, the task failed further down > > the stack. > > > > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which > > should allow `sys.executable` to continue working and will allow any tasks > > users have running which have come to rely on Thermos setting it for them > > to behave as expected. The problem is, I haven't had time to figure out > > what we should set $PATH to yet ;) (anyone have any thoughts?). > > > > I know this is probably more info than you bargained for when you opened > > what seemed like a simple pull request. I'm not opposed to accepting this > > patch (with a TODO to restore `sys.executable` when we figure out what to > > do about setting $PATH) if it unblocks your use case, but can you confirm > > that you're actually able to run the Mesos agent with > > `--executor_environment_variables='{}'` and still launch tasks? > > Pierre Cheynier wrote: > Hi, > > Sorry for my delay on opening this review on apache.org. > Actually, I started to use Aurora and faced this issue cause my setup use > `--executor_environment_variables`. > I first investigated in the Python default setup on CentOS, the pants/pex > build system and then the differences between the vagrant box provided in the > repo and mine before discovering that this is the origin of the issue. > > I admit this is a simple fix, I just tried to understand the reason of > using this mechanism and found that it was changed 2 years ago. > In 0.27.2 it works well by using this patch (I'm now able to run > something). > > Joshua Cohen wrote: > So he's the behavior I see when I run the Mesos agent with > `--executor_environment_variables='{}'`. Without this patch, as you note, > tasks fail to start with a permission denied error (due to `sys.executable` > being unset). After applying this patch, tasks start up, but processes fail > to fork: > https://www.dropbox.com/s/7zzy574xdy02ukq/Screen%20Shot%202016-03-24%20at%203.49.24%20PM.png?dl=0. > > Are you by chance running Docker tasks? I'm running tasks on the Mesos containerizer for now. Your problem is not purely a side-effect of not passing any environment variables ? (your tasks themselves maybe need PATH or LD_LIBRARY_PATH ?) - Pierre --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/#review124697 --- On mars 22, 2016, 9:50 matin, Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45104/ > --- > > (Updated mars 22, 2016, 9:50 matin) > > > Review request for Aurora, Joshua Cohen and Bill Farner. > > > Repository: aurora > > > Description > --- > > When using `--executor_environment_variables` without explicitely > passing LD_LIBRARY_PATH, `sys.executable` returns an empty string > resulting in a '[Errno 13] Permission denied' error for every launched > task. > > Moreover, it seems that this feature is coming in 0.30: "Executors no > longer inherit environment variables from the agent". > > This patch partially revert back 07ce21d where chmod_x method was > removed in favor of using sys.executable. > > > Diffs > - > > src/main/python/apache/aurora/executor/thermos_task_runner.py > 3896e3841562600379705dbf78a6f62728246348 > > Diff: https://reviews.apache.org/r/45104/diff/ > > > Testing > --- > > Make Aurora executor run on CentOS7, while running mesos agent with > `--executor_environment_variables` option and no excplicit $PATH set. > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
> On March 22, 2016, 1:02 a.m., Joshua Cohen wrote: > > Sorry for the delay on this. After you filed the pull request, I > > investigated a bit what will be required once Mesos 0.30.0 lands: > > https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes > > beyond the failure to find `sys.executable` when $PATH is not set. As even > > after switching back to chmod+x on the runner, the task failed further down > > the stack. > > > > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which > > should allow `sys.executable` to continue working and will allow any tasks > > users have running which have come to rely on Thermos setting it for them > > to behave as expected. The problem is, I haven't had time to figure out > > what we should set $PATH to yet ;) (anyone have any thoughts?). > > > > I know this is probably more info than you bargained for when you opened > > what seemed like a simple pull request. I'm not opposed to accepting this > > patch (with a TODO to restore `sys.executable` when we figure out what to > > do about setting $PATH) if it unblocks your use case, but can you confirm > > that you're actually able to run the Mesos agent with > > `--executor_environment_variables='{}'` and still launch tasks? > > Pierre Cheynier wrote: > Hi, > > Sorry for my delay on opening this review on apache.org. > Actually, I started to use Aurora and faced this issue cause my setup use > `--executor_environment_variables`. > I first investigated in the Python default setup on CentOS, the pants/pex > build system and then the differences between the vagrant box provided in the > repo and mine before discovering that this is the origin of the issue. > > I admit this is a simple fix, I just tried to understand the reason of > using this mechanism and found that it was changed 2 years ago. > In 0.27.2 it works well by using this patch (I'm now able to run > something). So he's the behavior I see when I run the Mesos agent with `--executor_environment_variables='{}'`. Without this patch, as you note, tasks fail to start with a permission denied error (due to `sys.executable` being unset). After applying this patch, tasks start up, but processes fail to fork: https://www.dropbox.com/s/7zzy574xdy02ukq/Screen%20Shot%202016-03-24%20at%203.49.24%20PM.png?dl=0. Are you by chance running Docker tasks? - Joshua --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/#review124697 --- On March 22, 2016, 9:50 a.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45104/ > --- > > (Updated March 22, 2016, 9:50 a.m.) > > > Review request for Aurora, Joshua Cohen and Bill Farner. > > > Repository: aurora > > > Description > --- > > When using `--executor_environment_variables` without explicitely > passing LD_LIBRARY_PATH, `sys.executable` returns an empty string > resulting in a '[Errno 13] Permission denied' error for every launched > task. > > Moreover, it seems that this feature is coming in 0.30: "Executors no > longer inherit environment variables from the agent". > > This patch partially revert back 07ce21d where chmod_x method was > removed in favor of using sys.executable. > > > Diffs > - > > src/main/python/apache/aurora/executor/thermos_task_runner.py > 3896e3841562600379705dbf78a6f62728246348 > > Diff: https://reviews.apache.org/r/45104/diff/ > > > Testing > --- > > Make Aurora executor run on CentOS7, while running mesos agent with > `--executor_environment_variables` option and no excplicit $PATH set. > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/ --- (Updated March 22, 2016, 9:50 a.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Repository: aurora Description --- When using `--executor_environment_variables` without explicitely passing LD_LIBRARY_PATH, `sys.executable` returns an empty string resulting in a '[Errno 13] Permission denied' error for every launched task. Moreover, it seems that this feature is coming in 0.30: "Executors no longer inherit environment variables from the agent". This patch partially revert back 07ce21d where chmod_x method was removed in favor of using sys.executable. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 3896e3841562600379705dbf78a6f62728246348 Diff: https://reviews.apache.org/r/45104/diff/ Testing (updated) --- Make Aurora executor run on CentOS7, while running mesos agent with `--executor_environment_variables` option and no excplicit $PATH set. Thanks, Pierre Cheynier
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
> On March 22, 2016, 1:02 a.m., Joshua Cohen wrote: > > Sorry for the delay on this. After you filed the pull request, I > > investigated a bit what will be required once Mesos 0.30.0 lands: > > https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes > > beyond the failure to find `sys.executable` when $PATH is not set. As even > > after switching back to chmod+x on the runner, the task failed further down > > the stack. > > > > I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which > > should allow `sys.executable` to continue working and will allow any tasks > > users have running which have come to rely on Thermos setting it for them > > to behave as expected. The problem is, I haven't had time to figure out > > what we should set $PATH to yet ;) (anyone have any thoughts?). > > > > I know this is probably more info than you bargained for when you opened > > what seemed like a simple pull request. I'm not opposed to accepting this > > patch (with a TODO to restore `sys.executable` when we figure out what to > > do about setting $PATH) if it unblocks your use case, but can you confirm > > that you're actually able to run the Mesos agent with > > `--executor_environment_variables='{}'` and still launch tasks? Hi, Sorry for my delay on opening this review on apache.org. Actually, I started to use Aurora and faced this issue cause my setup use `--executor_environment_variables`. I first investigated in the Python default setup on CentOS, the pants/pex build system and then the differences between the vagrant box provided in the repo and mine before discovering that this is the origin of the issue. I admit this is a simple fix, I just tried to understand the reason of using this mechanism and found that it was changed 2 years ago. In 0.27.2 it works well by using this patch (I'm now able to run something). - Pierre --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/#review124697 --- On March 21, 2016, 1:21 p.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45104/ > --- > > (Updated March 21, 2016, 1:21 p.m.) > > > Review request for Aurora, Joshua Cohen and Bill Farner. > > > Repository: aurora > > > Description > --- > > When using `--executor_environment_variables` without explicitely > passing LD_LIBRARY_PATH, `sys.executable` returns an empty string > resulting in a '[Errno 13] Permission denied' error for every launched > task. > > Moreover, it seems that this feature is coming in 0.30: "Executors no > longer inherit environment variables from the agent". > > This patch partially revert back 07ce21d where chmod_x method was > removed in favor of using sys.executable. > > > Diffs > - > > src/main/python/apache/aurora/executor/thermos_task_runner.py > 3896e3841562600379705dbf78a6f62728246348 > > Diff: https://reviews.apache.org/r/45104/diff/ > > > Testing > --- > > Make Aurora run on CentOS7 > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/#review124697 --- Sorry for the delay on this. After you filed the pull request, I investigated a bit what will be required once Mesos 0.30.0 lands: https://issues.apache.org/jira/browse/AURORA-1632. I think the problem goes beyond the failure to find `sys.executable` when $PATH is not set. As even after switching back to chmod+x on the runner, the task failed further down the stack. I suspect the fix for Mesos 0.30.0 will be to set our own $PATH which should allow `sys.executable` to continue working and will allow any tasks users have running which have come to rely on Thermos setting it for them to behave as expected. The problem is, I haven't had time to figure out what we should set $PATH to yet ;) (anyone have any thoughts?). I know this is probably more info than you bargained for when you opened what seemed like a simple pull request. I'm not opposed to accepting this patch (with a TODO to restore `sys.executable` when we figure out what to do about setting $PATH) if it unblocks your use case, but can you confirm that you're actually able to run the Mesos agent with `--executor_environment_variables='{}'` and still launch tasks? - Joshua Cohen On March 21, 2016, 1:21 p.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45104/ > --- > > (Updated March 21, 2016, 1:21 p.m.) > > > Review request for Aurora, Joshua Cohen and Bill Farner. > > > Repository: aurora > > > Description > --- > > When using `--executor_environment_variables` without explicitely > passing LD_LIBRARY_PATH, `sys.executable` returns an empty string > resulting in a '[Errno 13] Permission denied' error for every launched > task. > > Moreover, it seems that this feature is coming in 0.30: "Executors no > longer inherit environment variables from the agent". > > This patch partially revert back 07ce21d where chmod_x method was > removed in favor of using sys.executable. > > > Diffs > - > > src/main/python/apache/aurora/executor/thermos_task_runner.py > 3896e3841562600379705dbf78a6f62728246348 > > Diff: https://reviews.apache.org/r/45104/diff/ > > > Testing > --- > > Make Aurora run on CentOS7 > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/#review124582 --- Ship it! Ship It! - Bill Farner On March 21, 2016, 6:21 a.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45104/ > --- > > (Updated March 21, 2016, 6:21 a.m.) > > > Review request for Aurora, Joshua Cohen and Bill Farner. > > > Repository: aurora > > > Description > --- > > When using `--executor_environment_variables` without explicitely > passing LD_LIBRARY_PATH, `sys.executable` returns an empty string > resulting in a '[Errno 13] Permission denied' error for every launched > task. > > Moreover, it seems that this feature is coming in 0.30: "Executors no > longer inherit environment variables from the agent". > > This patch partially revert back 07ce21d where chmod_x method was > removed in favor of using sys.executable. > > > Diffs > - > > src/main/python/apache/aurora/executor/thermos_task_runner.py > 3896e3841562600379705dbf78a6f62728246348 > > Diff: https://reviews.apache.org/r/45104/diff/ > > > Testing > --- > > Make Aurora run on CentOS7 > > > Thanks, > > Pierre Cheynier > >
Re: Review Request 45104: Use chmod+x to make termos_runner.pex executable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/#review124544 --- Master (b24619b) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing "@ReviewBot retry" - Aurora ReviewBot On March 21, 2016, 1:21 p.m., Pierre Cheynier wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/45104/ > --- > > (Updated March 21, 2016, 1:21 p.m.) > > > Review request for Aurora, Joshua Cohen and Bill Farner. > > > Repository: aurora > > > Description > --- > > When using `--executor_environment_variables` without explicitely > passing LD_LIBRARY_PATH, `sys.executable` returns an empty string > resulting in a '[Errno 13] Permission denied' error for every launched > task. > > Moreover, it seems that this feature is coming in 0.30: "Executors no > longer inherit environment variables from the agent". > > This patch partially revert back 07ce21d where chmod_x method was > removed in favor of using sys.executable. > > > Diffs > - > > src/main/python/apache/aurora/executor/thermos_task_runner.py > 3896e3841562600379705dbf78a6f62728246348 > > Diff: https://reviews.apache.org/r/45104/diff/ > > > Testing > --- > > Make Aurora run on CentOS7 > > > Thanks, > > Pierre Cheynier > >
Review Request 45104: Use chmod+x to make termos_runner.pex executable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45104/ --- Review request for Aurora. Repository: aurora Description --- When using `--executor_environment_variables` without explicitely passing LD_LIBRARY_PATH, `sys.executable` returns an empty string resulting in a '[Errno 13] Permission denied' error for every launched task. Moreover, it seems that this feature is coming in 0.30: "Executors no longer inherit environment variables from the agent". This patch partially revert back 07ce21d where chmod_x method was removed in favor of using sys.executable. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 3896e3841562600379705dbf78a6f62728246348 Diff: https://reviews.apache.org/r/45104/diff/ Testing --- Make Aurora run on CentOS7 Thanks, Pierre Cheynier