Re: Review Request 27009: Use virtualenv to build pants instead of pex.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/#review57837 --- pants https://reviews.apache.org/r/27009/#comment98712 Why do we need to do this? - Zameer Manji On Oct. 22, 2014, 10:31 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/ --- (Updated Oct. 22, 2014, 10:31 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-876 https://issues.apache.org/jira/browse/AURORA-876 Repository: aurora Description --- Use virtualenv to build pants instead of pex. The goal of this change is to reduce CI flakiness by using `pip install` rather than `pex` to bootstrap pants. This still reaches out to external servers (and thus has the potential to be flaky) but is at least configurable (via `--default-timeout`). In a future review we can consider mirroring to svn.apache.org and disabling PyPI lookups. This requires a change to production executor code to use `sys.executable` rather than chmod+x-ing the runner PEX. Failure to do this results in the pants virtualenv's `site-packages` shadowing the dependencies in the `pex` (fairly obviously as there is a test failure related to psutil1/2 conflicts). I'm sure I'm papering over a PEX bug here but IMO this is fine as the only reason to execute the thermos runner PEX under a different python interpreter than the executor is using (based on the hash-bang in the pex) is not a configuration we need to support IMO. Diffs - pants 168419d7d8f5d89a0d87bfc6ac9f06c9e8e3dbc2 src/main/python/apache/aurora/executor/thermos_task_runner.py f17910826c31c5bf754b43eb72500de639652f37 Diff: https://reviews.apache.org/r/27009/diff/ Testing --- git clean -fdx ./pants src/test/python:all Thanks, Kevin Sweeney
Re: Review Request 27009: Use virtualenv to build pants instead of pex.
On Oct. 22, 2014, 10:59 a.m., Zameer Manji wrote: pants, line 40 https://reviews.apache.org/r/27009/diff/1/?file=728491#file728491line40 Why do we need to do this? We don't - removed. - Kevin --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/#review57837 --- On Oct. 22, 2014, 11:05 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/ --- (Updated Oct. 22, 2014, 11:05 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-876 https://issues.apache.org/jira/browse/AURORA-876 Repository: aurora Description --- Use virtualenv to build pants instead of pex. The goal of this change is to reduce CI flakiness by using `pip install` rather than `pex` to bootstrap pants. This still reaches out to external servers (and thus has the potential to be flaky) but is at least configurable (via `--default-timeout`). In a future review we can consider mirroring to svn.apache.org and disabling PyPI lookups. This requires a change to production executor code to use `sys.executable` rather than chmod+x-ing the runner PEX. Failure to do this results in the pants virtualenv's `site-packages` shadowing the dependencies in the `pex` (fairly obviously as there is a test failure related to psutil1/2 conflicts). I'm sure I'm papering over a PEX bug here but IMO this is fine as the only reason to execute the thermos runner PEX under a different python interpreter than the executor is using (based on the hash-bang in the pex) is not a configuration we need to support IMO. Diffs - pants 168419d7d8f5d89a0d87bfc6ac9f06c9e8e3dbc2 src/main/python/apache/aurora/executor/thermos_task_runner.py f17910826c31c5bf754b43eb72500de639652f37 Diff: https://reviews.apache.org/r/27009/diff/ Testing --- git clean -fdx ./pants src/test/python:all Thanks, Kevin Sweeney
Re: Review Request 27009: Use virtualenv to build pants instead of pex.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/ --- (Updated Oct. 22, 2014, 11:05 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- Removed debugging statements. Bugs: AURORA-876 https://issues.apache.org/jira/browse/AURORA-876 Repository: aurora Description --- Use virtualenv to build pants instead of pex. The goal of this change is to reduce CI flakiness by using `pip install` rather than `pex` to bootstrap pants. This still reaches out to external servers (and thus has the potential to be flaky) but is at least configurable (via `--default-timeout`). In a future review we can consider mirroring to svn.apache.org and disabling PyPI lookups. This requires a change to production executor code to use `sys.executable` rather than chmod+x-ing the runner PEX. Failure to do this results in the pants virtualenv's `site-packages` shadowing the dependencies in the `pex` (fairly obviously as there is a test failure related to psutil1/2 conflicts). I'm sure I'm papering over a PEX bug here but IMO this is fine as the only reason to execute the thermos runner PEX under a different python interpreter than the executor is using (based on the hash-bang in the pex) is not a configuration we need to support IMO. Diffs (updated) - pants 168419d7d8f5d89a0d87bfc6ac9f06c9e8e3dbc2 src/main/python/apache/aurora/executor/thermos_task_runner.py f17910826c31c5bf754b43eb72500de639652f37 Diff: https://reviews.apache.org/r/27009/diff/ Testing --- git clean -fdx ./pants src/test/python:all Thanks, Kevin Sweeney
Re: Review Request 27009: Use virtualenv to build pants instead of pex.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/#review57843 --- Ship it! Please file a PEX bug about the behaviour you noticed as well. - Zameer Manji On Oct. 22, 2014, 11:05 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/ --- (Updated Oct. 22, 2014, 11:05 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-876 https://issues.apache.org/jira/browse/AURORA-876 Repository: aurora Description --- Use virtualenv to build pants instead of pex. The goal of this change is to reduce CI flakiness by using `pip install` rather than `pex` to bootstrap pants. This still reaches out to external servers (and thus has the potential to be flaky) but is at least configurable (via `--default-timeout`). In a future review we can consider mirroring to svn.apache.org and disabling PyPI lookups. This requires a change to production executor code to use `sys.executable` rather than chmod+x-ing the runner PEX. Failure to do this results in the pants virtualenv's `site-packages` shadowing the dependencies in the `pex` (fairly obviously as there is a test failure related to psutil1/2 conflicts). I'm sure I'm papering over a PEX bug here but IMO this is fine as the only reason to execute the thermos runner PEX under a different python interpreter than the executor is using (based on the hash-bang in the pex) is not a configuration we need to support IMO. Diffs - pants 168419d7d8f5d89a0d87bfc6ac9f06c9e8e3dbc2 src/main/python/apache/aurora/executor/thermos_task_runner.py f17910826c31c5bf754b43eb72500de639652f37 Diff: https://reviews.apache.org/r/27009/diff/ Testing --- git clean -fdx ./pants src/test/python:all Thanks, Kevin Sweeney
Re: Review Request 27009: Use virtualenv to build pants instead of pex.
On Oct. 22, 2014, 2:06 p.m., Bill Farner wrote: pants, line 16 https://reviews.apache.org/r/27009/diff/2/?file=728833#file728833line16 Why allow unset? Virtualenv's activate script doesn't work under -o nounset. - Kevin --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/#review57896 --- On Oct. 22, 2014, 11:05 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/ --- (Updated Oct. 22, 2014, 11:05 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-876 https://issues.apache.org/jira/browse/AURORA-876 Repository: aurora Description --- Use virtualenv to build pants instead of pex. The goal of this change is to reduce CI flakiness by using `pip install` rather than `pex` to bootstrap pants. This still reaches out to external servers (and thus has the potential to be flaky) but is at least configurable (via `--default-timeout`). In a future review we can consider mirroring to svn.apache.org and disabling PyPI lookups. This requires a change to production executor code to use `sys.executable` rather than chmod+x-ing the runner PEX. Failure to do this results in the pants virtualenv's `site-packages` shadowing the dependencies in the `pex` (fairly obviously as there is a test failure related to psutil1/2 conflicts). I'm sure I'm papering over a PEX bug here but IMO this is fine as the only reason to execute the thermos runner PEX under a different python interpreter than the executor is using (based on the hash-bang in the pex) is not a configuration we need to support IMO. Diffs - pants 168419d7d8f5d89a0d87bfc6ac9f06c9e8e3dbc2 src/main/python/apache/aurora/executor/thermos_task_runner.py f17910826c31c5bf754b43eb72500de639652f37 Diff: https://reviews.apache.org/r/27009/diff/ Testing --- git clean -fdx ./pants src/test/python:all Thanks, Kevin Sweeney
Re: Review Request 27009: Use virtualenv to build pants instead of pex.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27009/ --- (Updated Oct. 22, 2014, 2:14 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- Fix grammar in description. Bugs: AURORA-876 https://issues.apache.org/jira/browse/AURORA-876 Repository: aurora Description (updated) --- Use virtualenv to build pants instead of pex. The goal of this change is to reduce CI flakiness by using `pip install` rather than `pex` to bootstrap pants. This still reaches out to external servers (and thus has the potential to be flaky) but is at least configurable (via `--default-timeout`). In a future review we can consider mirroring to svn.apache.org and disabling PyPI lookups. This requires a change to production executor code to use `sys.executable` rather than chmod+x-ing the runner PEX. Failure to do this results in the pants virtualenv's `site-packages` shadowing the dependencies in the `pex` (fairly obviously as there is a test failure related to psutil1/2 conflicts). I'm sure I'm papering over a PEX bug here but IMO this is fine as the only reason for it is to execute the thermos runner PEX under a different python interpreter than the executor is using (based on the hash-bang in the pex). This is not a configuration we need to support IMO. Diffs - pants 168419d7d8f5d89a0d87bfc6ac9f06c9e8e3dbc2 src/main/python/apache/aurora/executor/thermos_task_runner.py f17910826c31c5bf754b43eb72500de639652f37 Diff: https://reviews.apache.org/r/27009/diff/ Testing --- git clean -fdx ./pants src/test/python:all Thanks, Kevin Sweeney
