One python_library per exported setup.py project This is a mostly red code diff and makes the pants build work similar to how it already works in an IDE environment while minimizing duplicated information. This change also removes the confusing versions of `thermos` and `thermos_observer` in favor of new ones. Because of the way "covering dependencies" were required it was easier to do this refactor than more tactically create a working `setup_py` target for `apache.aurora.tools` without cycles and without duplicated files.
* Remove the `apache.thermos` package. * Rename the `apache.gen.aurora` package to `apache.aurora.thrift`. * Rename the `apache.gen.thermos` package to `apache.thermos.thrift`. * Introduce a new `apache.aurora.tools` package with the `thermos` and `thermos_observer` binaries. * Create apache.thermos.runner package, rename `thermos/bin` to `thermos/runner`. * Remove all `*-packaged` and virtual dependency trees, as well as the phrase "covering dependencies." * Use `_`-prefix naming convention for private target names. * Replace manual list of targets in `make-python-sdists` with automatically-generated one (using new convention). * Introduce a new `apache.aurora.kerberos` package with the `kaurora` and `kaurora_admin` binaries. * Remove all `BUILD` files in `src/main` that don't contain an exported `setup.py` library (except one mentioned in TODO). * Use dictionary syntax in `with_binaries`. * Remove unused/unreachable binaries (most `bin/` dirs under `thermos/`). * Create blank `__init__.py` files in directories that pants warns aren't packages. * Added `thermos` command to vagrant image. Testing Done: ./pants test.pytest --no-fast src/test/python:: ./build-support/python/make-python-sdists ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh ./build-support/python/make-pycharm-virtualenv Manually checked that `thermos status` works. I don't have a test environment for the debian package or the spec file. Documentation pushed to https://github.com/kevints/aurora/blob/36972/docs/build-system.md Bugs closed: AURORA-1381 Reviewed at https://reviews.apache.org/r/36972/ Project: http://git-wip-us.apache.org/repos/asf/aurora/repo Commit: http://git-wip-us.apache.org/repos/asf/aurora/commit/8fa4fb0f Tree: http://git-wip-us.apache.org/repos/asf/aurora/tree/8fa4fb0f Diff: http://git-wip-us.apache.org/repos/asf/aurora/diff/8fa4fb0f Branch: refs/heads/master Commit: 8fa4fb0f37e2bda12063615efcae7620a7d0ad80 Parents: 7f81411 Author: Kevin Sweeney <[email protected]> Authored: Mon Aug 3 11:41:33 2015 -0700 Committer: Kevin Sweeney <[email protected]> Committed: Mon Aug 3 11:41:33 2015 -0700 ---------------------------------------------------------------------- NEWS | 11 + api/src/main/thrift/org/apache/aurora/gen/BUILD | 18 +- api/src/main/thrift/org/apache/thermos/BUILD | 4 +- build-support/packaging/debian/rules | 8 +- build-support/packaging/rpm/aurora.spec | 6 +- build-support/python/make-pycharm-virtualenv | 8 +- build-support/release/make-python-sdists | 20 +- docs/README.md | 3 +- docs/build-system.md | 100 +++++++++ examples/vagrant/aurorabuild.sh | 10 +- src/main/python/apache/aurora/admin/BUILD | 88 +------- .../python/apache/aurora/admin/aurora_admin.py | 2 +- src/main/python/apache/aurora/client/BUILD | 64 +++--- src/main/python/apache/aurora/client/api/BUILD | 169 --------------- src/main/python/apache/aurora/client/cli/BUILD | 73 ------- .../python/apache/aurora/client/cli/client.py | 2 +- .../python/apache/aurora/client/hooks/BUILD | 25 --- src/main/python/apache/aurora/common/BUILD | 85 +------- src/main/python/apache/aurora/common/auth/BUILD | 36 ---- .../apache/aurora/common/auth/kerberos.py | 39 ---- src/main/python/apache/aurora/config/BUILD | 28 +-- .../python/apache/aurora/config/schema/BUILD | 23 --- src/main/python/apache/aurora/executor/BUILD | 113 +++-------- .../python/apache/aurora/executor/bin/BUILD | 47 ----- .../python/apache/aurora/executor/common/BUILD | 142 ------------- src/main/python/apache/aurora/kerberos/BUILD | 57 ++++++ .../python/apache/aurora/kerberos/__init__.py | 13 ++ .../apache/aurora/kerberos/auth_module.py | 38 ++++ src/main/python/apache/aurora/tools/BUILD | 73 +++++-- src/main/python/apache/thermos/BUILD | 33 --- src/main/python/apache/thermos/bin/BUILD | 45 ---- src/main/python/apache/thermos/bin/__init__.py | 13 -- .../python/apache/thermos/bin/thermos_ckpt.py | 83 -------- .../python/apache/thermos/bin/thermos_runner.py | 203 ------------------- src/main/python/apache/thermos/cli/BUILD | 33 ++- src/main/python/apache/thermos/cli/__init__.py | 13 ++ src/main/python/apache/thermos/cli/bin/BUILD | 21 -- .../python/apache/thermos/cli/bin/thermos.py | 23 --- .../python/apache/thermos/cli/commands/BUILD | 113 ----------- .../apache/thermos/cli/commands/__init__.py | 13 ++ src/main/python/apache/thermos/common/BUILD | 49 +---- src/main/python/apache/thermos/config/BUILD | 15 +- .../apache/thermos/config/bin/config_load.py | 62 ------ .../apache/thermos/config/bin/config_repl.py | 19 -- src/main/python/apache/thermos/core/BUILD | 61 +----- src/main/python/apache/thermos/monitoring/BUILD | 70 +------ src/main/python/apache/thermos/observer/BUILD | 55 ++--- .../python/apache/thermos/observer/bin/BUILD | 28 --- .../apache/thermos/observer/bin/__init__.py | 13 -- .../thermos/observer/bin/thermos_observer.py | 59 ------ .../python/apache/thermos/observer/http/BUILD | 90 -------- src/main/python/apache/thermos/runner/BUILD | 48 +++++ .../python/apache/thermos/runner/__init__.py | 13 ++ .../apache/thermos/runner/thermos_runner.py | 203 +++++++++++++++++++ src/main/python/apache/thermos/testing/BUILD | 10 +- src/test/python/apache/aurora/BUILD | 4 +- src/test/python/apache/aurora/admin/BUILD | 19 +- src/test/python/apache/aurora/client/BUILD | 8 +- src/test/python/apache/aurora/client/api/BUILD | 49 +++-- src/test/python/apache/aurora/client/cli/BUILD | 46 ++--- .../python/apache/aurora/client/hooks/BUILD | 9 +- src/test/python/apache/aurora/common/BUILD | 20 +- src/test/python/apache/aurora/config/BUILD | 2 +- src/test/python/apache/aurora/executor/BUILD | 30 +-- .../python/apache/aurora/executor/bin/BUILD | 2 +- .../python/apache/aurora/executor/common/BUILD | 44 ++-- .../aurora/executor/test_thermos_executor.py | 2 +- .../aurora/executor/test_thermos_task_runner.py | 2 +- src/test/python/apache/thermos/cli/BUILD | 4 +- .../python/apache/thermos/cli/commands/BUILD | 2 +- src/test/python/apache/thermos/common/BUILD | 8 +- src/test/python/apache/thermos/config/BUILD | 2 +- src/test/python/apache/thermos/core/BUILD | 20 +- src/test/python/apache/thermos/monitoring/BUILD | 10 +- src/test/python/apache/thermos/observer/BUILD | 2 +- .../python/apache/thermos/observer/http/BUILD | 5 +- 76 files changed, 859 insertions(+), 2124 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/NEWS ---------------------------------------------------------------------- diff --git a/NEWS b/NEWS index 0a8500c..6c0e20f 100644 --- a/NEWS +++ b/NEWS @@ -1,3 +1,14 @@ +0.10.0 +------ +- The Python build layout has changed: + * The apache.thermos package has been removed. + * The apache.gen.aurora package has been renamed to apache.aurora.thrift. + * The apache.gen.thermos package has been renamed to apache.thermos.thrift. + * A new apache.thermos.runner package has been introduced, providing the thermos_runner binary. + * A new apache.aurora.kerberos package has been introduced, containing the Kerberos-supporting + versions of aurora and aurora_admin (kaurora and kaurora_admin). + * Most BUILD targets under src/main have been removed, see http://s.apache.org/b8z for details. + 0.9.0 ----- http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/api/src/main/thrift/org/apache/aurora/gen/BUILD ---------------------------------------------------------------------- diff --git a/api/src/main/thrift/org/apache/aurora/gen/BUILD b/api/src/main/thrift/org/apache/aurora/gen/BUILD index d196fef..2dcc46d 100644 --- a/api/src/main/thrift/org/apache/aurora/gen/BUILD +++ b/api/src/main/thrift/org/apache/aurora/gen/BUILD @@ -17,34 +17,34 @@ import os TEST_DATA = globs('*test.thrift') python_thrift_library( - name = 'py-thrift', + name = '_api', sources = [ 'api.thrift', ], ) python_thrift_library( - name = 'py-thrift-test', + name = '_test', sources = TEST_DATA, ) python_thrift_library( - name = 'py-thrift-storage', + name = '_storage', sources = ['storage.thrift'], dependencies = [ - ':py-thrift', + ':_api', ], ) python_library( - name = 'py-thrift-packaged', + name = 'gen', dependencies = [ - ':py-thrift', - ':py-thrift-test', - ':py-thrift-storage', + ':_api', + ':_test', + ':_storage', ], provides = setup_py( - name = 'apache.gen.aurora', + name = 'apache.aurora.thrift', version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), description = 'Autogenerated Aurora thrift schemas.', ) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/api/src/main/thrift/org/apache/thermos/BUILD ---------------------------------------------------------------------- diff --git a/api/src/main/thrift/org/apache/thermos/BUILD b/api/src/main/thrift/org/apache/thermos/BUILD index d0d789a..6b13949 100644 --- a/api/src/main/thrift/org/apache/thermos/BUILD +++ b/api/src/main/thrift/org/apache/thermos/BUILD @@ -15,10 +15,10 @@ import os python_thrift_library( - name = 'py-thrift', + name = 'thermos', sources = ['thermos_internal.thrift'], provides = setup_py( - name = 'apache.gen.thermos', + name = 'apache.thermos.thrift', version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), description = 'Autogenerated Thermos thrift schemas.', ) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/build-support/packaging/debian/rules ---------------------------------------------------------------------- diff --git a/build-support/packaging/debian/rules b/build-support/packaging/debian/rules index 17e00c0..d981400 100755 --- a/build-support/packaging/debian/rules +++ b/build-support/packaging/debian/rules @@ -31,12 +31,12 @@ __gradle_build: __pants_build: mkdir -p third_party - $(pants) binary src/main/python/apache/aurora/admin:kaurora_admin - $(pants) binary src/main/python/apache/aurora/client/cli:kaurora - $(pants) binary src/main/python/apache/aurora/executor/bin:thermos_executor + $(pants) binary src/main/python/apache/aurora/executor:thermos_executor + $(pants) binary src/main/python/apache/aurora/kerberos:kaurora + $(pants) binary src/main/python/apache/aurora/kerberos:kaurora_admin $(pants) binary src/main/python/apache/aurora/tools:thermos - $(pants) binary src/main/python/apache/thermos/bin:thermos_runner $(pants) binary src/main/python/apache/aurora/tools:thermos_observer + $(pants) binary src/main/python/apache/thermos/runner:thermos_runner build-support/embed_runner_in_executor.py override_dh_installinit: http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/build-support/packaging/rpm/aurora.spec ---------------------------------------------------------------------- diff --git a/build-support/packaging/rpm/aurora.spec b/build-support/packaging/rpm/aurora.spec index 7cf8de6..3003ced 100644 --- a/build-support/packaging/rpm/aurora.spec +++ b/build-support/packaging/rpm/aurora.spec @@ -178,12 +178,12 @@ popd # Builds Aurora client PEX binaries. ./pants binary src/main/python/apache/aurora/admin:aurora_admin -./pants binary src/main/python/apache/aurora/client/cli:aurora +./pants binary src/main/python/apache/aurora/client:aurora # Builds Aurora Thermos and GC executor PEX binaries. -./pants binary src/main/python/apache/aurora/executor/bin:thermos_executor +./pants binary src/main/python/apache/aurora/executor:thermos_executor ./pants binary src/main/python/apache/aurora/tools:thermos -./pants binary src/main/python/apache/thermos/bin:thermos_runner +./pants binary src/main/python/apache/thermos/runner:thermos_runner ./pants binary src/main/python/apache/aurora/tools:thermos_observer # Packages the Thermos runner within the Thermos executor. http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/build-support/python/make-pycharm-virtualenv ---------------------------------------------------------------------- diff --git a/build-support/python/make-pycharm-virtualenv b/build-support/python/make-pycharm-virtualenv index 05a16d0..d7bd418 100755 --- a/build-support/python/make-pycharm-virtualenv +++ b/build-support/python/make-pycharm-virtualenv @@ -22,8 +22,8 @@ PROJECT_NAME=${PWD##*/} pushd "$BUILDROOT" ./pants dependencies --external-only src/test/python:: | sort -u > requirements.txt - ./pants setup-py api/src/main/thrift/org/apache/aurora/gen:py-thrift-packaged - ./pants setup-py api/src/main/thrift/org/apache/thermos:py-thrift + ./pants setup-py api/src/main/thrift/org/apache/aurora/gen + ./pants setup-py api/src/main/thrift/org/apache/thermos source build-support/pants.venv/bin/activate pytest_requirement=$(python -m pip freeze | grep pytest==) pytest_cov_requirement=$(python -m pip freeze | grep pytest-cov==) @@ -35,8 +35,8 @@ pushd "$BUILDROOT" # the namespace packages in twitter.common. python -m pip install --egg -r requirements.txt python -m pip install --egg -f dist \ - apache.gen.aurora==$(< .auroraversion) \ - apache.gen.thermos==$(<.auroraversion) + apache.aurora.thrift==$(< .auroraversion) \ + apache.thermos.thrift==$(<.auroraversion) # Use same py.test version embedded into pants python -m pip install $pytest_requirement $pytest_cov_requirement deactivate http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/build-support/release/make-python-sdists ---------------------------------------------------------------------- diff --git a/build-support/release/make-python-sdists b/build-support/release/make-python-sdists index 9608f68..51e11db 100755 --- a/build-support/release/make-python-sdists +++ b/build-support/release/make-python-sdists @@ -26,21 +26,9 @@ # aurora_admin --help set -o errexit -TARGETS=( - src/main/python/apache/aurora/admin:admin-packaged - src/main/python/apache/aurora/client:client-packaged - src/main/python/apache/aurora/common - src/main/python/apache/aurora/config:config-packaged - src/main/python/apache/aurora/executor:executor-packaged - src/main/python/apache/thermos - src/main/python/apache/thermos/common - src/main/python/apache/thermos/config - src/main/python/apache/thermos/core - src/main/python/apache/thermos/monitoring - src/main/python/apache/thermos/observer - api/src/main/thrift/org/apache/aurora/gen:py-thrift-packaged - api/src/main/thrift/org/apache/thermos:py-thrift -) +# Use the convention of a python_library exporting a setup_py alongside each public target +# to generate the list of setup_pys our project provides. +TARGETS=($(./pants filter --type=python_binary src/main:: | sed 's@:.*@@')) cd "`git rev-parse --show-toplevel`" # Run in a virtualenv so that we use a consistent version of setuptools to generate the sdist. @@ -54,5 +42,5 @@ fi source build-support/make-python-sdists.venv/bin/activate for t in "${TARGETS[@]}"; do - ./pants setup-py $t + ./pants setup-py --recursive $t done http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/docs/README.md ---------------------------------------------------------------------- diff --git a/docs/README.md b/docs/README.md index 9893763..fe115dc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -31,6 +31,7 @@ We encourage you to ask questions on the [Aurora developer list](http://aurora.a * [Developing the Aurora Scheduler](developing-aurora-scheduler.md) * [Developing the Aurora Client](developing-aurora-client.md) * [Committers Guide](committers.md) + * [Build System](build-system.md) ## Additional Resources - * [Presentation videos and slides](presentations.md) \ No newline at end of file + * [Presentation videos and slides](presentations.md) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/docs/build-system.md ---------------------------------------------------------------------- diff --git a/docs/build-system.md b/docs/build-system.md new file mode 100644 index 0000000..39c231d --- /dev/null +++ b/docs/build-system.md @@ -0,0 +1,100 @@ +The Python components of Aurora are built using [Pants](https://pantsbuild.github.io). + +Python Build Conventions +======================== +The Python code is laid out according to the following conventions: + +1. 1 `BUILD` per 3rd level directory. For a list of current top-level packages run: + + % find src/main/python -maxdepth 3 -mindepth 3 -type d |\ + while read dname; do echo $dname |\ + sed 's@src/main/python/\(.*\)/\(.*\)/\(.*\).*@\1.\2.\3@'; done + +2. Each `BUILD` file exports 1 + [`python_library`](https://pantsbuild.github.io/build_dictionary.html#bdict_python_library) + that provides a + [`setup_py`](https://pantsbuild.github.io/build_dictionary.html#setup_py) + containing each + [`python_binary`](https://pantsbuild.github.io/build_dictionary.html#python_binary) + in the `BUILD` file, named the same as the directory it's in so that it can be referenced + without a ':' character. The `sources` field in the `python_library` will almost always be + `rglobs('*.py')`. + +3. Other BUILD files may only depend on this single public `python_library` + target. Any other target is considered a private implementation detail and + should be prefixed with an `_`. + +4. `python_binary` targets are always named the same as the exported console script. + +5. `python_binary` targets must have identical `dependencies` to the `python_library` exported + by the package and must use `entry_point`. + + The means a PEX file generated by pants will contain exactly the same files that will be + available on the `PYTHONPATH` in the case of `pip install` of the corresponding library + target. This will help our migration off of Pants in the future. + +Annotated example - apache.thermos.runner +----------------------------------------- +``` +% find src/main/python/apache/thermos/runner +src/main/python/apache/thermos/runner +src/main/python/apache/thermos/runner/__init__.py +src/main/python/apache/thermos/runner/thermos_runner.py +src/main/python/apache/thermos/runner/BUILD +% cat src/main/python/apache/thermos/runner/BUILD +# License boilerplate omitted +import os + + +# Private target so that a setup_py can exist without a circular dependency. Only targets within +# this file should depend on this. +python_library( + name = '_runner', + # The target covers every python file under this directory and subdirectories. + sources = rglobs('*.py'), + dependencies = [ + '3rdparty/python:twitter.common.app', + '3rdparty/python:twitter.common.log', + # Source dependencies are always referenced without a ':'. + 'src/main/python/apache/thermos/common', + 'src/main/python/apache/thermos/config', + 'src/main/python/apache/thermos/core', + ], +) + +# Binary target for thermos_runner.pex. Nothing should depend on this - it's only used as an +# argument to ./pants binary. +python_binary( + name = 'thermos_runner', + # Use entry_point, not source so the files used here are the same ones tests see. + entry_point = 'apache.thermos.bin.thermos_runner', + dependencies = [ + # Notice that we depend only on the single private target from this BUILD file here. + ':_runner', + ], +) + +# The public library that everyone importing the runner symbols uses. +# The test targets and any other dependent source code should depend on this. +python_library( + name = 'runner', + dependencies = [ + # Again, notice that we depend only on the single private target from this BUILD file here. + ':_runner', + ], + # We always provide a setup_py. This will cause any dependee libraries to automatically + # reference this library in their requirements.txt rather than copy the source files into their + # sdist. + provides = setup_py( + # Conventionally named and versioned. + name = 'apache.thermos.runner', + version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), + ).with_binaries({ + # Every binary in this file should also be repeated here. + # Always use the dict-form of .with_binaries so that commands with dashes in their names are + # supported. + # The console script name is always the same as the PEX with .pex stripped. + 'thermos_runner': ':thermos_runner', + }), +) +``` http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/examples/vagrant/aurorabuild.sh ---------------------------------------------------------------------- diff --git a/examples/vagrant/aurorabuild.sh b/examples/vagrant/aurorabuild.sh index fbaa6ae..8850e93 100755 --- a/examples/vagrant/aurorabuild.sh +++ b/examples/vagrant/aurorabuild.sh @@ -33,12 +33,12 @@ function upstart_update { } function build_client { - ./pants binary src/main/python/apache/aurora/client/cli:kaurora + ./pants binary src/main/python/apache/aurora/kerberos:kaurora sudo ln -sf $DIST_DIR/kaurora.pex /usr/local/bin/aurora } function build_admin_client { - ./pants binary src/main/python/apache/aurora/admin:kaurora_admin + ./pants binary src/main/python/apache/aurora/kerberos:kaurora_admin sudo ln -sf $DIST_DIR/kaurora_admin.pex /usr/local/bin/aurora_admin } @@ -58,8 +58,8 @@ function build_scheduler { } function build_executor { - ./pants binary src/main/python/apache/aurora/executor/bin:thermos_executor - ./pants binary src/main/python/apache/thermos/bin:thermos_runner + ./pants binary src/main/python/apache/aurora/executor:thermos_executor + ./pants binary src/main/python/apache/thermos/runner:thermos_runner # Package runner within executor. build-support/embed_runner_in_executor.py @@ -69,6 +69,8 @@ function build_executor { function build_observer { ./pants binary src/main/python/apache/aurora/tools:thermos_observer + ./pants binary src/main/python/apache/aurora/tools:thermos + sudo ln -sf $DIST_DIR/thermos.pex /usr/local/bin/thermos upstart_update aurora-thermos-observer } http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/admin/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/admin/BUILD b/src/main/python/apache/aurora/admin/BUILD index 22bf3f9..b5d37f7 100644 --- a/src/main/python/apache/aurora/admin/BUILD +++ b/src/main/python/apache/aurora/admin/BUILD @@ -14,62 +14,16 @@ import os python_library( - name = 'admin', - sources = ['admin.py', 'admin_util.py'], + name = '_admin', + sources = rglobs('*.py'), dependencies = [ - ':help', - ':util', '3rdparty/python:twitter.common.app', '3rdparty/python:twitter.common.log', '3rdparty/python:twitter.common.quantity', - 'src/main/python/apache/aurora/client/api', - 'src/main/python/apache/aurora/client:base', + 'api/src/main/thrift/org/apache/aurora/gen', + 'src/main/python/apache/aurora/client', 'src/main/python/apache/aurora/common', - 'src/main/python/apache/aurora/client:config', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'help', - sources = ['help.py'], - dependencies = [ - '3rdparty/python:twitter.common.app', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/client:base' - ] -) - -python_library( - name = 'host_maintenance', - sources = ['host_maintenance.py'], - dependencies = [ - ':util', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - 'src/main/python/apache/aurora/client:api', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'maintenance', - sources = ['maintenance.py'], - dependencies = [ - ':util', - ':host_maintenance', - '3rdparty/python:twitter.common.app', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/client:base', - 'src/main/python/apache/aurora/common:clusters', - ] -) - -python_library( - name = 'util', - sources = ['admin_util.py'], - dependencies = [ - 'src/main/python/apache/aurora/client:base', + 'src/main/python/apache/aurora/config', ] ) @@ -77,39 +31,19 @@ python_binary( name = 'aurora_admin', entry_point = 'apache.aurora.admin.aurora_admin:proxy_main', dependencies = [ - ':aurora_admin_lib' + ':_admin', ] ) -# TODO(maxim): consider making this default ("aurora_admin") target along with kerberos auth. -python_binary( - name = 'kaurora_admin', - entry_point = 'apache.aurora.admin.aurora_admin:proxy_main', - dependencies = [ - ':aurora_admin_lib', - 'src/main/python/apache/aurora/common/auth:kerberos', - ], -) - python_library( - name = 'aurora_admin_lib', - sources = [ 'aurora_admin.py' ], - dependencies = [ - ':admin', - ':maintenance', - ] -) - -python_library( - name = 'admin-packaged', + name = 'admin', dependencies = [ - ':aurora_admin_lib', - 'src/main/python/apache/aurora/config:config-packaged', + ':_admin', ], provides = setup_py( name = 'apache.aurora.admin', version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), - ).with_binaries( - aurora_admin = ':aurora_admin', - ) + ).with_binaries({ + 'aurora_admin': ':aurora_admin', + }), ) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/admin/aurora_admin.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/admin/aurora_admin.py b/src/main/python/apache/aurora/admin/aurora_admin.py index 1c39717..470b2d2 100644 --- a/src/main/python/apache/aurora/admin/aurora_admin.py +++ b/src/main/python/apache/aurora/admin/aurora_admin.py @@ -29,7 +29,7 @@ def main(): app.help() try: - from apache.aurora.common.auth.kerberos import KerberosAuthModule + from apache.aurora.kerberos.auth_module import KerberosAuthModule register_auth_module(KerberosAuthModule()) except ImportError: # Use default auth implementation if kerberos is not available. http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/client/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/client/BUILD b/src/main/python/apache/aurora/client/BUILD index e73cd52..8424237 100644 --- a/src/main/python/apache/aurora/client/BUILD +++ b/src/main/python/apache/aurora/client/BUILD @@ -11,62 +11,48 @@ # See the License for the specific language governing permissions and # limitations under the License. # - import os -# Create an alias for the previous target -python_library( - name = 'api', - dependencies = [ - 'src/main/python/apache/aurora/client/api' - ] -) python_library( - name = 'base', - sources = ['base.py'], + name = '_client', + sources = rglobs('*.py'), dependencies = [ + '3rdparty/python:pex', + '3rdparty/python:pystachio', + '3rdparty/python:requests', '3rdparty/python:twitter.common.app', + '3rdparty/python:twitter.common.decorators', + '3rdparty/python:twitter.common.lang', '3rdparty/python:twitter.common.log', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', + '3rdparty/python:twitter.common.quantity', + '3rdparty/python:twitter.common.zookeeper', + 'api/src/main/thrift/org/apache/aurora/gen', 'src/main/python/apache/aurora/common', - ] -) - -python_library( - name = 'binding_helper', - sources = ['binding_helper.py'], - dependencies = [ - '3rdparty/python:twitter.common.lang', - ] + 'src/main/python/apache/aurora/config', + 'src/main/python/apache/thermos/common', + 'src/main/python/apache/thermos/config', + 'src/main/resources/apache/aurora/client/cli:auroraversion', + ], ) -python_library( - name = 'config', - sources = ['config.py'], +python_binary( + name='aurora', + entry_point = 'apache.aurora.client.cli.client:proxy_main', dependencies = [ - ':base', - ':binding_helper', - '3rdparty/python:pystachio', - '3rdparty/python:twitter.common.app', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/config', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] + ':_client' + ], ) python_library( - name = 'client-packaged', + name = 'client', dependencies = [ - 'src/main/python/apache/aurora/common', - 'src/main/python/apache/aurora/config:config-packaged', - 'src/main/python/apache/thermos/common', + ':_client', ], provides = setup_py( name = 'apache.aurora.client', version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), - ).with_binaries( - aurora = 'src/main/python/apache/aurora/client/cli:aurora', - ) + ).with_binaries({ + 'aurora': ':aurora', + }), ) - http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/client/api/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/client/api/BUILD b/src/main/python/apache/aurora/client/api/BUILD deleted file mode 100644 index a030a67..0000000 --- a/src/main/python/apache/aurora/client/api/BUILD +++ /dev/null @@ -1,169 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_library( - name = 'api', - sources = ['__init__.py'], - dependencies = [ - ':restarter', - ':scheduler_client', - ':sla', - ':updater', - '3rdparty/python:twitter.common.lang', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/client:base', - 'src/main/python/apache/aurora/common', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'command_runner', - sources = ['command_runner.py'], - dependencies = [ - ':api', - '3rdparty/python:pystachio', - 'src/main/python/apache/thermos/config:schema', - 'src/main/python/apache/aurora/common:cluster', - 'src/main/python/apache/aurora/config:schema', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'job_monitor', - sources = ['job_monitor.py'], - dependencies = [ - ':task_util', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'restarter', - sources = ['restarter.py'], - dependencies = [ - ':instance_watcher', - ':updater_util', - '3rdparty/python:twitter.common.log', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'scheduler_client', - sources = ['scheduler_client.py'], - dependencies = [ - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - '3rdparty/python:twitter.common.zookeeper', - 'src/main/python/apache/aurora/common/auth', - 'src/main/python/apache/aurora/common:cluster', - 'src/main/python/apache/aurora/common:transport', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'error_handling_thread', - sources = ['error_handling_thread.py'], - dependencies = [ - '3rdparty/python:twitter.common.decorators', - '3rdparty/python:twitter.common.quantity', - '3rdparty/python:twitter.common.log', - ] -) - -python_library( - name = 'scheduler_mux', - sources = ['scheduler_mux.py'], - dependencies = [ - ':error_handling_thread', - '3rdparty/python:twitter.common.quantity', - '3rdparty/python:twitter.common.log', - ] -) - -python_library( - name = 'task_util', - sources = ['task_util.py'], - dependencies = [ - ':scheduler_mux', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/client:base', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'instance_watcher', - sources = ['instance_watcher.py', 'health_check.py'], - dependencies = [ - ':scheduler_client', - ':scheduler_mux', - ':task_util', - '3rdparty/python:twitter.common.lang', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/common:http_signaler', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'quota_check', - sources = ['quota_check.py'], - dependencies = [ - ':scheduler_client', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/client:base', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'sla', - sources = ['sla.py'], - dependencies = [ - ':scheduler_client', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/client:base', - 'src/main/python/apache/aurora/common', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'updater', - sources = ['updater.py'], - dependencies = [ - ':scheduler_client', - ':instance_watcher', - ':job_monitor', - ':updater_util', - ':quota_check', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/client:base', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'updater_util', - sources = ['updater_util.py'], - dependencies = [ - '3rdparty/python:twitter.common.log', - ] -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/client/cli/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/client/cli/BUILD b/src/main/python/apache/aurora/client/cli/BUILD deleted file mode 100644 index 973d059..0000000 --- a/src/main/python/apache/aurora/client/cli/BUILD +++ /dev/null @@ -1,73 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_binary( - name='aurora', - entry_point = 'apache.aurora.client.cli.client:proxy_main', - dependencies = [ - ':client_lib' - ], -) - -# TODO(maxim): consider making this default ("aurora") target along with kerberos auth. -python_binary( - name='kaurora', - entry_point = 'apache.aurora.client.cli.client:proxy_main', - dependencies = [ - ':client_lib', - 'src/main/python/apache/aurora/common/auth:kerberos', - ], -) - -python_library( - name = 'client_lib', - sources = [ 'client.py' ], - dependencies = [ - ':cli', - '3rdparty/python:pex', - ] -) - -python_library( - name='cli', - sources = [ - '__init__.py', - 'client.py', - 'config.py', - 'context.py', - 'command_hooks.py', - 'cron.py', - 'jobs.py', - 'options.py', - 'quota.py', - 'sla.py', - 'task.py', - 'update.py', - ], - dependencies = [ - '3rdparty/python:pex', - '3rdparty/python:requests', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - 'src/main/python/apache/aurora/client/api:command_runner', - 'src/main/python/apache/aurora/client/api:job_monitor', - 'src/main/python/apache/aurora/client/api:updater', - 'src/main/python/apache/aurora/client/hooks', - 'src/main/python/apache/aurora/client:base', - 'src/main/python/apache/aurora/client:config', - 'src/main/python/apache/aurora/common', - 'src/main/resources/apache/aurora/client/cli:auroraversion', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ], -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/client/cli/client.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/client/cli/client.py b/src/main/python/apache/aurora/client/cli/client.py index c0974f3..c17f857 100644 --- a/src/main/python/apache/aurora/client/cli/client.py +++ b/src/main/python/apache/aurora/client/cli/client.py @@ -66,7 +66,7 @@ class AuroraAuthConfigurationPlugin(ConfigurationPlugin): def before_execution(self, context): try: - from apache.aurora.common.auth.kerberos import KerberosAuthModule + from apache.aurora.kerberos.auth_module import KerberosAuthModule register_auth_module(KerberosAuthModule()) except ImportError: # Use default auth implementation if kerberos is not available. http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/client/hooks/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/client/hooks/BUILD b/src/main/python/apache/aurora/client/hooks/BUILD deleted file mode 100644 index ddf813b..0000000 --- a/src/main/python/apache/aurora/client/hooks/BUILD +++ /dev/null @@ -1,25 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_library( - name = 'hooks', - sources = ['__init__.py', 'hooked_api.py'], - dependencies = [ - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/client:api', - 'src/main/python/apache/aurora/client:config', - 'src/main/python/apache/aurora/common:aurora_job_key', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/common/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/common/BUILD b/src/main/python/apache/aurora/common/BUILD index abc122b..5fce3d0 100644 --- a/src/main/python/apache/aurora/common/BUILD +++ b/src/main/python/apache/aurora/common/BUILD @@ -11,94 +11,21 @@ # See the License for the specific language governing permissions and # limitations under the License. # - import os -python_library( - name = 'aurora_job_key', - sources = ['aurora_job_key.py'], - dependencies = [ - '3rdparty/python:twitter.common.lang', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'cluster', - sources = ['cluster.py'], - dependencies = [ - '3rdparty/python:pystachio', - ] -) - -python_library( - name = 'clusters', - sources = ['clusters.py'], - dependencies = [ - ':cluster', - '3rdparty/python:pystachio', - '3rdparty/python:twitter.common.collections', - ] -) - -python_library( - name = 'cluster_option', - sources = ['cluster_option.py'], - dependencies = [ - '3rdparty/python:twitter.common.lang', - ] -) - -python_library( - name = 'http_signaler', - sources = ['http_signaler.py'], - dependencies = [ - '3rdparty/python:twitter.common.lang', - '3rdparty/python:twitter.common.log', - ] -) python_library( - name = 'pex_version', - sources = ['pex_version.py'], + name = 'common', + sources = rglobs('*.py'), dependencies = [ '3rdparty/python:pex', - ] -) - -python_library( - name = 'shellify', - sources = ['shellify.py'], - dependencies = [ - '3rdparty/python:twitter.common.lang', - ], -) - -python_library( - name = 'transport', - sources = ['transport.py'], - dependencies = [ + '3rdparty/python:pystachio', '3rdparty/python:requests', '3rdparty/python:thrift', + '3rdparty/python:twitter.common.collections', + '3rdparty/python:twitter.common.lang', '3rdparty/python:twitter.common.log', - ], -) - -python_library( - name = 'common', - dependencies = [ - ':aurora_job_key', - ':cluster', - ':cluster_option', - ':clusters', - ':pex_version', - ':http_signaler', - ':shellify', - ':transport', - 'src/main/python/apache/aurora/common/auth', - - # covering dependency on gen.* - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift-packaged', + 'api/src/main/thrift/org/apache/aurora/gen', ], provides = setup_py( name = 'apache.aurora.common', http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/common/auth/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/common/auth/BUILD b/src/main/python/apache/aurora/common/auth/BUILD deleted file mode 100644 index 0abac94..0000000 --- a/src/main/python/apache/aurora/common/auth/BUILD +++ /dev/null @@ -1,36 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_library( - name = 'auth', - sources = [ - '__init__.py', - 'auth_module.py', - 'auth_module_manager.py', - ], - dependencies = [ - '3rdparty/python:twitter.common.lang', - '3rdparty/python:twitter.common.log', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'kerberos', - sources = ['kerberos.py'], - dependencies = [ - '3rdparty/python:requests-kerberos', - ':auth' - ] -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/common/auth/kerberos.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/common/auth/kerberos.py b/src/main/python/apache/aurora/common/auth/kerberos.py deleted file mode 100644 index 2d782b6..0000000 --- a/src/main/python/apache/aurora/common/auth/kerberos.py +++ /dev/null @@ -1,39 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -from requests_kerberos import DISABLED, HTTPKerberosAuth - -from .auth_module import AuthModule - - -class KerberosAuthModule(AuthModule): - @property - def mechanism(self): - return 'KERBEROS' - - def payload(self): - """NOTE: until AURORA-1229 is addressed, using "Kerberized" client in production in a backwards - compatible way will require a new custom module that will override this method to - return the currently used payload (security blob used in SessionKey). - """ - return '' - - def auth(self): - # While SPNEGO supports mutual authentication of the response, it does not assert the validity - # of the response payload, only the identity of the server. Thus the scheduler will not set - # the WWW-Authenticate response header and the client will disable mutual authentication. - # In order to achieve communication with the scheduler subject to confidentiality and integrity - # constraints the client must connect to the scheduler API via HTTPS. Kerberos is thus only - # used to authenticate the client to the server. - return HTTPKerberosAuth(mutual_authentication=DISABLED) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/config/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/config/BUILD b/src/main/python/apache/aurora/config/BUILD index 0a3a93f..8c763c3 100644 --- a/src/main/python/apache/aurora/config/BUILD +++ b/src/main/python/apache/aurora/config/BUILD @@ -11,41 +11,17 @@ # See the License for the specific language governing permissions and # limitations under the License. # - import os -# Alias for src/main/python/apache/aurora/config/schema -python_library( - name = 'schema', - dependencies = [ - 'src/main/python/apache/aurora/config/schema', - ] -) python_library( name = 'config', - sources = ( - '__init__.py', - 'loader.py', - 'port_resolver.py', - 'thrift.py', - ), + sources = rglobs('*.py'), dependencies = [ '3rdparty/python:pystachio', '3rdparty/python:twitter.common.lang', + 'api/src/main/thrift/org/apache/aurora/gen', 'src/main/python/apache/aurora/common', - 'src/main/python/apache/aurora/config/schema', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ], - -) - -python_library( - name = 'config-packaged', - dependencies = [ - ':config', - - # covering dependencies 'src/main/python/apache/thermos/config', ], provides = setup_py( http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/config/schema/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/config/schema/BUILD b/src/main/python/apache/aurora/config/schema/BUILD deleted file mode 100644 index 171f42a..0000000 --- a/src/main/python/apache/aurora/config/schema/BUILD +++ /dev/null @@ -1,23 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_library( - name = 'schema', - sources = ['base.py'], - dependencies = [ - '3rdparty/python:pystachio', - 'src/main/python/apache/thermos/config:schema', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/executor/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/executor/BUILD b/src/main/python/apache/aurora/executor/BUILD index 891cbe6..0be65fc 100644 --- a/src/main/python/apache/aurora/executor/BUILD +++ b/src/main/python/apache/aurora/executor/BUILD @@ -14,112 +14,59 @@ import os + python_library( - name = 'thermos_task_runner', - sources = ['thermos_task_runner.py'], + name = '_executor', + sources = rglobs('*.py'), dependencies = [ '3rdparty/python:mesos.interface', + '3rdparty/python:pex', # To prevent an alpha version of protobuf from being pulled down by pants we # specify protobuf here. See AURORA-1128 for more details. '3rdparty/python:protobuf', - '3rdparty/python:twitter.common.dirutil', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - 'src/main/python/apache/thermos/common', - 'src/main/python/apache/thermos/config:schema', - 'src/main/python/apache/thermos/core', - 'src/main/python/apache/thermos/monitoring:monitor', - 'src/main/python/apache/aurora/executor/common:status_checker', - 'src/main/python/apache/aurora/executor/common:task_info', - 'src/main/python/apache/aurora/executor/common:task_runner', - ':http_lifecycle', - ] -) - -python_library( - name = 'executor_vars', - sources = ['executor_vars.py'], - dependencies = [ - '3rdparty/python:pex', '3rdparty/python:psutil', + '3rdparty/python:pystachio', + '3rdparty/python:twitter.common.app', + '3rdparty/python:twitter.common.concurrent', + '3rdparty/python:twitter.common.dirutil', '3rdparty/python:twitter.common.exceptions', + '3rdparty/python:twitter.common.lang', + '3rdparty/python:twitter.common.log', '3rdparty/python:twitter.common.metrics', '3rdparty/python:twitter.common.quantity', '3rdparty/python:twitter.common.string', - ] -) - -python_library( - name = 'http_lifecycle', - sources = ['http_lifecycle.py'], - dependencies = [ - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - 'src/main/python/apache/aurora/common:http_signaler', - 'src/main/python/apache/aurora/executor/common:task_runner', - ] -) - -python_library( - name = 'status_manager', - sources = ['status_manager.py'], - dependencies = [ - '3rdparty/python:twitter.common.exceptions', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - 'src/main/python/apache/aurora/executor/common:status_checker', - ] + '3rdparty/python:twitter.common.zookeeper', + 'api/src/main/thrift/org/apache/aurora/gen', + 'src/main/python/apache/aurora/common', + 'src/main/python/apache/aurora/config', + 'src/main/python/apache/thermos/common', + 'src/main/python/apache/thermos/config', + 'src/main/python/apache/thermos/core', + 'src/main/python/apache/thermos/monitoring', + ], ) -python_library( - name = 'executor_base', - sources = ['executor_base.py'], +python_binary( + name = 'thermos_executor', + entry_point = 'apache.aurora.executor.bin.thermos_executor_main:proxy_main', + always_write_cache = True, dependencies = [ - '3rdparty/python:mesos.interface', - # To prevent an alpha version of protobuf from being pulled down by pants we - # specify protobuf here. See AURORA-1128 for more details. - '3rdparty/python:protobuf', - '3rdparty/python:twitter.common.log', + ':_executor', + # TODO(ksweeney): This violates the rule that every library export the same deps as its + # contained binaries. + '3rdparty/python:mesos.native', ] ) python_library( - name = 'aurora_executor', - sources = ['aurora_executor.py'], + name = 'executor', dependencies = [ - ':status_manager', - ':executor_base', - '3rdparty/python:mesos.interface', - # To prevent an alpha version of protobuf from being pulled down by pants we - # specify protobuf here. See AURORA-1128 for more details. - '3rdparty/python:protobuf', - '3rdparty/python:pystachio', - '3rdparty/python:twitter.common.app', - '3rdparty/python:twitter.common.concurrent', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - 'src/main/python/apache/aurora/executor/common:kill_manager', - 'src/main/python/apache/aurora/executor/common:sandbox', - 'src/main/python/apache/aurora/executor/common:task_info', - 'src/main/python/apache/aurora/executor/common:task_runner', - ] -) - -python_library( - name = 'executor-packaged', - dependencies = [ - # Covering dependencies - 'src/main/python/apache/aurora/common', - 'src/main/python/apache/aurora/config', - 'src/main/python/apache/thermos/common', - 'src/main/python/apache/thermos/config', - 'src/main/python/apache/thermos/core', - 'src/main/python/apache/thermos/monitoring', + ':_executor', ], provides = setup_py( name = 'apache.aurora.executor', version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), ).with_binaries( - thermos_executor = 'src/main/python/apache/aurora/executor/bin:thermos_executor', + thermos_executor = ':thermos_executor', ) ) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/executor/bin/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/executor/bin/BUILD b/src/main/python/apache/aurora/executor/bin/BUILD deleted file mode 100644 index b23429a..0000000 --- a/src/main/python/apache/aurora/executor/bin/BUILD +++ /dev/null @@ -1,47 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_library( - name = 'thermos_executor_source', - sources = ['thermos_executor_main.py'], - dependencies = [ - # To prevent an alpha version of protobuf from being pulled down by pants we - # specify protobuf here. See AURORA-1128 for more details. - '3rdparty/python:protobuf', - '3rdparty/python:twitter.common.app', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.metrics', - 'src/main/python/apache/aurora/executor/common:announcer', - 'src/main/python/apache/aurora/executor/common:executor_detector', - 'src/main/python/apache/aurora/executor/common:executor_timeout', - 'src/main/python/apache/aurora/executor/common:health_checker', - 'src/main/python/apache/aurora/executor/common:path_detector', - 'src/main/python/apache/aurora/executor/common:resource_manager', - 'src/main/python/apache/aurora/executor/common:sandbox', - 'src/main/python/apache/aurora/executor:executor_vars', - 'src/main/python/apache/aurora/executor:aurora_executor', - 'src/main/python/apache/aurora/executor:thermos_task_runner', - 'src/main/python/apache/thermos/common:path', - ] -) - -python_binary( - name = 'thermos_executor', - entry_point = 'apache.aurora.executor.bin.thermos_executor_main:proxy_main', - always_write_cache = True, - dependencies = [ - '3rdparty/python:mesos.native', - ':thermos_executor_source', - ] -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/executor/common/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/executor/common/BUILD b/src/main/python/apache/aurora/executor/common/BUILD deleted file mode 100644 index 3d94792..0000000 --- a/src/main/python/apache/aurora/executor/common/BUILD +++ /dev/null @@ -1,142 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_library( - name = 'executor_detector', - sources = ['executor_detector.py'], - dependencies = [ - '3rdparty/python:twitter.common.string', - ] -) - -python_library( - name = 'path_detector', - sources = ['path_detector.py'], - dependencies = [ - 'src/main/python/apache/thermos/monitoring:detector', - ':executor_detector', - ] -) - -python_library( - name = 'status_checker', - sources = ['status_checker.py'], - dependencies = [ - '3rdparty/python:mesos.interface', - # To prevent an alpha version of protobuf from being pulled down by pants we - # specify protobuf here. See AURORA-1128 for more details. - '3rdparty/python:protobuf', - '3rdparty/python:twitter.common.lang', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.metrics', - ] -) - -python_library( - name = 'task_runner', - sources = ['task_runner.py'], - dependencies = [ - ':status_checker', - '3rdparty/python:twitter.common.lang', - ] -) - -python_library( - name = 'health_checker', - sources = ['health_checker.py'], - dependencies = [ - ':status_checker', - ':task_info', - '3rdparty/python:mesos.interface', - # To prevent an alpha version of protobuf from being pulled down by pants we - # specify protobuf here. See AURORA-1128 for more details. - '3rdparty/python:protobuf', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.exceptions', - 'src/main/python/apache/aurora/common:http_signaler', - ] -) - -python_library( - name = 'announcer', - sources = ['announcer.py'], - dependencies = [ - '3rdparty/python:twitter.common.concurrent', - '3rdparty/python:twitter.common.exceptions', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.metrics', - '3rdparty/python:twitter.common.quantity', - '3rdparty/python:twitter.common.zookeeper', - ':status_checker', - ':task_info', - ] -) - -python_library( - name = 'executor_timeout', - sources = ['executor_timeout.py'], - dependencies = [ - '3rdparty/python:twitter.common.exceptions', - '3rdparty/python:twitter.common.log', - '3rdparty/python:twitter.common.quantity', - ] -) - -python_library( - name = 'kill_manager', - sources = ['kill_manager.py'], - dependencies = [ - '3rdparty/python:mesos.interface', - # To prevent an alpha version of protobuf from being pulled down by pants we - # specify protobuf here. See AURORA-1128 for more details. - '3rdparty/python:protobuf', - ':status_checker', - ] -) - -python_library( - name = 'sandbox', - sources = ['sandbox.py'], - dependencies = [ - '3rdparty/python:twitter.common.dirutil', - '3rdparty/python:twitter.common.log', - ] -) - -python_library( - name = 'task_info', - sources = ['task_info.py'], - dependencies = [ - '3rdparty/python:pystachio', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/config', - 'src/main/python/apache/aurora/config/schema', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - ] -) - -python_library( - name = 'resource_manager', - sources = ['resource_manager.py'], - dependencies = [ - '3rdparty/python:psutil', - '3rdparty/python:twitter.common.dirutil', - '3rdparty/python:twitter.common.exceptions', - '3rdparty/python:twitter.common.log', - 'api/src/main/thrift/org/apache/aurora/gen:py-thrift', - 'src/main/python/apache/aurora/executor/common:status_checker', - 'src/main/python/apache/aurora/executor/common:task_info', - 'src/main/python/apache/thermos/monitoring', - ] -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/kerberos/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/kerberos/BUILD b/src/main/python/apache/aurora/kerberos/BUILD new file mode 100644 index 0000000..44eda13 --- /dev/null +++ b/src/main/python/apache/aurora/kerberos/BUILD @@ -0,0 +1,57 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +import os + + +python_library( + name = '_kerberos', + sources = rglobs('*.py'), + dependencies = [ + '3rdparty/python:requests-kerberos', + 'src/main/python/apache/aurora/admin', + 'src/main/python/apache/aurora/client', + 'src/main/python/apache/aurora/common', + ], +) + +python_binary( + name = 'kaurora', + entry_point = 'apache.aurora.client.cli.client:proxy_main', + dependencies = [ + ':_kerberos', + ], +) + +python_binary( + name = 'kaurora_admin', + entry_point = 'apache.aurora.admin.aurora_admin:proxy_main', + dependencies = [ + ':_kerberos', + ], +) + +python_library( + name = 'kerberos', + dependencies = [ + ':_kerberos', + ], + provides = setup_py( + name = 'apache.aurora.kerberos', + version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), + ).with_binaries({ + # TODO(ksweeney): Reconsider these as separate binaries in favor of extras_require or plugins. + 'kaurora': ':kaurora', + 'kaurora_admin': ':kaurora_admin', + }), +) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/kerberos/__init__.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/kerberos/__init__.py b/src/main/python/apache/aurora/kerberos/__init__.py new file mode 100644 index 0000000..0663a9a --- /dev/null +++ b/src/main/python/apache/aurora/kerberos/__init__.py @@ -0,0 +1,13 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/kerberos/auth_module.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/kerberos/auth_module.py b/src/main/python/apache/aurora/kerberos/auth_module.py new file mode 100644 index 0000000..fade005 --- /dev/null +++ b/src/main/python/apache/aurora/kerberos/auth_module.py @@ -0,0 +1,38 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +from requests_kerberos import DISABLED, HTTPKerberosAuth + +from apache.aurora.common.auth_module import AuthModule + + +class KerberosAuthModule(AuthModule): + @property + def mechanism(self): + return 'KERBEROS' + + def payload(self): + """NOTE: until AURORA-1229 is addressed, using "Kerberized" client in production in a backwards + compatible way will require a new custom module that will override this method to + return the currently used payload (security blob used in SessionKey). + """ + return '' + + def auth(self): + # While SPNEGO supports mutual authentication of the response, it does not assert the validity + # of the response payload, only the identity of the server. Thus the scheduler will not set + # the WWW-Authenticate response header and the client will disable mutual authentication. + # In order to achieve communication with the scheduler subject to confidentiality and integrity + # constraints the client must connect to the scheduler API via HTTPS. Kerberos is thus only + # used to authenticate the client to the server. + return HTTPKerberosAuth(mutual_authentication=DISABLED) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/aurora/tools/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/aurora/tools/BUILD b/src/main/python/apache/aurora/tools/BUILD index 9dfeb6c..e5ac758 100644 --- a/src/main/python/apache/aurora/tools/BUILD +++ b/src/main/python/apache/aurora/tools/BUILD @@ -11,29 +11,74 @@ # See the License for the specific language governing permissions and # limitations under the License. # +import os + + +THERMOS_SOURCE = 'thermos.py' +OBSERVER_SOURCE = 'thermos_observer.py' + +python_library( + name = '_tools', + sources = rglobs('*.py') - [THERMOS_SOURCE] - [OBSERVER_SOURCE], + dependencies = [ + '3rdparty/python:CherryPy', + '3rdparty/python:twitter.common.app', + '3rdparty/python:twitter.common.exceptions', + '3rdparty/python:twitter.common.log', + 'src/main/python/apache/aurora/executor', + 'src/main/python/apache/thermos/cli', + 'src/main/python/apache/thermos/common', + 'src/main/python/apache/thermos/monitoring', + 'src/main/python/apache/thermos/observer', + ], +) + +python_library( + name = '_thermos', + sources = [THERMOS_SOURCE], + dependencies = [ + ':_tools', + ], +) python_binary( name = 'thermos', - source = 'thermos.py', + # TODO(ksweeney): Investigate making this use entry_point instead. + source = THERMOS_SOURCE, dependencies = [ - 'src/main/python/apache/thermos/cli:common', - 'src/main/python/apache/thermos/cli:main', - 'src/main/python/apache/aurora/executor/common:path_detector', + ':_tools', ] ) +python_library( + name = '_thermos_observer', + sources = [OBSERVER_SOURCE], + dependencies = [ + ':_tools', + ], +) + python_binary( name = 'thermos_observer', - source = 'thermos_observer.py', + # TODO(ksweeney): Investigate making this use entry_point instead. + source = OBSERVER_SOURCE, dependencies = [ - '3rdparty/python:CherryPy', - '3rdparty/python:twitter.common.app', - '3rdparty/python:twitter.common.exceptions', - '3rdparty/python:twitter.common.log', - 'src/main/python/apache/aurora/executor/common:path_detector', - 'src/main/python/apache/thermos/common:constants', - 'src/main/python/apache/thermos/monitoring:detector', - 'src/main/python/apache/thermos/observer:task_observer', - 'src/main/python/apache/thermos/observer/http:configure', + ':_tools', ] ) + +python_library( + name = 'tools', + dependencies = [ + ':_thermos', + ':_thermos_observer', + ':_tools', + ], + provides = setup_py( + name = 'apache.aurora.tools', + version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), + ).with_binaries({ + 'thermos': ':thermos', + 'thermos_observer': ':thermos_observer', + }), +) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/BUILD b/src/main/python/apache/thermos/BUILD deleted file mode 100644 index 8221aa0..0000000 --- a/src/main/python/apache/thermos/BUILD +++ /dev/null @@ -1,33 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -import os - -page(name = 'readme', source = 'README.md') - -python_library( - name = 'thermos', - dependencies = [ - 'src/main/python/apache/thermos/core', - 'src/main/python/apache/thermos/monitoring', - ], - provides = setup_py( - name = 'apache.thermos', - version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), - descripton = 'The Apache Aurora Thermos runtime.', - ).with_binaries( - thermos = 'src/main/python/apache/thermos/cli/bin:thermos', - thermos_runner = 'src/main/python/apache/thermos/bin:thermos_runner', - ) -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/bin/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/bin/BUILD b/src/main/python/apache/thermos/bin/BUILD deleted file mode 100644 index 4f0cb9b..0000000 --- a/src/main/python/apache/thermos/bin/BUILD +++ /dev/null @@ -1,45 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_binary( - name = 'thermos_ckpt', - source = 'thermos_ckpt.py', - dependencies = [ - '3rdparty/python:twitter.common.app', - '3rdparty/python:twitter.common.recordio', - 'src/main/python/apache/thermos/common', - 'api/src/main/thrift/org/apache/thermos:py-thrift' - ] -) - -python_library( - name = 'thermos_runner_source', - sources = ['thermos_runner.py'], - dependencies = [ - 'src/main/python/apache/thermos/common:planner', - 'src/main/python/apache/thermos/common:statuses', - 'src/main/python/apache/thermos/config:schema', - 'src/main/python/apache/thermos/core', - ], -) - -python_binary( - name = 'thermos_runner', - entry_point = 'apache.thermos.bin.thermos_runner', - dependencies = [ - '3rdparty/python:twitter.common.app', - '3rdparty/python:twitter.common.log', - ':thermos_runner_source', - ], -) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/bin/__init__.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/bin/__init__.py b/src/main/python/apache/thermos/bin/__init__.py deleted file mode 100644 index 0663a9a..0000000 --- a/src/main/python/apache/thermos/bin/__init__.py +++ /dev/null @@ -1,13 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/bin/thermos_ckpt.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/bin/thermos_ckpt.py b/src/main/python/apache/thermos/bin/thermos_ckpt.py deleted file mode 100644 index 8ed69e0..0000000 --- a/src/main/python/apache/thermos/bin/thermos_ckpt.py +++ /dev/null @@ -1,83 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -from __future__ import print_function - -import pprint -import sys -import time - -from twitter.common import app -from twitter.common.recordio import RecordIO, ThriftRecordReader - -from apache.thermos.common.ckpt import CheckpointDispatcher - -from gen.apache.thermos.ttypes import RunnerCkpt, RunnerState, TaskState - -app.add_option( - "--checkpoint", - dest="ckpt", - metavar="CKPT", - help="read checkpoint from CKPT") - -app.add_option( - "--assemble", - dest="assemble", - metavar="CKPT", - default=True, - help="whether or not to replay the checkpoint records.") - - -def main(args): - values = app.get_options() - - if len(args) > 0: - print("ERROR: unrecognized arguments: %s\n" % (" ".join(args)), file=sys.stderr) - app.help() - sys.exit(1) - - if not values.ckpt: - print("ERROR: must supply --checkpoint", file=sys.stderr) - app.help() - sys.exit(1) - - fp = file(values.ckpt, "r") - rr = ThriftRecordReader(fp, RunnerCkpt) - wrs = RunnerState(processes={}) - dispatcher = CheckpointDispatcher() - try: - for wts in rr: - print('Recovering: %s' % wts) - if values.assemble is True: - dispatcher.dispatch(wrs, wts) - except RecordIO.Error as err: - print('Error recovering checkpoint stream: %s' % err, file=sys.stderr) - return - print('\n\n\n') - if values.assemble: - print('Recovered Task Header') - pprint.pprint(wrs.header, indent=4) - - print('\nRecovered Task States') - for task_status in wrs.statuses: - print(' %s [pid: %d] => %s' % ( - time.asctime(time.localtime(task_status.timestamp_ms / 1000.0)), - task_status.runner_pid, - TaskState._VALUES_TO_NAMES[task_status.state])) - - print('\nRecovered Processes') - pprint.pprint(wrs.processes, indent=4) - - -app.main() http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/bin/thermos_runner.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/bin/thermos_runner.py b/src/main/python/apache/thermos/bin/thermos_runner.py deleted file mode 100644 index bd8cf7f..0000000 --- a/src/main/python/apache/thermos/bin/thermos_runner.py +++ /dev/null @@ -1,203 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -import functools -import getpass -import os -import pwd -import signal -import sys -import traceback - -from twitter.common import app, log - -from apache.thermos.common.options import add_port_to -from apache.thermos.common.planner import TaskPlanner -from apache.thermos.common.statuses import ( - INTERNAL_ERROR, - INVALID_TASK, - TERMINAL_TASK, - UNKNOWN_ERROR, - UNKNOWN_USER -) -from apache.thermos.config.loader import ThermosConfigLoader -from apache.thermos.core.process import Process -from apache.thermos.core.runner import TaskRunner - -app.add_option( - "--thermos_json", - dest="thermos_json", - default=None, - help="read a thermos Task from a serialized json blob") - - -app.add_option( - "--sandbox", - dest="sandbox", - metavar="PATH", - default=None, - help="the sandbox in which this task should run") - - -app.add_option( - "--checkpoint_root", - dest="checkpoint_root", - metavar="PATH", - default=None, - help="the path where we will store checkpoints") - - -app.add_option( - "--task_id", - dest="task_id", - metavar="STRING", - default=None, - help="The id to which this task should be bound, created if it does not exist.") - - -app.add_option( - "--setuid", - dest="setuid", - metavar="USER", - default=None, - help="setuid tasks to this user, requires superuser privileges.") - - -app.add_option( - "--enable_chroot", - dest="chroot", - default=False, - action='store_true', - help="chroot tasks to the sandbox before executing them.") - - -app.add_option( - "--port", - type='string', - nargs=1, - action='callback', - callback=add_port_to('prebound_ports'), - dest='prebound_ports', - default={}, - metavar="NAME:PORT", - help="bind a numbered port PORT to name NAME") - - -app.add_option( - "--hostname", - default=None, - dest="hostname", - help="The hostname to advertise in ZooKeeper and the thermos observer instead of " - "the locally-resolved hostname.") - - -def get_task_from_options(opts): - tasks = ThermosConfigLoader.load_json(opts.thermos_json) - if len(tasks.tasks()) == 0: - app.error("No tasks specified!") - if len(tasks.tasks()) > 1: - app.error("Multiple tasks in config but no task name specified!") - task = tasks.tasks()[0] - if not task.task.check().ok(): - app.error(task.task.check().message()) - return task - - -def runner_teardown(runner, sig=signal.SIGUSR1, frame=None): - """Destroy runner on SIGUSR1 (kill) or SIGUSR2 (lose)""" - op = 'kill' if sig == signal.SIGUSR1 else 'lose' - log.info('Thermos runner got signal %s, shutting down.' % sig) - log.info('Interrupted frame:') - if frame: - for line in ''.join(traceback.format_stack(frame)).splitlines(): - log.info(line) - runner.close_ckpt() - log.info('Calling runner.%s()' % op) - getattr(runner, op)() - sys.exit(0) - - -class CappedTaskPlanner(TaskPlanner): - TOTAL_RUN_LIMIT = 100 - - -def proxy_main(args, opts): - assert opts.thermos_json and os.path.exists(opts.thermos_json) - assert opts.sandbox - assert opts.checkpoint_root - - thermos_task = get_task_from_options(opts) - prebound_ports = opts.prebound_ports - missing_ports = set(thermos_task.ports()) - set(prebound_ports) - - if missing_ports: - log.error('ERROR! Unbound ports: %s' % ' '.join(port for port in missing_ports)) - sys.exit(INTERNAL_ERROR) - - if opts.setuid: - user = opts.setuid - else: - user = getpass.getuser() - - # if we cannot get the uid, this is an unknown user and we should fail - try: - pwd.getpwnam(user).pw_uid - except KeyError: - log.error('Unknown user: %s' % user) - sys.exit(UNKNOWN_USER) - - task_runner = TaskRunner( - thermos_task.task, - opts.checkpoint_root, - opts.sandbox, - task_id=opts.task_id, - user=opts.setuid, - portmap=prebound_ports, - chroot=opts.chroot, - planner_class=CappedTaskPlanner, - hostname=opts.hostname, - ) - - for sig in (signal.SIGUSR1, signal.SIGUSR2): - signal.signal(sig, functools.partial(runner_teardown, task_runner)) - - try: - task_runner.run() - except TaskRunner.InternalError as err: - log.error('Internal error: %s' % err) - sys.exit(INTERNAL_ERROR) - except TaskRunner.InvalidTask as err: - log.error('Invalid task: %s' % err) - sys.exit(INVALID_TASK) - except TaskRunner.StateError as err: - log.error('Checkpoint error: %s' % err) - sys.exit(TERMINAL_TASK) - except Process.UnknownUserError as err: - log.error('User ceased to exist: %s' % err) - sys.exit(UNKNOWN_USER) - except KeyboardInterrupt: - log.info('Caught ^C, tearing down runner.') - runner_teardown(task_runner) - except Exception as e: - log.error('Unknown exception: %s' % e) - for line in traceback.format_exc().splitlines(): - log.error(line) - sys.exit(UNKNOWN_ERROR) - - -def main(args, opts): - return proxy_main(args, opts) - - -app.main() http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/cli/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/cli/BUILD b/src/main/python/apache/thermos/cli/BUILD index 2b9bc62..a4932b2 100644 --- a/src/main/python/apache/thermos/cli/BUILD +++ b/src/main/python/apache/thermos/cli/BUILD @@ -11,27 +11,26 @@ # See the License for the specific language governing permissions and # limitations under the License. # +import os + python_library( - name = 'common', - sources = ['common.py'], + name = 'cli', + sources = rglobs('*.py'), dependencies = [ + '3rdparty/python:pystachio', '3rdparty/python:twitter.common.app', + '3rdparty/python:twitter.common.dirutil', '3rdparty/python:twitter.common.log', + '3rdparty/python:twitter.common.recordio', + 'api/src/main/thrift/org/apache/thermos', + 'src/main/python/apache/thermos/common', 'src/main/python/apache/thermos/config', - 'src/main/python/apache/thermos/core:runner', - 'src/main/python/apache/thermos/monitoring:detector', - ] -) - -python_library( - name = 'main', - sources = ['main.py'], - dependencies = [ - '3rdparty/python:twitter.common.app', - ':common', - 'src/main/python/apache/thermos/cli/commands', - 'src/main/python/apache/thermos/common:constants', - 'src/main/python/apache/thermos/monitoring:detector', - ] + 'src/main/python/apache/thermos/core', + 'src/main/python/apache/thermos/monitoring', + ], + provides = setup_py( + name = 'apache.thermos.cli', + version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), + ) ) http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/cli/__init__.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/cli/__init__.py b/src/main/python/apache/thermos/cli/__init__.py new file mode 100644 index 0000000..0663a9a --- /dev/null +++ b/src/main/python/apache/thermos/cli/__init__.py @@ -0,0 +1,13 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/cli/bin/BUILD ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/cli/bin/BUILD b/src/main/python/apache/thermos/cli/bin/BUILD deleted file mode 100644 index f33c7f8..0000000 --- a/src/main/python/apache/thermos/cli/bin/BUILD +++ /dev/null @@ -1,21 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -python_binary( - name = 'thermos', - source = 'thermos.py', - dependencies = [ - 'src/main/python/apache/thermos/cli:main', - ] -) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/aurora/blob/8fa4fb0f/src/main/python/apache/thermos/cli/bin/thermos.py ---------------------------------------------------------------------- diff --git a/src/main/python/apache/thermos/cli/bin/thermos.py b/src/main/python/apache/thermos/cli/bin/thermos.py deleted file mode 100644 index fcef38d..0000000 --- a/src/main/python/apache/thermos/cli/bin/thermos.py +++ /dev/null @@ -1,23 +0,0 @@ -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -from twitter.common import app - -from apache.thermos.cli.main import register_commands, register_options - -register_commands(app) -register_options(app) - - -app.main()
