Somewhat complementary work - this change would make the generated packages equivalent to the ultimate goal of replacing these top-level BUILD files directly with setup.py files. Users could begin testing deployment with the pants-generated sdists and seamlessly drop in setup.py-generated replacements. Users wishing to deploy via PEX could use the pex tool on the generated sdists rather than the pants pex plugin.
On Thu, Jul 30, 2015 at 4:13 PM, Bill Farner <wfar...@apache.org> wrote: > We've talked in the past about switching build tools. Just to keep that in > context - how would you weigh this effort against a tool change? > > -=Bill > > On Thu, Jul 30, 2015 at 4:00 PM, Kevin Sweeney <kevi...@apache.org> wrote: > > > I propose a simplification of the Python BUILD layout as follows and a > set > > of new conventions, as follows: > > > > 1) 1 BUILD per 3rd level directory. These are currently > > > > ``` > > % find src/main/python -maxdepth 3 -mindepth 3 -type d |while read > dirname; > > do echo $dirname | sed 's@src > /main/python/\(.*\)/\(.*\)/\(.*\).*@\1.\2.\3@ > > '; > > done > > apache.aurora.client > > apache.aurora.common > > apache.aurora.tools > > apache.aurora.admin > > apache.aurora.executor > > apache.aurora.config > > apache.thermos.monitoring > > apache.thermos.common > > apache.thermos.cli > > apache.thermos.testing > > apache.thermos.core > > apache.thermos.runner > > apache.thermos.observer > > apache.thermos.config > > ``` > > > > 2) Each BUILD file exports 1 python_library that provides a setup_py > > containing each python_binary in the BUILD file, named the same as the > > directory it's in so that it can be referenced without a ':' character. > The > > sources field in the python_binary will always be rglobs('*.py'). > > > > 3) Other BUILD files may only depend on this single public python_library > > target. Any other target is considered a private implementation detail > and > > should be prefixed with an _. > > > > 4) python_binary targets are always named the same as the exported > console > > script. > > > > 5) python_binary targets must have identical dependencies to the > > python_library exported by the package and must use entry_point. > > > > The advantage of this change is that a PEX file generated by pants will > > contain exactly the same files that will be available on the PYTHONPATH > in > > the case of pip-installation of the corresponding library target. This > will > > help our migration off pants in the future. > > > > Annotated example: apache.thermos.runner (renamed thermos/bin -> > > thermos/runner) > > > > ``` > > % find src/main/python/apache/thermos/runner > > src/main/python/apache/thermos/runner > > src/main/python/apache/thermos/runner/__init__.py > > src/main/python/apache/thermos/runner/thermos_runner.py > > src/main/python/apache/thermos/runner/BUILD > > % cat src/main/python/apache/thermos/runner/BUILD > > # License boilerplate omitted > > import os > > > > > > # Private target so that a setup_py can exist without a circular > > dependency. Only targets within this file should depend on > > # this. > > python_library( > > name = '_runner', > > # The target covers every python file under this directory and > > subdirectories. > > sources = rglobs('*.py'), > > dependencies = [ > > '3rdparty/python:twitter.common.app', > > '3rdparty/python:twitter.common.log', > > # Source dependencies are always referenced without a ':'. > > 'src/main/python/apache/thermos/common', > > 'src/main/python/apache/thermos/config', > > 'src/main/python/apache/thermos/core', > > ], > > ) > > > > # Binary target for thermos_runner.pex. Nothing should depend on this - > > it's only used as an argument to ./pants binary. > > python_binary( > > name = 'thermos_runner', > > # Use entry_point, not source so the files used here are the same ones > > tests see. > > entry_point = 'apache.thermos.bin.thermos_runner', > > dependencies = [ > > # Notice that we depend only on the single private target from this > > BUILD file here. > > ':_runner', > > ], > > ) > > > > # The public library that everyone importing the runner symbols uses. > > # The test targets and any other dependent source code should depend on > > this. > > python_library( > > name = 'runner', > > dependencies = [ > > # Again, notice that we depend only on the single private target from > > this BUILD file here. > > ':_runner', > > ], > > # We always provide a setup_py. This will cause any dependee libraries > to > > automatically reference this library > > # in their requirements.txt rather than copy the source files into > their > > sdist. > > provides = setup_py( > > # Conventionally named and versioned. > > name = 'apache.thermos.runner', > > version = open(os.path.join(get_buildroot(), > > '.auroraversion')).read().strip().upper(), > > ).with_binaries({ > > # Every binary in this file should also be repeated here. > > # Always use the dict-form of .with_binaries so that commands with > > dashes in their names are supported. > > # The console script name is always the same as the PEX with .pex > > stripped. > > 'thermos_runner': ':thermos_runner', > > }), > > ) > > ``` > > > > Let me know what you think, if y'all agree I'll prepare a patch shortly. > > > -- Kevin Sweeney @kts