Hi all,
> The sdist step creates a package that should be installed into each > tox environment. If the tox environment has cython when this apache > beam package is installed, it should be used. Nose (or whatever) > should then run the tests. > I spent some time this weekend trying to understand the Beam python build process, and here’s an overview of what I’ve learned: - the :sdks:python:sdist gradle task creates the source tarball (no surprises there) - the protobuf stubs are generated during this process - the sdist is provided to tox, which installs it into the the virtualenv for that task - for *-cython tasks, tox installs the cython dep and, as Ahmet asserted, python setup.py nosetests performs the cythonization. - this cythonized build overrides the one installed by tox Here’s what I learned about the current status of tests wrt cython: - cython tox tasks *are* using cython (good!) - non-cython tox tasks *are not* using cython (good!) - none of the GCP or integration tests are using cython (bad?) - This is because the build is only cythonized when python setup.py nosetests is used in conjunction with tox (tox installs cython, python setup.py nosetests compiles it). - GCP tests don't install cython. ITs don't use tox. To confirm my understanding of this, I created a PR [1] to assert that a cythonized or pure-python build is being used. A cythonized build is expected by default on linux systems unless a special flag is provided to inform the test otherwise. It appears as though the portable tests passed (i.e. used cython), but I forgot to add the assertion for those; like the other ITs they are not using cython. *Questions:* - Is the lack of cython for ITs expected and/or desired? - Why aren't ITs using tox? It's quite possible to pass arguments into tox to control it's behavior. For example, it seems reasonable that run_integration_test.sh could be inside tox *Next Steps:*There has been some movement in the python community to solve problems around build dependencies [2] and toolchains [3]. I hope to have a proposal for how to simplify this process soon. [1] https://github.com/apache/beam/pull/10058 [2] https://www.python.org/dev/peps/pep-0517/ [3] https://www.python.org/dev/peps/pep-0518/ -chad