autophagy opened a new pull request, #26897: URL: https://github.com/apache/flink/pull/26897
## What is the purpose of the change With [FLINK-37775](https://issues.apache.org/jira/browse/FLINK-37775) and [FLINK-36900](https://issues.apache.org/jira/browse/FLINK-36900) we have started to use uv for managing Python testing environments and installing linting tools, as well as defining test/lint/typecheck dependencies in pyproject.toml. However, the CI/CD scripts and developer documentation is in a bit of an inbetween state. We use uv for creating Python virtual environments for tool installation, but uv supports these natively with `uv run` disregarding the need for a custom made virtual environment, for example. This PR does the following: - Uses `uv` in our CI/CD scripts and developer documentation where possible. When it comes to lint stages such as flake8 and mypy, for example, the dependencies for those checks are managed by `uv` using `uv run`. This means the install steps for those tools in `lint-python.sh` can be removed, as this is managed by uv. - Used the `tox-uv` extension to `tox` so that `tox` uses `uv` to create the correct python environment for testing against various python versions, rather than relying on premade virtual environments. This also means the old `install-command.sh` script is no longer needed. - Replaced instances where the `apache-flink` and `apache-flink-libraries` packages were being built via `python setup.py` to instead use `uv build`, taking advantage of build isolation and automatic build dependency management. - Migrated static package metadata for `apache-flink` and `apache-flink-libraries` into their own `pyproject.toml` files, so they are viewed as concrete projects by `uv`. - Added `./apache-flink-libraries` as a uv source so that, during development, the `apache-flink-libraries` package is automatically built (for example, when doing `uv pip install -e .` in the `flink-python` project). This sidesteps the need for building and installing the `apache-flink-libraries` dependency manually from source when doing local development. - Changed the `build-wheels.sh` script to build the pyflink wheels using `uv build --python <python-version>`. This, coupled with the tox changes, means that the `py_env` step of `lint-python.sh` (where we create venvs for supported python versions) can be removed. ## Brief change log - Migrated the `lint-python.sh` script to use `uv run` for running lint, testing, typechecking and docs building steps. - Added `tox-uv` and bumped the `tox` dependency so that it can create the virtualenvs that it needs to run tests as needed with `uv`. - Updated building and testing scripts to use `uv build`, `uv run` and `uv pip` where possible. - Added section to developer docs about building the PyFlink project using `uv`. ## Verifying this change This change is already covered by existing tests, such as PyFlink unit tests, end-to-end tests and running the `build-wheels.sh` script. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
