oresttokovenko opened a new pull request, #4208: URL: https://github.com/apache/iceberg-python/pull/4208
# What This change replaces `Makefile` and `.pre-commit-config.yaml` with a single `noxfile.py`, and leaves the `justfile` as a thin entry point with one command line per recipe. CI now calls the same nox sessions that a developer runs locally, so there is one place to look when a check fails. - `noxfile.py` defines 24 sessions, six of which form the gate while the rest are test suites and local actions. - `justfile` holds no logic and is 141 lines long. - Tool versions live in `pyproject.toml` and `uv.lock` only, so each session installs the dependency group it needs by name. - The Python matrix comes from the trove classifiers in `pyproject.toml` rather than from a workflow matrix. - The three workflows that ran `make` now run `just`, and `python-ci.yml` calls the sessions directly. This is not the shortest way to write this file, because I copied the old behaviour first: the pre-commit hooks let `pre-commit` choose the files to check, which nox does not do, so `noxfile.py` carries helpers that walk the tree, read `.gitignore`, and separate text files from binary ones. A native tool does that work itself, and if the hooks stay in `pre-commit` or `prek`, or if each tool scans its own paths, I expect the file to shrink to about half. That is not this pull request, whose goal is narrow: show that one file describes the same checks as the two old files while behaving the same way. The shorter version can follow once this lands. Two dependencies move into the project as a result. `mypy` and its four stub packages become part of a `types` dependency group, because the pre-commit hook used to install them in an environment of its own and nothing else made them available. The `ty` configuration is untouched, since the type-checker work lands separately in #4207, which adds the `tests/` exclusion and the stricter rules; until that merges, `just typecheck-ty` reports more diagnostics than it will afterwards. # Why Three files described the same checks in three different ways, so learning what CI runs meant opening the `Makefile`, then `.pre-commit-config.yaml`, and then the workflow. Versions lived in two places for the same reason, since the hook configuration pinned every tool with `rev:`. Now one file describes each check and CI runs that same session, so a version bump touches `pyproject.toml` and `uv.lock` and nothing else. # Where the old recipes and checks live now Makefile targets: | Old target | Command now | Session or recipe | | --- | --- | --- | | `help` | `just --list` | default recipe | | `install-uv` | `just install-uv` | justfile, uv must exist before nox starts | | `install` | `just install` | `install` | | `check-license` | `just check-license` | justfile, `dev/check-license` unchanged | | `lint` | `just check`, `just fix` | the six gate sessions | | `test` | `just test` | `tests-3.11` to `tests-3.14` | | `test-integration` | `just test-integration` | `integration` | | `test-integration-setup` | part of `just test-integration` | `_start_services` | | `test-integration-exec` | part of `just test-integration` | `_pytest` | | `test-integration-cleanup` | part of `just test-integration` | `_stop_services` | | `test-integration-rebuild` | `just test-integration-rebuild` | `integration-rebuild` | | `test-s3` | `just test-s3` | `s3` | | `test-adls` | `just test-adls` | `adls` | | `test-gcs` | `just test-gcs` | `gcs` | | `test-coverage` | `just test-coverage` | `tests`, `integration`, `s3`, `adls`, `gcs`, then `coverage` | | `coverage-report` | `just coverage-report` | `coverage` | | `test-notebook` | `just test-notebook` | `test-notebook` | | `docs-serve` | `just docs-serve` | `docs-serve` | | `docs-build` | `just docs-build` | `docs-build` | | `notebook` | `just notebook` | `notebook` | | `notebook-infra` | `just notebook-infra` | `notebook-infra` | | `clean` | `just clean` | `clean` | Pre-commit hooks: | Old hook | Session | | --- | --- | | `trailing-whitespace` | `hygiene` | | `end-of-file-fixer` | `hygiene` | | `check-yaml` | `hygiene` | | `check-ast` | `hygiene` | | `ruff` | `lint` | | `ruff-format` | `format` | | `nbqa-ruff` | `lint` | | `mypy` | `typecheck-mypy` | | `ty` | `typecheck-ty`, local and experimental | | `markdownlint` | `markdownlint` | | `codespell` | `codespell` | | `uv-lock` | `just lock-check` | # Behaviour changes - There are no Git hooks any more, so run `just check` before you push, which is exactly what CI runs. - `just check` writes no files, while `just fix` applies the fixes. - Choose an interpreter with nox's own flag, `just install --force-python 3.12`. - The gate is whatever lacks `default=False`, so a new check session joins it by default instead of needing a second registration. - Markdown linting runs on `rumdl` from PyPI, which reads the existing `.markdownlint.yaml` and avoids a Node.js install. Rule `MD057` stays off because the `reference/` pages are generated by mkdocstrings at build time. - `check-yaml` runs with `--unsafe` because `mkdocs/mkdocs.yml` uses the `!!python/name:` tag. - `mypy` checks every tracked Python file again, 215 in total and with `--scripts-are-modules`; an earlier draft of this change had narrowed it to the package alone. - The Windows job keeps the previous workflow's exclusion of the `hive-kerberos` extra, which pulls `kerberos`, a package that publishes no Windows wheels and would need MIT Kerberos headers the runner does not have. - Each test session rebuilds the in-place Cython extension when its ABI tag is missing, because a cached editable build for one interpreter otherwise leaves another one unable to import `pyiceberg.avro.decoder_fast`. - `notebook-infra` stops the Docker services when JupyterLab exits, unless `KEEP_COMPOSE=1` asks to keep them. # Small fixes the port needed - `.gitignore` ships in the sdist, because `noxfile.py` reads it and release verification runs from an unpacked sdist. - `.coverage.*` is ignored now, since parallel coverage runs write one data file per interpreter. - `format-noxfile` formats the noxfile without going through nox, so it works before the file can be imported. # Tests `just check` runs six sessions in about 13 seconds and is green. `just test` runs the four interpreters in parallel: 37 seconds on a cold tree, and the full unit suite on 3.13 takes a minute. `just install --force-python 3.12 -- --dry-run` prints `uv sync --all-extras --python 3.12` while the plain form leaves the environment alone, and `just docs-build` builds in seven seconds. `just clean` then `just install` rebuilds from an empty tree, an unpacked sdist lets nox read the noxfile, and a deliberately dirty Markdown file fails `just check` while a clean one passes. The integration and cloud suites, CI on GitHub, and `dev/release/verify_rc.sh` end to end were not run, because they need Docker services and a real runner. `typecheck-ty` is tagged experimental and stays out of the gate; on the current base it reports 894 diagnostics, which #4207 reduces once its exclusions and rules are in place. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
