ashb opened a new pull request #7421: [AIRFLOW-XXXX] Speed up mypy runs. URL: https://github.com/apache/airflow/pull/7421 This PR does two things: 1. It enables the mypy cache (default folder name .mypy_cache) so that repeated runs locally are quicker 2. It _disables_ passing only the changed files in. Point 2 seems counter-intuitave, but in my testing running with all files (airflow docs tests) was about twice as fast as without. My hypothesis for why this happens is that when mypy is checking file x, it has to check dependencies/imports for it too, and when we have pass_filenames set runs multiple processes in parallel, and each of them have to do this work! Timings before and after: - Before: For all files ``` ❯ time pre-commit run mypy -a Run mypy................................................................. Forcing response 'no' to 'rebuild CI?' Passed pre-commit run mypy -a 0.31s user 0.07s system 2% cpu 17.140 total ``` With only a single file ``` ❯ time pre-commit run mypy --files airflow/configuration.py Run mypy................................................................. Forcing response 'no' to 'rebuild CI?' Passed pre-commit run mypy --files airflow/configuration.py 0.30s user 0.07s system 5% cpu 6.724 total ``` - After: With a clean cache (`rm -rf .mypy_cache`): ``` $ time pre-commit run mypy Run mypy................................................................. Forcing response 'no' to 'rebuild CI?' Passed pre-commit run mypy -a 0.26s user 0.10s system 2% cpu 17.226 total ``` Repeated run (cache folder exists): ``` $ time pre-commit run mypy Run mypy................................................................. Forcing response 'no' to 'rebuild CI?' Passed pre-commit run mypy -a 0.28s user 0.07s system 7% cpu 4.721 total ``` I didn't test with a single file because even with `--files` passed to pre-commit, because we have specified `pass_filenames: false` it doesn't pass _any_ files. But since it's about the same time (quicker in my testing!) I think this is fine :) --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup> - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
