potiuk opened a new pull request #14332: URL: https://github.com/apache/airflow/pull/14332
This change attempts to stabilize pylint checks. Since we have recently added self-hosted runners with multiple CPUS, seems that re-enabling parallel mode for pylint makes perfect sense as we will finally be able to use the parallelism and speed up static checks significantly. Previously the tests were run in single-processor mode in attempt to avoid random mistakes where different files were processed in different processes. This led to random pylint or mypy problems and aither false-positives or false negatives before especially when it came to circular dependencies. but since we are now past heavy refactoring, this should be no problem for future changes and occasional false positive/negative is less disruptive than long checks. The attempt is made to apply sorting order in the files passed to pylint. This should provide more stability in the results of running the tests in PR and in master. We had some custom pylint plugins that prevented using of pylint parallelism. For now we are giving up on one of the plugins (no asserts use) and we rely on committer's review on that (we have a rule in place to only use asserts in tests). The other plugin was replaced by coming back to separation of "main code" and "test code" and applying different rules to those - we have now two different configuration files|: pylintrc and pylintrc-tests to control settings for those two different cases. Mypy and flake8 have been parallelized at the level of pre-commits. By implementing those changes we are likely to speed up the tests on self-hosted runners 6x-8x times. <!-- Thank you for contributing! Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation. Feel free to ping committers for the review! In case of existing issue, reference it using one of the following: closes: #ISSUE related: #ISSUE How to write a good git commit message: http://chris.beams.io/posts/git-commit/ --> --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
