potiuk commented on a change in pull request #8383: Remove steps from travis
URL: https://github.com/apache/airflow/pull/8383#discussion_r409041753
##########
File path: .github/workflows/static-tests.yml
##########
@@ -0,0 +1,36 @@
+name: "static tests"
+on: [pull_request, push]
+env:
+ TRAVIS_BRANCH: "master"
+jobs:
+ statics-tests:
+ name: Static tests
+ runs-on: ubuntu-latest
+ env:
+ PYTHON_VERSION: 3.6
+ AIRFLOW_MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
+ TRAVIS_JOB_NAME: "Static"
+ strategy:
+ matrix:
+ scripts: [ci_run_static_checks_pylint_tests.sh,
+ ci_mypy.sh,
+ ci_fix_ownership.sh,
Review comment:
Been there done that @dimberman .
I think this will not increase the speed. It might even slow it down.
There is quite an overhead introduced by creating a new runner (it works
similarly in Travis and GA last time I checked). Vast majority of the tests -
including mypy, llake8, bat_tests are fast and it makes no sense to spin
separate worker just to run the single static check.. The ones that take vast
majority (like 90%) of time are pylint_main and pylint_test), I experimented
with it before and the best overall execution time was what was there before:
1, job - run all tests but pylint_tests
2, job - Run pylint_tests
Further fragmentation introduces much more overhead for worker creation and
elapsed time to execute the builds was much longer. With this approach you can
gain at most single seconds in the best case.
What would ultimately help if is we finish all pylint tests and introduce
paralllelism in pylint tests (the tests can run muich faster but currently we
have uncertainty of cycles introduced because some of our code is excluded from
pylint and contains cycles, Those cycles are randomly appearing when people
make their PRs so Pylint tests are now set to run without parallelism.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
