potiuk commented on pull request #10959:
URL: https://github.com/apache/airflow/pull/10959#issuecomment-694116848
I hope this will be green this time.
Some more context and explanation why this change is important @kaxil @ashb
@turbaszek @mik-laj @feluelle @houqp @zhongjiajie @dimberman - this
innocuously looking change might be our gateway to a single biggest improvement
in waiting time for most contributors and committers in our CI.
It allows for full programmatic control over which tests are executed for
every PR. Together with the decision on fully separating out providers
(implementation in progress - #10822), which will result in removing all
dependencies to providers from the Airflow core. This has been enabled by
AIP-21 (separating providers) and the CI refactoring (#10422).
We will be able to finally implement #10507 which IMHO is a real
game-changer for our CI waiting time (I have it already POC'd implemented in my
separate branch).
The effect of this will be:
* for PRs changing only one or few providers (the vast majority of the
commits from "casual contributors") we will only run tests for those providers
(and no tests for the "core' Airflow. We will only run them for one database
(Postgres ?) as there is virtually no difference, which DB is used for the
provider tests. There are a few edge cases (like 'MySQL' and 'Postgres'
providers) but since the selection of tests will be fully programmable, we can
have a very flexible approach here and make sure that when the provider
requires Postgres/MySQL, the tests for this backend will also be running for
those backends
* we will still have the safety net - all the tests for all the backends
will be running after we merge a PR to master, so we will still see if there is
a case we missed and we will have a chance to fix it.
* for PRs changing "airflow" core - we will continue running all tests for
all the backends.
* The final result of it will be that:
- we will have far less number of "Test" jobs run for most of the PRs
- those "Test" jobs in most cases will run much, much faster (minutes
rather than half an hour)
- in most cases, the "feedback" time for those "casual" contributors
will be < 10 minutes rather than the current >40 minutes.
- the "core" PRs (mostly done by committers) will run even more
comprehensive tests (all backends + all types of tests always) but then they
will have many more jobs available in CI to run as those will not be blocked by
the "casual" changes to providers.
I honestly think this is a game-changer and I will try to implement it
quickly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
