potiuk commented on PR #27946: URL: https://github.com/apache/airflow/pull/27946#issuecomment-1329369078
> Ultimately it doesn't matter, so long as it's tested somewhere. (If I was writing it I would have put it in unit tests, but I'm not so it doesn't really matter where it is ) BTW. There are projects (we are not one of them) where there are real, fast, super-fast unit tests (taking a fraction of second to run), people often also run (at least those fast tests) in pre-commits - this saves a lot of iteration time on CI with very limited impact on the developer productivity - whether the commit takes 100ms longer does not really matter, but saving a CI iteration feedback to find out that things do not work is a lot of saved time and focus. So ultimately for me the choice is: * fast -> pre-commit * slow -> tests in CI Getting feedback as soon as possible without unnecessary steps is very important. Maybe even one day we could move some of the "fast" tests to be run as pre-commit (especially when we separate out providers and separate DB-requiring tests from those that do not require DB. Then we should be able to not only run those tests in a local venv (like most projects have), use tox (like lot of project hs) and run the unit tests as part of pre-commit. That would be my dream setup after we separate providers out. Then on CI we would only have to run all the "heavy tests" that will be left (and if we apply technique from this PR to generate "true" API interface of Airflow for Providers, then we might even not have to run provider compatibility tests for Airlfow. And are not that far from it, I believe. If we apply similar approach to define and separate Airflow core API that Airlfow exposes to providers, and add mechanism to control changes to those APIs in similar way to this PR, then we will have the final step to be able to attempt the split of providers (if we decide we want it of course). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
