potiuk commented on issue #12983: URL: https://github.com/apache/airflow/issues/12983#issuecomment-762430800
Surely this is all subjective. And it's soooo easy to forget about things like werkzeug drama (and a number of other problems). And about alll the issues flooding us when one day suddenly airflow could not be installed because dependencies broken. Or about mypy/flake errors that a number of people could not repreoduce or about failing kubernetes tests because the scripts combined tox, docker image and the K8S setup was failing randomly (which was far more complex than current tests TBH and no-one understood it then anyway). Or about conflicting constraints and totally outdated dependencies and base python images we had for the docker image (which are getting improved) Most of the complexity of the system we have now is the result of fixing those problems (in the way that prevents them from reoccurring - that's why all those things are constantly running on our CI and double-checking). The complexity is emerging from the complexity of the system we have, not "reason" on its own. This is often the case that you fail to see the things that "just work" and forget how much work was put into getting to that "just works" state. It's really, really easy to be overhwelmed by a number of issues like that. If you do not continuously work on fighting such issues (and yest sometimes it means increasing complexity) you might one day wake up in the state that you have *just* enough capacity to fight with the on-going issues. It's super easy to underestimate importance of it and put a blind eye on it. I've seen many CI systems that took that slippery road - the problem with it is that once you let it happen it goes down with accelerating pattern. System of that size and that many people contributing people is inevitable to take quite some maintenance time. I wonder if you've thought about the number changes (when CI works reasonable) that we are able to handle because of all the systems we have in place now rather than "just code". If you look at that chart below, we have hardly more active people working on Airflow as 'active commiters' than 1.5 years ago, yet we are able to handle many more changes:  This is compound change of many things - including the fact that most of us can focus on contribution to product, because there are people who constantly keep the machinery not only going but also improving and rolling the wheel better. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
