potiuk commented on issue #33241: URL: https://github.com/apache/airflow/issues/33241#issuecomment-1670917849
Thanks a lot for that. That explanation will definitely help me to make my talk better focus on dispelling some misconceptions. > Thanks for your detailed answer. I had not read the explanation about constraints from issue https://github.com/apache/airflow/issues/33169 from two days ago, I wish it were around when I struggled with trying to get airflow 2.2 mwaa deployed last year for the same exact reason. This is why I am giving the talk on the Summit. I hope it will help. > The misconception starts from the very name of the file--constraints.txt. Why would you install something with constraints into an environment and immediately afterwards go and violate those constraints? Knowing that it is meant to be used per-command and not on the entire environment at all times makes it all self-explanatory. Yet the name invites you to use it with every command, or worse yet, trying to include the constraints file when you create your entire virtual env from a requirements.txt file. Well actualy I believe (the name is from `pip` ) this is quite different I see constraint as name. **REQUIREMENTS** are hard, **constraints** are softer. The whole idea of constraints is that they just "constraint" you to a subset of things while you run particular operation and they are not "required". > For people who have to work with an SRE team, getting the deploy wrong on AWS means letting down other people, lots of frustration all around, potential downtime for users, lost revenue, etc. What if it deploys but then things just start breaking because of version incompatibilities between python libraries and constraints file? People who think too much and don't understand the illogicalness of violating constraints.txt will have these thoughts, yet once you learn the constraints are per-command and don't apply to the environment overall, it all seems so obvious. Yes - but this is where SRE people have take responsibility. If they don't then, well, what else their job is? A bit philosophical rant .... You can't have cake and it eat it too. Airflow has 650 dependencies. About 30% of them have also extras (which means there are other dependencies and limits they apply optionally when you specify them). It's IMPOSSIBLE to satisfy all the combinations of those with a single file. What I recommend instead if you want to keep control is to make an environment as SRE is to build the environment, add your dependencies, run all your tests if all is right (you do have some way to test it) and [produce your own constraints](https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#using-your-own-constraints) for your environment. We as community can give you constraints that help to install airflow in a reproducible way to start with - and you know all this has been tested before. But if you add your own variations - we cannot test all combinations of all variations for our users. This is where community role ends. This is where impossible task (test al l the possible combinations in the community) has to be turned into possible task (I test MY combination of extra dependencies I have). Also if you consider that you are paid to do the SRE job, and the testing done in the community is done mostly by 2500 volunteers who spend their nights and weekends, and that you get the software for free, then you will likely understand that what you are requesting is for somone to do for free many order of magnitude complex work that you can do in your paid time. Just being brutally honest - trying to push that responsibility down to the open-source community is quite a bit unfair game. > You know what would help? Adding the instructions to the top of the constraints.txt file. That's a VERY GOOD suggestion. Thank you! I will do it right away. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
