erg commented on issue #33241: URL: https://github.com/apache/airflow/issues/33241#issuecomment-1670588487
Thanks for your detailed answer. I had not read the explanation about constraints from issue #33169 from two days ago, I wish it were around when I struggled with trying to get airflow 2.2 mwaa deployed last year for the same exact reason. The misconception starts from the very name of the file--`constraints.txt`. Why would you install something with constraints into an environment and immediately afterwards go and violate those constraints? Knowing that it is meant to be used per-command and not on the entire environment at all times makes it all self-explanatory. Yet the name invites you to use it with every command, or worse yet, trying to include the constraints file when you create your entire virtual env from a requirements.txt file. For people who have to work with an SRE team, getting the deploy wrong on AWS means letting down other people, lots of frustration all around, potential downtime for users, lost revenue, etc. What if it deploys but then things just start breaking because of version incompatibilities between python libraries and constraints file? People who think too much and don't understand the illogicalness of violating constraints.txt will have these thoughts, yet once you learn the constraints are per-command and don't apply to the environment overall, it all seems so obvious. You know what would help? Adding the instructions to the top of the constraints.txt file. constraints.txt: ``` # # This constraints file was automatically generated on 2023-01-18T18:46:04Z # via "eager-upgrade" mechanism of PIP. For the "v2-5-test" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. # # Those constraints are actually those that regular users use to install released version of Airflow. # We also use those constraints after "apache-airflow" is released and the constraints are tagged with # "constraints-X.Y.Z" tag to build the production image for that version. # # This constraints file is meant to be used only in the "apache-airflow" installation command and not # in all subsequent pip commands. By using a constraints.txt file, we ensure that solely the Airflow installation step # is reproducible. Subsequent pip commands may install packages that would have violated the constraints # required for Airflow installation step. Finally, pip commands that might change the installed version of apache-airflow # should include `apache-airflow==2.5.1` in the list of install targets to preserve the Airflow version. APScheduler==3.6.3 Authlib==1.2.0 Babel==2.11.0 ConfigUpdater==3.1.1 Deprecated==1.2.13 ... ``` Places that constraints.txt files are not adequately explained as per-command rather than per-environment: - https://pip.pypa.io/en/stable/user_guide/#constraints-files - https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html - literally everywhere else besides #33169 and now this ticket I have wasted so much time with python/pip/poetry/conda/miniconda/anaconda/venv/python2... Anyway, thanks for your help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
