potiuk commented on code in PR #25780: URL: https://github.com/apache/airflow/pull/25780#discussion_r956385787
########## docs/apache-airflow/best-practices.rst: ########## @@ -619,3 +621,221 @@ Prune data before upgrading --------------------------- Some database migrations can be time-consuming. If your metadata database is very large, consider pruning some of the old data with the :ref:`db clean<cli-db-clean>` command prior to performing the upgrade. *Use with caution.* + + +Handling Python dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Airflow has many Python dependencies and sometimes the Airflow dependencies are conflicting with dependencies that your +task code expects. Since - by default - Airflow environment is just a single set of Python dependencies and single +Python environment, often there might also be cases that some of your tasks require different dependencies than other tasks +and the dependencies basically conflict between those tasks. + +If you are using pre-defined Airflow Operators to talk to external services, there is not much choice, but usually those +operators will have dependencies that are not conflicting with basic Airflow dependencies. Airflow uses constraints mechanism +which means that you have a "fixed" set of dependencies that the community guarantees that Airflow can be installed with +(including all community providers) without triggering conflicts. However you can upgrade the providers +independently and their constraints do not limit you so the chance of a conflicting dependency is lower (you still have +to test those dependencies). Therefore when you are using pre-defined operators, chance is that you will have +little, to no problems with conflicting dependencies. + +However, when you are approaching Airflow in a more "modern way", where you use TaskFlow Api and most of +your operators are written using custom python code, or when you want to write your own Custom Operator, Review Comment: I changed it to "pythonic" from modern -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
