kaxil commented on issue #6515: [AIRFLOW-XXX] GSoD: How to make DAGs production ready URL: https://github.com/apache/airflow/pull/6515#issuecomment-551736975 > @KKcorps I think one of the important things to mention in this document is database access during parsing time and especially avoiding to use Airflow Variables in the DAGs (they are still ok to use in "execute" method). > > This is a known problem that people are complaining a lot that scheduler opens and closes a lot of connections to the database - because every time the file is parsed, and variable is reached, a database connection is opened and query executed. > > I think there are lots of examples around with using variables in the DAGs but this is not really a good practice and I think this is a perfect place to mention it. > > I believe environment variables are better way to share common configuration. > > @kaxil @ashb @mik-laj - what are your thoughts about it ? Agree. So using lots of Airflow variables in the file outside of task code will cause many DB connections. It is fine to use it in a deferred way in Jinja templated field. For using it outside task file use Environment Variables.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
