You are right, it's a sure way to saturate db connections, as a connection
is established every few seconds when the DAGs are parsed. The same happens
when you use variables in __init__ of an operator. Os environment variable
would be safer for your need.


On Mon, 22 Oct 2018, 08:34 Pramiti Goel, <pramitigoe...@gmail.com> wrote:

> Hi,
> We want to make owner and email Id general, so we don't want to put in
> airflow dag. Using variables will help us in changing the email/owner
> later, if there are lot of dags of same owner.
> For example:
> default_args = {
>     'owner': Variable.get('test_owner_de'),
>     'depends_on_past': False,
>     'start_date': datetime(2018, 10, 17),
>     'email': Variable.get('de_infra_email'),
>     'email_on_failure': True,
>     'email_on_retry': True,
>     'retries': 2,
>     'retry_delay': timedelta(minutes=1)}
> Looking into the code of Airflow, it is making connection session everytime
> the variable is created, and then close it. (Let me know if I understand
> wrong). If there are many dags with variables in default args running
> parallel, querying variable table in MySQL, will it have any sort of
> limitation on number of sessions of SQLAlchemy ? Will that make dag slow as
> there will be many queries to mysql for each dag? is the above approach
> good ?
>  >using Airlfow 1.9
> Thanks,
> Pramiti.

Reply via email to