[ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Zhu updated AIRFLOW-2371: ------------------------------ Description: Basically we need a long term solution for https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor is a workaround, however it slows down execution of subdags. Looking for a better solution. I also see that when running subdags, db connection pools are not shared either (since they are separate processes), which causes db disconnection issues like https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db because there can be too many connections. I hope it can be fixed too when subdag operator works with pooling.. was: Basically we need a long term solution for https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor is a workaround, however it slows down execution of subdags. Looking for a better solution. I also see that when running subdags, db connection pools are not shared either (since they are separate processes), which causes db disconnection issues like https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db because there can be too many connections. I hope it can be fixed when subdag operator can work with pooling! > SubdagOperator ignores pooling when not using SequentialExecutor > ---------------------------------------------------------------- > > Key: AIRFLOW-2371 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2371 > Project: Apache Airflow > Issue Type: Bug > Components: subdag > Affects Versions: Airflow 1.9.0 > Environment: Airflow 1.9.0 > Reporter: Xiao Zhu > Priority: Major > > Basically we need a long term solution for > https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor > is a workaround, however it slows down execution of subdags. > Looking for a better solution. > I also see that when running subdags, db connection pools are not shared > either (since they are separate processes), which causes db disconnection > issues like https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db > because there can be too many connections. I hope it can be fixed too when > subdag operator works with pooling.. -- This message was sent by Atlassian JIRA (v7.6.3#76005)