[
https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Zhu updated AIRFLOW-2371:
------------------------------
Description:
Basically we need a long term solution for
https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor is
a workaround, however it slows down execution of subdags.
Looking for a better solution.
I also see that when running subdags, db connection pools are not shared either
(since they are separate processes), which causes db disconnection issues like
https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db because there can
be too many connections. I hope it can be fixed too when subdag operator works
with pooling..
was:
Basically we need a long term solution for
https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor is
a workaround, however it slows down execution of subdags.
Looking for a better solution.
I also see that when running subdags, db connection pools are not shared either
(since they are separate processes), which causes db disconnection issues like
https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db because there can
be too many connections. I hope it can be fixed when subdag operator can work
with pooling!
> SubdagOperator ignores pooling when not using SequentialExecutor
> ----------------------------------------------------------------
>
> Key: AIRFLOW-2371
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2371
> Project: Apache Airflow
> Issue Type: Bug
> Components: subdag
> Affects Versions: Airflow 1.9.0
> Environment: Airflow 1.9.0
> Reporter: Xiao Zhu
> Priority: Major
>
> Basically we need a long term solution for
> https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor
> is a workaround, however it slows down execution of subdags.
> Looking for a better solution.
> I also see that when running subdags, db connection pools are not shared
> either (since they are separate processes), which causes db disconnection
> issues like https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db
> because there can be too many connections. I hope it can be fixed too when
> subdag operator works with pooling..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
