Vincent Castaneda created AIRFLOW-4346:
------------------------------------------
Summary: Kubernetes Executor Fails for Large Wide DAGs
Key: AIRFLOW-4346
URL: https://issues.apache.org/jira/browse/AIRFLOW-4346
Project: Apache Airflow
Issue Type: Bug
Components: DAG, executor, kubernetes
Affects Versions: 1.10.3, 1.10.2
Reporter: Vincent Castaneda
Attachments: configmap-airflow-share.yaml, wide_dag_bash_test.py,
wide_dag_test_100_300.py, wide_dag_test_300_300.py
When running large DAGs–those with parallelism of over 100 task instances to be
running concurrently--several tasks fail on the executor and are reported to
the database, but the scheduler is never aware of them failing.
Attached are:
- A test DAG that we can use to replicate the issue.
- The configmap-airflow.yaml file
I will be available to answer any other questions that are raised about our
configuration. We are running this on GKE and giving the scheduler and web pod
a base 100m for execution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
