Barend created AIRFLOW-6996:
-------------------------------
Summary: Control KubernetesExecutor delete_worker_pods setting per
task
Key: AIRFLOW-6996
URL: https://issues.apache.org/jira/browse/AIRFLOW-6996
Project: Apache Airflow
Issue Type: Improvement
Components: executor-kubernetes
Affects Versions: 1.10.9, 2.0.0
Reporter: Barend
Assignee: Daniel Imberman
h3. Context
The global {{airflow.cfg}} defines a boolean flag that controls whether the
KubernetesExecutor should delete worker pods:
{code:none}
[kubernetes]
# If True (default), worker pods will be deleted upon termination
delete_worker_pods = True
{code}
You generally want this flag to be {{True}}, unless diagnosing specific kinds
of task launch problems.
It's currently an all-or-nothing setting that affects all tasks in all DAGs.
h3. Improvement
When including a task in a DAG, I have the option to provide executor
configuration specifically for that task:
{code:python}
sometask = DummyOperator(
task_id="example",
executor_config={ "KubernetesExecutor": { "image": "..." } }
)
{code}
This does not currently (v1.10.9) give me the option to override pod deletion
for that specific task. I think this would be an improvement for two reasons:
# you're generally stopping deletion to debug the launch of a specifc task,
making this the tightest possible scope where you'd want to control this
behaviour
# you can control the setting without restarting Airflow, by reloading the DAG
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
