yunduz opened a new issue #11603:
URL: https://github.com/apache/airflow/issues/11603
**Apache Airflow version**: 1.10.12
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`): Openshift 3.11 (Kubernetes 1.11)
**Environment**:
- **Cloud provider or hardware configuration**: Openshift 3.11
- **OS** (e.g. from /etc/os-release): Red Hat 7.7
**What happened**:
Switched from Airflow 1.10.7 to 1.10.12 and one of the tasks fails to
launch, scheduler shows an error and Airflow becomes unusable (ex can't trigger
dag) and has to be restarted. Operator is a PythonOperator, executor is
KubernetesExecutor, `executor_config` is specified which includes "volumes" and
"volume_mounts". If "volumes" and "volume_mounts" are not specified in the
`executor_config`, the issue does not occur.
**What you expected to happen**: Pod for a task is launched, Airflow
continues to work.
**How to reproduce it**: I modified one of the example dags included with
Airflow which reproduces the problem.
```
import os
from airflow.models import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
default_args = {
'owner': 'Airflow',
'start_date': days_ago(2)
}
with DAG(
dag_id='test_yunduz',
default_args=default_args,
schedule_interval=None
) as dag:
def test_volume_mount():
with open('/foo/volume_mount_test.txt', 'w') as foo:
foo.write('Hello')
return_code = os.system("cat /foo/volume_mount_test.txt")
assert return_code == 0
second_task = PythonOperator(
task_id="four_task",
python_callable=test_volume_mount,
executor_config={
"KubernetesExecutor": {
"volumes": [
{
"name": "example-kubernetes-test-volume",
"hostPath": {"path": "/tmp/"},
},
],
"volume_mounts": [
{
"mountPath": "/foo/",
"name": "example-kubernetes-test-volume",
},
]
}
}
)
second_task
```
**Anything else we need to know**:
<details><summary>error in airflow scheduler log</summary>
```
[2020-10-16 19:03:54,972] {scheduler_job.py:1401} ERROR - Exception when
executing execute_helper
Traceback (most recent call last):
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py",
line 105, in _mutate_pod_backcompat
new_pod = PodGenerator.reconcile_pods(pod, dummy_pod)
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
line 410, in reconcile_pods
client_pod_cp.spec = PodGenerator.reconcile_specs(base_pod.spec,
client_pod_cp.spec)
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
line 459, in reconcile_specs
merged_spec = extend_object_field(base_spec, client_spec, 'volumes')
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
line 657, in extend_object_field
appended_fields = _merge_list_of_objects(base_obj_set, client_obj_set)
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
line 668, in _merge_list_of_objects
client_obj_set[k] = merge_objects(v, client_obj_set[k])
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py",
line 621, in merge_objects
for base_key in base_obj.to_dict().keys():
AttributeError: 'dict' object has no attribute 'to_dict'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py",
line 108, in _mutate_pod_backcompat
settings.pod_mutation_hook(pod)
File "/app/airflow/config/airflow_local_settings.py", line 62, in
pod_mutation_hook
log.info(f"Running command: {pod.cmds} ")
AttributeError: 'V1Pod' object has no attribute 'cmds'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py",
line 1399, in _execute
self._execute_helper()
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py",
line 1470, in _execute_helper
if not
self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py",
line 1532, in _validate_and_run_task_instances
self.executor.heartbeat()
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/executors/base_executor.py",
line 134, in heartbeat
self.sync()
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py",
line 831, in sync
self.kube_scheduler.run_next(task)
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py",
line 460, in run_next
self.launcher.run_pod_async(pod,
**self.kube_config.kube_client_request_args)
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py",
line 77, in run_pod_async
pod = self._mutate_pod_backcompat(pod)
File
"/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py",
line 111, in _mutate_pod_backcompat
raise Exception([e, e2])
Exception: [AttributeError("'dict' object has no attribute 'to_dict'",),
AttributeError("'V1Pod' object has no attribute 'cmds'",)]
[2020-10-16 19:03:54,974] {helpers.py:325} INFO - Sending Signals.SIGTERM to
GPID 170
[2020-10-16 19:03:54,976] {helpers.py:291} INFO - Process
psutil.Process(pid=1153, status='terminated', started='19:03:54') (1153)
terminated with exit code None
[2020-10-16 19:03:55,004] {helpers.py:291} INFO - Process
psutil.Process(pid=170, status='terminated', exitcode=0, started='19:02:44')
(170) terminated with exit code 0
[2020-10-16 19:03:55,004] {helpers.py:291} INFO - Process
psutil.Process(pid=1151, status='terminated', started='19:03:54') (1151)
terminated with exit code None
[2020-10-16 19:03:55,005] {scheduler_job.py:1404} INFO - Exited execute loop
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
