scauglog opened a new issue #12760:
URL: https://github.com/apache/airflow/issues/12760
**Apache Airflow version**: 1.10.12 and 2.0.0b3
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`): 1.19.2
**Environment**:
- **Cloud provider or hardware configuration**: minikube
- **OS** (e.g. from /etc/os-release): Ubuntu 20.04
- **Kernel** (e.g. `uname -a`): 5.4.0-53
**What happened**:
With the kubernetePodOperator and KubernetesExecutor when I try to mount
secret as volume the pod (the one in kubernetesPodOperator) didn't launch and
the task return as failed. The error occur with either V1Volume object or
airflow Secret object.
Persistent volume claim work perfectly.
**What you expected to happen**:
airflow launch a worker pod on kubernetes and the worker pod launch a pod on
kubernetes
**How to reproduce it**:
```python
import airflow
from airflow import DAG
from kubernetes.client import models as k8s
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import
KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.kubernetes.secret import Secret
default_args = {
'owner': 'debug',
'depends_on_past': False,
'start_date': airflow.utils.dates.days_ago(1),
}
pvc_volume = k8s.V1Volume(
name='ml-data',
persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='ml-data')
)
secret_volume = k8s.V1Volume(
name='deploy-key',
secret=k8s.V1SecretVolumeSource(default_mode=600,
secret_name="dvc-deploy-key")
)
pvc_volume_mount = k8s.V1VolumeMount(
name='ml-data', mount_path='/data/', sub_path=None, read_only=False
)
secret_volume_mount = k8s.V1VolumeMount(
name='deploy-key', mount_path='/root/.ssh', sub_path=None, read_only=True
)
secret_file = Secret(deploy_type='volume',
deploy_target='/root/.ssh/',
secret='dvc-deploy-key')
dag = DAG(
"testing",
default_args=default_args,
description='indexation train pipeline',
schedule_interval=None
)
start = DummyOperator(
task_id="start",
dag=dag
)
t1 = KubernetesPodOperator(
task_id=f't1',
name=f't1',
namespace='airflow',
image="busybox",
cmds=["echo", "main"],
volume_mounts=[pvc_volume_mount],
volumes=[pvc_volume],
is_delete_operator_pod=False,
get_logs=True,
dag=dag,
)
t10 = KubernetesPodOperator(
task_id=f't10',
name=f't10',
namespace='airflow',
image="busybox",
cmds=["echo", "main"],
secrets=secret_file,
volume_mounts=[pvc_volume_mount],
volumes=[pvc_volume],
is_delete_operator_pod=False,
get_logs=True,
dag=dag,
)
t2 = KubernetesPodOperator(
task_id=f't2',
name=f't2',
namespace='airflow',
image="busybox",
cmds=["echo", "main"],
volume_mounts=[secret_volume_mount],
volumes=[secret_volume],
is_delete_operator_pod=False,
get_logs=True,
dag=dag,
)
t3 = KubernetesPodOperator(
task_id=f't3',
name=f't3',
namespace='airflow',
image="busybox",
cmds=["echo", "main"],
volume_mounts=[pvc_volume_mount, secret_volume_mount],
volumes=[pvc_volume, secret_volume],
is_delete_operator_pod=False,
get_logs=True,
dag=dag,
)
start >> [t1, t10, t2, t3]
```
**Anything else we need to know**:
I'm not sure if it's an airflow issue or a kubernetes-client issue
How often does this problem occur? Once? Every time etc?
This problem occur every time
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>worker.log</summary> airflow@testingpulldata:/opt/airflow$
airflow tasks run testing pull_data "2020-12-02T15:12:29.757041+00:00" --local
--pool default_pool --subdir /opt/airflow/dags/test.py
[2020-12-02 15:30:19,007] {dagbag.py:440} INFO - Filling up the DagBag from
/opt/airflow/dags/test.py
/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26
DeprecationWarning: This module is deprecated. Please use `kub
ernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27
DeprecationWarning: This module is deprecated. Please use `kub
ernetes.client.models.V1VolumeMount`.
Running <TaskInstance: testing.pull_data 2020-12-02T15:12:29.757041+00:00
[success]> on host testingpulldata
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
