artur-v-zaripov opened a new pull request #19157:
URL: https://github.com/apache/airflow/pull/19157
**ISSUE**
During the upgrade from Airflow 1.10.15 to Airflow 2.1.4 we've faced the
following error:
```python
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line
40, in main
args.func(args)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py",
line 48, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line
92, in wrapper
return f(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py",
line 292, in task_run
_run_task_by_selected_method(args, dag, ti)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py",
line 105, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py",
line 163, in _run_task_by_local_task_job
run_job.run()
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py",
line 245, in run
self._execute()
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/local_task_job.py",
line 134, in _execute
self.heartbeat()
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py",
line 226, in heartbeat
self.heartbeat_callback(session=session)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py",
line 67, in wrapper
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/local_task_job.py",
line 209, in heartbeat_callback
raise AirflowException("PID of job runner does not match")
airflow.exceptions.AirflowException: PID of job runner does not match
```
Also, we tried to reproduce it on clean install using Helm charts from the
repository and Airflow version 2.2 - the same issue occurs.
**SAMPLE DAG**
We are using `KubernetesExecutor`, DAGs with `KubernetesPodOperator`, and
`run_as_user` setting.
To reproduce the bug, we've created a sample dag:
```python
from airflow import DAG
from datetime import datetime, timedelta
from airflow.providers.cncf.kubernetes.operators import kubernetes_pod
from airflow.operators.dummy_operator import DummyOperator
from kubernetes.client import models as k8s
YESTERDAY = datetime.now() - timedelta(days=1)
default_args = {
'depends_on_past': False,
'retries': 0,
'start_date': datetime(2019, 1, 1),
'namespace': "airflow-new",
'image_pull_policy': "Always",
'is_delete_operator_pod': False,
'in_cluster': True,
'startup_timeout_seconds': 300,
'hostnetwork': True,
'run_as_user': "airflow"
}
with DAG(
dag_id='chart_test_dag',
schedule_interval="30 0 * * *",
max_active_runs=1,
catchup=False,
start_date=YESTERDAY,
default_args=default_args,
) as dag:
start = DummyOperator(task_id="start", dag=dag)
pro = kubernetes_pod.KubernetesPodOperator(
task_id="pro",
dag=dag,
name="pro",
cmds=["bash", "-cx"],
arguments=["pwd"],
env_vars=[k8s.V1EnvVar(name="EMR_CONFIG_ID", value="DEV")],
image="ubuntu:latest",
namespace="airflow-new",
)
start >> pro
```
**ENVIRONMENT**
Python 3.8
Kubernetes 1.19
**RELATED TO**
Probably, related issues:
https://github.com/apache/airflow/issues/17394
https://github.com/apache/airflow/issues/18041
**PR**
This PR contains the fix that helped to avoid this issue. Any other
solutions are much appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
