eduardchai commented on issue #18041:
URL: https://github.com/apache/airflow/issues/18041#issuecomment-1014385675
We started having this issue after we upgraded to v2.2.3. We did not
experience this issue when we were at v2.0.2.
Here is the sample dag that we used:
```
from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
with DAG(
dag_id="stress-test-kubepodoperator",
schedule_interval=None,
catchup=False,
start_date=datetime(2021, 1, 1),
) as dag:
for i in range(1000):
KubernetesPodOperator(
name="airflow-test-pod",
namespace="airflow-official",
image="ubuntu:latest",
cmds=["bash", "-cx"],
arguments=["r=$(( ( RANDOM % 120 ) + 1 )); sleep ${r}s"],
labels={"foo": "bar"},
task_id="task_" + str(i),
is_delete_operator_pod=True,
startup_timeout_seconds=300,
service_account_name="airflow-worker",
get_logs=True,
resources={"request_memory": "128Mi", "request_cpu": "100m"},
queue="kubernetes",
)
```
Error message:
```
[2022-01-17, 18:41:48 +08] {local_task_job.py:212} WARNING - State of this
instance has been externally set to scheduled. Terminating instance.
[2022-01-17, 18:41:48 +08] {process_utils.py:124} INFO - Sending
Signals.SIGTERM to group 16. PIDs of all processes in the group: [16]
[2022-01-17, 18:41:48 +08] {process_utils.py:75} INFO - Sending the signal
Signals.SIGTERM to group 16
[2022-01-17, 18:41:48 +08] {taskinstance.py:1408} ERROR - Received SIGTERM.
Terminating subprocesses.
[2022-01-17, 18:41:48 +08] {taskinstance.py:1700} ERROR - Task failed with
exception
```
Environment information:
- Installed with official helm charts
- No. of scheduler replicas: 3 (with 3 CPU and 6Gi memory) request each
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
