vrumjantsev opened a new issue, #32084:
URL: https://github.com/apache/airflow/issues/32084
### Apache Airflow version
2.6.2
### What happened
I'm running airflow 2.6.2 installed in GKE private cluster, deployed via
official helm chart 1.9.0
Aifflow image also includes following dependencies:
```
apache-airflow-providers-google==10.1.1
apache-airflow-providers-cncf-kubernetes==7.0.0
```
Connecting to k8s fails with RecursionError
logs of scheduled job pod:
```
git-sync-init INFO: detected pid 1, running init handler
git-sync-init I0623 06:27:43.737796 12 main.go:389] "level"=0
"msg"="starting up" "pid"=12 "args"=["/git-sync"]
git-sync-init I0623 06:27:43.744509 12 main.go:934] "level"=0
"msg"="cloning repo" "origin"="..." "path"="/git"
git-sync-init I0623 06:27:44.260208 12 main.go:748] "level"=0
"msg"="syncing git" "rev"="HEAD"
"hash"="4a9c81598a57a458d2f99a95265058d6d4d6cd97"
git-sync-init I0623 06:27:45.209088 12 main.go:783] "level"=0
"msg"="adding worktree" "path"="/git/4a9c81598a57a458d2f99a95265058d6d4d6cd97"
"branch"="origin/master"
git-sync-init I0623 06:27:45.212610 12 main.go:844] "level"=0
"msg"="reset worktree to hash"
"path"="/git/4a9c81598a57a458d2f99a95265058d6d4d6cd97"
"hash"="4a9c81598a57a458d2f99a95265058d6d4d6cd97"
git-sync-init I0623 06:27:45.212634 12 main.go:849] "level"=0
"msg"="updating submodules"
Stream closed EOF for analytics/vlad-test-meltano-test-task-id-clv8htbg
(git-sync-init)
stream logs failed container "base" in pod
"vlad-test-meltano-test-task-id-clv8htbg" is waiting to start: PodInitializing
for analytics/vlad-test-meltano-test-task-id-clv8htbg (base)
base
base [2023-06-23T06:27:50.820+0000] {dagbag.py:541} INFO - Filling up the
DagBag from /opt/airflow/dags/repo/dags/test_vlad/test_meltano_DAG.py
base [2023-06-23T06:27:54.093+0000] {task_command.py:410} INFO - Running
<TaskInstance: Vlad_Test_Meltano.test-task-id
manual__2023-06-23T06:27:32.634896+00:00 [queued]> on host
vlad-test-meltano-test-task-id-clv8htbg
base [2023-06-23T06:27:54.132+0000] {taskinstance.py:1103} INFO -
Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance:
Vlad_Test_Meltano.test-task-id manual__2023-06-23T06:27:32.634896+00:00
[queued]>
base [2023-06-23T06:27:54.190+0000] {taskinstance.py:1103} INFO -
Dependencies all met for dep_context=requeueable deps ti=<TaskInstance:
Vlad_Test_Meltano.test-task-id manual__2023-06-23T06:27:32.634896+00:00
[queued]>
base [2023-06-23T06:27:54.191+0000] {taskinstance.py:1308} INFO - Starting
attempt 1 of 1
base [2023-06-23T06:27:54.208+0000] {taskinstance.py:1327} INFO - Executing
<Task(GKEStartPodOperator): test-task-id> on 2023-06-23 06:27:32.634896+00:00
base [2023-06-23T06:27:54.215+0000] {standard_task_runner.py:57} INFO -
Started process 24 to run task
base [2023-06-23T06:27:54.217+0000] {standard_task_runner.py:84} INFO -
Running: ['airflow', 'tasks', 'run', 'Vlad_Test_Meltano', 'test-task-id',
'manual__2023-06-23T06:27:32.634896+00:00', '--job-id', '95567', '--raw',
'--subdir', 'DAGS_FOLDER/test_vlad/test_meltano_DAG.py', '--cfg-path',
'/tmp/tmpkw6flv6m']
base [2023-06-23T06:27:54.218+0000] {standard_task_runner.py:85} INFO - Job
95567: Subtask test-task-id
base [2023-06-23T06:27:54.246+0000] {task_command.py:410} INFO - Running
<TaskInstance: Vlad_Test_Meltano.test-task-id
manual__2023-06-23T06:27:32.634896+00:00 [running]> on host
vlad-test-meltano-test-task-id-clv8htbg
base [2023-06-23T06:27:54.303+0000] {pod_generator.py:529} WARNING - Model
file /opt/airflow/pod_templates/pod_template_file.yaml does not exist
base [2023-06-23T06:27:54.334+0000] {taskinstance.py:1545} INFO - Exporting
env vars: AIRFLOW_CTX_DAG_OWNER='airflow'
AIRFLOW_CTX_DAG_ID='Vlad_Test_Meltano' AIRFLOW_CTX_TASK_ID='test-task-id'
AIRFLOW_CTX_EXECUTION_DATE='2023-06-23T06:27:32.634896+00:00'
AIRFLOW_CTX_TRY_NUMBER='1'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-06-23T06:27:32.634896+00:00'
base [2023-06-23T06:27:54.343+0000] {base.py:73} INFO - Using connection ID
'google_cloud_default' for task execution.
base [2023-06-23T06:27:54.349+0000] {kubernetes_engine.py:288} INFO -
Fetching cluster (project_id=..., location=europe-west2,
cluster_name=prod-cluster-k8s-1)
base object address : 0x7f20800bf4c0
base object refcount : 4
base object type : 0x7f208b625160
base object type name: RecursionError
base object repr :
base lost sys.stderr
Stream closed EOF for analytics/vlad-test-meltano-test-task-id-clv8htbg
(base)
```
### What you think should happen instead
_No response_
### How to reproduce
1) install airflow to GKE private cluster using helm
2) run following DAG
```
from airflow.kubernetes.secret import Secret
from airflow.providers.google.cloud.operators.kubernetes_engine import (
GKEStartPodOperator,
)
from kubernetes.client import V1ResourceRequirements
from kubernetes.client.models import (
V1LocalObjectReference,
V1Affinity,
V1NodeAffinity,
V1PreferredSchedulingTerm,
V1NodeSelectorTerm,
V1NodeSelectorRequirement
)
import datetime
from airflow.models import DAG
with DAG(
dag_id='Vlad_Test_Meltano',
start_date=datetime.datetime(2023, 6, 22),
default_args={
'owner': 'airflow',
},
description='test meltano pipeline',
tags=['test', ],
schedule_interval='0 12 20 * *',
catchup=False
) as dag:
pod_start_operator = GKEStartPodOperator(
task_id="test-task-id",
name="test-task-name",
project_id='...',
location='europe-west2',
cluster_name='prod-cluster-k8s-1',
is_delete_operator_pod=True,
get_logs=True,
in_cluster=True,
use_internal_ip=True,
reattach_on_restart=True,
namespace="analytics",
image="...",
image_pull_secrets=[V1LocalObjectReference('gitlab-docker-registry')],
cmds=["meltano"],
arguments=["run tap-mysql target-bigquery"],
secrets=[
Secret(
...
),
Secret(
...
)
],
startup_timeout_seconds=60,
image_pull_policy="IfNotPresent",
container_resources=V1ResourceRequirements(
requests={
"memory": "4Gi",
},
limits={
"memory": "8Gi",
},
),
do_xcom_push=True,
)
pod_start_operator
```
### Operating System
Container-Optimized OS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
