tommyhutcheson commented on issue #34644:
URL: https://github.com/apache/airflow/issues/34644#issuecomment-1761957965
Hi @hussein-awala
I have tried quite a few different configurations at this point but there
just seems to be an issue here.
when running the below dag the only task that completes is the
deferrable-false task, the other two look to be running the code and output
hello-world with deferrable set and I see the dag status change to purple
however the runs are failing still with the error below, I have checked the
kube-config file and I can see there is a key contexts. I have re-opened by
Google support case to asking their product team to test the dag themselves and
install composer-2.4.3-airflow-2.5.3. If there is another suggests please let
me know.
### ERROR
```
[2023-10-13, 14:38:27 UTC] {standard_task_runner.py:100} ERROR - Failed to
execute job 48004 for task deferrable-true-extended-conf (Invalid kube-config
file. Expected key contexts in kube-config;
743598)
```
### TESTING DAG
```
from airflow.providers.cncf.kubernetes.operators.pod import
KubernetesPodOperator
import airflow
from airflow import DAG
from datetime import timedelta
default_args = {
'start_date': airflow.utils.dates.days_ago(0),
'retries': 1,
'retry_delay': timedelta(minutes=5)
}
with DAG(
'tommy_test_kub_simple_dag',
default_args=default_args,
description='liveness monitoring dag',
schedule_interval='*/10 * * * *',
max_active_runs=2,
catchup=False,
dagrun_timeout=timedelta(minutes=10),
) as dag:
task1 = KubernetesPodOperator(
name="deferrable-true",
image="python:3.11-slim",
cmds=['python', '-c', "print('hello world')"],
task_id="deferrable-true",
config_file="/home/airflow/composer_kube_config",
deferrable=True,
in_cluster=False
)
task2 = KubernetesPodOperator(
name="deferrable-false",
image="python:3.11-slim",
cmds=['python', '-c', "print('hello world')"],
task_id="deferrable-false",
config_file="/home/airflow/composer_kube_config",
deferrable=False,
in_cluster=False
)
task3 = KubernetesPodOperator(
name="deferrable-true-extended-conf",
image="python:3.11-slim",
cmds=['python', '-c', "print('hello world')"],
task_id="deferrable-true-extended-conf",
kubernetes_conn_id="kubernetes_default",
deferrable=True,
in_cluster=False,
cluster_context="gke_my_orchestrater_id",
config_file="/home/airflow/composer_kube_config",
)
task1
task2
task3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
