amoghrajesh commented on code in PR #46926:
URL: https://github.com/apache/airflow/pull/46926#discussion_r1964976888
##########
kubernetes_tests/test_kubernetes_executor.py:
##########
@@ -75,15 +69,6 @@ def test_integration_run_dag_with_scheduler_failure(self):
timeout=300,
)
- self.monitor_task(
- host=self.host,
- dag_run_id=dag_run_id,
- dag_id=dag_id,
- task_id="other_namespace_task",
- expected_final_state="success",
- timeout=300,
- )
Review Comment:
Residue of the above explanation
##########
airflow/example_dags/example_kubernetes_executor.py:
##########
@@ -149,18 +149,6 @@ def non_root_task():
print_stuff()
third_task = non_root_task()
-
- executor_config_other_ns = {
- "pod_override": k8s.V1Pod(
- metadata=k8s.V1ObjectMeta(namespace="test-namespace",
labels={"release": "stable"})
- )
- }
-
- @task(executor_config=executor_config_other_ns)
- def other_namespace_task():
- print_stuff()
Review Comment:
This runs workers in another namespace: `test-namespace` which runs the task
with this as entrypoint:
https://github.com/apache/airflow/blob/main/task_sdk/src/airflow/sdk/execution_time/execute_workload.py#L56-L69,
observe that the "server" is
"http://airflow-api-server.airflow.svc.cluster.local:9091/execution/"; which
points to the airflow api server in the "airflow" namespace.
This airflow api server has network policy configured too, where `from: []`
and if from: [] remains empty, then no pod inside or outside the namespace can
contact the API server pod.
There is no real value in changing default in helm to fix this since we
already test couple of different examples with different executor configs.
Error example:
```
2025-02-20T12:23:22.156384794Z stdout F [2m2025-02-20 12:23:22.153945[0m
[[32m[1minfo [0m] [1mExecuting workload in Kubernetes[0m
[[0m[1m[34m__main__[0m][0m [36mworkload[0m=[35mExecuteTask(token='',
ti=TaskInstance(id=UUID('0195234f-a9f9-73fe-8c4a-17e7c0f49df2'),
task_id='other_namespace_task', dag_id='example_kubernetes_executor',
run_id='manual__2025-02-20T12:22:58.278654+00:00', try_number=1, map_index=-1,
pool_slots=1, queue='default', priority_weight=4, executor_config=None),
dag_rel_path=PurePosixPath('example_kubernetes_executor.py'),
bundle_info=BundleInfo(name='dags-folder', version=None),
log_path='dag_id=example_kubernetes_executor/run_id=manual__2025-02-20T12:22:58.278654+00:00/task_id=other_namespace_task/attempt=1.log',
kind='ExecuteTask')[0m
2025-02-20T12:23:22.260257326Z stdout F [2m2025-02-20 12:23:22.259874[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 1st time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:23:23.267525996Z stdout F [2m2025-02-20 12:23:23.267326[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 2nd time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:23:25.041348243Z stdout F [2m2025-02-20 12:23:25.041120[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 3rd time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:23:26.350582937Z stdout F [2m2025-02-20 12:23:26.350390[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 4th time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:23:31.195242541Z stdout F [2m2025-02-20 12:23:31.195069[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 5th time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:23:33.618349782Z stdout F [2m2025-02-20 12:23:33.616686[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 6th time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:23:59.289110163Z stdout F [2m2025-02-20 12:23:59.288950[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 7th time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:24:52.427356184Z stdout F [2m2025-02-20 12:24:52.427177[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 8th time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:24:59.489998585Z stdout F [2m2025-02-20 12:24:59.489770[0m
[[33m[1mwarning [0m] [1mStarting call to
'airflow.sdk.api.client.Client.request', this is the 9th time calling it.[0m
[[0m[1m[34mairflow.sdk.api.client[0m][0m
2025-02-20T12:25:44.392850125Z stdout F [2m2025-02-20 12:25:44.392633[0m
[[36m[1mdebug [0m] [1mWorkload process exited [0m
[[0m[1m[34msupervisor[0m][0m [36mexit_code[0m=[35m<Negsignal.SIGKILL:
-9>[0m
2025-02-20T12:25:44.392963234Z stdout F [2m2025-02-20 12:25:44.392819[0m
[[32m[1minfo [0m] [1mProcess exited [0m
[[0m[1m[34msupervisor[0m][0m [36mexit_code[0m=[35m<Negsignal.SIGKILL:
-9>[0m [36mpid[0m=[35m15[0m [36msignal[0m=[35mSIGKILL[0m
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
