yovio-rca opened a new issue, #58736:
URL: https://github.com/apache/airflow/issues/58736
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.17.0
apache-airflow-providers-apprise==2.2.0
apache-airflow-providers-celery==3.13.1
**apache-airflow-providers-cncf-kubernetes==10.10.0**
apache-airflow-providers-common-compat==1.9.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-docker==4.4.5
apache-airflow-providers-fab==1.5.3
apache-airflow-providers-ftp==3.13.0
apache-airflow-providers-github==2.9.4
apache-airflow-providers-grpc==3.8.3
apache-airflow-providers-http==5.3.0
apache-airflow-providers-imap==3.9.0
apache-airflow-providers-microsoft-winrm==3.12.0
apache-airflow-providers-mongo==5.2.3
apache-airflow-providers-mysql==6.3.5
apache-airflow-providers-opsgenie==5.9.3
apache-airflow-providers-postgres==6.4.1
apache-airflow-providers-redis==4.3.3
apache-airflow-providers-samba==4.11.1
apache-airflow-providers-sftp==5.4.2
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-sqlite==4.1.0
apache-airflow-providers-ssh==4.1.6
apache-airflow-providers-standard==1.9.2
### Apache Airflow version
2.11.0
### Operating System
Debian GNU/Linux 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
Due to New feature in version 10.10.0: KubernetesPodTriggerer reads pod logs
instead of KubernetesPodOperator (#57531)
Now airflow triggerer will read logs from airflow, but I suspect part of log
read also listing Kubernetes events.
Meanwhile airflow-airflow-pod-log-reader-role created by helm chart does
have list events permission.
Earlier version works well because log read was handled by
KubernetesPodOperator which run in worker that uses
airflow-airflow-pod-launcher-role and has list events permission.
### What you think should happen instead
We should fix helm chart template for airflow-airflow-pod-log-reader-role
### How to reproduce
Create a dag with a task using KubernetesPodOperator, make it
defferable=true.
We will get error like:
kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <CIMultiDictProxy('Audit-Id':
'63f99090-5c09-45e5-9716-76cb8cb9e006', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff',
'X-Kubernetes-Pf-Flowschema-Uid': '92570511-6de3-4f0c-b903-ef06f04261e2',
'X-Kubernetes-Pf-Prioritylevel-Uid': '271a661f-e33f-4d2d-8831-5979edf49286',
'Date': 'Wed, 26 Nov 2025 16:40:58 GMT', 'Content-Length': '299')>
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"events
is forbidden: User \"system:serviceaccount:airflow:airflow-triggerer\" cannot
list resource \"events\" in API group \"\" in the namespace
\"ra-batch\"","reason":"Forbidden","details":{"kind":"events"},"code":403}
### Anything else
Full stack of the error:
[2025-11-26, 10:42:08 CST] {taskinstance.py:3336} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py",
line 776, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py",
line 742, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py",
line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py",
line 1816, in resume_execution
return execute_callable(context)
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 933, in trigger_reentry
raise AirflowException(message)
airflow.exceptions.AirflowException: Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py",
line 158, in run
state = await self._wait_for_pod_start()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py",
line 251, in _wait_for_pod_start
await asyncio.gather(pod_start_task, events_task)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 1013, in watch_pod_events
await watch_pod_events(pod_manager=self, pod=pod,
check_interval=check_interval)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 122, in watch_pod_events
events = await pod_manager.read_pod_events(pod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py",
line 189, in async_wrapped
return await copy(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py",
line 111, in __call__
do = await self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py",
line 153, in iter
result = await action(retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/_utils.py", line
99, in inner
return call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
420, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
187, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in
result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py",
line 114, in __call__
result = await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 1006, in read_pod_events
return await self._hook.get_pod_events(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py",
line 950, in get_pod_events
events: CoreV1EventList = await v1_api.list_namespaced_event(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/api_client.py",
line 192, in __call_api
raise e
File
"/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/api_client.py",
line 185, in __call_api
response_data = await self.request(
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/rest.py",
line 212, in GET
return (await self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/rest.py",
line 206, in request
raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <CIMultiDictProxy('Audit-Id':
'63f99090-5c09-45e5-9716-76cb8cb9e006', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff',
'X-Kubernetes-Pf-Flowschema-Uid': '92570511-6de3-4f0c-b903-ef06f04261e2',
'X-Kubernetes-Pf-Prioritylevel-Uid': '271a661f-e33f-4d2d-8831-5979edf49286',
'Date': 'Wed, 26 Nov 2025 16:40:58 GMT', 'Content-Length': '299')>
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"events
is forbidden: User \"system:serviceaccount:airflow:airflow-triggerer\" cannot
list resource \"events\" in API group \"\" in the namespace
\"ra-batch\"","reason":"Forbidden","details":{"kind":"events"},"code":403}
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
