snjypl commented on PR #28394:
URL: https://github.com/apache/airflow/pull/28394#issuecomment-1386891079
> @snjypl That solved the previous issue, and the execution is attempted
now. However, manual runs still fail because the service account gets rejected
from creating a pod for the execution:
>
> ```
> [2023-01-18T07:20:26.236+0000] {kubernetes_executor.py:527} INFO - Start
Kubernetes executor
> [2023-01-18T07:20:26.261+0000] {kubernetes_executor.py:130} INFO - Event:
and now my watch begins starting at resource_version: 0
> [2023-01-18T07:20:26.395+0000] {kubernetes_executor.py:476} INFO - Found 0
queued task instances
> [2023-01-18T07:20:26.438+0000] {base_executor.py:95} INFO - Adding to
queue: ['airflow', 'tasks', 'run', 'x-y-z', 'X-Y',
'scheduled__2023-01-18T07:00:00+00:00', '--ignore
> -all-dependencies', '--ignore-dependencies', '--local', '--pool',
'default_pool', '--subdir', 'DAGS_FOLDER/airflow-dags-sap/X-Y/x-y-z.py']
> [2023-01-18T07:20:26.438+0000] {base_executor.py:215} INFO - task
TaskInstanceKey(dag_id='x-y-z', task_id='x-y-z',
run_id='scheduled__2023-01-18T07:00:00+00:00', try_number
> =3, map_index=-1) is still running
> [2023-01-18T07:20:26.508+0000] {kubernetes_executor.py:339} INFO -
Creating kubernetes pod for job is TaskInstanceKey(dag_id='x-y-z',
task_id='X-Y', run_id='scheduled__20
> 23-01-18T07:00:00+00:00', try_number=3, map_index=-1), with pod name
x-y-z-78e3092210f94420bb0e98a744969f29
> [2023-01-18T07:20:26.538+0000] {kubernetes_executor.py:274} ERROR -
Exception when attempting to create Namespaced Pod: {
> .
> .
> .
> kubernetes.client.exceptions.ApiException: (403)
> Reason: Forbidden
> HTTP response headers: HTTPHeaderDict({'Audit-Id':
'ddc99dcf-9d70-4f88-8c7c-77f543879844', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-
> Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid':
'e7834783-2050-421a-b99e-0615f85f6e92', 'X-Kubernetes-Pf-Prioritylevel-Uid':
'e9e2e589-5d4c-442b-8568-f7bfbdbfaafd', 'Date': '
> Wed, 18 Jan 2023 07:20:26 GMT', 'Content-Length': '315'})
> HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"system:serviceaccount:airflow-test-ns:airflow-test-webse
> rver\" cannot create resource \"pods\" in API group \"\" in the namespace
\"airflow-test-ns\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
> ```
>
> Which is strange, because this SA is allowed to create pods, and other
tasks are being executed by the same service account. I trimmed out the pod
definition, but nothing looks particularly out of place there. Any idea why
this fails only on a manual run attempt?
this issue is caused by `airflow-webserver` not haveing pod-launcher-role,
you can try the fix in this PR: https://github.com/apache/airflow/pull/29012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
