dmndru opened a new issue, #37795:
URL: https://github.com/apache/airflow/issues/37795
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes 8.0.0
apache-airflow-providers-google 10.15.0
### Apache Airflow version
2.7.3
### Operating System
Debian 11
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
GKE cluster version 1.26.13
### What happened
We are using the GKEStartPodOperator to run a pod in our GKE clusters and
getting the error:
> kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'677bd23e-3885-4057-84cb-cbcfd8bcb4d2', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff',
'X-Kubernetes-Pf-Flowschema-Uid': '43677b2c-e54a-4a50-ae9f-79d579f5d98c',
'X-Kubernetes-Pf-Prioritylevel-Uid': '9ac4bb3e-9b09-4aca-912b-6b01d3f002b1',
'Date': 'Tue, 27 Feb 2024 10:48:33 GMT', 'Content-Length': '337'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"7843619910037672257843\" cannot create resource \"pods\"
in API group \"\" in the namespace \"staging\": requires one of
[\"container.pods.create\"]
permission(s).","reason":"Forbidden","details":{"kind":"pods"},"code":403}
<details><summary>trace</summary>
[2024-02-27, 10:48:33 UTC] {pod_manager.py:329} ERROR - Exception when
attempting to create Namespaced Pod: { "apiVersion": "v1", "kind": "Pod",
"metadata": { "annotations": {}, "labels": { "tier": "staging",
"dag_id": "batch", "task_id": "start", "run_id":
"manual__2024-02-27T104509.7089670000-57e30e5c0",
"kubernetes_pod_operator": "True", "try_number": "1",
"airflow_version": "2.7.3", "airflow_kpo_in_cluster": "False" },
"name": "batch-9ef11196", "namespace": "staging" }, "spec": {
"affinity": {}, "containers": [ { "args": [
"--name", "batch", "--batch_pg_run_id",
"manual__2024-02-27T10:45:09.708967+00:00", "--batch_pg",
"input/", "--batch_pg_use_proxy", "True",
"--batch_pg_dag_execution_date", "2024-02-27",
"--batch_pg_always_use_selenium", "False", "--batch_pg_s
elenium_on_scrapy_error", "False",
"--batch_pg_batch_size", "1000", "--batch_pg",
"False" ], "command": [], "env": [], "envFrom":
[], "image": "staging_latest", "imagePullPolicy": "Always",
"name": "base", "ports": [], "terminationMessagePolicy":
"File", "volumeMounts": [] } ], "hostNetwork": false,
"initContainers": [], "restartPolicy": "Never", "securityContext": {},
"volumes": [] } }
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 324, in run_pod_async
resp = self._client.create_namespaced_pod(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py",
line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py",
line 7455, in create_namespaced_pod_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 391, in request
return self.rest_client.POST(url,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py",
line 279, in POST
return self.request("POST", url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py",
line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'677bd23e-3885-4057-84cb-cbcfd8bcb4d2', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff',
'X-Kubernetes-Pf-Flowschema-Uid': '43677b2c-e54a-4a50-ae9f-79d579f5d98c',
'X-Kubernetes-Pf-Prioritylevel-Uid': '9ac4bb3e-9b09-4aca-912b-6b01d3f002b1',
'Date': 'Tue, 27 Feb 2024 10:48:33 GMT', 'Content-Length': '337'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"7843619910037672257843\" cannot create resource \"pods\"
in API group \"\" in the namespace \"staging\": requires one of
[\"container.pods.create\"]
permission(s).","reason":"Forbidden","details":{"kind":"pods"},"code":403}
[2024-02-27, 10:48:33 UTC] {pod.py:1109} ERROR - 'NoneType' object has no
attribute 'metadata'
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 578, in execute_sync
self.pod = self.get_or_create_pod( # must set `self.pod` for `on_kill`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 538, in get_or_create_pod
self.pod_manager.create_pod(pod=pod_request_obj)
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
289, in wrapped_f
return self(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
379, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
314, in iter
return fut.result()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in
result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
382, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 354, in create_pod
return self.run_pod_async(pod)
^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 332, in run_pod_async
raise e
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 324, in run_pod_async
resp = self._client.create_namespaced_pod(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py",
line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py",
line 7455, in create_namespaced_pod_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 391, in request
return self.rest_client.POST(url,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py",
line 279, in POST
return self.request("POST", url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py",
line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'677bd23e-3885-4057-84cb-cbcfd8bcb4d2', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff',
'X-Kubernetes-Pf-Flowschema-Uid': '43677b2c-e54a-4a50-ae9f-79d579f5d98c',
'X-Kubernetes-Pf-Prioritylevel-Uid': '9ac4bb3e-9b09-4aca-912b-6b01d3f002b1',
'Date': 'Tue, 27 Feb 2024 10:48:33 GMT', 'Content-Length': '337'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"7843619910037672257843\" cannot create resource \"pods\"
in API group \"\" in the namespace \"staging\": requires one of
[\"container.pods.create\"]
permission(s).","reason":"Forbidden","details":{"kind":"pods"},"code":403}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 937, in patch_already_checked
name=pod.metadata.name,
^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'metadata'
[2024-02-27, 10:48:33 UTC] {taskinstance.py:1937} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 578, in execute_sync
self.pod = self.get_or_create_pod( # must set `self.pod` for `on_kill`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 538, in get_or_create_pod
self.pod_manager.create_pod(pod=pod_request_obj)
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
289, in wrapped_f
return self(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
379, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
314, in iter
return fut.result()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in
result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
382, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 354, in create_pod
return self.run_pod_async(pod)
^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 332, in run_pod_async
raise e
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 324, in run_pod_async
resp = self._client.create_namespaced_pod(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py",
line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py",
line 7455, in create_namespaced_pod_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py",
line 391, in request
return self.rest_client.POST(url,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py",
line 279, in POST
return self.request("POST", url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py",
line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'677bd23e-3885-4057-84cb-cbcfd8bcb4d2', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff',
'X-Kubernetes-Pf-Flowschema-Uid': '43677b2c-e54a-4a50-ae9f-79d579f5d98c',
'X-Kubernetes-Pf-Prioritylevel-Uid': '9ac4bb3e-9b09-4aca-912b-6b01d3f002b1',
'Date': 'Tue, 27 Feb 2024 10:48:33 GMT', 'Content-Length': '337'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"7843619910037672257843\" cannot create resource \"pods\"
in API group \"\" in the namespace \"staging\": requires one of
[\"container.pods.create\"]
permission(s).","reason":"Forbidden","details":{"kind":"pods"},"code":403}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/google/cloud/operators/kubernetes_engine.py",
line 548, in execute
return super().execute(context)
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 570, in execute
return self.execute_sync(context)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 629, in execute_sync
self.cleanup(
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 839, in cleanup
raise AirflowException(
airflow.exceptions.AirflowException: Pod batch-pagegrabber-crawler-9ef11196
returned a failure.
remote_pod: None
[2024-02-27, 10:48:33 UTC] {taskinstance.py:1400} INFO - Marking task as
FAILED. dag_id=batch, task_id=start, execution_date=20240227T104509,
start_date=20240227T104831, end_date=20240227T104833
[2024-02-27, 10:48:33 UTC] {standard_task_runner.py:104} ERROR - Failed to
execute job 19 for task start (Pod batch-9ef11196 returned a failure.
remote_pod: None; 24)
</details>
### What you think should happen instead
_No response_
### How to reproduce
1. create a GKE cluster
2. install Airflow 2.7.3 into the cluster
3. create a namespace in the cluster
4. create a GCP service account and grant the _Kubernetes Engine Viewer_
role to the service account
5. grant permission to create pods in the namespace to the service account
by creating role and rolebinding
6. run a pod on the cluster using the GKEStartPodOperator:
```
pod_batch = GKEStartPodOperator(
task_id="start_batch",
gcp_conn_id="my_gcp_conn",
name="batch",
image="myimage",
image_pull_policy="Always",
startup_timeout_seconds=1800,
is_delete_operator_pod=True,
project_id="cluster1_project_id",
cluster_name="cluster1",
location=GCP_LOCATION,
namespace="staging",
labels={"tier": "staging"},
)
```
### Anything else
The error could be fixed by granting the _Kubernetes Engine Developer_ to
the service account, but it is cluster-wide, and we need to grant permissions
to a single namespace.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
