applevladko commented on issue #21605:
URL: https://github.com/apache/airflow/issues/21605#issuecomment-1593001867
Seems like we faced similar issue on the later version:
Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==6.1.0
kubernetes==23.6.0
kubernetes-asyncio==24.2.3
Apache Airflow version
2.6.0
Operating System
Debian VERSION="11 (bullseye)"
Seems like after empty string timestamp error (that appears during long
query) log parses stops and fail into 404.
the issue is not permanent and appears sometimes
`[2023-06-14, 15:16:36 UTC] {pod_manager.py:342} INFO - SELECT * FROM
columns_to_select
[2023-06-14, 15:16:36 UTC] {pod_manager.py:342} INFO - -- /* {"app": "dbt",
"dbt_version": "1.5.0", "profile_name": "athena", "target_name": "athena",
"node_id": "model.data_models.stg_event_wallet_balance_changed"} */
[2023-06-14, 15:17:28 UTC] {pod_manager.py:410} ERROR - Error parsing
timestamp (no timestamp in message ''). Will continue execution but won't
update timestamp
[2023-06-14, 15:17:28 UTC] {pod_manager.py:342} INFO -
[2023-06-14, 15:17:31 UTC] {pod.py:905} ERROR - (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'fc6d1076-bb5e-4c1d-ad92-d568c53bfd3e', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid':
'b6307941-9fe6-4061-9796-76ce2ad8cd8b', 'X-Kubernetes-Pf-Prioritylevel-Uid':
'3dbd0f1c-d64d-450a-805b-741194c49f71', 'Date': 'Wed, 14 Jun 2023 15:17:31
GMT', 'Content-Length': '288'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
\"dbt-dm-data-vault--*pod name*-b9k0b0t1\" not
found","reason":"NotFound","details":{"name":"dbt-dm-data-vault--*pod
name*-b9k0b0t1","kind":"pods"},"code":404}
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 557, in execute_sync
self.remote_pod = self.pod_manager.await_pod_completion(self.pod)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 394, in await_pod_completion
remote_pod = self.read_pod(pod)
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
289, in wrapped_f
return self(f, *args, **kw)
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
379, in __call__
do = self.iter(retry_state=retry_state)
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
325, in iter
raise retry_exc.reraise()
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
158, in reraise
raise self.last_attempt.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in
result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
382, in __call__
result = fn(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 490, in read_pod
return self._client.read_namespaced_pod(pod.metadata.name,
pod.metadata.namespace)
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py",
line 23483, in read_namespaced_pod
return self.read_namespaced_pod_with_http_info(name, namespace,
**kwargs) # noqa: E501
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py",
line 23570, in read_namespaced_pod_with_http_info
return self.api_client.call_api(
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 348, in call_api
return self.__call_api(resource_path, method,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 180, in __call_api
response_data = self.request(
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 373, in request
return self.rest_client.GET(url,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/rest.py",
line 240, in GET
return self.request("GET", url,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/rest.py",
line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'caf17afd-96b2-4b53-b9ff-cef1ada8b553', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid':
'b6307941-9fe6-4061-9796-76ce2ad8cd8b', 'X-Kubernetes-Pf-Prioritylevel-Uid':
'3dbd0f1c-d64d-450a-805b-741194c49f71', 'Date': 'Wed, 14 Jun 2023 15:17:31
GMT', 'Content-Length': '288'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
\"dbt-dm-data-vault--*pod name*-b9k0b0t1\" not
found","reason":"NotFound","details":{"name":"dbt-dm-data-vault--*pod
name*-b9k0b0t1","kind":"pods"},"code":404}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 745, in patch_already_checked
self.client.patch_namespaced_pod(
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py",
line 19662, in patch_namespaced_pod
return self.patch_namespaced_pod_with_http_info(name, namespace, body,
**kwargs) # noqa: E501
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py",
line 19777, in patch_namespaced_pod_with_http_info
return self.api_client.call_api(
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 348, in call_api
return self.__call_api(resource_path, method,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 180, in __call_api
response_data = self.request(
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 407, in request
return self.rest_client.PATCH(url,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/rest.py",
line 295, in PATCH
return self.request("PATCH", url,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/rest.py",
line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'fc6d1076-bb5e-4c1d-ad92-d568c53bfd3e', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid':
'b6307941-9fe6-4061-9796-76ce2ad8cd8b', 'X-Kubernetes-Pf-Prioritylevel-Uid':
'3dbd0f1c-d64d-450a-805b-741194c49f71', 'Date': 'Wed, 14 Jun 2023 15:17:31
GMT', 'Content-Length': '288'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
\"dbt-dm-data-vault--*pod name*-b9k0b0t1\" not
found","reason":"NotFound","details":{"name":"dbt-dm-data-vault--*pod
name*-b9k0b0t1","kind":"pods"},"code":404}
[2023-06-14, 15:17:31 UTC] {pod.py:721} INFO - Deleting pod:
dbt-dm-data-vault--*pod name*-b9k0b0t1
[2023-06-14, 15:17:31 UTC] {taskinstance.py:1847} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 557, in execute_sync
self.remote_pod = self.pod_manager.await_pod_completion(self.pod)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 394, in await_pod_completion
remote_pod = self.read_pod(pod)
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
289, in wrapped_f
return self(f, *args, **kw)
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
379, in __call__
do = self.iter(retry_state=retry_state)
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
325, in iter
raise retry_exc.reraise()
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
158, in reraise
raise self.last_attempt.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in
result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.10/site-packages/tenacity/__init__.py", line
382, in __call__
result = fn(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 490, in read_pod
return self._client.read_namespaced_pod(pod.metadata.name,
pod.metadata.namespace)
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py",
line 23483, in read_namespaced_pod
return self.read_namespaced_pod_with_http_info(name, namespace,
**kwargs) # noqa: E501
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py",
line 23570, in read_namespaced_pod_with_http_info
return self.api_client.call_api(
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 348, in call_api
return self.__call_api(resource_path, method,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 180, in __call_api
response_data = self.request(
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/api_client.py",
line 373, in request
return self.rest_client.GET(url,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/rest.py",
line 240, in GET
return self.request("GET", url,
File
"/home/airflow/.local/lib/python3.10/site-packages/kubernetes/client/rest.py",
line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'caf17afd-96b2-4b53-b9ff-cef1ada8b553', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid':
'b6307941-9fe6-4061-9796-76ce2ad8cd8b', 'X-Kubernetes-Pf-Prioritylevel-Uid':
'3dbd0f1c-d64d-450a-805b-741194c49f71', 'Date': 'Wed, 14 Jun 2023 15:17:31
GMT', 'Content-Length': '288'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
\"dbt-dm-data-vault--*pod name*-b9k0b0t1\" not
found","reason":"NotFound","details":{"name":"dbt-dm-data-vault--*pod
name*-b9k0b0t1","kind":"pods"},"code":404}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/airflow/dags/operators/dbt_operator.py", line 181, in execute
dbt.execute(context=context)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 529, in execute
return self.execute_sync(context)
File
"/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 559, in execute_sync
self.cleanup(
File "/opt/airflow/dags/operators/k8s_pod.py", line 30, in cleanup
status = next((x for x in statuses if x.name ==
self.base_container_name), None)
TypeError: 'NoneType' object is not iterable
[2023-06-14, 15:17:31 UTC] {taskinstance.py:1368} INFO - Marking task as
UP_FOR_RETRY. dag_id=dbt_dm_data_vault,
task_id=data_vault.stg_event_wallet_balance_changed.run,
execution_date=20230613T205009, start_date=20230614T151608,
end_date=20230614T151731
[2023-06-14, 15:17:31 UTC] {standard_task_runner.py:104} ERROR - Failed to
execute job 9356814 for task data_vault.stg_event_wallet_balance_changed.run
('NoneType' object is not iterable; 12158)
[2023-06-14, 15:17:31 UTC] {local_task_job_runner.py:232} INFO - Task exited
with return code 1
[2023-06-14, 15:17:31 UTC] {taskinstance.py:2674} INFO - 0 downstream tasks
scheduled from follow-on schedule check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
