ziliangpeng commented on issue #13129:
URL: https://github.com/apache/airflow/issues/13129#issuecomment-789324654
I got a similar issue where it tries to re-attach to a pod while it's
initializing..
```
[2021-03-02 21:22:00,796] {taskinstance.py:1455} ERROR - (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'23926992-dc19-4a0c-8d4b-8f17cdc91bd2', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'Date': 'Tue, 02 Mar 2021 21:22:00 GMT',
'Content-Length': '280'})
HTTP response body:
b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container
\\"base\\" in pod
\\"afj-prem-range-image-test-job-utils-20210302102909-dump-da7wf1s.0cf16bbe75a0401786514f8629164efe\\"
is waiting to start: ContainerCreating","reason":"BadRequest","code":400}\n'
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line
1112, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File
"/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line
1285, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File
"/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line
1315, in _execute_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 335, in execute
labels, try_numbers_match, launcher, pod_list.items[0]
File
"/usr/local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 374, in handle_pod_overlap
final_state, result = self.monitor_launched_pod(launcher, pod)
File
"/usr/local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 513, in monitor_launched_pod
(final_state, result) = launcher.monitor_pod(pod, get_logs=self.get_logs)
File
"/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py",
line 132, in monitor_pod
logs = self.read_pod_logs(pod, timestamps=True,
since_seconds=read_logs_since_sec)
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line
329, in wrapped_f
return self.call(f, *args, **kw)
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line
409, in call
do = self.iter(retry_state=retry_state)
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line
368, in iter
raise retry_exc.reraise()
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line
186, in reraise
raise self.last_attempt.result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in
result
return self.__get_result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in
__get_result
raise self._exception
File "/usr/local/lib/python3.7/site-packages/tenacity/__init__.py", line
412, in call
result = fn(*args, **kwargs)
File
"/usr/local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py",
line 222, in read_pod_logs
**additional_kwargs,
File
"/usr/local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py",
line 19199, in read_namespaced_pod_log
(data) = self.read_namespaced_pod_log_with_http_info(name, namespace,
**kwargs) # noqa: E501
File
"/usr/local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py",
line 19305, in read_namespaced_pod_log_with_http_info
collection_formats=collection_formats)
File
"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line
345, in call_api
_preload_content, _request_timeout)
File
"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line
176, in __call_api
_request_timeout=_request_timeout)
File
"/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line
366, in request
headers=headers)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py",
line 241, in GET
query_params=query_params)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py",
line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'23926992-dc19-4a0c-8d4b-8f17cdc91bd2', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'Date': 'Tue, 02 Mar 2021 21:22:00 GMT',
'Content-Length': '280'})
HTTP response body:
b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container
\\"base\\" in pod
\\"afj-prem-range-image-test-job-utils-20210302102909-dump-da7wf1s.0cf16bbe75a0401786514f8629164efe\\"
is waiting to start: ContainerCreating","reason":"BadRequest","code":400}\n'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
