raphaelauv opened a new issue #19369:
URL: https://github.com/apache/airflow/issues/19369
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==2.0.2
### Apache Airflow version
2.1.2
### Operating System
GCP Container-Optimized OS
### Deployment
Composer
### Deployment details
_No response_
### What happened
```
[2021-11-02 14:16:27,086] {pod_launcher.py:193} ERROR - Error parsing
timestamp. Will continue execution but won't update timestamp
[2021-11-02 14:16:27,086] {pod_launcher.py:149} INFO - rpc error: code =
NotFound desc = an error occurred when try to find container
"8f8c2f3dce295f70ba5d60175ff847854e05ab288f7efa3ce6d0bd976d0378ea": not found
[2021-11-02 14:16:28,152] {taskinstance.py:1503} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1158, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File
"/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1333, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File
"/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py",
line 1363, in _execute_task
result = task_copy.execute(context=context)
File
"/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 367, in execute
final_state, remote_pod, result =
self.create_new_pod_for_operator(labels, launcher)
File
"/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 521, in create_new_pod_for_operator
final_state, remote_pod, result = launcher.monitor_pod(pod=self.pod,
get_logs=self.get_logs)
File
"/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py",
line 154, in monitor_pod
if not self.base_container_is_running(pod):
File
"/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py",
line 215, in base_container_is_running
status = next(iter(filter(lambda s: s.name == 'base',
event.status.container_statuses)), None)
TypeError: 'NoneType' object is not iterable
```
in the GKE logs I see ```airflow-XXX.XXXXX.b14f7e38312c4e83984d1d7a3987655a
"Pod ephemeral local storage usage exceeds the total limit of containers
10Mi. "
```
So the pod failed because I set to low limit for local storage , but the
airflow operator should not fail and catch the problem.
### What you expected to happen
The KubernetesPodOperator should managed this kind of error
### How to reproduce
launch KubernetesPodOperator with a very small local disk limit and runa
container using more than this limit
```python
KubernetesPodOperator(
task_id="XXX",
name="XXXXXXXXX",
namespace="default",
resources={'limit_memory': "512M",
'limit_cpu': "250m",
'limit_ephemeral_storage': "200M"},
is_delete_operator_pod=True,
image="any_image_using_more_than_limit_ephemeral_storage")
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
