collinmcnulty opened a new issue #22267: URL: https://github.com/apache/airflow/issues/22267
### Apache Airflow version 2.2.3 ### What happened On celery executor, a KubernetesPodOperator task was launched. At some point, the Airflow task failed, but the pod continued. I believe the task failed as a result of the worker pod dying suddenly. On the second attempt, the task reattached to the running pod and retrieved the logs, but all log lines that predate the re-attachment have the timestamp of when the task re-attached, not when the log was originally generated. This means that several hours of logs all have the same timestamp. The logs of the first attempt are empty, as we are using S3 remote logging and the worker died before sending logs to S3. ### What you think should happen instead The logs for the second attempt should have the timestamp from when they were generated, not pod reattachment. We retrieve the timestamp but then don't use it for this purpose. https://github.com/apache/airflow/blob/602abe8394fafe7de54df7e73af56de848cdf617/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L202 ### How to reproduce 1. Set up S3 remote logging and celery executor 2. Launch a KPO that will run for a while and emit logs. Set retries > 0. 3. Kill the celery worker before the KPO finishes. ### Operating System Debian Bullseye ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==2.4.0 ### Deployment Astronomer ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
