michalmisiewicz opened a new issue #11324:
URL: https://github.com/apache/airflow/issues/11324
KubernetesPodOperator stops tailing logs after 4 hours in long living task.
When log tailing is interrupted, task log is flooded by `Event: pod had an
event of type Running` messages. Messages are generated every 2 seconds.
Log tail interruption occurs every time after 4 hours. Interruption can be
related to timeout on AKS control plane. Nevertheless, KubernetesPodOperator
should continue to tail logs after interruption . I'm going to submit a fix.
**Apache Airflow version**: 1.10.12
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`): 1.17.7
**Environment**: Azure Kubernetes Service
```
[2020-09-23 13:35:43,077] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 600:
percentage: 2.00\n'
[2020-09-23 13:51:01,477] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 900:
percentage: 3.00\n'
[2020-09-23 14:05:50,749] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 1200:
percentage: 4.00\n'
[2020-09-23 14:20:42,323] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 1500:
percentage: 5.00\n'
[2020-09-23 14:35:42,412] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 1800:
percentage: 6.00\n'
[2020-09-23 14:50:52,880] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 2100:
percentage: 7.00\n'
[2020-09-23 15:06:05,138] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 2400:
percentage: 8.00\n'
[2020-09-23 15:21:24,700] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 2700:
percentage: 9.00\n'
[2020-09-23 15:36:52,563] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3000:
percentage: 10.00\n'
[2020-09-23 15:52:32,510] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3300:
percentage: 11.00\n'
[2020-09-23 16:08:53,489] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3600:
percentage: 12.00\n'
[2020-09-23 16:24:13,732] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3900:
percentage: 13.00\n'
[2020-09-23 16:24:14,633] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3900:
percentage: 13.00\n'
[2020-09-23 16:24:14,690] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3900:
percentage: 13.00\n'
[2020-09-23 16:24:14,883] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 3900:
percentage: 13.00\n'
[2020-09-23 16:39:40,816] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 4200:
percentage: 14.00\n'
[2020-09-23 16:55:21,781] {pod_launcher.py:156} INFO - b'\x1b[2K\ron 4500:
percentage: 15.00\n'
[2020-09-23 17:10:58,901] {pod_launcher.py:171} INFO - Event: some-pod had
an event of type Running
[2020-09-23 17:10:58,927] {pod_launcher.py:166} INFO - Pod some-pod has
state running
[2020-09-23 17:11:01,148] {pod_launcher.py:171} INFO - Event: some-pod had
an event of type Running
[2020-09-23 17:11:01,154] {pod_launcher.py:166} INFO - Pod some-pod has
state running
[2020-09-23 17:11:01,154] {pod_launcher.py:166} INFO - Pod some-pod has
state running
...
the same message for next 10 hours every 2 seconds
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
