vincbeck commented on issue #33634: URL: https://github.com/apache/airflow/issues/33634#issuecomment-1690619969
I just thought of a potential solution. Here the issue is `start_time` right? When tasks are rerun, `task_instance.start_date` is the lastest run and as such, previous logs are ignored. What if we do not set `start_time`? I know that the optimization is actually done on the `end_time`, when `end_time` is not provided, very old tasks can take a lot of time to load logs because CLoudWatch scans since now. But if we provide only `end_time`, then the optimization is still in place and the bug would be fixed because no `start_time` is provided. I'll create a PR, discussion can happen there or here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
