rushabh-lokhande opened a new issue, #32897: URL: https://github.com/apache/airflow/issues/32897
### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened MWAA Version: 2.4.3 Airflow Version: 2.4.3 Airflow Logs currently do not fetch logs from Cloudwatch without time range, so when the cloudwatch logs are large and CloudWatch log streams are OLD, the airflow UI cannot display logs with error message: *** Reading remote log from Cloudwatch log_group: airflow-cdp-airflow243-XXXX-Task log_stream: dag_id=<DAG_NAME>/run_id=scheduled__2023-07-27T07_25_00+00_00/task_id=<TASK_ID>/attempt=1.log. Could not read remote logs from log_group: airflow-cdp-airflow243-XXXXXX-Task log_group: airflow-cdp-airflow243-XXXX-Task log_stream: dag_id=<DAG_NAME>/run_id=scheduled__2023-07-27T07_25_00+00_00/task_id=<TASK_ID>/attempt=1.log The Airflow API need to pass start and end timestamps to GetLogEvents API from Amazon CloudWatch to resolve this error and it also improves performance of fetching logs. This is critical issue for customers when they would like to fetch logs to investigate failed pipelines form few days to weeks old ### What you think should happen instead The Airflow API need to pass start and end timestamps to GetLogEvents API from Amazon CloudWatch to resolve this error. This should also improve performance of fetching logs. ### How to reproduce This issue is intermittent and happens mostly on FAILD tasks. 1. Log onto Amazon MWAA Service 2. Open Airflow UI 3. Select DAG 4. Select the Failed Tasks 5. Select Logs You should see error message like below in the logs: *** Reading remote log from Cloudwatch log_group: airflow-cdp-airflow243-XXXX-Task log_stream: dag_id=<DAG_NAME>/run_id=scheduled__2023-07-27T07_25_00+00_00/task_id=<TASK_ID>/attempt=1.log. Could not read remote logs from log_group: airflow-cdp-airflow243-XXXXXX-Task log_group: airflow-cdp-airflow243-XXXX-Task log_stream: dag_id=<DAG_NAME>/run_id=scheduled__2023-07-27T07_25_00+00_00/task_id=<TASK_ID>/attempt=1.log ### Operating System Running with Amazon MWAA ### Versions of Apache Airflow Providers apache-airflow-providers-amazon==8.3.1 apache-airflow==2.4.3 ### Deployment Amazon (AWS) MWAA ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
