seanmuth opened a new issue, #33085: URL: https://github.com/apache/airflow/issues/33085
### Apache Airflow version 2.6.3 ### What happened locally (`astro dev start`): ``` *** Could not read served logs: Client error '404 NOT FOUND' for url 'http://d19d312ee147:8794/log/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:08:36.472735+00:00/task_id=kpo_task/map_index=0/attempt=1.log.trigger.4.log' ``` then after the KPO completes, that line changes to ``` *** Found local files: *** * /usr/local/airflow/logs/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:08:36.472735+00:00/task_id=kpo_task/map_index=0/attempt=1.log *** * /usr/local/airflow/logs/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:08:36.472735+00:00/task_id=kpo_task/map_index=0/attempt=1.log.trigger.4.log ``` and I can see the one line from the triggerer.4.log in the UI: ``` [2023-08-02T15:09:41.104+0000] {triggerer_job_runner.py:608} INFO - Trigger example_dynamic_task_mapping_async/manual__2023-08-02T15:08:36.472735+00:00/kpo_task/0/1 (ID 17) fired: TriggerEvent<{'status': 'done', 'namespace': 'default', 'pod_name': 'kpo-task-74ru0c3a'}> ``` on AWS: ``` *** Could not read served logs: Client error '404 NOT FOUND' for url 'http://172.21.210.239:8794/log/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:23:10.766353+00:00/task_id=kpo_task/map_index=1/attempt=1.log.trigger.122.log' ``` 172.21.210.239 is the IP of the triggerer pod. exec’d onto it and there is indeed no file at that path, there’s not even a dag_id... dir but no mention of the trigger log line upon task completion: ``` 172.21.153.159 *** Found logs in s3: *** * s3://airflow-logs-clj7iifjn00i301muxstkk3qt/clklbvzr17992245s0cxefxljjr/dag_id=example_dynamic_task_mapping_async/run_id=manual__2023-08-02T15:23:10.766353+00:00/task_id=kpo_task/map_index=1/attempt=1.log ``` On GCP, during task execution I get the same `*** Could not read served logs: Client error '404 NOT FOUND'` error, but once the task completes I can then see ``` 172.21.4.107 *** Found remote logs: *** * gs://airflow-logs-cliuf5mwb01o901l4vf3lcihz/clktwflrz4100034zvyen3adssu/dag_id=example_dynamic_task_mapping_async/run_id=scheduled__2023-08-01T15:50:58.184208+00:00/task_id=kpo_task/map_index=0/attempt=1.log *** * gs://airflow-logs-cliuf5mwb01o901l4vf3lcihz/clktwflrz4100034zvyen3adssu/dag_id=example_dynamic_task_mapping_async/run_id=scheduled__2023-08-01T15:50:58.184208+00:00/task_id=kpo_task/map_index=0/attempt=1.log.trigger.4.log ``` with the single trigger message: ``` [2023-08-02, 15:52:35 UTC] {triggerer_job_runner.py:616} INFO - Trigger example_dynamic_task_mapping_async/scheduled__2023-08-01T15:50:58.184208+00:00/kpo_task/0/1 (ID 9) fired: TriggerEvent<{'status': 'done', 'namespace': 'traditional-waxing-1670', 'pod_name': 'kpo-task-7o06jv52'}> ``` ### What you think should happen instead Per PR #27758 the task logs view in the UI should show the trigger log output both while the logs are local and the task is in-flight, and after completion, once logs are uploaded to whatever `remote_logging` provider is being used. ### How to reproduce Attached an example dag with mapped and deferred KPOA [example_dynamic_mapping_dag_async.py.txt](https://github.com/apache/airflow/files/12254508/example_dynamic_mapping_dag_async.py.txt) ### Operating System debian ### Versions of Apache Airflow Providers ``` apache-airflow-providers-amazon==8.3.1 apache-airflow-providers-celery==3.2.1 apache-airflow-providers-cncf-kubernetes==7.3.0 apache-airflow-providers-common-sql==1.6.0 apache-airflow-providers-datadog==3.3.1 apache-airflow-providers-elasticsearch==4.5.1 apache-airflow-providers-ftp==3.4.2 apache-airflow-providers-google==10.0.0 apache-airflow-providers-http==4.5.0 apache-airflow-providers-imap==3.2.2 apache-airflow-providers-microsoft-azure==6.2.1 apache-airflow-providers-postgres==5.5.2 apache-airflow-providers-redis==3.2.1 apache-airflow-providers-sqlite==3.4.2 ``` ### Deployment Astronomer ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
