jd878-gif commented on issue #70314: URL: https://github.com/apache/airflow/issues/70314#issuecomment-5183375486
> [@jd878-gif](https://github.com/jd878-gif) it looks like it is open, as far as I can tell. Thank you for considering working on it. Although, I'm not really sure whether somebody is already working on it or not. > > I've checked open PRs and it looks like this issue may be somewhat similar to these ones: > > * [Fix remote processor injection happening before dictConfig runs in configure_logging #66633](https://github.com/apache/airflow/pull/66633) > * [Fix CloudWatch remote logging for ephemeral lifecycle executor #68779](https://github.com/apache/airflow/pull/68779) > > At least, based on function names mentioned in the descriptions. Maybe, it is the same problem, but observed in a different use case. > > [@jason810496](https://github.com/jason810496) [@korex-f](https://github.com/korex-f) [@ferruzzi](https://github.com/ferruzzi) [@o-nikolas](https://github.com/o-nikolas) [@seanghaeli](https://github.com/seanghaeli) it looks like you've been working on something similar recently. Can you take a look whether it is the same issue or something different, please? Thanks for the detailed root-cause writeup. I traced this through the codebase and it matches exactly: configure_logging() in the Task SDK (task-sdk/src/airflow/sdk/log.py) accesses .processors on the remote log handler — which lazily builds the watchtower.CloudWatchLogHandler — before calling the shared configure_logging(), which runs logging.config.dictConfig() and closes every handler in logging._handlerList, including that one. This was already fixed in #68779 ("Fix CloudWatch remote logging for ephemeral lifecycle executor"), merged into main and released in apache-airflow-providers-amazon==9.32.0. The fix makes CloudWatchRemoteLogIO.handler self-heal by rebuilding the watchtower handler if it detects shutting_down=True, rather than trying to prevent dictConfig from touching it. There's a regression test covering this exact scenario: test_log_message_after_handler_closed_by_dictconfig. I confirmed this directly using your own repro DAG and docker-compose setup, bumping only the provider pin to apache-airflow-providers-amazon==9.32.0. The trigger's CloudWatch log stream (.../task_id=test_triggerer/attempt=1.log.trigger.1.log) now contains the full trigger lifecycle that was previously missing entirely: trigger ... starting trigger starting 109 seconds remaining; sleeping 10 seconds ... yielding event with payload DateTime(...) Trigger fired event trigger completed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
