namanjain24-sudo commented on issue #73285: URL: https://github.com/apache/airflow/issues/73285#issuecomment-5761337628
Some observations from the current code that may help when choosing the shape (no preference between the options from my side): - In deferrable mode `AwsTaskLogFetcher` is not what forwards the logs for ECS: `TaskDoneTrigger._forward_logs` polls CloudWatch itself inside the triggerer, and a callable cannot be handed to a trigger since its kwargs are stored serialized. For Batch, `execute_complete` builds a fetcher and reads the last messages once after the job has ended (`get_last_log_messages(10000)`), so a callback there would receive everything at once instead of while the job runs. An `on_log_event` parameter would therefore only behave as described with `deferrable=False`, unless it takes a serializable form such as an import path, or the limitation is documented. - `AwsTaskLogFetcher.run()` calls `self.logger.log(...)` in a loop without exception handling. An exception raised by a user callback would end that background thread without failing the task, and log fetching would silently stop, so the callback call would need to be wrapped and its errors logged. --- Drafted-by: a Gen-AI coding assistant (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
