seanghaeli opened a new issue, #77055: URL: https://github.com/apache/airflow/issues/77055
With #76527, each asset watcher logs to `triggers/trigger_id=<id>/triggerer_job_id=<id>/trigger.log` on its triggerer. With remote logging on, each rotated part is uploaded as soon as the file reaches `[triggerer] watcher_log_max_bytes`, but the current part only when the watcher leaves that triggerer or the triggerer stops. A killed triggerer's current parts are uploaded when a triggerer next starts on the same disk (`WatcherLogs` in `airflow-core/src/airflow/utils/log/watcher_log.py`). So a watcher that stays on one triggerer for weeks has only its rotated parts in remote storage. Its newest lines, up to 2 MiB by default, are only on the triggerer's disk, and a quiet watcher may have nothing uploaded at all. If the triggerer is killed and its disk goes with it, as with a pod's local volume, those lines are lost apart from the copy in the triggerer's output, if that's kept. Suggestion: also rotate the current file on a configurable timer, not only by size, whenever it has new lines, so the existing rotated-part upload takes it to remote storage. Uploading the open file directly isn't safe, since several remote handlers empty or delete a file after uploading it. Handlers that send each line as it's written (CloudWatch, Cloud Logging) don't need this. Done when: a watcher's newest lines reach remote storage within the configured interval while it stays on one triggerer, without uploading when nothing new was written, with tests. This needs #76527 to merge first. I'd recommend building on top of #76527's branch and rebasing once it's merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
