m8719-github commented on PR #68370: URL: https://github.com/apache/airflow/pull/68370#issuecomment-4686191323
> However the entire pre-fetch might be moot now. This is caching to a single supervisor process which shuts down when the task finishes. There's not much point in trying to pre-cache as it will always fail now? > > However this leads to an interesting problem -- some loggers such as GCP's StackDriver or AWS's Cloudwatch Logs want to upload logs as they come in, not in a batch at the end, and for those to work we need to know that we weren't able to get logs and try again once the task has started running and we can access connections. > > (Which isn't to say we shouldn't have a tactical fix like this to help most loggers, but the main issue is not resolved until we fix it for all) Yes, this occurred to me as well but I wanted to be as tactical with the fix as possible. I'll take a closer look at GCP StackDriver and AWS Cloudwatch remote logging sub-systems in a separate branch and see what can be done there to fix this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
