derkuci commented on issue #20500: URL: https://github.com/apache/airflow/issues/20500#issuecomment-1002189756
> > This will change the growth of buffer from exponential to linear, and the users may hit the recursive stack limit before the out-of-memory error, and would more easily figure out the issue. > > Unfortunately this would change the logging behaviour. which we do not want to do as it would affect legitimately written logs and use extra memory when not needed. What we do with log writtng is correct - we should not use extra buffer nor strip messages. The problems is that by the time you write log to stderr, stderr is directly routed back to the stream writer that propagates the logs further. It's not intended to redirect to itself. Yet by adding StreamHandler it happens. Please be noted that the stripping is in the current airflow code; I copied the behavior to keep things the same when nothing bad happens (and if we don't strip, we get extra blank lines in the log). And since Python uses references, I don't see any "extra" buffering here. The only changed part is when the buffer (that StreamLogWriter keeps log messages) is cleared, before or after the recursion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
