derkuci commented on issue #20500:
URL: https://github.com/apache/airflow/issues/20500#issuecomment-1001698462
I have no problem with calling this a feature, but would suggest a few
things to help (others) debug when the circular logging indeed happens.
* `StreamLogWriter.write()` could be slightly modified to swap the buffer
before sending another message to `_propagate_log`. Something like
```python
buf, self._buffer = self._buffer, ''
self._propagate_log(buf.rstrip())
```
This will change the growth of buffer from exponential to linear, and the
users may hit the recursive stack limit before the out-of-memory error, and
would more easily figure out the issue.
* There are cases where users don't have much control over the code (tasks)
run in airflow. It's also quite possible some weird import statement pulls in
the unwanted redirection. If airflow could detect such misuse, or even better,
could separate its own log from users (sorry, I don't know how), that would be
much nicer.
Anyway, just hope this would save others some time when they deploy the new
versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
