[issue36193] Redirected stderr not reset properly when using logging

[Andrius Laukavičius]

Andrius Laukavičius  added the comment:

Ok. Thanks for the clarification.

On Thu, Mar 7, 2019, 08:13 Vinay Sajip  wrote:

>
> Vinay Sajip  added the comment:
>
> The StreamHandler allows you to specify *exactly* which stream you want to
> use to log to; sys.stderr is provided as a convenient default argument
> because that's what a lot of people want a lot of the time.
>
> This is typically done at logging configuration time, or whenever a
> StreamHandler is created. This is done implicitly by your logging.warning()
> call (as documented, this calls logging.basicConfig(), which adds a
> StreamHandler using whatever sys.stderr is set to at the time the
> StreamHandler is instantiated). Also documented is that basicConfig() is
> only effective once (it will not do anything if a handler has already been
> added to the root logger - it is only meant to be use for simple one-off
> scripts). The documentation for basicConfig() is clear:
>
> "Does basic configuration for the logging system by creating a
> StreamHandler with a default Formatter and adding it to the root logger.
> The functions debug(), info(), warning(), error() and critical() will call
> basicConfig() automatically if no handlers are defined for the root logger.
>
> This function does nothing if the root logger already has handlers
> configured for it."
>
> If you want to use the real console streams, don't use logging.warning(),
> but instead explicitly call basicConfig() using __stderr__, as Peter says.
> Alternatively, use the approach suggested in the cookbook for
> context-sensitive logging:
>
>
> https://docs.python.org/3/howto/logging-cookbook.html#using-a-context-manager-for-selective-logging
>
> Closing, as this is not a bug in logging.
>
> --
> resolution:  -> not a bug
> stage:  -> resolved
> status: open -> closed
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[Vinay Sajip]

Vinay Sajip  added the comment:

The StreamHandler allows you to specify *exactly* which stream you want to use 
to log to; sys.stderr is provided as a convenient default argument because 
that's what a lot of people want a lot of the time.

This is typically done at logging configuration time, or whenever a 
StreamHandler is created. This is done implicitly by your logging.warning() 
call (as documented, this calls logging.basicConfig(), which adds a 
StreamHandler using whatever sys.stderr is set to at the time the StreamHandler 
is instantiated). Also documented is that basicConfig() is only effective once 
(it will not do anything if a handler has already been added to the root logger 
- it is only meant to be use for simple one-off scripts). The documentation for 
basicConfig() is clear:

"Does basic configuration for the logging system by creating a StreamHandler 
with a default Formatter and adding it to the root logger. The functions 
debug(), info(), warning(), error() and critical() will call basicConfig() 
automatically if no handlers are defined for the root logger.

This function does nothing if the root logger already has handlers configured 
for it."

If you want to use the real console streams, don't use logging.warning(), but 
instead explicitly call basicConfig() using __stderr__, as Peter says. 
Alternatively, use the approach suggested in the cookbook for context-sensitive 
logging:

https://docs.python.org/3/howto/logging-cookbook.html#using-a-context-manager-for-selective-logging

Closing, as this is not a bug in logging.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[Andrius Laukavičius]

Andrius Laukavičius  added the comment:

Peter,
well I don't have deep experience with logging lib to know all ins and outs. So 
I might not be correct person to define what should be changed specifically. 
But I think it should be consistent on all standard libraries how it handles 
stdout/stderr, unless there is a reason to do it differently?

For example `print` to me seems to behave how I expect, because if stdout is 
redirected elsewhere, print uses redirected stdout without a problem. 

Now with logging, it does feel strange that on first call of `logging`, it sets 
internally stderr where it is pointing at that moment and forgets about it. You 
call logging again, and it naively expects stderr to be where it was on initial 
call, when in reality it might be redirected elsewhere (as in my case).

Maybe you know the reasoning why logging even needs to behave this way? Though 
if logging behaves this way, why print behaves differently?

So to summarize this, I think all libraries should respect stdout/stderr where 
it is and have some unified way/interface (like I expected 
redirect_stdout/redirect_stderr to be) to specify that stdout/stderr has 
changed and related libraries must update their state regarding that (logging 
library case) or simply rely on where stdout/stderr is pointed, so nothing 
needs to be done on that library once stdout/stderr is redirected.


To comment on your view about `dump_stuff(dest)`. Well I think this suits well, 
when you have something to dump already, but what if something to dump is what 
is produced via stdout/stderr and you need to capture it first? How can I do 
that without first redirecting stdout where I need to?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[Peter Otten]

Peter Otten <__pete...@web.de> added the comment:

[Andrius]
> So it is not possible to consistently manage stderr when it involves > 
> logging library without explicitly "manage" it?

That's what I think, at least from within a script. To me redirect_xxx() always 
looked like a workaround to have old code write to a destination that was 
unforeseen in the initial design. I prefer

dump_stuff(dest)

to

with redirect_stdout(dest):
   dump_stuff()

That said -- do you have a suggestion how the logging library could be adapted 
to work smoothly with redirect_xxx()?

(A compelling use case would also be nice, but note that I'm not the one who 
makes the decisions.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I find issue6333 slightly relatable to this issue where it talks about external 
modules possibly closing sys.stdout/stderr.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[Andrius Laukavičius]

Andrius Laukavičius  added the comment:

Peter Otten thanks for examples,
though with capture_output function, I had a little bit different intentions.

At first I actually did similarly as you described in case 2, where I called 
BasicConfig explicitly. Then I could specify where to redirect stream and what 
to do after I was done with capturing stderr (just reset back where it would be 
on default).

Though I actually wanted to just capture stdout and stderr indirectly, meaning 
I don't want to specifically initiate logging object or modify it.

I mean, I want to just capture any stdout/stderr that would be outputed while 
running some function that was passed via `target` argument (in my 
capture_output function). In this case, some function that I capture output 
from, might have some different way to handle sys.stderr (e.g. different 
basicConfig or some other implementation where sys.stderr is handled?).

So it is not possible to consistently manage stderr when it involves logging 
library without explicitly "manage" it?

P.S. don't know if I'm clear enough, so please ask me to clarify anything if 
its not clear..:)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[Peter Otten]

Peter Otten <__pete...@web.de> added the comment:

I see various options to address this.

(1) Change basicConfig() to use __stderr__ instead of stderr to avoid 
redirections from within the script.

(2) Change your script to call basicConfig() explicitly, before the temporary 
redirection takes place. 

In both cases logging output will always go to the "real" stderr.

(3) Write a special handler that fetches sys.stderr lazily. Here's a sketch:

class StderrHandler(logging.StreamHandler):
def __init__(self):
super().__init__()

def get_stream(self):
return sys.stderr

def set_stream(self, _value):
pass

stream = property(get_stream, set_stream)

logging.basicConfig(handlers=[StderrHandler()])

As written this is likely to fail with multiple threads...

My personal favourite is option (2).

--
nosy: +peter.otten

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

A little simplified reproducer :  

import io, contextlib, logging

message = 'dummy test'

with io.StringIO() as sio:
with contextlib.redirect_stdout(sio), contextlib.redirect_stderr(sio):
logging.warning(message)
output = sio.getvalue()
print(output)

logging.warning(message)


$ python3.7 /tmp/foo.py
WARNING:root:dummy test

--- Logging error ---
Traceback (most recent call last):
  File 
"/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/__init__.py",
 line 1036, in emit
stream.write(msg)
ValueError: I/O operation on closed file
Call stack:
  File "/tmp/foo.py", line 11, in 
logging.warning(message)
Message: 'dummy test'
Arguments: ()

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[SilentGhost]

Change by SilentGhost :


--
nosy: +vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36193] Redirected stderr not reset properly when using logging

[Andrius Laukavičius]

New submission from Andrius Laukavičius :

It looks like logging library uses always first assigned stderr object and 
won't change it even if it was reset.

capture_output function redirects stdout and stderr to io.StringIO object, 
saves what was captured in string and returns it. And then context manager 
resets back stdout and stderr where it should be originally. Though it seems 
logging library ignores that.

Here is the code I encountered issue with:

import io
import sys
import contextlib
from typing import Optional
import logging

def capture_output(
target: object,
args: Optional[tuple] = None,
kwargs: Optional[dict] = None) -> str:
"""Redirect stdout and stderr into string buffer and capture it.

target object is executed with optional args, kwargs and all stdout/
stderr that is captured, returned in string form.

Args:
target: object to execute, usually a function.
args: target positional arguments (default: {None})
kwargs: target keyword arguments (default: {None})
"""
if not args:
args = ()
if not kwargs:
kwargs = {}
with io.StringIO() as sio:
with contextlib.redirect_stdout(sio), contextlib.redirect_stderr(sio):
target(*args, **kwargs)
output = sio.getvalue()

print(output)


def dummy():
print('dummy test')
logging.warning('dummy test')

def dummy2():
print('dummy2 test')


capture_output(dummy)  # works
capture_output(dummy)  # won't work anymore.

capture_output(dummy2)  # works
capture_output(dummy2)  # works

Here is what I get running this code:

dummy test
WARNING:root:dummy test

dummy test
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 982, in emit
stream.write(msg)
ValueError: I/O operation on closed file
Call stack:
  File "tt.py", line 43, in 
capture_output(dummy)  # won't work anymore.
  File "tt.py", line 28, in capture_output
target(*args, **kwargs)
  File "tt.py", line 36, in dummy
logging.warning('dummy test')
Message: 'dummy test'
Arguments: ()

dummy2 test

dummy2 test


P.S. here is original question I asked on stack overflow: 
https://stackoverflow.com/questions/54999650/python3-redirected-stderr-not-reset-properly.
 I got suggestion to report it as a bug.

--
components: Library (Lib)
messages: 337178
nosy: Andrius Laukavičius
priority: normal
severity: normal
status: open
title: Redirected stderr not reset properly when using logging
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com