[issue9338] argparse optionals with nargs='?', '*' or '+' can't be followed by positionals
Change by Walter Doekes : -- nosy: +wdoekes ___ Python tracker <https://bugs.python.org/issue9338> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23010] unclosed file warning when defining unused logging FileHandler in dictConfig
Walter Doekes added the comment: No worries. I know how it is ;) Thanks for the update. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23010 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23010] unclosed file warning when defining unused logging FileHandler in dictConfig
Walter Doekes added the comment:
The handlers are AFAIK referenced - if you peek at logging._handlerList or
logging._handlers you should see them in there.
Aha. But that's the point. They aren't. If they were, I wouldn't have any
problems with this. But I have this problem because the warnings are generated
at startup of my program, not at cleanup.
Look at this:
$ PYTHONWARNINGS=default python3 problem.py 21 | sed -e 's/^//'
2014-12-09 08:58:45.982258: starting
2014-12-09 08:58:46.011788: after dictconfig
/usr/lib/python3.4/importlib/_bootstrap.py:321: ResourceWarning: unclosed
file _io.FileIO name='/tmp/debug.log' mode='ab'
return f(*args, **kwds)
imported once
2014-12-09 08:58:46.019327: after error
2014-12-09 08:58:51.024376: after sleep
_handlerList: []
_handlers: []
If what you say is true, the following wouldn't be:
- _handlerList is non-empty (wrong, see above)
- the warning would be generated at application shutdown (no, it's generated at
importlib time, see the timestamps)
(See source below.)
You could just leave out the handlers, since you're never using them; it's
not logging's job to be overly restrictive about this sort of thing.
I could agree with you there. But in that case I still think we should call it
illegal (and perhaps raise a more sensible warning).
Obviously you spot the problem immediately in my toy example. But in a large
LOGGING dictionary, items may be shuffled around and changed. When that
happens, and these warnings turn up out of the blue, that's distracting to put
it mildly.
I expect that you'll reconsider your strict opinion after seeing that
logging._handlers is indeed empty and the handlers are not referenced (and
therefore should have been destroyed properly or not created at all).
Cheers,
Walter
Example source:
$ cat problem.py | sed -e 's/^//'
from datetime import datetime
from time import sleep
import sys
if __name__ == '__main__':
LOGGING = {
'version': 1,
'handlers': {
'logfile': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/tmp/debug.log',
},
},
}
print('%s: starting' % (datetime.now(),))
from logging.config import dictConfig
dictConfig(LOGGING)
print('%s: after dictconfig' % (datetime.now(),))
sys.stdout.flush()
# using importlib on a new file triggers the warnings
import importlib, shutil, os
shutil.copy(__file__, __file__ + '.new')
os.unlink(__file__)
os.rename(__file__ + '.new', __file__)
importlib.import_module('problem')
sys.stderr.flush()
print('%s: after error' % (datetime.now(),))
sys.stdout.flush()
sleep(5)
print('%s: after sleep' % (datetime.now(),))
# Vinay Sajip wrote:
# The handlers are AFAIK referenced - if you peek at
# logging._handlerList or logging._handlers you should see them in
# there.
from logging import _handlerList, _handlers
print('_handlerList:', _handlerList)
print('_handlers:', [i for i in _handlers.values()])
else:
print('imported once')
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23010
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23010] unclosed file warning when defining unused logging FileHandler in dictConfig
New submission from Walter Doekes:
If you define a FileHandler in your logging dictConfig but fail to use it in
the same configuration, the file handle is leaked and a ResourceWarning is
thrown.
Versions:
$ python3 -V
Python 3.4.0
$ apt-cache show python3.4 | grep ^Version
Version: 3.4.0-2ubuntu1
Expected:
$ PYTHONWARNINGS=default,ignore::DeprecationWarning \
python3 problem.py
imported once
Observed:
$ PYTHONWARNINGS=default,ignore::DeprecationWarning \
python3 problem.py
.../lib/python3.4/importlib/_bootstrap.py:656:
ResourceWarning: unclosed file
_io.FileIO name='/tmp/debug.log' mode='ab'
code = marshal.loads(data)
imported once
To reproduce, save below as problem.py:
if __name__ == '__main__':
LOGGING = {
'version': 1,
'handlers': {
'logfile': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/tmp/debug.log',
},
},
}
from logging.config import dictConfig
dictConfig(LOGGING)
# using importlib on a new file triggers the warnings
import importlib, shutil, os
shutil.copy(__file__, __file__ + '.new')
os.unlink(__file__)
os.rename(__file__ + '.new', __file__)
importlib.import_module('problem')
else:
print('imported once')
Fixed by:
--- /usr/lib/python3.4/logging/config.py.orig 2014-12-08
14:06:24.911460799 +0100
+++ /usr/lib/python3.4/logging/config.py2014-12-08 14:16:17.519460799
+0100
@@ -637,6 +637,16 @@ class DictConfigurator(BaseConfigurator)
except Exception as e:
raise ValueError('Unable to configure root '
'logger: %s' % e)
+
+# Close unused loggers
+used_handlers = set()
+for logger in logging.root.manager.loggerDict.values():
+if hasattr(logger, 'handlers') and logger.handlers:
+used_handlers.add(*logger.handlers)
+for name, handler in handlers.items():
+if handler not in used_handlers:
+if hasattr(handler, 'close'):
+handler.close()
finally:
logging._releaseLock()
The fix above might not be the best fix, but it proves where the problem is.
Perhaps the handlers shouldn't be instantiated until the first user of such a
handler is instantiated.
This issue was found by using Django 1.7 with Python 3.4 when enabling the
PYTHONWARNINGS.
Cheers,
Walter Doekes
OSSO B.V.
--
components: Library (Lib)
messages: 232302
nosy: wdoekes
priority: normal
severity: normal
status: open
title: unclosed file warning when defining unused logging FileHandler in
dictConfig
versions: Python 3.4
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23010
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23010] unclosed file warning when defining unused logging FileHandler in dictConfig
Walter Doekes added the comment: Thanks for the quick response! Well, it's possible that you could configure handlers in the configuration for later use (i.e. at some point after the dictConfig() call returns). After the dictConfig call returns, the StreamHandler/FileHandler is not referenced by anyone anymore. That's what causes the ResourceWarning. Unless I'm severely mistaken, there is no way to reach that old FileHandler instance. If you want to avoid opening the file until it's actually needed, you can specify delay=True. I am aware of that, but that's a workaround, not a fix. (And it has drawbacks of its own, for example in forking and/or setuid situations.) Unless there's some reason you can't do that, or there's something I've overlooked, I think I should close this as invalid. You could do that, but -- barring me overlooking something here -- I think that would only be correct if the dictionary that I passed to dictConfig is judged as being illegal, because it contains unused handlers. The ResourceWarning thrown is hard to understand because of where it is raised (at random points in a different modules), so like it is now, it may dissuade users from enabling more (visual) warnings. I'd rather see a warning raised earlier from dictConfig() that I configured an unused handler, so I have a better indication of what to fix. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23010 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2980] Pickle stream for unicode object may contain non-ASCII characters.
Walter Doekes walter+pyt...@wjd.nu added the comment: Same issue with Django here ;-) I wouldn't mind a protocol 3 that does 128 ascii only. If only because debugging base64'd zlib'd protocol-2 data is not particularly convenient. -- nosy: +wdoekes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2980 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1222] locale.format bug if thousand separator is space (french separator as example)
Walter Doekes [EMAIL PROTECTED] added the comment: @Antoine Pitrou Regarding # XXX is the trailing space a bug?: I'm inclined to believe that it is. Man 7 locale does not mention that p_sep_by_space should be used for non-int currency symbols, nor that it shouldn't. However, it does say: char *int_curr_symbol; /* First three chars are a currency symbol from ISO 4217. Fourth char is the separator. Fifth char is ’ ’. */ I haven't seen that fifth character, and in the libc sources I can't find one either: glibc-2.7/localedata/locales/nl_NL:66:int_curr_symbol U0045U0055U0052U0020 I do however see a separator. In libc documentation I found here ( http://www.chemie.fu-berlin.de/chemnet/use/info/libc/libc_19.html#SEC325 ), it says the following: In other words, treat all nonzero values alike in these members. These members apply only to currency_symbol. When you use int_curr_symbol, you never print an additional space, because int_curr_symbol itself contains the appropriate separator. The POSIX standard says that these two members apply to the int_curr_symbol as well as the currency_symbol. But an example in the ANSI C standard clearly implies that they should apply only to the currency_symbol---that the int_curr_symbol contains any appropriate separator, so you should never print an additional space. Based on what we know now, we recommend you ignore these members when printing international currency symbols, and print no extra space. This is probably not right either, because this forces you to use an n_sign_posn and p_sign_posn that have the symbol on the same side of the value. (Although that might not be such an awful assumption.) And, more importantly, a grep through the sources reveal that no language has a preceding space. (But they do, I assume, have *_sign_posn's that want a trailing symbol.) glibc-2.7/localedata/locales$ grep ^int_curr_symbol * | grep -vE '(U0020| )' | wc -l 0 That leaves us with two more options. Option three: the fourth character is optional and defines what the separator is but not _where_ it should be. I.e. you might have to move it according to what *_sign_posn says. And finally, option four: international formatting should ignore all of *_cs_precedes, *_sep_by_space and *_sign_posn. Locale(7) explicitly mentions currency_symbol, not int_cur_symbol. Perhaps one should assume that international notation is universal. (I would guess that most common is: int_curr_symbolspaceoptional_minusnummon_thousands_sepnummon_decimal_pointnum) Personally I would go with option three. It has the least impact on formatting because it only cleans up spaces. I'm guessing however that option four is the Right One. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1222 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1222] locale.format bug if thousand separator is space (french separator as example)
Changes by Walter Doekes [EMAIL PROTECTED]: -- nosy: +wdoekes ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1222 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
