[issue22676] Creating the string representation of a module is slower
Georg Brandl added the comment: HasAttr would just call GetAttr and discard the exception. @ OP: what objects are you pickling? You can give them a __module__ attribute to save the lookup in sys.modules. -- nosy: +georg.brandl ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: There's no doubt that whichmodule() has grown more complex since 3.2. We probably cannot eliminate the O(number of modules) component, but could at least make it faster. -- stage: - needs patch title: Creating the string representation of a module is slower - _pickle's whichmodule() is slow versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22683] bisect index out of bounds issue
New submission from Paul Ianas:
The precondition for all the bisect functions is implemented like this:
if lo 0:
raise ValueError('lo must be non-negative')
if hi is None:
hi = len(a)
Now, of course, if hi is given, and hi = 2 * len(a), then we get an
IndexError. In case hi 0, we always get 0 as a result (even if the element is
there).
I think it would be better to treat the hi in the precondition in the same way
as the lo parameter: that means, raise a ValueError in case hi has an illegal
value.
Disclaimer: of course, it makes no sense to give an illegal argument to that
function; still, since lo is treated against illegal values, maybe it's better
to do the same for hi.
At the same time, maybe moving the precondition code in a separate function
(which raises a ValueError in case precondition is not met) makes more sense,
for not repeating the same code in all bisect functions.
A small snippet which reproduces this:
from bisect import bisect_left
a = [1, 2, 3, 4]
idx = bisect_left(a, 2, 0, 10) # 10 2 * 4
print(idx)
--
components: Library (Lib)
messages: 229750
nosy: Paul.Ianas
priority: normal
severity: normal
status: open
title: bisect index out of bounds issue
type: enhancement
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22683
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: Note the problem is easily reproduce. For example we can take numpy where some ufuncs (i.e. function-like objects implemented in C) don't have a __module__. $ PYTHONHASHSEED=0 python3.4 -m timeit -s import pickle, numpy; d=numpy.add pickle.dumps(d) 1000 loops, best of 3: 280 usec per loop $ PYTHONHASHSEED=0 python3.4 -m timeit -s import pickle, numpy; d=numpy.diff pickle.dumps(d) 10 loops, best of 3: 2.74 usec per loop We see that pickling numpy.add (which doesn't have a __module__) is 100x slower than numpy.diff (which has a __module__)! Note I'm forcing PYTHONHASHSEED for consistent results, since whichmodule() uses dict iteration. For comparison, 2.7 is fast enough: $ PYTHONHASHSEED=0 python2.7 -m timeit -s import cPickle as pickle, numpy; d=numpy.add pickle.dumps(d) 10 loops, best of 3: 6.12 usec per loop $ PYTHONHASHSEED=0 python2.7 -m timeit -s import cPickle as pickle, numpy; d=numpy.diff pickle.dumps(d) 10 loops, best of 3: 2.35 usec per loop (varying PYTHONHASHSEED didn't produce any slower results) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Georg Brandl added the comment: Attached patch addresses the repeated __repr__ calling and gives a 4x speedup here. -- keywords: +patch Added file: http://bugs.python.org/file36985/whichmodule_speedup.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: Actually, numpy.add takes a different path. Sorry. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: So, a proper way to reproduce it is through Ellipsis (which does go through whichmodule()): $ PYTHONHASHSEED=0 python3.4 -m timeit -s import pickle; d=Ellipsis pickle.dumps(d) 1000 loops, best of 3: 201 usec per loop -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
STINNER Victor added the comment: For example we can take numpy where some ufuncs (i.e. function-like objects implemented in C) don't have a __module__. Oh. I was not aware of that. Is there a way to fix numpy to set the __module__ attribute? Maybe we should warn users that serialiaing objects without __module__ is much slower? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
STINNER Victor added the comment: whichmodule_speedup.diff: I proposed once a more generic solution which is much more complex to implement. Lazy formatting of Exception message: in most cases, the message is not used. Replace AttributeError(message) with AttributeError(attr=name), only format when str(exc) is called. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: Victor, see https://github.com/numpy/numpy/issues/4952 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Antoine Pitrou added the comment: Attached patch produces a 8x speedup on Ellipsis. -- Added file: http://bugs.python.org/file36986/whichmodule.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22676] _pickle's whichmodule() is slow
Georg Brandl added the comment: I didn't have numpy anyway, I tested on a function with __module__ set to None manually. But I see the same speedup for Ellipsis. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22676 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22683] bisect index out of bounds issue
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +rhettinger, tim.peters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22683 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22660] Review ssl docs for security recommendations
Roundup Robot added the comment: New changeset 6f6e56bb10aa by Antoine Pitrou in branch '2.7': Issue #22660: update various mentions in the ssl module documentation. https://hg.python.org/cpython/rev/6f6e56bb10aa -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22660 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22660] Review ssl docs for security recommendations
Changes by Antoine Pitrou pit...@free.fr: -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22660 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22607] find by dichotomy the failing test
Xavier de Gaye added the comment:
The attached patch adds the '-X' and '-Y' options to the regrtest tool,
allowing to select a range of tests and a range of their subtests. The patch
is missing the test cases for the moment.
Limitation:
Does not work very well with nested subtest (nested subtests are currently only
used by the unittest test suite itself): subtest numbers are flattened across
all the nested subtests.
Here is the sequence of commands that would have allowed to find the subtest
responsible for the crash in test_capi at issue 22588, without modifying any
code:
./python -m test -X [] test_capi # Get the test count: 18
tests.
./python -m test -X range(1,10) -R 23:23 test_capi# pass
./python -m test -X range(10,15) -R 23:23 test_capi # pass
./python -m test -X range(15,17) -R 23:23 test_capi # pass
./python -m test -X [17] -R 23:23 test_capi # pass
./python -m test -X [18] -R 23:23 test_capi # fail
./python -m test -X [18] -Y [] test_capi# Test 18 has 35 subtests.
./python -m test -X [18] -Y range(1,19) -R 23:23 test_capi # fail
./python -m test -X [18] -Y range(1,10) -R 23:23 test_capi # fail
./python -m test -X [18] -Y range(1,6) -R 23:23 test_capi # pass
./python -m test -X [18] -Y range(6,8) -R 23:23 test_capi # pass
./python -m test -X [18] -Y [8] -R 23:23 test_capi # pass
./python -m test -X [18] -Y [9] -R 23:23 test_capi # fail
Output of the last command:
Test# 18: test__testcapi
subTest# 9: internal, {'name': 'test_incref_decref_API'}
--
Added file: http://bugs.python.org/file36987/regrest_XY_options.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22607
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22684] message.as_bytes() produces recursion depth exceeded
New submission from Pas: Please see the attached traceback (or this http://pastebin.com/WYinRGie for fancy colors). It depends on message size, we're trying to send Multipart MIME messages (a PDF attached, that has an image embedded). After editing flask_mail.py to use the fallback ( message().as_string().encode(self.charset or 'utf-8') ) things work again. If anyone could help confirm if this is a bug, or help me understand how I misuse the library, I'd be grateful. Thanks! -- components: email files: py34-email.message.as_bytes-recursion-depth-exceeded.txt messages: 229762 nosy: barry, pas, r.david.murray priority: normal severity: normal status: open title: message.as_bytes() produces recursion depth exceeded type: behavior versions: Python 3.4 Added file: http://bugs.python.org/file36988/py34-email.message.as_bytes-recursion-depth-exceeded.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22684 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
New submission from wabu: using `p = create_subprocess_exec(..., stdout=subprocess.PIPE, limit=...)`, p.stdout has not transport set, so the underlying protocol is unable to pause the reading of the transport, resulting in high memory usage when slowly consuming input from p.stdout, even if the limit parameter is passed. A workaround is to set the transport manually after creating the subprocess: `p.stdout.set_transport(p._transport.get_pipe_transport(1))`, but this should happen inside the create_subprocess call. -- components: asyncio messages: 229763 nosy: gvanrossum, haypo, wabu, yselivanov priority: normal severity: normal status: open title: memory leak: no transport for pipes by create_subprocess_exec/shell versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22684] message.as_bytes() produces recursion depth exceeded
R. David Murray added the comment: It looks like a bug, but I'm not sure why as_bytes would trigger it but not as_string. Can you supply a copy of the message that fails? The as_string version (assuming the content was all ascii) should be enough to reproduce the issue, since it appears to be happening in the header folding step. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22684 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22686] random.randint does not include endpoint
New submission from FH: Upon inspection, random.randint(a,b) does not include the endpoint b, contrary to documentation, and contrary to the argumentation in Issue7009. To see this, run e.g. sum(np.random.randint(0,1,1000)) which will return 0 repeatedly (statistically very unlikely). I tried both on Kubuntu 14.04/python 2.7 and pythoneverwhere.org Within random.py, randint is (in both 2.7 and 3.5) implemented as def randint(self, a, b): Return random integer in range [a, b], including both end points. return self.randrange(a, b+1) which falsely seems to include the endpoint (as randrange excludes the endpoint). However, upon running it does not. -- components: Library (Lib) messages: 229765 nosy: georg.brandl, orsenthil, sciencebuggy priority: normal severity: normal status: open title: random.randint does not include endpoint type: behavior versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22686 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22686] random.randint does not include endpoint
Mark Dickinson added the comment: You seem to be confusing `np.random.randint`, which is a function from NumPy (not part of core Python), with `random.randint` from the standard library. NumPy's np.random.randint does not include the endpoint. Python's does. As far as I can tell, the documentation is correct for both. Python 2.7.8 (default, Oct 15 2014, 22:04:42) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin Type help, copyright, credits or license for more information. import random sum(random.randint(0, 1) for _ in range(1000)) 501 Closing as 'not a bug'. -- nosy: +mark.dickinson resolution: - not a bug status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22686 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
STINNER Victor added the comment: p = create_subprocess_exec(..., stdout=subprocess.PIPE, limit=...) I don't see yield from. Do you wait until the coroutine completes? See the documentation: https://docs.python.org/dev/library/asyncio-subprocess.html#subprocess-using-streams Please write a more complete example to demontrate your issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22687] horrible performance of textwrap.wrap() with a long word
New submission from Piotr Engelking: Wrapping a paragraph containing a long word takes a lot of time: $ time python3 -c 'import textwrap; textwrap.wrap(a * 2 ** 16)' real3m14.923s user3m14.792s sys 0m0.016s $ A straightforward replacement is 5000 times faster: $ time python3 -c '(.join(x) for x in zip(*[iter(a * 2 ** 16)] * 70))' real0m0.053s user0m0.032s sys 0m0.016s $ Tested on Debian with python3.4 3.4.2-1 and python2.7 2.7.8-10. -- messages: 229768 nosy: inkerman priority: normal severity: normal status: open title: horrible performance of textwrap.wrap() with a long word type: performance versions: Python 2.7, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22687 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22687] horrible performance of textwrap.wrap() with a long word
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +georg.brandl, serhiy.storchaka versions: +Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22687 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22688] Use the subprocess module in the uuid module
New submission from Serhiy Storchaka: There is a proposition (msg229163) to use the subprocess module instead of os.popen() in the uuid module. I hesitate how classify this issue (behavior, security or enhancement) and for which versions target it. May be this is a dependency of issue17293. -- components: Library (Lib) messages: 229769 nosy: haypo, neologix, serhiy.storchaka priority: normal severity: normal stage: needs patch status: open title: Use the subprocess module in the uuid module ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22688 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22687] horrible performance of textwrap.wrap() with a long word
Serhiy Storchaka added the comment: This particular case is related to the behavior of the wordsep_re regular expression in worst case. When text contains long sequence of words characters which is not ended by a hypen, or long sequence of non-word and non-space characters (and in some other cases), computational complexity of this regular expression matching is quadratic. This is a peculiarity of current implementation of regular expression engine. May be it is possible to rewrite the regular expression so that quadratic complexity will gone, but this is not so easy. The workaround -- use break_on_hyphens=False. -- assignee: - serhiy.storchaka priority: normal - low stage: - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22687 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ram Rachum added the comment: Thanks for the fix Ethan. Will you also push this to the backport on PyPI? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22688] Use the subprocess module in the uuid module
STINNER Victor added the comment: Did you see my issue which has a patch? http://bugs.python.org/issue22637 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22688 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ethan Furman added the comment: Thanks for the reminder! ... Done. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ram Rachum added the comment: Thanks! Two questions: 1. Is there a GitHub repo for enum34? Couldn't find a link to it on the PyPI page. 2. Aren't 'name' and 'value' already included in `added_behavior`? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ram Rachum added the comment: Also, aren't you excluding a lot of important magic methods from that `dir`? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22688] Use the subprocess module in the uuid module
Serhiy Storchaka added the comment: Oh, I missed it. Thank you for opening the issue and writting the patch. -- resolution: - duplicate stage: needs patch - resolved status: open - closed superseder: - avoid using a shell in uuid: replce os.popen with subprocess.Popen ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22688 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22637] avoid using a shell in uuid: replce os.popen with subprocess.Popen
Serhiy Storchaka added the comment: Besides few nitpicks the patch LGTM. -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22637 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
wabu added the comment: Sorry for the confusion, yes i do the yield from. The stdout stream for the process is actually producing data as it should. The subprocess produces a high amount of data (pbzip2), but is only consumed slowly. Normally when the buffer limit is reached for a stream reader, it calls pause_reading on the transport inside the feed_data method (see https://code.google.com/p/tulip/source/browse/asyncio/streams.py#365), but here this is not happening, as the returned reader has no transport set (p.stdout._transport == None). So it fills up all the memory. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
Guido van Rossum added the comment: Hm... It does look like there's nothing that tells stdout (which is a StreamReader) about its transport. Wabu, could you experiment with a change to asyncio/subprocess.py where SubprocessStreamProtocol.connection_made() calls self.stdout.set_transport(transport) right after creating self.stdout? On Tue, Oct 21, 2014 at 12:55 PM, wabu rep...@bugs.python.org wrote: wabu added the comment: Sorry for the confusion, yes i do the yield from. The stdout stream for the process is actually producing data as it should. The subprocess produces a high amount of data (pbzip2), but is only consumed slowly. Normally when the buffer limit is reached for a stream reader, it calls pause_reading on the transport inside the feed_data method (see https://code.google.com/p/tulip/source/browse/asyncio/streams.py#365), but here this is not happening, as the returned reader has no transport set (p.stdout._transport == None). So it fills up all the memory. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Ethan Furman added the comment: 1. Is there a GitHub repo for enum34? Couldn't find a link to it on the PyPI page. The repo is at: https://bitbucket.org/stoneleaf/enum34 2. Aren't 'name' and 'value' already included in `added_behavior`? They didn't used to be, but they are now. I'll fix that (eventually). Also, aren't you excluding a lot of important magic methods from that `dir`? We decided the dunder methods were not interesting, so declined to include them in the listing. And no, we are not changing our minds on that. ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
wabu added the comment: Here's a more complete example @coroutine put_data(filename, queue, chunksize=16000): pbzip2 = yield from asyncio.create_subprocess_exec( 'pbzip2', '-cd', filename, stdout=asyncio.subprocess.PIPE, limit=self.chunksize*2) while not pbzip2.stdout.at_eof(): data = yield from pbzip2.stdout.read(chunksize) yield from queue.put(data) adding the workaround after createing the stream fixes the issue: pbzip2.stdout.set_transport(pbzip2._transport.get_pipe_transport(1)) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
Guido van Rossum added the comment: Can you confirm that this patch fixes the problem (without you needing the workaround in your own code)? -- keywords: +patch Added file: http://bugs.python.org/file36989/fix22685.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22637] avoid using a shell in uuid: replce os.popen with subprocess.Popen
Roundup Robot added the comment: New changeset 8ee63d0bd7b8 by Victor Stinner in branch 'default': Issue #22637: avoid using a shell in uuid https://hg.python.org/cpython/rev/8ee63d0bd7b8 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22637 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
wabu added the comment: thanks a lot, the fix works! On 21.10.2014 22:16, Guido van Rossum wrote: Guido van Rossum added the comment: Can you confirm that this patch fixes the problem (without you needing the workaround in your own code)? -- keywords: +patch Added file: http://bugs.python.org/file36989/fix22685.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22506] `dir` on Enum subclass doesn't expose parent class attributes
Roundup Robot added the comment: New changeset 9038b63dad52 by Ethan Furman in branch 'default': Issue22506: remove name value from __dir__ as they now show up automatically https://hg.python.org/cpython/rev/9038b63dad52 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22506 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
Guido van Rossum added the comment: Victor, do you think this needs a unittest? It seems kind of difficult to test for whether memory fills up (the machine may get wedged if it does :-). On Tue, Oct 21, 2014 at 1:39 PM, wabu rep...@bugs.python.org wrote: wabu added the comment: thanks a lot, the fix works! On 21.10.2014 22:16, Guido van Rossum wrote: Guido van Rossum added the comment: Can you confirm that this patch fixes the problem (without you needing the workaround in your own code)? -- keywords: +patch Added file: http://bugs.python.org/file36989/fix22685.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
STINNER Victor added the comment: Oh ok, now I understood the issue :-) Attached patch should fix it. It implements Guido's suggestion. There is not unit test yet. -- Added file: http://bugs.python.org/file36990/set_transport.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22689] Posix getenv makes no guarantee of lifetime of returned string
New submission from Aidan Hobson Sayers:
Posix says the following on the subject of getenv:
The returned string pointer might be invalidated or the string content might
be overwritten by a subsequent call to getenv()
(http://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html)
Unfortunately, in Modules/getpath.c:
static void
calculate_path(void)
{
[...]
char *_rtpypath = Py_GETENV(PYTHONPATH); /* XXX use wide version on
Windows */
wchar_t *rtpypath = NULL;
wchar_t *home = Py_GetPythonHome();
char *_path = getenv(PATH);
So 3 potential getenv calls in quick succession, meaning _rtpypath and home can
become junk before they get used and Python crashes before it can start up (it
becomes unable to find the site module).
Unfortunately it looks like the assumption that getenv pointers will remain
safe forever is used in a few places in python.
Explicit notes on the correct use of getenv:
https://www.securecoding.cert.org/confluence/display/seccode/ENV34-C.+Do+not+store+pointers+returned+by+certain+functions
Someone's apparently seen this before (but didn't report it?) -
http://sourceforge.net/p/edk2/mailman/edk2-devel/thread/66bd57653246d24e9698b0a6509545a86ddb8...@orsmsx109.amr.corp.intel.com/
--
components: Interpreter Core
messages: 229788
nosy: aidanhs
priority: normal
severity: normal
status: open
title: Posix getenv makes no guarantee of lifetime of returned string
type: crash
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22689
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
STINNER Victor added the comment: test_pause_reading.py: manual test for pause reading. It should pause reading stdout pipe 4 times on UNIX. -- Added file: http://bugs.python.org/file36991/test_pause_reading.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
STINNER Victor added the comment: Ok, I just missed completly the Guido posted a patch before me... Maybe we wrote the patch at the same time, I didn't refresh my browser :-) Victor, do you think this needs a unittest? It seems kind of difficult to test for whether memory fills up (the machine may get wedged if it does :-). Since the pause/resume thing is very tricky, it would be much better to have a unit test. We can probably mock a lot of things, we just need to check that the stream reader calls transport.pause_reading() when feed_data() is called with too much data. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22689] Posix getenv makes no guarantee of lifetime of returned string
Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo versions: -Python 3.2, Python 3.3, Python 3.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22689 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
wabu added the comment: On 21.10.2014 22:41, Guido van Rossum wrote: Guido van Rossum added the comment: Victor, do you think this needs a unittest? It seems kind of difficult to test for whether memory fills up (the machine may get wedged if it does :-). You could setup a the subprocess with asyncio.async(asyncio.create_subprocess_exec(...)) and then let the asyncio loop run for a limited time with loop.run_until_complete(asyncio.sleep(.1)), watching carefully for higher memory usage after each sleep. But still it's difficult to create a reliable unit-test with this ... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22687] horrible performance of textwrap.wrap() with a long word
Changes by Ben Roberts bjr.robe...@gmail.com: -- nosy: +roippi ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22687 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22689] Posix getenv makes no guarantee of lifetime of returned string
Aidan Hobson Sayers added the comment: In case it matters, I'm compiling using Emscripten which implements getenv like so: https://github.com/kripken/emscripten/blob/1.25.2/src/library.js#L3323 (I personally think it's a bizarre way to do it, but technically I think it's ok?) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22689 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22685] memory leak: no transport for pipes by create_subprocess_exec/shell
Guido van Rossum added the comment: Victor, your fix is identical to mine except for the variable names. I like your version better. :-) So now it's just about the unittest. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22685 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22690] importing Gtk breaks strptime
New submission from Sigz: I usually convert date string to time with strptime formating. I noticed importing Gtk breaks strptime parsing for abbreviated month name and weekday name : import time time.strptime(Mon, 20 Oct 2014 08:00:32 +, %a, %d %b %Y %H:%M:%S %z) time.struct_time(tm_year=2014, tm_mon=10, tm_mday=20, tm_hour=8, tm_min=0, tm_sec=32, tm_wday=0, tm_yday=293, tm_isdst=-1) from gi.repository import Gtk time.strptime(Mon, 20 Oct 2014 08:00:32 +, %a, %d %b %Y %H:%M:%S %z) Traceback (most recent call last): File stdin, line 1, in module File /usr/lib/python3.4/_strptime.py, line 494, in _strptime_time tt = _strptime(data_string, format)[0] File /usr/lib/python3.4/_strptime.py, line 337, in _strptime (data_string, format)) ValueError: time data 'Mon, 20 Oct 2014 08:00:32 +' does not match format '%a, %d %b %Y %H:%M:%S %z' Is there a workaround for Gtk + time ? -- components: Extension Modules messages: 229794 nosy: sigzegv priority: normal severity: normal status: open title: importing Gtk breaks strptime type: crash versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22690 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22690] importing Gtk breaks strptime
R. David Murray added the comment: What locale are you using? Is it possible importing Gtk makes that locale operative? (In which case, that would be a bug in either Gtk or its wrapper). -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22690 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22691] A Better Help File
New submission from James: Just the General Help that is in Python, doesn't really help. Here's what would help, if every Module, had an example in code of how it was used instead of the Trees. I mean, word trees, well that's what the writing reminds me of, is word trees like you'd produce in an English class. But, a short working piece of code, does allot more for me, than a lecture, you'd have to take a course to understand because of it's use of it's own writing style. It doesn't help to type help() and hit enter. -- assignee: docs@python components: Documentation messages: 229796 nosy: FCK, docs@python priority: normal severity: normal status: open title: A Better Help File type: enhancement versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22691 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22691] A Better Help File
James added the comment: Just the General Help that is in Python, doesn't really help. Here's what would help, if every Module, had an example in code of how it was used instead of the Trees. I mean, word trees, well that's what the writing reminds me of, is word trees like you'd produce in an English class. But, a short working piece of code, does allot more for me, than a lecture, you'd have to take a course to understand because of it's use of it's own writing style. It doesn't help to type help() and hit enter. Honestly, the old old version of Quick Basic, that once came with every version of windows back in the 90s. It's help, is a template that I think Python, should follow. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22691 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
