[issue26499] http.client.IncompleteRead from HTTPResponse read after part reading file

[Martin Panter]

Martin Panter added the comment:

Thanks for the report and the patch. It looks okay as far as it goes, but I 
think there are other related bugs:

## read1() doesn’t update length either ##

>>> handle = urlopen("https://www.python.org/;)
>>> len(handle.read1())
7374
>>> handle.read()
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/proj/python/cpython/Lib/http/client.py", line 462, in read
s = self._safe_read(self.length)
  File "/home/proj/python/cpython/Lib/http/client.py", line 614, in _safe_read
raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(39583 bytes read, 7374 more expected)

## readline() and read1() blindly read beyond Content-Length ##

>>> conn = HTTPSConnection("www.python.org")
>>> conn.request("GET", "/")
>>> resp = conn.getresponse()
>>> resp.readlines()  # Hangs until the connection is closed

I wonder if anyone has considered implementing HTTPResponse by wrapping or 
subclassing BufferedReader; that way you get well-tested readline() etc 
functionality for free. The HTTP protocol (Connection: close, Content-Length, 
chunked decoding) could be handled by an internal RawIOBase.readinto() method.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21042] ctypes.util.find_library() should return full pathname instead of filename in linux

[Martin Panter]

Martin Panter added the comment:

It looks like the ldconfig parsing isn’t working for some ABIs. See the 
following buildbot failures:

* Cortex A15 armv7l: 
http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.x/builds/3721/steps/test/logs/stdio
* ppc64le POWER8: 
http://buildbot.python.org/all/builders/PPC64LE%20Fedora%203.x/builds/769/steps/test/logs/stdio

I presume there are other flags in the ABI string, perhaps like 
(libc6,hard-float) on ARM. The code that produces these strings seems to be 
here: 
.

Looking closer at the find_library() implementation, I also realize it is not 
correct to say an absolute path is always returned. If the ldconfig check 
fails, it falls back to _get_soname(). I think we have the following options:

* Adjust the documentation to say an absolute path is only returned if the 
ldconfig call works
* Figure out how to get the right ldconfig flags for ARM and PPC
* Use the old parsing code on ARM and PPC platforms, and only return a full 
path on x86 or other platforms
* Revert the whole change

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21910] File protocol should document if writelines must handle generators sensibly

[Dan Haffey]

Dan Haffey added the comment:

+1, I just lost an hour-plus compute job to this. It sure violates POLA. I've 
been passing large generators to file.writelines since about as long as 
generators have existed, so I never would have guessed that a class named 
"StreamWriter" of all things wouldn't, you know, stream its writelines argument.

--
nosy: +dhaffey

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26515] Update extending/embedding docs to new way to build modules in C

[Brett Cannon]

New submission from Brett Cannon:

https://docs.python.org/3/extending/extending.html#a-simple-example uses 
PyModule_Create() instead of PyModuleDef_Init().

--
assignee: docs@python
components: Documentation
messages: 261398
nosy: brett.cannon, docs@python, eric.snow, ncoghlan
priority: normal
severity: normal
stage: needs patch
status: open
title: Update extending/embedding docs to new way to build modules in C
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26508] Infinite crash leading to DoS

[Brett Cannon]

Brett Cannon added the comment:

Victor's point is that this isn't a bug because you are manipulating the C API 
to make this occur. That means there is no safety guarantee and thus this is 
not a bug but simply a mis-use of the C API.

--
nosy: +brett.cannon
resolution: works for me -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26508] Infinite crash leading to DoS

[pablo sacristan]

Changes by pablo sacristan :


--
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26508] Infinite crash leading to DoS

[pablo sacristan]

pablo sacristan added the comment:

New content for crash.py:

import ctypes, struct, sys, os
inner = ()
outer = (inner,)
c_outer = (ctypes.c_char * sys.getsizeof(outer)).from_address(id(outer))
inner_index = c_outer[:].find(struct.pack('P', id(inner)))
c_outer[inner_index:inner_index+struct.calcsize('P')] = struct.pack('P', 
id(outer))
print outer

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26508] Infinite crash leading to DoS

[pablo sacristan]

pablo sacristan added the comment:

Then it is no bug that it crashes python? You don't have to put it in a loop, 
but by looping it I am adding the part that makes it take more time but will 
keep on crashing infinitely, take away the while loop but not what it has 
inside and then also delete the os.system() and you get python to crash once. 
Is a crash not a bug?

--
resolution: not a bug -> works for me

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18156] Add an 'attr' attribute to AttributeError

[Xiang Zhang]

Xiang Zhang added the comment:

The concerns mentioned by haypo seems to have existed in PEP473.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21042] ctypes.util.find_library() should return full pathname instead of filename in linux

[Roundup Robot]

Roundup Robot added the comment:

New changeset 3092cf163eb4 by Martin Panter in branch 'default':
Issue #21042: Return full path in ctypes.util.find_library() on Linux
https://hg.python.org/cpython/rev/3092cf163eb4

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26439] ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX)

[Michael Felt]

Michael Felt added the comment:

The patch works when installed on top of pre-compiled version (e.g., copy the 
two files to /opt/lib/python2.7/ctypes/

but there seems to be a nasty side-effect when trying to build.

make completes, but make install fails during a compile step.

Please assist with howto see how/why ctypes/util is impacting this part.

And - for my first adventure into python please provide the needed (expected) 
feedback.

Michael

make install DESTDIR=/var/aixtools/aixtools/python/2.7.11.2 > 
.buildaix/install.out

Listing /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax ...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/__init__.py ...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/_exceptions.py 
...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/expatreader.py 
...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/handler.py ...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/saxutils.py ...
Compiling 
/var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/xmlreader.py 
...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xmllib.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xmlrpclib.py 
...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/zipfile.py 
...

So, zipfile.py seems to exit non-zero. :(

--
keywords: +patch
Added file: http://bugs.python.org/file42094/python.Lib.ctypes.160309.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26466] could not build python 2.7.11 on AIX

[Michael Felt]

Michael Felt added the comment:

Hope it is okay for me to close this. As it is working with the additional flag 
to configure.

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26513] platform.win32_ver() broken in 2.7.11

[Steve Dower]

Steve Dower added the comment:

Nothing has changed here as far as I'm aware.

The part of platform.py in use here has potentially different behavior if you 
have pywin32 installed. Is that the case? Maybe you updated that as well or 
haven't installed it for 2.7.11 yet?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26514] Object defines '__ne__' as 'not __eq__' if '__ne__' is not implemented

[Eryk Sun]

Changes by Eryk Sun :


--
resolution:  -> works for me
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25458] ftplib: command response shift - mismatch

[Peter Pan]

Peter Pan added the comment:

The problem in my example is ftplib reports a "226" response to command "NOOP" 
which is nonsense. ftplib received "226" before "ftp.sendcmd('NOOP')" was 
called.

Since "transfercmd()" returns a socket, ftplib was planned to allow for manual 
transfer socket handling, but it is currently unable to handle the asynchronous 
responses from the server when the transfer is done. This is a bug or design 
error.


Multiple changes are needed to support manual transfer socket handling. I 
suggest the following:

Since asynchronous responses from the server are possible on the command 
socket, "set_debuglevel(1)" must report them at once, but this would require 
multithreading. A good compromise is to debug print and clear any buffered 
status right before sending the next command.
We also need a method to check the last status code, in order to know the 
result of the last manual transfer. ftplib has to store it separately as an 
attribute.


New attribute
-

this.last_status_code = None #has no effect to any command or debug output
this.last_status_message

New internal method
---

#loop: look for buffered status response; if present, print debug and assign 
this.last_status = buffer.pop()
.unbuffer():
 ...


New user methods


#Set last status to None so we can use "get_last_status" to check/poll for the 
next one.
.clear_last_status():
 this.last_status_code = None
 this.last_status_message = ""
 return

#Return the last status received from the server; if there is one on the 
buffer, unbuffer and return that.
.get_last_status():
 this.unbuffer()
 return [this.last_status_code, this.last_status_message]

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25228] Regression in cookie parsing with brackets and quotes

[Collin Anderson]

Collin Anderson added the comment:

It should be safe to hard split on semicolon. `name="some;value"` is not valid, 
even though it's quoted. I think raw double quotes, commas, semicolons and 
backslashes are _always_ invalid characters in cookie values.

>From https://tools.ietf.org/html/rfc6265:

{{{
 cookie-value  = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
 cookie-octet  = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
   ; US-ASCII characters excluding CTLs,
   ; whitespace DQUOTE, comma, semicolon,
   ; and backslash
}}}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26514] Object defines '__ne__' as 'not __eq__' if '__ne__' is not implemented

[Josh Rosenberg]

Josh Rosenberg added the comment:

It's not exactly what was asked for, but it's actually better (in that the 
__ne__ default implementation handles NotImplemented correctly). Per the docs 
at https://docs.python.org/3/reference/datamodel.html#object.__eq__ :

"By default, __ne__() delegates to __eq__() and inverts the result unless it is 
NotImplemented."

This feature was never backported to 2.7, but it's existed in one form or 
another for the entire 3.x line.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26514] Object defines '__ne__' as 'not __eq__' if '__ne__' is not implemented

[Ethan Furman]

Ethan Furman added the comment:

The following behaviour is from 3.5:

--> class Huh:
...   def __eq__(self, other):
... return other == 'blah'
... 
--> h = Huh()
--> h == 'ew'
False
--> h != 'ew'
True
--> h == 'blah'
True
--> h != 'blah'
False

Which seems to be exactly what you want.  Do you have a counter-example?

--
nosy: +ethan.furman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26514] Object defines '__ne__' as 'not __eq__' if '__ne__' is not implemented

[Marc Guetg]

New submission from Marc Guetg:

I propose to change __ne__ of `object` in the following form:

class NewObject(object):
def __ne__(self, other):
return not self.__eq__(other)

Currently overwriting the `__eq__` method requires also overwriting `__ne__`. 
In a vast majority of cases this results in some boilerplate code as:

(a == b) ^ (a != b) == True

to reduce surprises. Changing the default behavior still allows for the limited 
number of use cases where we want to implement __ne__ differently.


In short I propose the same behavior than __str__ and __repr__ have for __eq__ 
and __ne__.
(https://docs.python.org/3/reference/datamodel.html#object.__str__)

--
components: Interpreter Core
messages: 261385
nosy: Marc Guetg
priority: normal
severity: normal
status: open
title: Object defines '__ne__' as 'not __eq__' if '__ne__' is not implemented
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26513] platform.win32_ver() broken in 2.7.11

[Florian Roth]

New submission from Florian Roth:

Using Python 2.7.9 and 2.7.10 (32bit) on a Windows Server 2008 R2 64bit System 
platform.win32_ver() shows the following, which is correct:

('2008ServerR2', '6.1.7601', 'SP1', u'Multiprocessor Free')
 
Using Python version 2.7.11 (32bit) on the same Windows 2008 R2 server shows:

('7', '6.1.7601', 'SP1', u'Multiprocessor Free')

which is wrong. 
Platform OS detection seems to be broken with version 2.7.11.

--
components: Windows
messages: 261384
nosy: Florian Roth, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: platform.win32_ver() broken in 2.7.11
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26512] Vocabulary: Using "integral" in library/stdtypes.html

[Alexander Belopolsky]

Alexander Belopolsky added the comment:

In the contexts that you mentioned, "integral" is a synonym of "integer."

--
nosy: +belopolsky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15068] fileinput requires two EOF when reading stdin

[Roundup Robot]

Roundup Robot added the comment:

New changeset 88d6742aa99a by Serhiy Storchaka in branch '2.7':
Issue #15068: Avoid creating a reference loop in fileinput.
https://hg.python.org/cpython/rev/88d6742aa99a

New changeset a0de41b46aa6 by Serhiy Storchaka in branch '3.5':
Issue #15068: Avoid creating a reference loop in fileinput.
https://hg.python.org/cpython/rev/a0de41b46aa6

New changeset 27c9849ba5f3 by Serhiy Storchaka in branch 'default':
Issue #15068: Avoid creating a reference loop in fileinput.
https://hg.python.org/cpython/rev/27c9849ba5f3

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15068] fileinput requires two EOF when reading stdin

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ah, thanks Martin. I forgot that assigning an attribute to a bound method 
creates a reference loop.

This can be fixed without performance lost by using a clever trick.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26465] Upgrade OpenSSL shipped with python installers

[Steve Dower]

Changes by Steve Dower :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26512] Vocabulary: Using "integral" in library/stdtypes.html

[Julien]

New submission from Julien:

o/

TL;DR: I think the various usages of "Integral" in 
https://docs.python.org/3.5/library/stdtypes.html#numeric-types-int-float-complex
 are either wrong or too hard to understand and should be rewriten as "int".


While translating the documentation I found the sentence "x truncated to 
Integral" in 
https://docs.python.org/3.5/library/stdtypes.html#numeric-types-int-float-complex
 for the `math.floor(x)` method.

By "Integral" I assume you're speaking about the ABC `numbers.Integral`, whish 
is already not clear, but the capital I helps.

According to pydoc and PEP3141, the `numbers.Integral` ABC inherits from 
Rational, which inherits from Real which inherits from Complex which inherits 
from Number. So number is the broader ABC and integral the opposite, the 
representation of an integer.

I do _NOT_ assume that someone reading the stdtypes.html section is aware of 
those ABCs, so I don't think it's nice for them to use them without linking to 
them.

So I infer than "x truncated to integral" means "x truncated to the nearest 
integer". Which is far more readable.

Two lines after, I found "The greatest integral float <= x" for `math.floor(x)` 
which is less readable, no capital I to `integral` and what is an `integral 
float` ?

Also the documentation of `math.floor` states an `int` is returned (as the doc 
for math.trunc states an Integral is returned).

So there's two possibilities here:

 - The doc uses vocabulary and phrase structures I don't understand and 
"integral float" actually mean something.
 - The doc is not clear

In both cases, I suggest to either transform "integral" usages to a link to 
:class:`numbers.Integral` or simply speaking of "int" here.

Context: I'm translating the documentation in French, I achieved 
`tutorial.html` and `functions.html` and it's until here the hardest sentence 
to understand (others were simply wrong like in 
http://bugs.python.org/issue26029) or understandable, so even if those 
structures are semantically and gramatically correct, they're clearly one of 
the hardest to understand, which is sad as we're only speaking of rounding 
numbers. I mean, if there's a goal to sand away spikes in doc readability, this 
one is a big one.

So, can an english native speaker can enlighten me about this ?

Bests,

--
assignee: docs@python
components: Documentation
messages: 261380
nosy: docs@python, sizeof
priority: normal
severity: normal
status: open
title: Vocabulary: Using "integral" in library/stdtypes.html
type: enhancement
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15068] fileinput requires two EOF when reading stdin

[Martin Panter]

Martin Panter added the comment:

It seems this change is causing some (intermittent?) buildbot failures in 2.7:

http://buildbot.python.org/all/builders/s390x%20RHEL%202.7/builds/273/steps/test/logs/stdio

==
FAIL: test_saveall (test.test_gc.GCTests)
--
Traceback (most recent call last):
  File 
"/home/dje/cpython-buildarea/2.7.edelsohn-rhel-z/build/Lib/test/test_gc.py", 
line 199, in test_saveall
self.assertEqual(gc.garbage, [])
AssertionError: Lists differ: [

Diff is 1461 characters long. Set self.maxDiff to None to see it.

==
FAIL: test_create_read (test.test_csv.TestLeaks)
--
Traceback (most recent call last):
  File 
"/home/dje/cpython-buildarea/2.7.edelsohn-rhel-z/build/Lib/test/test_csv.py", 
line 1103, in test_create_read
self.assertEqual(gc.garbage, [])
AssertionError: Lists differ: [

Diff is 1461 characters long. Set self.maxDiff to None to see it.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26465] Upgrade OpenSSL shipped with python installers

[Steve Dower]

Steve Dower added the comment:

I didn't have nasm on the path when preparing, so it made empty asm files.

I've committed the correctly build ASM files, and I'll trigger the buildbots to 
clean and patch the prepare script to fail if nasm is not available.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26456] import _tkinter + TestForkInThread leaves zombie with stalled thread

[Martin Panter]

Martin Panter added the comment:

The change to the test seems to have the desired effect. The buildbots are no 
longer timing out (tests are failing for other reasons).

--
resolution:  -> fixed
stage: commit review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26465] Upgrade OpenSSL shipped with python installers

[Zachary Ware]

Zachary Ware added the comment:

Steve: This broke 64-bit Windows, I haven't tried digging into what's going on 
yet.

See: 
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/7331/steps/compile/logs/stdio

--
resolution: fixed -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25933] Unhandled exception (TypeError) with ftplib in function retrbinary/retrlines causes inoperable behavior without crashing

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

It looks to me that forcing self.voidresp() in case the callback raises an 
exception is a bad idea: it may easily end up hanging if the server is not 
supposed to send a response at that point (see: the transfer is not finished).

As for the issue per se: I think there's nothing wrong with the current 
behavior except perhaps we could provide a better error message. But in order 
to do that we should check file mode before hand and relying on the file mode 
is also not a good idea IMO because the file object can be anything.

--
nosy: +giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25458] ftplib: command response shift - mismatch

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

I'm not sure I understood the problem. Perhaps if you provide a patch that 
would make things more clear and you'd have more chances to see this issue 
fixed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25459] EAGAIN errors in Python logging module

[Henrique Andrade]

Henrique Andrade added the comment:

Vinay, apologies for the long time you took me to respond (the notification
was stuck in my Spam folder, thanks Gmail!). Indeed, both of your comments
are spot on. In my particular case, I ended up implementing what your first
suggestion alludes to (a retry capped by a maximum).

The reason for my reporting this is because the existing (mainlined)
implementation has shortcomings, which might be affecting others out there.

I contacted the module developer, but no response. So assuming we have a
consensus from whomever "owns" the module, I can volunteer to do a proper
patch.

On the issue of maximum retries, I would go with a reasonable
(configurable) max as well as a (configurable) back-off sleep.

On Sun, Feb 7, 2016 at 1:57 PM, Vinay Sajip  wrote:

>
> Vinay Sajip added the comment:
>
> The problem with your suggested fix is that if you keep getting EAGAIN,
> the handler will appear to hang. If you want to suggest that maybe we limit
> the number of tries, then what number of retries would you pick, and why?
> Also, a caller might want to know about EAGAIN, so I'm not sure you'd want
> to (effectively) swallow it in the handler.
>
> Perhaps a different strategy (than just trying for ever) would be better.
> If you know the application is unusual in having a SIGPIPE handler and you
> think this is a contributory factor, why not just subclass the handler, add
> a more resilient emit() and use that in your application?
>
> --
> status: open -> pending
>
> ___
> Python tracker 
> 
> ___
>

-- 

Henrique Andrade | +1-530-426-2123 | h...@unscrambl.com |
http://unscrambl.com

--
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17446] doctest test finder doesnt find line numbers of properties

[Michael Cuthbert]

Michael Cuthbert added the comment:

looks like we're stuck on a style change (backslash to parens; ironic: I chose 
backslash to match surrounding code; I never use them myself). tuxtimo - could 
you fix this? (or I'll do once the move to github is done).  Thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14102] argparse: add ability to create a man page

[Ben Finney]

Changes by Ben Finney :


--
versions: +Python 3.6 -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14102] argparse: add ability to create a man page

[Ben Finney]

Ben Finney added the comment:

Oz Tiram wrote:

> I doubt if the correct place for formatting a man page should be in 
> argparse.py itself. My solution is an extension of Andial's brecht solution

Thank you for that, Oz.

I see an easy seam to refactor this work into separate concerns.

The formatting of a man page from the parser for the command, can be coupled to 
the ‘argparse’ module since it is dependent only on an ArgumentParser plus some 
extra properties for populating the man page.

The building of one man page should not, IMO, be so tightly coupled to a Python 
distribution. There may be several different commands installed by one 
distribution. Each man page's fields should mostly come from the parser for the 
command, not from the distribution metadata. Instead, the build process should 
be told which parser to use for generating each of the man pages.

I propose to work on Oz's module and produce a man page formatter for exactly 
one ArgumentParser, as an enhancement for ‘argparse’.

--
nosy: +bignose

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25911] Regression: os.walk now using os.scandir() breaks bytes filenames on windows

[Roundup Robot]

Roundup Robot added the comment:

New changeset 757159fb4847 by Serhiy Storchaka in branch '3.5':
Issue #25911: Tring to silence deprecation warnings in bytes path walk tests.
https://hg.python.org/cpython/rev/757159fb4847

New changeset ecf4e51c222f by Serhiy Storchaka in branch 'default':
Issue #25911: Tring to silence deprecation warnings in bytes path walk tests.
https://hg.python.org/cpython/rev/ecf4e51c222f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26263] Serialize array.array to JSON by default

[shakur shams Mullick]

Changes by shakur shams Mullick :


--
nosy: +shakur shams Mullick

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26465] Upgrade OpenSSL shipped with python installers

[Steve Dower]

Changes by Steve Dower :


--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26465] Upgrade OpenSSL shipped with python installers

[Roundup Robot]

Roundup Robot added the comment:

New changeset 195ce1f4f54b by Steve Dower in branch '2.7':
Issue #26465: Update Windows builds to use OpenSSL 1.0.2g.
https://hg.python.org/cpython/rev/195ce1f4f54b

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26465] Upgrade OpenSSL shipped with python installers

[Roundup Robot]

Roundup Robot added the comment:

New changeset 6d7f0421b71f by Steve Dower in branch '3.5':
Issue #26465: Update Windows builds to use OpenSSL 1.0.2g.
https://hg.python.org/cpython/rev/6d7f0421b71f

New changeset e045f26ea74a by Steve Dower in branch 'default':
Issue #26465: Update Windows builds to use OpenSSL 1.0.2g.
https://hg.python.org/cpython/rev/e045f26ea74a

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26387] Crash calling sqlite3_close with invalid pointer

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This may be similar to issue26494.

As for the patch, _pysqlite_seterror() shouldn't be called when GIL is 
released. And setting self->db not guarded with GIL looks dangerous.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15068] fileinput requires two EOF when reading stdin

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Committed in changesets 5fbd16326353 (2.7), 9ead3a6c5f81 (3.5), and 
fefedbaac640 (default). Due to SMTP failure there is no Roundup report.

Warnings are not emitted in maintained releases.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Add open_file_descriptor parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)

[Hristo Venev]

Hristo Venev added the comment:

Here is the OFD patch, I'll open another issue for the overflow bug.

--
versions: +Python 3.6
Added file: 
http://bugs.python.org/file42093/0002-fcntl-Support-Linux-open-file-descriptor-locks.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26511] Add link to id() built-in in comparison operator documentation for "is"

[Mike Vertolli]

New submission from Mike Vertolli:

Here's the doc: https://docs.python.org/2/reference/expressions.html#is

It would be great if this line:
"The operators is and is not test for object identity: x is y is true if and 
only if x and y are the same object. x is not y yields the inverse truth 
value.[7]"

Could link to the id() builtin doc:
https://docs.python.org/2/library/functions.html#id

One way to rephrase this could be:
"The operators is and is not test for object identity: x is y is true if and 
only if x and y are the same object or id(x) == id(y). x is not y yields the 
inverse truth value.[7]"

And, the id's can be hyperlinks.

It would also be nice if that reference linked to a page on integer caching in 
Python.

--
assignee: docs@python
components: Documentation
messages: 261364
nosy: Mike Vertolli, docs@python
priority: normal
severity: normal
status: open
title: Add link to id() built-in in comparison operator documentation for "is"
type: enhancement
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26387] Crash calling sqlite3_close with invalid pointer

[Filipp Andjelo]

Filipp Andjelo added the comment:

Hi Serhiy,

unfortunately not, we have one heavy application which is crashing with the 
given error non deterministically after about 25 hours of running on 40 CPUs 
with 280GiB memory (!) in use. We already made all kind of test to get it 
reproducible "in a small", but it just doesn't crash. If I have more 
information, I'll post it here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Fragmentation of the heap memory in the Python parser

[A. Skrobov]

A. Skrobov added the comment:

@Serhiy: if your build is 32-bit, then every node is half the size, as it 
mostly consists of pointers.

The amount of heap fragmentation can also depend on gcc/glibc version.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25911] Regression: os.walk now using os.scandir() breaks bytes filenames on windows

[Roundup Robot]

Roundup Robot added the comment:

New changeset 19a3e0e664af by Serhiy Storchaka in branch '3.4':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/19a3e0e664af

New changeset f9e22717722d by Serhiy Storchaka in branch '3.5':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/f9e22717722d

New changeset da020e408c7f by Serhiy Storchaka in branch 'default':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/da020e408c7f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23808] Symlink to directory on Windows 8

[Roundup Robot]

Roundup Robot added the comment:

New changeset 19a3e0e664af by Serhiy Storchaka in branch '3.4':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/19a3e0e664af

New changeset f9e22717722d by Serhiy Storchaka in branch '3.5':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/f9e22717722d

New changeset da020e408c7f by Serhiy Storchaka in branch 'default':
Issues #23808, #25911: Trying to fix walk tests on Windows.
https://hg.python.org/cpython/rev/da020e408c7f

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26510] [argparse] Add required argument to add_subparsers

[Memeplex]

Changes by Memeplex :


--
nosy: +bethard

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26510] [argparse] Add required argument to add_subparsers

[Memeplex]

Changes by Memeplex :


--
title: Add required argument to add_subparsers -> [argparse] Add required 
argument to add_subparsers

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26387] Crash calling sqlite3_close with invalid pointer

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you please provide a test Filipp?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26510] Add required argument to add_subparsers

[Memeplex]

New submission from Memeplex:

It's useful in combination with dest. Currently you have to set required as an 
attribute. This is not only inconsistent but not even documented as a valid 
usage (in general, it's not clear from the docs whether attribute setting -vs 
argument passing- is a valid usage).

--
components: Library (Lib)
messages: 261358
nosy: memeplex
priority: normal
severity: normal
status: open
title: Add required argument to add_subparsers
type: enhancement
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26502] traceback.extract_tb breaks compatibility by returning FrameSummary

[Berker Peksag]

Berker Peksag added the comment:

This is basically a duplicate of issue 25573. We need to add the examples in 
traceback module documentation to test_traceback.py.

> Should __len__ be added as well?

IMO, the new FrameSummary API (and its friends) shouldn't change the return 
value of the existing traceback functions like extract_tb() so I'd say yes.

--
nosy: +berker.peksag
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2202] urllib2 fails against IIS 6.0 (No support for MD5-sess auth)

[Mathieu Dupuy]

Mathieu Dupuy added the comment:

Much better indeed. Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26502] traceback.extract_tb breaks compatibility by returning FrameSummary

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +rbcollins

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18156] Add an 'attr' attribute to AttributeError

[STINNER Victor]

STINNER Victor added the comment:

> try:
>   cls.meth()
> except AttributeError as exc:
>   if exc.attr != 'meth':
> raise

What if cls.__getattr__('meth') calls a completly unrelated function which also 
raises AttributeError(attr='meth') but on a different object? I would also 
expect an "obj" attribute, a reference to the object which doesn't have 
attribute.

But AttributeError can be raised manually without setting attr and/or obj 
attribute. So the best test would look to something like:

if exc.obj is cls and exc.attr is not None and exc.attr != 'meth':
raise

The test is much more complex than expected :-/ Maybe it's simpler to split the 
code to first get the bounded method?

try:
   meth = cls.meth
except AttributeError:
   ...
meth()

Or check first if cls has the attribute 'meth' with hasattr(cls, 'meth')?

--

About attr/obj attribute not set, an alternative is to add a new 
BetterAttributeError(obj, attr) exception which requires obj and attr to be 
set, it inherits from AttributeError.

class BetterAttributeError(AttributeError):
   def __init__(self, obj, attr):
   super().__init__('%r has no attribute %r' % (obj, attr)
   self.obj = obj
   self.attr = attr

It would allow a smoother transition from "legacy" AttributeError to the new 
BetterAttributeError.

The major issue with keeping a strong reference to the object is that Python 3 
introduced Exception.__traceback__ which can create a reference cycle if an 
exception is stored in a local variable somewhere in the traceback. It's a 
major pain point in asyncio. In asyncio, the problem is more likely since 
exceptions are stored in Future objects to be used later.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Add open_file_descriptor parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Add open_file_descriptor parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)

[STINNER Victor]

Changes by STINNER Victor :


--
title: Add F_OFD_SETLK, etc support to fcntl.lockf -> Add open_file_descriptor 
parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Add F_OFD_SETLK, etc support to fcntl.lockf

[STINNER Victor]

STINNER Victor added the comment:

Strange, Rietveld (the tool linked with the [review] button) doesn't show the 
change on the Modules/fcntlmodule.c file.

By the way, I suggest to not include the Modules/clinic/fcntlmodule.c.h change 
in the patch.

--
title: Please add F_OFD_SETLK, etc support to fcntl.lockf -> Add F_OFD_SETLK, 
etc support to fcntl.lockf

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Please add F_OFD_SETLK, etc support to fcntl.lockf

[STINNER Victor]

STINNER Victor added the comment:

> This implements the open_file_descriptor argument and fixes a bug with 
> converting to int when off_t is 64-bit but long is 32-bit.

Please extract the fix into a different patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26470] Make OpenSSL module compatible with OpenSSL 1.1.0

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26493] Bad formatting in WinError 193 when using subprocess.check_call

[STINNER Victor]

STINNER Victor added the comment:

See also issue #25585: "DLL load failed: %1 is not a valid Win32 application" 
message of ImportError exception. Hum, so maybe a function is better, since 
ImportError doesn't inherit from OSError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25585] Bad path leads to: ImportError: DLL load failed: %1 is not a valid Win32 application.

[STINNER Victor]

STINNER Victor added the comment:

See also issue #26493.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26493] Bad formatting in WinError 193 when using subprocess.check_call

[STINNER Victor]

STINNER Victor added the comment:

> A new issue should be raised to fix the FormatMessage calls in the standard 
> library that mistakenly leave out FORMAT_MESSAGE_IGNORE_INSERTS.

Do you suggest to modify OSError constructor to modify the call to 
FormatMessageW(): don't pass the FORMAT_MESSAGE_IGNORE_INSERTS flag?

I prefer "%1 is not a valid Win32 application" message than "".

Currently, PyErr_SetFromErrnoWithFilenameObjects() doesn't pass any argument to 
FormatMessageW(). The problem is that it looks like the expected argument 
depends a lot of the failing system call. I don't think that it's ok to always 
pass the filename (or two filenames when we get two filenames, ex: os.rename) 
to FormatMessageW().

For example, _winapi.CreateProcess() doesn't pass any filename to OSError 
constructor.


> It would be possible for subprocess to replace "%1" with the filename parsed 
> from the command line and then re-raise the exception.

I like the idea of formatting the error message in the subprocess module. IMHO 
it's much safer to reformat the error message from the function which raises 
the exception, since the function knows the system call, has all parameters to 
the system call, and may expect some specific system calls.

There is no need to re-raise the exception: the "strerror" attribute contains 
the error message and it can be modified.

We need an helper function or add a new method to the OSError class. Example of 
method:

def reformat_strerror(self, *args):
   sef.strerror = FormatMessageW(..., self.strerror, ..., args)

For this specific issue, I don't know if reformat_strerror() should be called 
from subprocess.py or CreateProcess function of Modules/_winapi.c.

I understand that we should call reformat_strerror() with application_name 
(first parameter of CreateProcess, "executable" in subprocess), but only if the 
error message is 193.

Since there are "a lot" of error messages and a lot of functions calling system 
calls, I don't think that it will be possible to handle all possible error 
messages in all Python functions. I suggest to only call reformat_strerror() 
when an user complains, only promise best effort ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Fragmentation of the heap memory in the Python parser

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

On my computer peak memory usage in non-debug build is about 450 MB. Peak 
memory usage in debug build is about 730 MB. I'm not sure this is due to memory 
fragmentation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26494] Double deallocation on iterator exhausting

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Fixed test for OrderedDict. Tests for list and tuple are moved up to 
seq_tests.py.

--
Added file: http://bugs.python.org/file42092/free_after_iterating_3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26493] Bad formatting in WinError 193 when using subprocess.check_call

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Fragmentation of the heap memory in the Python parser

[STINNER Victor]

STINNER Victor added the comment:

libapr library of Apache is designed to group all memory allocations required 
to handle an HTTP request. It is possible to release *all* memory allocations 
of the request at once.

It looks like the the "pool" object:

* https://apr.apache.org/docs/apr/2.0/group__apr__pools.html
* https://apr.apache.org/docs/apr/2.0/group__apr__allocator.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Fragmentation of the heap memory in the Python parser

[STINNER Victor]

STINNER Victor added the comment:

My misc notes about memory fragmentation: 
https://haypo-notes.readthedocs.org/heap_fragmentation.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Fragmentation of the heap memory in the Python parser

[STINNER Victor]

Changes by STINNER Victor :


--
title: Out of memory, trying to parse a 35MB dict -> Fragmentation of the heap 
memory in the Python parser

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Out of memory, trying to parse a 35MB dict

[STINNER Victor]

STINNER Victor added the comment:

> So, apparently, it's not the nodes themselves taking up a disproportionate 
> amount of memory -- it's the heap getting so badly fragmented that 89% of its 
> memory allocation is wasted.

Yeah, the Python parser+compiler badly uses the memory allocator. See my 
"benchmark" for the memory allocator: python_memleak.py.

The classical pattern of memory fragmentation is:

* allocate a lot of small objects
* allocate a few objects
* allocate more small objects
* free *all* small objects

All objects must allocated on the heap, not mmap(). So the maximum size of a 
single object must be 128 KB (usual threshold used in malloc() to switch 
beetween the heap memory and mmap).

We can try various hacks to reduce the fragmentation, but IMHO the only real 
fix is to use a different memory allocator for the compiler and then free 
everything allocated by the parser+compiler at once.

We already have an "arena" memory allocator: Include/pyarena.h, 
Python/pyarena.c. It is already used by the parser+compiler, but it's only used 
for AST objects in practice. The parser uses the PyMem allocator API (ex: 
PyMem_Malloc).

--
nosy: +haypo
Added file: http://bugs.python.org/file42091/python_memleak.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26492] Exhausted array iterator should left exhausted

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

TODO: After resolving issue26494 add similar test for array.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26494] Double deallocation on iterator exhausting

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

TODO: After resolving issue26492 add the test for array.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26492] Exhausted array iterator should left exhausted

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thanks Victor. Answered you comments.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26508] Infinite crash leading to DoS

[STINNER Victor]

STINNER Victor added the comment:

> "construct and print a self-referencing tuple"

I don't understand the use case here. In pure Python, you cannot build a 
self-referecing tuple!?

Yeah, using the C API or by modifying bytes directly in the memory, you can do 
that. But if you start to use the C API, you must be careful. The C API is 
designed for speed, not to prevent bugs.

I don't think that we should fix anything in Python. I don't consider that it's 
a bug in Python.


> run it and wait around 10 sec for it to happen, but once it starts you will 
> be forced to force it to shut because it will keep on crashing infinitely 
> which is extremely annoying,

Basically, crash.py is a fork-bomb: the program starts itself in a loop. It's 
not a bug in Python.

--
nosy: +haypo
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26509] spurious ConnectionAbortedError logged on Windows

[STINNER Victor]

STINNER Victor added the comment:

Attached patch fixes the issue.

--
keywords: +patch
Added file: http://bugs.python.org/file42090/fatal_error_ignore.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26492] Exhausted array iterator should left exhausted

[STINNER Victor]

STINNER Victor added the comment:

I reviewed exhausted_array_iterator2.patch.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26492] Exhausted array iterator should left exhausted

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thanks for suggestion Benjamin.

seq_tests.py provides common tests for list, tuple, UserList and deque. This 
test doesn't work for tuple, since it is not mutable. It doesn't work for 
deque, since it can't be iterated after changing size. Common test for list and 
UserList can be moved to list_tests.py. In any case we need separate tests for 
bytearray, array and general sequence iterator.

I would be happy to not repeat the test, but tests for array and general 
sequence iterator have not much common with test for list.

Updated patch moves list test to list_tests.py (it now works for UserList) and 
reuses in for bytearray.

--
Added file: http://bugs.python.org/file42089/exhausted_array_iterator2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26486] Backport some tests to 2.7

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thanks Martin. This test just doesn't work in 2.7 on all platforms.

--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26486] Backport some tests to 2.7

[Roundup Robot]

Roundup Robot added the comment:

New changeset ff2368d5c1d3 by Serhiy Storchaka in branch '2.7':
Issue #26486: Removed pickle test that doesn't work in 2.7 on 64-bit platform.
https://hg.python.org/cpython/rev/ff2368d5c1d3

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26501] bytes splitlines() method returns strings without decoding

[Paul Moore]

Paul Moore added the comment:

Sigh. I had installed Python 2.7 at some point, and py.exe picked it up by 
default :-(

Isn't it about time that py.exe defaulted to the latest version installed, not 
the latest version of Python 2?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Out of memory, trying to parse a 35MB dict

[A. Skrobov]

A. Skrobov added the comment:

OK, I've now looked into it with a fresh build of 3.6 trunk on Linux x64.

Peak memory usage is about 3KB per node:

$ /usr/bin/time -v ./python -c 'import ast; ast.parse("0,"*100, 
mode="eval")'
Command being timed: "./python -c import ast; ast.parse("0,"*100, 
mode="eval")"
...
Maximum resident set size (kbytes): 3015552
...


Out of the 2945 MB total peak memory usage, only 330 MB are attributable to the 
heap use:

$ valgrind ./python -c 'import ast; ast.parse("0,"*100, mode="eval")'
==21232== ...
==21232== HEAP SUMMARY:
==21232== in use at exit: 3,480,447 bytes in 266 blocks
==21232==   total heap usage: 1,010,171 allocs, 1,009,905 frees, 348,600,304 
bytes allocated
==21232== ...


So, apparently, it's not the nodes themselves taking up a disproportionate 
amount of memory -- it's the heap getting so badly fragmented that 89% of its 
memory allocation is wasted.

gprof confirms that there are lots of mallocs/reallocs going on, up to 21 per 
node:

$ gprof python
Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self  self total   
 time   seconds   secondscalls   s/call   s/call  name
 17.82  0.31 0.31  220 0.00 0.00  PyParser_AddToken
 13.79  0.55 0.242 0.12 0.16  freechildren
 12.64  0.77 0.22 21039125 0.00 0.00  _PyMem_RawMalloc
  6.32  0.88 0.11 17000101 0.00 0.00  PyNode_AddChild
  5.75  0.98 0.10 28379846 0.00 0.00  visit_decref
  5.75  1.08 0.10  104 0.00 0.00  ast_for_expr
  4.60  1.16 0.08 2867 0.00 0.00  collect
  4.02  1.23 0.07 20023405 0.00 0.00  _PyObject_Free
  2.30  1.27 0.04  3031305 0.00 0.00  _PyType_Lookup
  2.30  1.31 0.04  3002234 0.00 0.00  
_PyObject_GenericSetAttrWithDict
  2.30  1.35 0.041 0.04 0.05  ast2obj_expr
  1.72  1.38 0.03 28366858 0.00 0.00  visit_reachable
  1.72  1.41 0.03 12000510 0.00 0.00  subtype_traverse
  1.72  1.44 0.03 3644 0.00 0.00  list_traverse
  1.44  1.47 0.03  3002161 0.00 0.00  _PyObjectDict_SetItem
  1.15  1.49 0.02 20022785 0.00 0.00  PyObject_Free
  1.15  1.51 0.02 15000763 0.00 0.00  _PyObject_Realloc


So, I suppose what needs to be done is to try reducing the number of reallocs 
involved in handling an AST node; the representation of the nodes themselves 
doesn't need to change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26498] _io.so flat namespace

[Larry]

Larry added the comment:

Ned, thank you for your suggestions.  

Just so you know, my HD recently crashed so I rebuilt the system with a new 
SSD.  I was previously running 2.7.9 so I installed 2.7.9 to minimize any 
potential conflicts with my code base.  All was well.

Now it was time to rev Python to 2.7.11 and that was when the problem occurred. 
 ...so I tried 2.7.10 and then 2.7.9.  I hadn't gotten around to virtualenv so 
all these Python installs were on top of each other in /Library/Frameworks/...

Following your suggestion, with 2.7.9 installed, I ran your commands and the 
output was as expected.  I did notice, however, a lot of [benign?] cruft in my 
.profile and .bashrc and thought it best to clean them up.  Ok, still good.  I 
then installed 2.7.11 and reran your commands.  Again, everything worked as 
expected.  Huh?

I restored back to my earlier versions of .profile and .bashrc (cruft 
included), and am still unable to reproduce.  :-/  I've tested and retested.  
All is still well, so sadly I'm unable to report the cause.  I suggest we close 
the report.


Changing the subject slightly...  I noticed that each time we do an install of 
Python, another path is prepended to PATH even if there is already an identical 
entry in .profile.  Did I mention cruft?  ;-)  Hard perhaps, but is there a way 
for the install script to mock execute .profile to devine the resulting PATH, 
inspecting it for the desired path rather than just blindly adding it to 
.profile?  ...just a thot.

Again, thanks for your help.  I'm off and running.

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26039] More flexibility in zipfile interface

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yes, using the lock in write() or writestr() is equally compatible.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com