[issue28165] The 'subprocess' module leaks 4 kiB memory for each thread

[STINNER Victor]

STINNER Victor added the comment:

Test-2.py has issues:

* it doesn't call Timer.join()
* it uses a weak synchronization between the main thread and the Timer thread: 
see msg276990 for an example using Event

If you use a better synchronization code, call timer.join() and call 
gc.collect(), the memory usage is very stable even after creating more than 100 
000 threads + subprocesses.

Xavion: "As I noted before the problem is not with subprocess leaking 4K memory 
*always*. The issue comes from the fact that subprocess seems to leak 4K memory 
per individual thread."

I'm unable to reproduce a *leak*.

Test-3.py output:

497 thread+subprocess
VmRSS:  9584 kB
986 thread+subprocess
VmRSS:  9596 kB  <== after the warmup, the usage seems stable
1490 thread+subprocess
VmRSS:  9596 kB
(...)
10361 thread+subprocess
VmRSS:  9596 kB
(...)
30282 thread+subprocess
VmRSS:  9596 kB
30695 thread+subprocess
VmRSS:  9672 kB
31160 thread+subprocess
VmRSS:  9684 kB  <=== memory usage decreases :-)
(...)
60768 thread+subprocess
VmRSS:  9684 kB
^C
---

If you really want to say that something is wrong: I don't understand why we 
must call gc.collect() to keep the memory usage stable. But I guess that the GC 
is not always called for performance.

Without the GC it's not that bad:
---
1083 thread+subprocess
VmRSS:  9764 kB
2097 thread+subprocess
VmRSS:  9888 kB
3136 thread+subprocess
VmRSS:  9888 kB
(...)
11750 thread+subprocess
VmRSS:  9888 kB
12668 thread+subprocess
VmRSS:  9940 kB
13705 thread+subprocess
VmRSS:  9940 kB
(...)
70948 thread+subprocess
VmRSS:  9940 kB
^C
---

There is no such "4k leak per function call".

I close the issue. It's a bug in your code, not in Python.

--
resolution:  -> not a bug
status: open -> closed
Added file: http://bugs.python.org/file44749/Test-3.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24416] Return a namedtuple from date.isocalendar()

[Baptiste Mispelon]

Baptiste Mispelon added the comment:

I updated the patch based on Raymond's feedback.

I don't know C at all and I tried to mimic the namedtuple usage of timemodule.c 
as much as I could (until the code compiled and the test suite passed).

I still have two questions:

* It seems that the Python implementation (Lib/datetime.py) is not tested when 
I run `./python -m test test_datetime` (I introduced a deliberate error in 
datetime.isocalendar() there but the test still succeeded). Is there a 
particular procedure to trigger those tests?

* I'm currently using the name `IsoCalendarDate` as suggested by user taleinat. 
However, shouldn't it be `ISOCalendarDate`?

Thanks.

--
Added file: http://bugs.python.org/file44750/issue24416_5.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28210] argparse with subcommands difference in python 2.7 / 3.5

[stephan]

New submission from stephan:

In python 2.7.12 I get an error if I do not pass
arguments, while in python 3.5.2 I do not get
the error (it fails silently).
Stumbled on this during my migration of my python 2.7 code to python 3.5 for 
django.

Here is the console output:

D:\util\python\test>py -3 test_argparse.py
3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)]
Use subparsers: True
Namespace(command=None)

D:\util\python\test>py -2 test_argparse.py
2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)]
Use subparsers: True
usage: test_argparse.py [-h] command ...
test_argparse.py: error: too few arguments

D:\util\python\test>

--
files: test_argparse.py
messages: 277017
nosy: stephan
priority: normal
severity: normal
status: open
title: argparse with subcommands difference in python 2.7 / 3.5
type: behavior
versions: Python 2.7, Python 3.5
Added file: http://bugs.python.org/file44751/test_argparse.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28211] Wrong return value type in the doc of PyMapping_Keys/Values/Items

[Xiang Zhang]

New submission from Xiang Zhang:

PyMapping_Keys/Values/Items can only return a list or tuple. Even in the case 
of a dict, it returns a list. But the doc tells a dictionary view will be 
returned in case of dict, which is wrong.

--
assignee: docs@python
components: Documentation
files: mapping_doc.patch
keywords: patch
messages: 277018
nosy: docs@python, xiang.zhang
priority: normal
severity: normal
stage: patch review
status: open
title: Wrong return value type in the doc of PyMapping_Keys/Values/Items
versions: Python 3.5, Python 3.6, Python 3.7
Added file: http://bugs.python.org/file44752/mapping_doc.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[Константин Волков]

New submission from Константин Волков:

When you close asyncio socket server it closes all connection sockets, remove 
its readers and do "cancel" for readers. Problem is that, that after this 
reader tasks are leaved alone in "cancelling" state. They wouldn`t be really 
cancelled, because they need a loop step for that. But they are not in loop 
now(they suggest execution throught selector, but they not presented there 
after removal). Cancelling step woldn`t be done and tasks wouldn`t be really 
finished, and no finishing actions can be done.
I think that It is good idea such tasks to "ready" queue, so they can be 
executed in normal task way after removing from selector.

--
components: asyncio
messages: 277019
nosy: gvanrossum, yselivanov, Константин Волков
priority: normal
severity: normal
status: open
title: Closing server in asyncio is not efficient
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28213] asyncio SSLProtocol _app_transport is private

[Константин Волков]

New submission from Константин Волков:

Seems that this field must not be private(or must have read-only property) as 
it is supposed to use outside of class.

I catched that, when implemented STARTTLS smtp process, when you must start SSL 
connection over existing socket connection. Currently it is not hard, its easy 
to add SSL layer as loop do:

self._tls_protocol = sslproto.SSLProtocol()
socket_transport = self.transport
socket_transport._protocol = self._tls_protocol
self.transport = self._tls_protocol._app_transport
self._tls_protocol.connection_made(socket_transport)

But here you must access to private property "app_transport". It must be public 
because its purpose to public "result" of SSL layer implementation. 

>From class BaseSelectorEventLoop:

def _make_ssl_transport(...):
...
return ssl_protocol._app_transport

--
components: asyncio
messages: 277020
nosy: gvanrossum, yselivanov, Константин Волков
priority: normal
severity: normal
status: open
title: asyncio SSLProtocol _app_transport is private
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28211] Wrong return value type in the doc of PyMapping_Keys/Values/Items

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Hmm, the documentation was changed in issue25909, but seems I was wrong. Since 
PyDict_Items() returns a list, but not a dict view, PyMapping_Items() can 
return only a list or a tuple.

--
nosy: +martin.panter, mine0901, orsenthil, python-dev, r.david.murray, 
serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28203] complex() gives wrong error when the second argument has an invalid type

[Soumya Sharma]

Soumya Sharma added the comment:

Changed error message to:

>>> complex({1:2},1)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: complex() first arg must be a string or a number, not 'dict'

>>> complex(1j, {1: 2})
Traceback (most recent call last):
  File "", line 1, in 
TypeError: complex() second arg must be a number, not 'dict'

Added tests to check the error raised.

--
Added file: http://bugs.python.org/file44753/Issue28203#2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28203] complex() gives wrong error when the second argument has an invalid type

[Soumya Sharma]

Changes by Soumya Sharma :


Removed file: http://bugs.python.org/file44753/Issue28203#2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28203] complex() gives wrong error when the second argument has an invalid type

[Soumya Sharma]

Soumya Sharma added the comment:

Apologies. This is the correct file.

--
Added file: http://bugs.python.org/file44754/Issue28203#3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1043134] Add preferred extensions for MIME types

[Tom Christie]

Tom Christie added the comment:

Confirming that I've also bumped into this for Python 3.5.

A docs update would seem to be the lowest-cost option to start with.

Right now `mimetypes.guess_extension()` isn't terribly useful, and it'd be 
better to at least know that upfront.

--
nosy: +Tom.Christie

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28203] complex() gives wrong error when the second argument has an invalid type

[Soumya Sharma]

Soumya Sharma added the comment:

Squashed the commits for better readability.
Also, change required in Python 3.4 as well

--
versions: +Python 3.4
Added file: http://bugs.python.org/file44755/Issue28203#4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28203] complex() gives wrong error when the second argument has an invalid type

[Berker Peksag]

Berker Peksag added the comment:

Thanks for the patch. We can't fix this in 3.4 because it's in 
security-fix-only mode: 
https://docs.python.org/devguide/index.html#status-of-python-branches

--
versions:  -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28214] Improve exception reporting for problematic __set_name__ attributes

[Tim Graham]

New submission from Tim Graham:

As requested by Nick [0], this is a usability issue against CPython 3.6 to 
provide a better chained TypeError in this case:

class _TokenType(tuple):
parent = None

def __getattr__(self, name):
new = _TokenType(self + (name,))
setattr(self, name, new)
new.parent = self
return new

Token = _TokenType()

Keyword = Token.Keyword

class KeywordCaseFilter(object):
ttype = Keyword


Traceback (most recent call last):
  File "test.py", line 14, in 
class KeywordCaseFilter(object):
TypeError: '_TokenType' object is not callable

The exception should report the specific object with the problematic 
__set_name__ attribute (rather than just passing along the underlying 
exception), as well as supporting __set_name__ = None to explicitly disable 
further lookup processing.

Follow up to #27366.

[0] https://github.com/andialbrecht/sqlparse/issues/286#issuecomment-248208900

--
components: Interpreter Core
messages: 277027
nosy: Tim.Graham, ncoghlan
priority: normal
severity: normal
status: open
title: Improve exception reporting for problematic __set_name__ attributes
type: behavior
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28210] argparse with subcommands difference in python 2.7 / 3.5

[Tim Graham]

Tim Graham added the comment:

The behavior change is from #10424. Do you believe the new behavior is 
incorrect?

--
nosy: +Tim.Graham

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28182] Expose OpenSSL verification results in SSLError

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

With this change: (tested with OpenSSL git-master)

@@ -632,20 +651,22 @@ newPySSLSocket(PySSLContext *sslctx, PyS
 SSL_set_bio(self->ssl, inbio->bio, outbio->bio);
 }
 mode = SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER;
 #ifdef SSL_MODE_AUTO_RETRY
 mode |= SSL_MODE_AUTO_RETRY;
 #endif
 SSL_set_mode(self->ssl, mode);
 
+if (server_hostname != NULL) {
 #if HAVE_SNI
-if (server_hostname != NULL)
 SSL_set_tlsext_host_name(self->ssl, server_hostname);
 #endif
+SSL_set1_host(self->ssl, server_hostname);
+}
 
 /* If the socket is in non-blocking mode or timeout mode, set the BIO
  * to non-blocking mode (blocking is the default)
  */
 if (sock && sock->sock_timeout >= 0) {
 BIO_set_nbio(SSL_get_rbio(self->ssl), 1);
 BIO_set_nbio(SSL_get_wbio(self->ssl), 1);
 }

When connecting to https://wrong.host.badssl.com/, the error is:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate 
verify failed: Hostname mismatch (_ssl.c:768)

With this change in mind, an idea is to drop the Python implementation of 
match_hostname and rely on OpenSSL's checking mechanism (`do_x509_check`). As a 
result:

* ssl.CertificateError can be either an alias of ssl.SSLCertVerificationError 
or a subclass of it
* When verify_result is X509_V_ERR_HOSTNAME_MISMATCH, the error message is 
formatted with more information following the current approach in 
`match_hostname` ("hostname XXX doesn't match YYY...")

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[R. David Murray]

R. David Murray added the comment:

You have to call wait_closed to complete the shutdown.

This should probably be mentioned directly in the socket server docs (at least 
in the examples) (assuming I'm not wrong...but I'm pretty sure I'm right) 
rather than by implicit reference to Server via the create_server comment.

--
assignee:  -> docs@python
components: +Documentation
nosy: +docs@python, r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[Константин Волков]

Константин Волков added the comment:

Thats not a thing I talking about. But thanks for "wait_closed", I will be
use it in future )

Im talking about reading tasks.When reader removed from selector, there is
reading task for it. Loop do "cancel" for it, but it not really cancelling
task. Its only marking task for be cancelled on next loop step. It must
raise Cancel exception, so task may do corresponding actions inside it (for
exemple try..except Cancel). But it will not happen, because there will be
no next step - task removed from selector and now is missed from loop. It
will freese in "cancelling" state forever if protocol implementation
wouldn`t take care about it itself. Some protocol implementations make
"_step" for such tasks manually to loop them for correct finishing, someone
makes "set_result(None)", but its all looks bad.
Seems it will be better put it in loop after removing from selector, so it
will go "cancelling way" itself as every task do.

2016-09-20 17:50 GMT+03:00 R. David Murray :

>
> R. David Murray added the comment:
>
> You have to call wait_closed to complete the shutdown.
>
> This should probably be mentioned directly in the socket server docs (at
> least in the examples) (assuming I'm not wrong...but I'm pretty sure I'm
> right) rather than by implicit reference to Server via the create_server
> comment.
>
> --
> assignee:  -> docs@python
> components: +Documentation
> nosy: +docs@python, r.david.murray
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[Guido van Rossum]

Guido van Rossum added the comment:

Can you please supply a complete example?

--Guido (mobile)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28182] Expose OpenSSL verification results in SSLError

[Christian Heimes]

Christian Heimes added the comment:

Yes, I'm planning to use the feature in 3.7. First I have to finish my PEP and 
get consents that I can drop support for OpenSSL 1.0.1 and earlier. We still 
support older versions but the feature is only available in 1.0.2+. I also need 
to come up with a solution to check IP addresses, too. It requires a different 
function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[Константин Волков]

Константин Волков added the comment:

Yes? may be a in a hour,

2016-09-20 18:12 GMT+03:00 Guido van Rossum :

>
> Guido van Rossum added the comment:
>
> Can you please supply a complete example?
>
> --Guido (mobile)
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24416] Return a namedtuple from date.isocalendar()

[Tal Einat]

Tal Einat added the comment:

Regarding the name "IsoCalendarDate", see for example this question on Stack 
Overflow[1] where both of the leading answers suggest beginning with "Iso" or 
"iso" rather than "ISO". However, taking a look at the relatively new module 
urllib.request[2], it uses names such as "HTTPHandler" rather than 
"HttpHandler". I'll leave this up to someone more knowledgeable about the 
conventions to settle.

.. [1]: http://stackoverflow.com/questions/15526107/acronyms-in-camelcase
.. [2]: 
https://docs.python.org/3.5/library/urllib.request.html#module-urllib.request

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28215] PyModule_AddIntConstant() wraps >=2^32 values when long is 4 bytes

[Kyle Altendorf]

New submission from Kyle Altendorf:

I am cross compiling Python 3.5.2 for use on a 32-bit ARM processor with Linux. 
 I use socket.CAN_EFF_FLAG and noticed that it is negative on the target 
despite being positive on my host (64-bit Intel Linux).

  Host:

altendky@tp:~$ uname -a
Linux tp 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 
x86_64 x86_64 GNU/Linux
altendky@tp:~$ python3 --version
Python 3.5.2
altendky@tp:~$ python3 -c 'import socket; print(socket.CAN_EFF_FLAG)'
2147483648

  ^^ Is expected


  Target:

root@rosi ~$ uname -a
Linux rosi 3.0.101-rt130-opusa3-2.1.0-2 #1 PREEMPT Tue Apr 12 13:49:26 CEST 
2016 armv6l GNU/Linux
root@rosi ~$ /opt/epc/bin/python3 --version
Python 3.5.2
root@rosi ~$ /opt/epc/bin/python3 -c 'import socket; print(socket.CAN_EFF_FLAG)'
-2147483648

  ^^ Is not expected to be negative


  Only CAN_EFF_FLAG reference in my source used to cross build Python:

Modules/socketmodule.c:PyModule_AddIntMacro(m, CAN_EFF_FLAG);


  Definition in cross compiler include:

altendky@tp:/opt/freescale/usr/local/gcc-4.6.2-glibc-2.13-linaro-multilib-2011.12/fsl-linaro-toolchain/arm-fsl-linux-gnueabi/multi-libs/default/usr/include$
 grep -r CAN_EFF_FLAG
linux/can.h:#define CAN_EFF_FLAG 0x8000U /* EFF/SFF is set in the MSB */


  For reference, here it is on the host system (looks the same to me):

altendky@tp:/usr/include$ grep -r CAN_EFF_FLAG
linux/can.h:#define CAN_EFF_FLAG 0x8000U /* EFF/SFF is set in the MSB */


  But perhaps this `long` type for the value is the issue?  If signed and only 
4-bytes as is the case on my target then this will misinterpret the 0x8000U 
literal resulting in the above observed -2147483648.

PyModule_AddIntConstant(PyObject *m, const char *name, long value)

  On my target system, printf("%d", sizeof(long)) yields 4.

  For now I just work around it in my application by reassigning it to be it's 
absolute value.

socket.CAN_EFF_FLAG = abs(socket.CAN_EFF_FLAG)

--
messages: 277036
nosy: altendky
priority: normal
severity: normal
status: open
title: PyModule_AddIntConstant() wraps >=2^32 values when long is 4 bytes
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28214] Improve exception reporting for problematic __set_name__ attributes

[Nick Coghlan]

Changes by Nick Coghlan :


--
nosy: +Martin Teichmann, Martin.Teichmann
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28182] Expose OpenSSL verification results in SSLError

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

That's great. OpenSSL plans to drop 1.0.1 branch support after 2016/12/31. [1] 
I guess it's OK to drop 1.0.1 support in 3.7.

Thanks for constantly improving SSL/TLS support in CPython!

[1] https://www.openssl.org/source/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28214] Improve exception reporting for problematic __set_name__ attributes

[Nick Coghlan]

Nick Coghlan added the comment:

The information we want to include in the chained exception:

1. The name of the offending attribute (since the traceback will point to the 
class header, not to the assignment)
2. The repr of the offending attribute (since the innner exception will refer 
to the return value from "attr.__set_name__" rather then the value assigned to 
the attribute)

The other idea I mentioned (allowing "__set_name__ = None" to prevent falling 
back to "__getattr__") needs a bit more consideration, as I don't quite recall 
where we're currently at in terms of expanding that idiom beyond "__hash__" and 
to the other one-shot ABCs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28182] Expose OpenSSL verification results in SSLError

[Christian Heimes]

Christian Heimes added the comment:

I'm familiar with the release cycles of OpenSSL. In fact I want to tie support 
for OpenSSL versions to the release cycle of OpenSSL. Python core dev is a bit 
... special. :) I can't just drop support. Some developers are opposing my 
plans and want to keep support for OpenSSL 1.0.1 and earlier. 

Enjoy this mail thread: 
https://mail.python.org/pipermail/python-dev/2016-August/145907.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28214] Improve exception reporting for problematic __set_name__ attributes

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Maybe there is a bug in using __set_name__. Usually special methods are looked 
in class dict. But __set_name__ is looked in instance dict. This causes to 
invoking __getattr__.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[Константин Волков]

Константин Волков added the comment:

Attached file with test example. There in "ping" task "except" and "finally" 
sections never called as expected (but code inside "try" will work).
Server is closed while client connection is active, and it just removes reading 
task from loop. You can still find it through Task.all_tasks(), but it is 
weakref array. So, sometime later GC will simply delete this task and no 
finalization will be performed.

--
Added file: http://bugs.python.org/file44756/task_example.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28182] Expose OpenSSL verification results in SSLError

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

> I'm familiar with the release cycles of OpenSSL.

Oh I shouldn't say something trivial :)

I know that thread. Hope I can help something on persuading others.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28214] Improve exception reporting for problematic __set_name__ attributes

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Proposed patch makes class creation code not looking up the __set_name__ 
attribute in instance dict.

--
keywords: +patch
stage: needs patch -> patch review
Added file: http://bugs.python.org/file44757/lookup___set_name__.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28215] PyModule_AddIntConstant() wraps >=2^32 values when long is 4 bytes

[Christian Heimes]

Christian Heimes added the comment:

The constant is an unsigned long but PyModule_AddIntConstant() takes a signed 
long. You have to write your own function that uses PyLong_FromUnsignedLong() 
and PyModule_AddObject().

Do you get a compiler warning because 0x8000U is larger than (1<<31)-1?

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28214] Improve exception reporting for problematic __set_name__ attributes

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In any case I suggest to raise an AttributeError for dunder names in 
__getattr__ implementation. Otherwise you will encounter similar issue with 
pickling.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28201] dict: perturb shift should be done when first conflict

[INADA Naoki]

Changes by INADA Naoki :


--
keywords: +patch
Added file: http://bugs.python.org/file44759/dict-perturb-shift.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28216] micro optimization for import_all_from

[Xiang Zhang]

New submission from Xiang Zhang:

Since PyMapping_Keys always return a list or tuple and most __all__s are list 
(all in stdlib), I think we can avoid calling PySequence_GetItem for every key 
and use PySequence_Fast* APIs instead. This doesn't help much since other 
operations involved are expensive.

With patch:

[bin]$ ./python3 -m perf timeit -s 'import _ast; code=compile("from _ast import 
*", "", "exec")' 'exec(code)'

Median +- std dev: 11.3 us +- 0.3 us
[bin]$ ./python3 -m perf timeit -s 'import zipfile; code=compile("from zipfile 
import *", "", "exec")' 'exec(code)'

Median +- std dev: 2.98 us +- 0.04 us

Without patch:

[bin]$ ./python3 -m perf timeit -s 'import _ast; code=compile("from _ast import 
*", "", "exec")' 'exec(code)'

Median +- std dev: 12.5 us +- 0.3 us
./python3 -m perf timeit -s 'import zipfile; code=compile("from zipfile import 
*", "", "exec")' 'exec(code)'

Median +- std dev: 3.09 us +- 0.06 us

--
components: Interpreter Core
files: import_all_from.patch
keywords: patch
messages: 277046
nosy: brett.cannon, haypo, serhiy.storchaka, xiang.zhang
priority: normal
severity: normal
stage: patch review
status: open
title: micro optimization for import_all_from
type: enhancement
versions: Python 3.7
Added file: http://bugs.python.org/file44758/import_all_from.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1602] windows console doesn't print or input Unicode

[Dāvis]

Dāvis added the comment:

Steve Dower (steve.dower)
> [...]
> Anything else requires a real console with a real person with a real keyboard.

FYI, not really, it is possible to fully automatically test console's 
output/input using WinAPI functions like WriteConsoleInput, 
GetConsoleScreenBufferInfo, ReadConsoleOutputCharacter

very recently I wrote such test, you can look at it as example 
http://review.source.kitware.com/gitweb?p=KWSys.git;a=blob;f=testConsoleBuf.cxx;hb=HEAD

it tests all 3 cases when output is actual console, redirected pipe and file.

--
nosy: +davispuh

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1602] windows console doesn't print or input Unicode

[Steve Dower]

Steve Dower added the comment:

Oh nice, I like that. We should definitely add some tests using that (though it 
seems like quite a big task... maybe I'll open a new issue for it).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28217] Add interactive console tests

[Steve Dower]

New submission from Steve Dower:

We can test the interactive console by opening new console buffers (CONIN$ and 
CONOUT$) in the test process, then creating a subprocess with those set as the 
standard handles. Now we can use WriteConsoleInput from the test process to 
simulate the user typing at the console.

This would be useful for verifying that Unicode support works properly and that 
the interactive mode behaves correctly.

--
components: Tests, Windows
messages: 277049
nosy: paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
stage: test needed
status: open
title: Add interactive console tests
type: enhancement
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1602] windows console doesn't print or input Unicode

[Steve Dower]

Steve Dower added the comment:

Created issue28217 for adding these tests.

--
superseder:  -> Add interactive console tests

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28215] PyModule_AddIntConstant() wraps >=2^32 values when long is 4 bytes

[Christian Heimes]

Christian Heimes added the comment:

Oh sorry, I missed the point that you are talking about an existing constant in 
the socket module. At first I thought that you were referring to a constant 
that you have added.

It sounds like a bug for constants >= 2**31.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28215] PyModule_AddIntConstant() wraps >=2^31 values when long is 4 bytes

[Kyle Altendorf]

Kyle Altendorf added the comment:

I do not seem to be getting a compiler warning.

arm-fsl-linux-gnueabi-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv 
-O3 -Wall -Wstrict-prototypes-Werror=declaration-after-statement   -I. 
-IInclude -I./Include 
-I"/epc/t/262/misc-build-arm-fsl-linux-gnueabi/sysroot/root/all//opt/epc/include"
   -DPy_BUILD_CORE  -c ./Modules/socketmodule.c -o Modules/socketmodule.o

To really encompass all cases I think you are correct that both a signed and an 
unsigned handler are needed.  Though, I have an idea for something nifty, I'll 
share if it works.

Regardless, shouldn't they use `intmax_t` and `uintmax_t` from stdtypes.h to 
make sure they handle anything that could be defined in the referenced C code?  
In my case simply switching from `long` to `intmax_t` would be sufficient.

Note that I am not worried about getting this fixed for my one case.  My 
workaround is fine for my application.

I also will hopefully be correcting the subject to >=2**31 if this change does 
what I think.  Good ol' off-by-one.

--
title: PyModule_AddIntConstant() wraps >=2^32 values when long is 4 bytes -> 
PyModule_AddIntConstant() wraps >=2^31 values when long is 4 bytes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28214] Improve exception reporting for problematic __set_name__ attributes

[Eric Snow]

Eric Snow added the comment:

I agree with Serhiy that __set_name__ should be looked up on the class like all 
other special methods.  Pickle is a great example why lookup (of __reduce__) on 
the instance is a pain.

--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28216] micro optimization for import_all_from

[Xiang Zhang]

Xiang Zhang added the comment:

As pointed out by Serhiy, PyObject_GetAttr may change __all__ so my proposal is 
not going to work. Close this and sorry for noise. :-(

--
resolution:  -> rejected
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28215] PyModule_AddIntConstant() wraps >=2^31 values when long is 4 bytes

[Kyle Altendorf]

Kyle Altendorf added the comment:

A little macro funny business gets a function the ability to know if the type 
passed to its wrapping macro is signed or not.

http://ideone.com/NZYs7u


  // http://stackoverflow.com/questions/7469915
  #define IS_UNSIGNED(v) (v >= 0 && ~v >= 0)
  #define F(v) f(IS_UNSIGNED(v), v, v)
  void f (bool is_unsigned, intmax_t s, uintmax_t u)

Looking in `Objects/longobject.c` suggests that perhaps the two functions that 
could be chosen from would be `PyLong_From[Unsigned]LongLong()` to avoid 
truncation.  Is there some reason not to use these?  I don't know the habits of 
CPython developers to know if there's a significant optimization going on here.

Just to throw it out there, in the case of macros, `PyLong_FromString()` might 
even be usable...


Included for quick reference:

int PyModule_AddIntConstant(PyObject *m, const char *name, long value)
  https://hg.python.org/cpython/file/tip/Python/modsupport.c#l566

PyModule_AddIntMacro(m, CAN_EFF_FLAG);
  https://hg.python.org/cpython/file/tip/Modules/socketmodule.c#l7098

PyObject * PyLong_FromLong(long ival)
  https://hg.python.org/cpython/file/tip/Objects/longobject.c#l231

#define PyModule_AddIntMacro(m, c) PyModule_AddIntConstant(m, #c, c)
  https://hg.python.org/cpython/file/tip/Include/modsupport.h#l80

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15797] bdist_msi does not pass -install/remove flags to install_script

[Petri Savolainen]

Petri Savolainen added the comment:

Any chance the patch could be processed?

--
nosy: +petri
versions: +Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28218] Windows docs have wrong versionadded description

[Steve Dower]

New submission from Steve Dower:

At https://docs.python.org/3.6/using/windows.html#finding-modules the 
versionadded description needs the reference to `sys.path file` removed.

--
assignee: steve.dower
messages: 277058
nosy: steve.dower
priority: normal
severity: normal
stage: needs patch
status: open
title: Windows docs have wrong versionadded description
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15797] bdist_msi does not pass -install/remove flags to install_script

[Petri Savolainen]

Petri Savolainen added the comment:

If I understood the patch correctly, while adding the '-install' and '-remove' 
arguments, the patch also removes sys.argv[0], ie. the (install) script name. 
Why? That also changes the the documented behavior.

The patch also appears to add an uninstall script. That would be a nice bonus. 
But should it be a separate issue & patch?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27923] PEP 467 -- Minor API improvements for binary sequences

[Elias Zamaria]

Elias Zamaria added the comment:

Ethan, by "Ned", I am guessing that you are referring to Ned Batchelder. Is 
that right? If so, do we need to put him on the nosy list or do anything else 
to bring this to his attention?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28219] Is order of argparse --help output officially defined?

[Barry A. Warsaw]

New submission from Barry A. Warsaw:

Sometimes we want to control the order of arguments printed by an argparse 
defined cli.  In practice, arguments are printed in the order in which they are 
added via add_argument(), but I don't think this is documented and thus should 
be considered an implementation detail.  It should be possible to enforce an 
order by the use of a formatter_class, but you'd probably need some extra 
metadata on arguments to do this, so it's not completely obvious how to write 
such a formatter class.

Instead, I propose to document and officially support that the order of 
add_argument() calls defines the order in which arguments are printed via 
--help.

--
assignee: docs@python
components: Documentation
messages: 277060
nosy: barry, docs@python
priority: normal
severity: normal
status: open
title: Is order of argparse --help output officially defined?
versions: Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28213] asyncio SSLProtocol _app_transport is private

[Guido van Rossum]

Changes by Guido van Rossum :


--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[Andrew Svetlov]

Andrew Svetlov added the comment:

It's a known annoying issue.
Honestly I don't know how to fix it properly.
`transport.close()` is affected also because `protocol.connection_lost()` is 
called on next loop iteration only.

Adding a small `asyncio.sleep()` between finishing all worn and loop closing 
solves the problem for me but the solution is not obvious for newbies.

--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[Константин Волков]

Константин Волков added the comment:

Seems that its not so hard - in loop.remove_reader add

self._ready.append(reader)

after reader.cancel()

May be its needed to check that its not already there, but I cant imagine
how it can be.

2016-09-20 23:16 GMT+03:00 Andrew Svetlov :

>
> Andrew Svetlov added the comment:
>
> It's a known annoying issue.
> Honestly I don't know how to fix it properly.
> `transport.close()` is affected also because `protocol.connection_lost()`
> is called on next loop iteration only.
>
> Adding a small `asyncio.sleep()` between finishing all worn and loop
> closing solves the problem for me but the solution is not obvious for
> newbies.
>
> --
> nosy: +asvetlov
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28212] Closing server in asyncio is not efficient

[Guido van Rossum]

Changes by Guido van Rossum :


--
nosy:  -gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28207] SQLite headers are not searched in custom locations

[Santiago Castro]

Santiago Castro added the comment:

I tried with pyenv (https://github.com/yyuu/pyenv): pyenv install 3.5.2. Maybe 
the error is from their side, but basically it downloads Python and compiles 
it: 
https://github.com/yyuu/pyenv/blob/master/plugins/python-build/install.sh#L24

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27955] getrandom() syscall returning EPERM make the system unusable.

[Roundup Robot]

Roundup Robot added the comment:

New changeset 41e9e711b9b5 by Victor Stinner in branch '3.5':
Cleanup random.c
https://hg.python.org/cpython/rev/41e9e711b9b5

New changeset ddc54f08bdfa by Victor Stinner in branch '3.5':
Catch EPERM error in py_getrandom()
https://hg.python.org/cpython/rev/ddc54f08bdfa

New changeset 27d05bb6f832 by Victor Stinner in branch '3.6':
(Merge 3.5) Catch EPERM error in py_getrandom()
https://hg.python.org/cpython/rev/27d05bb6f832

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27955] getrandom() syscall returning EPERM make the system unusable.

[STINNER Victor]

STINNER Victor added the comment:

I modified Python 3.5, 3.6 and 3.7 to fall back on reading /dev/urandom when 
getrandom() syscall fails with EPERM.

Thanks for the bug report iwings!

Note: Python 2.7 does not use getrandom() and so is not impacted.


Christian:
> Did you open a bug with your vendor, too? QNAP is clearly violating Kernel 
> APIs. getrandom() is not suppose to fail with EPERM.

I don't know if it can be seen as a violation of the Kernel API, but at least, 
it doesn't seem to be something smart to block getrandom() syscall. getrandom() 
was designed to enhance the security of applications ;-)


Nick:
> With #27778 implemented, there's also the question of how os.getrandom() will 
> react to security policies that restrict access to the getrandom syscalls (vs 
> just not having it available in the kernel).

This is no question: os.getrandom() of Python 3.6 is a thin wrapper on the 
syscall. If the syscall fails, the Python function raises an exception ;-) 
OSError(EPERM) on this case.

--
resolution:  -> fixed
status: open -> closed
versions: +Python 3.7 -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28213] asyncio SSLProtocol _app_transport is private

[Yury Selivanov]

Yury Selivanov added the comment:

-1 on exposing app_protocol. It's an implementation detail, starttls should be 
implemented in asyncio (and while it's not, it's ok to use '_app_protocol'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28220] argparse's add_mutually_exclusive_group() should accept title and description

[Barry A. Warsaw]

New submission from Barry A. Warsaw:

I'd love to sneak this into 3.6, but I can accept being too late.  In any case, 
_MutuallyExclusiveGroup.__init__() should accept title and description 
arguments and pass them to the super class.  Otherwise, you can't great a 
mutually exclusive group that also essentially acts as an argument group.

--
components: Library (Lib)
messages: 277067
nosy: barry
priority: normal
severity: normal
status: open
title: argparse's add_mutually_exclusive_group() should accept title and 
description
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28220] argparse's add_mutually_exclusive_group() should accept title and description

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

Hmm, it might be more complicated than that, so let's ignore 3.6

--
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27778] PEP 524: Add os.getrandom()

[Roundup Robot]

Roundup Robot added the comment:

New changeset d31b4de433b7 by Victor Stinner in branch '3.6':
Fix memleak in os.getrandom()
https://hg.python.org/cpython/rev/d31b4de433b7

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27778] PEP 524: Add os.getrandom()

[STINNER Victor]

STINNER Victor added the comment:

I pushed the fix for the issue #27955, os.urandom() now handles getrandom() 
failing with EPERM.

@Christian: Thanks for your review, I pushed a change fixing the two issues 
that you reported (memory leak and inefficient temporarily buffer).

I attached getrandom_errno.patch: a change proposing to document ENOSYS and 
EPERM. What do you think?

--
Added file: http://bugs.python.org/file44760/getrandom_errno.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28220] argparse's add_mutually_exclusive_group() should accept title and description

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

The workaround is to do something like:

group = parser.add_argument_group(title, description)
me_group = group.add_mutually_exclusive_group()
me_group.add_argument(...blah blah blah...)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27990] Provide a way to enable getrandom on Linux even when build system runs an older kernel

[STINNER Victor]

STINNER Victor added the comment:

I'm not excited by the idea of using hardcoded constants for getrandom(). There 
is a risk of using wrong constants depending on the architecture or the Linux 
kernel version.

The code is already very low-level: it calls directly the syscall using 
syscall() function. getrandom() has no nice glibc clean API yet:
https://sourceware.org/bugzilla/show_bug.cgi?id=17252

I suggest to close the issue as WONTFIX. It's ok to use a file descriptor and 
read /dev/urandom. getrandom() is nice to have, but it's not really a killer 
feature.

For Fedora: sure, you can use vendor patches to workaround your technical 
issue, builders using an old kernel :-) But it would be simpler to upgrade the 
builder, no? :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27292] Warn users that os.urandom() prior to 3.6 can return insecure values

[STINNER Victor]

STINNER Victor added the comment:

> Please ensure that the documentation properly warns users about these edge 
> cases.

I disagree. I don't think that the Python documentation is the right place to 
document the security level of system urandom.

It's just a mess, there are so many corner cases and it's very hard to provide 
a clear explanation for end users.

I suggest to keep the positive "suitable for cryptographic use". If you change 
this sentence, I only expect that users will use something WORSE. For example 
"os.urandom is not secure! we must use ssl.RAND_bytes!". No. Don't do that, 
ssl.RAND_bytes() has its own set of issues, like two processes with the same 
pid producing the same random sequence... (old known issue, very hard to fix)

Python cannot workaround OS limitations, we can only do our best to use the 
most secure source of entropy. That's why Python 3.5 now uses getrandom() on 
Linux. That's why Python 3.6 now calls getrandom() in blocking mode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22542] Use arc4random under OpenBSD for os.urandom() if /dev/urandom is not present

[STINNER Victor]

STINNER Victor added the comment:

"Trying to run the python interpreter in a chroot fails if /dev/urandom is not 
present."

The workaround is simple: fix your chroot to correctly expose /dev/urandom in 
the chroot. It's a common and known issue, no?

Since the issue is almost dead since 2 years and I don't know if arc4random() 
is suitable for os.urandom(), I close the issue. If you want to reopen it, 
please come back with more information on arc4random() security ;-)

--
resolution:  -> out of date
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27282] Raise BlockingIOError in os.urandom if kernel is not ready

[STINNER Victor]

STINNER Victor added the comment:

This idea is the PEP 522 which was superseded by the PEP 524 (accepted in 
Python 3.6) which proposed to make os.urandom() blocking.

--
nosy: +haypo
resolution:  -> rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25470] Random Malloc error raised

[STINNER Victor]

STINNER Victor added the comment:

Python 3.6 got a builtin debugging tool to detect buffer under- and overflow: 
PYTHONMALLOC=debug

https://docs.python.org/dev/whatsnew/3.6.html#pythonmalloc-environment-variable

In most cases, an error in PyObject_Free() is a hint of a buffer overflow. So I 
close the issue.

--
nosy: +haypo
resolution:  -> third party
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28165] The 'subprocess' module leaks memory when called in certain ways

[Xavion]

Xavion added the comment:

Firstly, you've misquoted me.  The quote you attributed to me in your latest 
post was actually made by 'ztane'.

Secondly, your extra thread/event code makes no difference here.  I will attach 
the memory usage logs in subsequent posts.

For consistency, I have removed all of the collateral stuff from your 
"Test-3.py" script and reattached it here as "Test-3a.py".

--
resolution: not a bug -> 
status: closed -> open
title: The 'subprocess' module leaks 4 kiB memory for each thread -> The 
'subprocess' module leaks memory when called in certain ways
Added file: http://bugs.python.org/file44761/Test-3a.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28165] The 'subprocess' module leaks memory when called in certain ways

[Xavion]

Changes by Xavion :


Added file: http://bugs.python.org/file44762/Test-3a-no-gc.log

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28165] The 'subprocess' module leaks memory when called in certain ways

[Xavion]

Changes by Xavion :


Added file: http://bugs.python.org/file44763/Test-3a-gc.log

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28217] Add interactive console tests

[Eryk Sun]

Eryk Sun added the comment:

Here's the ctypes code (mentioned on issue 1602) for writing to the input 
buffer and reading from the screen buffer. For output testing I also have a 
context manager to create and temporarily activate a new screen buffer with a 
given number of columns and rows and filled with a particular character (e.g. 
NUL), which makes it easier to read the output up to the current cursor 
position. 

Activating the new screen can be factored out if you're just testing a child 
writing to stdout. I added the screen activation part in order to read the 
output from a program that was stubbornly writing to CONOUT$, ignoring the pipe 
that was passed as stdout.

--
nosy: +eryksun
Added file: http://bugs.python.org/file44764/conin.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28217] Add interactive console tests

[Eryk Sun]

Changes by Eryk Sun :


Added file: http://bugs.python.org/file44765/conout.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28220] argparse's add_mutually_exclusive_group() should accept title and description

[Berker Peksag]

Berker Peksag added the comment:

Unless I'm missing something, this is a duplicate of issue 17218 :)

--
nosy: +berker.peksag
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> support title and description in argparse 
add_mutually_exclusive_group

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28213] asyncio SSLProtocol _app_transport is private

[Andrew Svetlov]

Andrew Svetlov added the comment:

-1, agree with Yury.
As an option it's possible to wrap `sslproto.SSLProtocol` by custom derived 
class which overrides `connection_made()` for storing a transport somewhere.
The solution looks like sub-optimal but it's backward-compatible.
The other problem is: `sslproto` is a very private module for working with SSL 
if `ssl.MemoryBIO` is available.
I believe that third-party code shouldn't rely on `sslproto.SSLProtocol` and 
even on `asyncio.sslproto` existence.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17218] support title and description in argparse add_mutually_exclusive_group

[Barry A. Warsaw]

Changes by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22431] Change format of test runner output

[Tim Graham]

Tim Graham added the comment:

Is there opposition to changing the default output as outlined in the first 
comment? If so, then I think this ticket should be closed or retitled to 
reflect the intent.

--
nosy: +Tim.Graham

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22431] Change format of test runner output

[Tim Graham]

Tim Graham added the comment:

Here's the patch if we make the change in Django instead: 
https://github.com/cjerdonek/django/commit/9c8d162f3f616e9d9768659a06fcf27bb389214b

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22431] Change format of test runner output

[R. David Murray]

R. David Murray added the comment:

My understanding of the comments is that the change to the default is OK (in a 
feature release, at least).

--
versions: +Python 3.7 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22431] Change format of test runner output

[Robert Collins]

Robert Collins added the comment:

+1 to changing the UI for 3.7 - just noting that if you're machine processing 
the output, the TUI isn't an appropriate channel.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28221] Unused indata in test_ssl.ThreadedTests.test_asyncore_server

[Martin Panter]

New submission from Martin Panter:

In r62273 (Apr 2008), method testAsyncoreServer() was added to the py3k branch 
with indata="FOO\n". In r64578 (Jun 2008), this test method was added to the Py 
2 branch, but with indata = "TEST MESSAGE of mixed case\n". Later, r80598 added 
the mixed case line to the Py 3 branch, but it is not used because the original 
FOO line overwrites it. Then revision 221a1f9155e2 backported the Py 3 code to 
2.7. So now both versions include the mixed case line but neither use it.

I haven’t investigated, but I presume either the mixed case version would test 
more stuff and should be enabled, or it would be overkill and should be dropped 
in favour of the simpler line.

--
assignee: christian.heimes
components: SSL, Tests
messages: 277085
nosy: christian.heimes, martin.panter
priority: normal
severity: normal
status: open
title: Unused indata in test_ssl.ThreadedTests.test_asyncore_server
type: enhancement
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

The problem is resolved if call PyGC_Collect() after PyDict_DelItemString(). Is 
it expected to call PyGC_Collect() here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28221] Unused indata in test_ssl.ThreadedTests.test_asyncore_server

[Martin Panter]

Martin Panter added the comment:

Actually in the Py 3 branch, I found an earlier revision that added 
indata="FOO\n": r59506 (Dec 2007).

Anyway, the server in the test case just does a simple lower() call on the 
data, so I think the simpler FOO line may be fine on its own.

--
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28201] dict: perturb shift should be done when first conflict

[Josh Rosenberg]

Josh Rosenberg added the comment:

General comment on the patch: I believe per PEP7, we're still sticking to ANSI 
C (aka C89), and specifically, "all declarations must be at the top of a block 
(not necessarily at the top of function".

The patch assumes lax standards compliance (or C99+ compliance), declaring 
variables in the for loop initializer section and midway through blocks. This 
should be changed to declare all variables at the top of blocks, and not in for 
loop initializer sections.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28201] dict: perturb shift should be done when first conflict

[Josh Rosenberg]

Josh Rosenberg added the comment:

Removing those unrelated changes looks like it would dramatically reduce the 
churn too, making review easier.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Josh Rosenberg]

Josh Rosenberg added the comment:

The fact that it's resolved by PyGC_Collect indicates there is a reference 
cycle somewhere. PyGC_Collect is just looking for cyclic garbage and breaking 
the cycles so it can be cleaned; it would happen eventually unless GC was 
explicitly disabled or the process exited before the next implicit GC 
invocation, so it means this bug is really about timing (and possibly cycles 
we'd prefer to avoid), not reference leaks.

--
nosy: +josh.r

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28222] test_distutils fails

[Xiang Zhang]

New submission from Xiang Zhang:

test_distutils consistently fails now:

./python -m test test_distutils
Run tests sequentially
0:00:00 [1/1] test_distutils
test test_distutils failed -- Traceback (most recent call last):
  File "/home/angwer/cpython/Lib/distutils/tests/test_check.py", line 122, in 
test_check_restructuredtext_with_syntax_highlight
self.assertEqual(cmd._warnings, 0)
AssertionError: 1 != 0

test_distutils failed

1 test failed:
test_distutils

Total duration: 1 sec
Tests result: FAILURE

--
components: Tests
messages: 277091
nosy: xiang.zhang
priority: normal
severity: normal
status: open
title: test_distutils fails
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13276] bdist_wininst-created installer does not run the postinstallation script when uninstalling

[Petri Savolainen]

Changes by Petri Savolainen :


--
nosy: +petri

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

Looks to me, there is NO reference cycle on the Simple object in the python 
test code. Why needs to call PyGC_Collect() here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28162] WindowsConsoleIO readall() fails if first line starts with Ctrl+Z

[Eryk Sun]

Eryk Sun added the comment:

For breaking out of the readall while loop, you only need to check if the 
current read is empty:

/* when the read is empty we break */
if (n == 0)
break;

Also, the logic is wrong here:

if (len == 0 || buf[0] == '\x1a' && _buflen(self) == 0) {
/* when the result starts with ^Z we return an empty buffer */
PyMem_Free(buf);
return PyBytes_FromStringAndSize(NULL, 0);
}

This is true when len is 0 or when buf[0] is Ctrl+Z and _buflen(self) is 0. 
Since buf[0] shouldn't ever be Ctrl+Z here (low-level EOF handling is 
abstracted in read_console_w), it's never checking the internal buffer. We can 
easily see this going wrong here:

>>> a = sys.stdin.buffer.raw.read(1); b = sys.stdin.buffer.raw.read()
Ā^Z
>>> a
b'\xc4'
>>> b
b''

It misses the remaining byte in the internal buffer.

This check can be simplified as follows:

rn = _buflen(self);

if (len == 0 && rn == 0) {
/* return an empty buffer */
PyMem_Free(buf);
return PyBytes_FromStringAndSize(NULL, 0);
}

After this the code assumes that len isn't 0, which leads to more 
WideCharToMultiByte failure cases. 

In the last conversion it's overwrite bytes_size without including rn. 

I'm not sure what's going on with _PyBytes_Resize(&bytes, n * sizeof(wchar_t)). 
ISTM, it should be resized to bytes_size, and make sure this includes rn.

Finally, _copyfrombuf is repeatedly overwriting buf[0] instead of writing to 
buf[n]. 

With the attached patch, the behavior seems correct now:

>>> sys.stdin.buffer.raw.read()
^Z
b''

>>> sys.stdin.buffer.raw.read()
abc^Z
^Z
b'abc\x1a\r\n'

Split U+0100:

>>> a = sys.stdin.buffer.raw.read(1); b = sys.stdin.buffer.raw.read()
Ā^Z
>>> a
b'\xc4'
>>> b
b'\x80'

Split U+1234:

>>> a = sys.stdin.buffer.raw.read(1); b = sys.stdin.buffer.raw.read()
ሴ^Z
>>> a
b'\xe1'
>>> b
b'\x88\xb4'

The buffer still can't handle splitting an initial non-BMP character, stored as 
a surrogate pair. Both codes end up as replacement characters because they 
aren't transcoded as a unit.

Split U+0001:

>>> a = sys.stdin.buffer.raw.read(1); b = sys.stdin.buffer.raw.read()
𐀀^Z
^Z
>>> a
b'\xef'
>>> b
b'\xbf\xbd\xef\xbf\xbd\x1a\r\n'

--
keywords: +patch
status: closed -> open
Added file: http://bugs.python.org/file44766/issue_28162_01.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Nick Coghlan]

Nick Coghlan added the comment:

While I agree with Raymond regarding the performance implications if this isn't 
handled carefully, I think we're also getting to a point where better 
accounting for signal handling latency in inner loops is something that could 
be considered for 3.7 - the benchmarking infrastructure being built out to 
measure performance optimisations would also allow overhead tuning of a 
"batched iteration" idiom where signals were checked for either every N 
thousand iterations, periodically based on time, or some combination of the two.

Benchmarking to measure the speed impact is going to be essential, though - 
this is a case where changing the behaviour is clearly possible, so the key 
questions are whether or not the resulting runtime overhead can be made low 
enough to be deemed acceptable, and whether or not it can be done in a way that 
doesn't make the affected parts of the code base effectively unreadable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Please output Py_REFCNT(py_module) and Py_REFCNT(py_dict) before deleting the 
module from sys.modules. Is there a difference between 3.5 and 3.6?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[Nick Coghlan]

Nick Coghlan added the comment:

As far as the "What's the benefit to users?" question goes, I think the main 
intended beneficiaries would be children and other folks playing at the command 
prompt and trying out different things.

The "no segfaults from normal Python code" rule aims to make that kind of 
exploration a significantly more positive experience than it is in a language 
like C - you're far more likely to get a traceback than you are to have the 
interpreter fall over completely. Tracebacks can be intimidating to new users, 
but they still give them new information to work with.

Infinite loops at the Python level are similarly about as friendly to ad hoc 
exploration as we can possibly make them: Ctrl-C will break you out of them 
with a traceback.

Implementation level infinite (or near-infinite, or 
finite-but-eating-all-of-RAM) loops by contrast are much closer to their 
traditional C level counterparts: your only way out is via destructive 
termination of the entire process.

So that's why I think this is an idea worth exploring further, even though it 
may still turn out to be impractical for code readability or runtime speed 
reasons.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28202] Python 3.5.1 C API, the global variable is not destroyed when delete the module

[Jack Liu]

Jack Liu added the comment:

@serhiy.storchaka, The reference counts before PyDict_DelItemString are same on 
Python 3.3, 3.5 and 3.6.
Py_REFCNT(py_module)1
Py_REFCNT(py_dict)4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26351] Occasionally check for Ctrl-C in long-running operations like sum

[STINNER Victor]

STINNER Victor added the comment:

Can someone work on a patch? Then we can benchmark it to take a
decision ;-) Maybe we might expose signalmodule.c internals to have a
cheaper check?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28210] argparse with subcommands difference in python 2.7 / 3.5

[stephan]

stephan added the comment:

I am not an expert with argparse since I just started using it
(and it has quite a lot of features which are 
not all self-explaining to me).

But if I didn't misunderstand something it should behave like the python 2.7 
version:
 If I do not pass any command I am told that I missed the parameter.

Or: 
 - are there some parameters I missed to make them behave equally?
 - what is the intention to fail silently?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com