[issue27145] long_add and long_sub might return a new int where _ints[x] could be returned

[Inada Naoki]

Change by Inada Naoki :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27145] long_add and long_sub might return a new int where _ints[x] could be returned

[Inada Naoki]

Inada Naoki  added the comment:


New changeset 036fe85bd3e6cd01093d836d71792a1966f961e8 by Inada Naoki 
(HongWeipeng) in branch 'master':
bpo-27145: small_ints[x] could be returned in long_add and long_sub (GH-15716)
https://github.com/python/cpython/commit/036fe85bd3e6cd01093d836d71792a1966f961e8


--
nosy: +inada.naoki

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38897] Example in socket documentation uses deprecated array.fromstring

[miss-islington]

Change by miss-islington :


--
keywords: +patch
pull_requests: +16868
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17386

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38913] Py_BuildValue("(s#O)", ...) segfaults if entered with exception raised

[danielen]

New submission from danielen :

The following code results in a segmentation fault in CPython 3.8 while 
executes fines (raises a SystemError) in CPython 3.7.

PyObject* debug(PyObject *self, PyObject *args)
{
const char * debug = "debug";
PyErr_SetString(PyExc_ValueError, "debug!");
return Py_BuildValue("(s#O)", debug, strlen(debug), Py_None);
}

It seems necessary for the format string to contain both an instance of "s#" 
and "O" to trigger the bug.

I'm attaching a C module that reproduces the problem.

--
components: Extension Modules
files: module.c
messages: 357485
nosy: danielen
priority: normal
severity: normal
status: open
title: Py_BuildValue("(s#O)", ...) segfaults if entered with exception raised
versions: Python 3.8
Added file: https://bugs.python.org/file48743/module.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38328] Speed up the creation time of constant list and set literals.

[Inada Naoki]

Change by Inada Naoki :


--
nosy:  -inada.naoki
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38328] Speed up the creation time of constant list and set literals.

[Inada Naoki]

Inada Naoki  added the comment:


New changeset 6dd9b64770af8905bef293c81d541eaaf8d8df52 by Inada Naoki (Brandt 
Bucher) in branch 'master':
bpo-38328: Speed up the creation time of constant list and set display. 
(GH-17114)
https://github.com/python/cpython/commit/6dd9b64770af8905bef293c81d541eaaf8d8df52


--
nosy: +inada.naoki

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38295] test_relative_path of test_py_compile fails on macOS 10.15 Catalina

[Bo Anderson]

Bo Anderson  added the comment:

For what it's worth, this is having an impact on some real code: 
https://github.com/Homebrew/homebrew-core/pull/45110

Perhaps a simpler way to reproduce is:

% cd /tmp
% python3 -c 'import os; open(os.path.relpath("/tmp/test.txt"), "w")'
Traceback (most recent call last):
  File "", line 1, in 
FileNotFoundError: [Errno 2] No such file or directory: '../../tmp/test.txt'

--
nosy: +Bo98

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34716] MagicMock.__divmod__ should return a pair

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38910] AssertionError: ElementTree not initialized, missing root

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

This seems more like an issue with lxml.

--
nosy: +scoder, xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38895] performance degradation creating a mock object (by factor 7-8)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

> This is already done in master branch ;)

Thanks, I noticed the speed improvement in master and 3.7 but assumed it's due 
to some other optimisation in master.

> Do you mind if I create PR for it? I would like to start contributing to 
> CPython ;)

Sure, I will be happy to review. I would like to know others thoughts on this 
too on deferring whether the attribute is async check to the mock's own 
creation in _get_child_mock instead of making the parent mock construction 
slightly expensive as per the current approach. We can at least backport the 
NonCallableMock.__init__ signature fix which seems safe to make an improvement.

Thanks

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38861] zipfile: Corrupts filenames containing non-UTF8 characters

[John Goerzen]

John Goerzen  added the comment:

Hi Jon,

I've read your article in the gist, the ZIP spec, and the article you linked 
to.  As the article you linked to 
(https://marcosc.com/2008/12/zip-files-and-encoding-i-hate-you/) states, 
"Implementers just encode file names however they want (usually byte for byte 
as they are in the OS".  That is certainly my observation.  CP437 has NEVER 
been guaranteed, *even on DOS*.  See 
https://en.wikipedia.org/wiki/Category:DOS_code_pages and 
https://www.aivosto.com/articles/charsets-codepages-dos.html for details on DOS 
code pages.  I do not recall any translation between DOS codepages being done 
in practice, or even possible - since the whole point of multiple codepages was 
the need for more than 256 symbols.  So (leaving aside utf-8 encodings for a 
second) no operating system or ZIP implementation I am aware of performs a 
translation to cp437, such translation is often not even possible, and they're 
just copying literal bytes to ZIP -- as the POSIX filesystem itself is.

So, from the above paragraph, it's clear that the assumption in zipfile that 
cp437 is in use is faulty.  Your claim that Python "fixes" a problem is also 
faulty.  Converting from a latin-1 character, using a cp437 codeset, and 
generating a filename with that cp437 character represented as a Unicode code 
point is wrong in many ways.  Python should not take an opinion on this; it 
should be agnostic and copy the bytes that represent the filename in the ZIP to 
bytes that represent the filename on the filesystem.

POSIX filenames contain any of 254 characters (only 0x00 and '/' are invalid).  
The filesystem is encoding-agnostic; POSIX filenames are just stream of bytes.  
There is no alternative but to treat ZIP filenames (without the Unicode flag) 
the same way.  Copy bytes to bytes.  It is not possible to identify the 
encoding of the filename in the absence of the Unicode flag.

zipfile should:

1) expose a bytes interface to filename
2) use byte-for-byte extraction when no Unicode flag is present
3) not make the assumption that cp437 was the original encoding

Your proposal only "works" cross-platform because it is broken on every 
platform!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Tzu-ping Chung]

Tzu-ping Chung  added the comment:

>> not enough since it’d break `flit install --python=py` because that’s give 
>> you the location of py.exe, not the actual interperter.
> This would be fine if you still run the process to get its sys.executable.

But then I need two separate workflows based on what is passed in. For py.exe I 
need to run it and get sys.executable. But for python.exe I *cannot* use 
sys.executable because that’s the base interepeter, not the venv path I want. 
And `if Path(arg).stem == "py"` just seems like a bug waiting to happen.


> Your specific example would never have worked, FWIW, as it always would have 
> picked up pythonA rather than the application one or the base one, unless you 
> were relying on python3/python3.7 not being available on Windows (which is no 
> longer true - they are included in the Store package now).

It is an illustration. I am fully aware of Windows not having version-named 
python executables. Thanks for the reminder anyway.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36375] PEP 499 implementation: "python -m foo" binds the main module as both __main__ and foo in sys.modules

[Cameron Simpson]

Cameron Simpson  added the comment:

On 25Nov2019 17:38, Python Bug Reports  wrote:
>Eric Snow  added the comment:
>
>FWIW, I have some feedback on the PEP.  (See msg357448.)  Can we discuss here 
>or should I open a mailing list thread?

Let's discuss it here unless it looks like we need wider input. This is 
related to issue37292 (_xxsubinterpreters: Can't unpickle objects 
defined in __main__), yes?

With PEP499, "python -m foo" should have __name__=='__main__' and 
__spec__.name=='foo'. Were you thinking the __module__ should come from 
__spec__.name (and that the PEP should make that clear)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38911] include __del__ in Generator ABC

[Guido van Rossum]

Guido van Rossum  added the comment:

I wouldn't have use a Generator subclass for that. Let's not destabilize the 
Generator class.

--
resolution:  -> wont fix
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38911] include __del__ in Generator ABC

[Matt McEwen]

Matt McEwen  added the comment:

My interpretation of issue 24018 was that the Generator ABC was trying to 
follow the PEP as much as possible, so that users were able to produce a custom 
generator object and have it behave just like a builtin generator object.

I know that subclassing Generator is unusual; In my case that is a custom 
object that needs to be an iterator and also enter context managers internally. 
This seemed to me a very similar problem that the `close` machinery for 
generators was aimed to address, and subclassing generator worked really 
nicely, with one exception.

It took me some time to understand why the context managers weren't being 
exited correctly when the generator was interrupted, and it was because my 
`close` wasn't being called when the object fell out of scope, as it would have 
been if my object was a builtin generator object. Manually implementing __del__ 
fixed this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38803] test_wait3 and test_wait4 leaked references on x86 Gentoo Refleaks 3.x

[Benjamin Peterson]

Benjamin Peterson  added the comment:


New changeset e4db1f05e9a5828f6b287f99067362fa0e5732e8 by Benjamin Peterson 
(Eddie Elizondo) in branch 'master':
closes bpo-38803: Fix leak in posixmodule. (GH-17373)
https://github.com/python/cpython/commit/e4db1f05e9a5828f6b287f99067362fa0e5732e8


--
nosy: +benjamin.peterson
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26730] SpooledTemporaryFile doesn't correctly preserve data for text (non-binary) SpooledTemporaryFile objects when Unicode characters are written

[Graham Coster]

Graham Coster  added the comment:

This may be a silly question, however, does SpooledTemporaryFile need to exist 
at all?

>From some testing on macOS, SpooledTemporaryFile appeared to never have a 
>performance advantage over OS file caching, but with max_size greater than 
>4GB, it was a significant disadvantage. So, if the purpose of 
>SpooledTemporaryFile is to increase performance, it may not work.

I found that the macOS built-in file cache was increasing in size as I wrote 
bigger TemporaryFile files, up to some limit the OS had decided. So, it seems 
the OS is automatically doing the same job as SpooledTemporaryFile.  Once the 
OS decided to write to disk, there was no sudden hit to performance, it just 
slowed down.

However, when SpooledTemporaryFile rolled-over large max_size files, there was 
a temporary big hit to performance, which then became a consistent slow down 
the same as a TemporaryFile that had exceeded the OS file cache.

A big issue came with very large SpooledTemporaryFile  max_sizes hogging RAM 
and causing the OS to start swapping all processes.  This caused a huge 
performance hit to my program and the system as a whole. Once my program did 
finish, it took the system considerable time to reclaim swap.

I’m guessing SpooledTemporaryFile may have benefits on light weight embedded 
OSes that have no, or poor, file caching.  However, tuning the max_size to work 
with embedded systems’ limited RAM could be tricky for developers and would be 
hardware dependent. So, perhaps leaving file caching to the underlying 
operating systems is actually a better, and safer, option than offering it in 
Python?

If there are no benefits to SpooledTemporaryFile, should it be deprecated? If 
so, as it is phasesd out, could it be patched to be a TemporaryFile wrapper, 
with no rollover functionality?

--
nosy: +graham.coster

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38625] SpooledTemporaryFile does not seek correctly after being rolled over

[Graham Coster]

Graham Coster  added the comment:

This may be a silly question, however, does SpooledTemporaryFile need to exist 
at all?

>From some testing on macOS, SpooledTemporaryFile appeared to never have a 
>performance advantage over OS file caching, but with max_size greater than 
>4GB, it was a significant disadvantage.   

I found that the macOS built-in file cache was increasing in size as I wrote 
bigger TemporaryFile files, up to some limit the OS had decided. So, it seems 
the OS is automatically doing the same job as SpooledTemporaryFile.  Once the 
OS decided to write to disk, there was no sudden hit to performance, it just 
slowed down.

However, when SpooledTemporaryFile rolled-over large max_size files, there was 
a temporary big hit to performance, which then became a consistent slow down 
the same as TemporaryFile.

A big issue came with very large SpooledTemporaryFile  max_sizes hogging RAM 
and causing the OS to start swapping all processes.  This caused a huge 
performance hit to my program and the system as a whole. Once my program did 
finish, it took the system considerable time to reclaim swap.

I’m guessing SpooledTemporaryFile may have benefits on light weight embedded 
OSes that have no, or poor, file caching.  However, tuning the max_size to work 
with embedded systems’ limited RAM could be tricky for developers and would be 
hardware dependent. So, perhaps leaving file caching to operating systems is 
actually a better, and safer, option than offering it in Python?

If there are no benefits to SpooledTemporaryFile, should it be deprecated? If 
so, as it is phasesd out, could it be patched to be a TemporaryFile wrapper, 
with no rollover functionality?

> On 25 Nov 2019, at 1:37 pm, Inada Naoki  wrote:
> 
> 
> Inada Naoki  added the comment:
> 
> SpooledTemporaryFile has very serious bug which causes data corruption 
> (#26730).  Please don't use it with text mode until it is fixed.
> 
> --
> 
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38911] include __del__ in Generator ABC

[Guido van Rossum]

Guido van Rossum  added the comment:

The PEP does not specify collections.Generator at all, so this is not just a 
matter of interpreting the PEP.

The presence of a __del__ method can cause subtle behavior changes to the GC, 
so I worry that adding __del__ to that class now is going to break 
currently-working code.

Not having seen the OP's Generator subclass, I wonder if they even meant to 
subclass collections.Generator?  It's pretty esoteric to subclass Generator -- 
usually it's better to subclass Iterator or Iterable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29167] Race condition in enum.py:_decompose()

[Ethan Furman]

Ethan Furman  added the comment:

The latest patch from issue38045 should make race-conditions non-existent.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38912] test_asyncio altered the execution environment

[Pablo Galindo Salgado]

New submission from Pablo Galindo Salgado :

https://buildbot.python.org/all/#/builders/260/builds/60/steps/5/logs/stdio


0:05:06 load avg: 6.65 [131/423] test_contextlib_async passed -- running: 
test_signal (1 min 53 sec), test_concurrent_futures (4 min 11 sec), 
test_zipfile (47.5 sec), test_statistics (48.6 sec), 
test_multiprocessing_forkserver (2 min 58 sec), test_multiprocessing_spawn (4 
min 54 sec)
beginning 6 repetitions
123456
Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
.Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
.Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
.Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
.Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
.Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
Task was destroyed but it is pending!
task: ()>>
.
0:05:06 load avg: 6.65 [132/423] test_ssl passed -- running: test_signal (1 min 
53 sec), test_concurrent_futures (4 min 11 sec), test_zipfile (47.6 sec), 
test_statistics (48.7 sec), test_multiprocessing_forkserver (2 min 58 sec), 
test_multiprocessing_spawn (4 min 54 sec)
beginning 6 repetitions
123456
/home/buildbot/buildarea/3.8.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/support/__init__.py:3323:
 ResourceWarning: unclosed 
  del self.thread
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Resource 'ipv6.google.com' is not available
./home/buildbot/buildarea/3.8.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/support/__init__.py:3323:
 ResourceWarning: unclosed 
  del self.thread
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Resource 'ipv6.google.com' is not available
./home/buildbot/buildarea/3.8.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/support/__init__.py:3323:
 ResourceWarning: unclosed 
  del self.thread
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Resource 'ipv6.google.com' is not available
./home/buildbot/buildarea/3.8.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/support/__init__.py:3323:
 ResourceWarning: unclosed 
  del self.thread
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Resource 'ipv6.google.com' is not available
./home/buildbot/buildarea/3.8.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/support/__init__.py:3323:
 ResourceWarning: unclosed 
  del self.thread
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Resource 'ipv6.google.com' is not available
./home/buildbot/buildarea/3.8.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/support/__init__.py:3323:
 ResourceWarning: unclosed 
  del self.thread
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Resource 'ipv6.google.com' is not available
.

beginning 6 repetitions
123456
Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 
2)
Dangling thread: <_MainThread(MainThread, started 140669094238016)>
Dangling thread: 
Unknown child process pid 3450080, will report returncode 255
Loop <_UnixSelectorEventLoop running=False closed=True debug=False> that 
handles pid 3450080 is closed
Unknown child process pid 3467752, will report returncode 255
Loop <_UnixSelectorEventLoop running=False closed=True debug=False> that 
handles pid 3467752 is closed
.Unknown child process pid 3468093, will report returncode 255
.

1 test altered the execution environment:
test_asyncio

--
components: Tests, asyncio
messages: 357470
nosy: asvetlov, pablogsal, yselivanov
priority: normal
severity: normal
status: open
title: test_asyncio altered the execution environment
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38911] include __del__ in Generator ABC

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Guido should decide this one.  The PEP isn't entirely clear whether __del__ is 
a CPython implementation detail, nor is it clear whether the intent was for the 
isinstance() to insist on __del__ being present.  Also, at one time __del__ 
interfered with garbage collection, but that may have been fixed.

--
assignee:  -> gvanrossum
nosy: +gvanrossum, pje, rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1812] doctest _load_testfile function -- newline handling seems incorrect

[Peter Donis]

Change by Peter Donis :


--
versions: +Python 3.9 -Python 2.7, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1812] doctest _load_testfile function -- newline handling seems incorrect

[Peter Donis]

Peter Donis  added the comment:

I have submitted pull request #17385 regarding this issue:

https://github.com/python/cpython/pull/17385

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1812] doctest _load_testfile function -- newline handling seems incorrect

[Peter Donis]

Change by Peter Donis :


--
pull_requests: +16867
pull_request: https://github.com/python/cpython/pull/17385

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38911] include __del__ in Generator ABC

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
pull_requests: +16866
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17384

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38911] include __del__ in Generator ABC

[Matt McEwen]

New submission from Matt McEwen :

The Generator ABC in the standard library lets users define objects that follow 
the Generator specification given in PEP342, and which can be used in the place 
of builtin generator objects. 

This was originally added in issue 24018

The ABC enforces that the methods `send`, `throw` and `close` are implemented, 
as per the spec. It doesn't enforce that `__del__` is implemented. PEP342 
specifies that `__del__` be a wrapper for `close`, such that when a generator 
object is garbage collected `close` is called.

For a user (like me) implementing such a generator object by inheriting the 
Generator ABC, the resulting objects do not have `close` called when they are 
garbage collected. Fixing this requires manually implementing a `__del__` that 
calls `close`. 

Ideally from the user perspective, inheriting from Generator should be enough 
to guarantee that the object behaves like a generator object, provided the 
abstract methods are correctly implemented. This could be easily achieved by 
implementing `__del__` concretely in the ABC as a wrapper for `close`:

def __del__(self):
self.close()

--
components: Library (Lib)
messages: 357467
nosy: Matt McEwen
priority: normal
severity: normal
status: open
title: include __del__ in Generator ABC
type: enhancement
versions: Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38910] AssertionError: ElementTree not initialized, missing root

[Joseph Reagle]

Joseph Reagle  added the comment:

Here's the error:

```
Traceback (most recent call last):
  File "", line 1, in 
  File "src/lxml/etree.pyx", line 2291, in lxml.etree._ElementTree.xpath
  File "src/lxml/etree.pyx", line 1869, in 
lxml.etree._ElementTree._assertHasRoot
AssertionError: ElementTree not initialized, missing root
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38910] AssertionError: ElementTree not initialized, missing root

[Joseph Reagle]

New submission from Joseph Reagle :

I've attached a simple XHTML file with which the 3.8 interpreter throws an 
error on the following code, but 3.7 does not. (You'll have to change the path 
in the code below.)

```
from io import StringIO, BytesIO
from lxml import etree
import os
import readline

HOME = os.path.expanduser("~")

ofile = HOME + '/data/2web/reagle.org/joseph/plan/plans/index.html'
plan_fd = open(ofile, 'r', encoding='utf-8', errors='replace')
plan_content = plan_fd.read()
plan_fd.close()
plan_tree = etree.parse(StringIO(plan_content),
etree.XMLParser(ns_clean=True, recover=True))
ul_found = plan_tree.xpath(
'''//x:div[@id='Done']/x:ul''',
namespaces={'x': 'http://www.w3.org/1999/xhtml'})
```

--
components: Library (Lib)
files: index.html
messages: 357465
nosy: joseph.reagle
priority: normal
severity: normal
status: open
title: AssertionError: ElementTree not initialized, missing root
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file48742/index.html

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38884] __import__ is not thread-safe on Python 3

[Valentyn Tymofieiev]

Valentyn Tymofieiev  added the comment:

@brett.cannon, looks like I inadvertently dropped you from nosy list, so in 
case you missed the updates - please take a look at issue38884.zip. Thank you.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38884] __import__ is not thread-safe on Python 3

[Valentyn Tymofieiev]

Change by Valentyn Tymofieiev :


--
nosy: +brett.cannon, eric.snow, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38884] __import__ is not thread-safe on Python 3

[Valentyn Tymofieiev]

Change by Valentyn Tymofieiev :


--
nosy: +Valentyn Tymofieiev -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38884] __import__ is not thread-safe on Python 3

[Valentyn Tymofieiev]

Change by Valentyn Tymofieiev :


--
nosy: +brett.cannon -Valentyn Tymofieiev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21063] Touch up one-line descriptions of modules for module index

[miss-islington]

miss-islington  added the comment:


New changeset 089387ed1f47d9443b5f28bb1863294eeac2de08 by Miss Islington (bot) 
in branch '3.8':
bpo-21063: Improve module synopsis for distutils (GH-17363)
https://github.com/python/cpython/commit/089387ed1f47d9443b5f28bb1863294eeac2de08


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21063] Touch up one-line descriptions of modules for module index

[miss-islington]

Change by miss-islington :


--
pull_requests: +16865
pull_request: https://github.com/python/cpython/pull/17383

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21063] Touch up one-line descriptions of modules for module index

[Brett Cannon]

Change by Brett Cannon :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21063] Touch up one-line descriptions of modules for module index

[miss-islington]

Change by miss-islington :


--
pull_requests: +16864
pull_request: https://github.com/python/cpython/pull/17382

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21063] Touch up one-line descriptions of modules for module index

[miss-islington]

Change by miss-islington :


--
pull_requests: +16863
pull_request: https://github.com/python/cpython/pull/17381

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21063] Touch up one-line descriptions of modules for module index

[Brett Cannon]

Brett Cannon  added the comment:


New changeset f8a6316778faff3991144c3aec4fa92d7b30a72b by Brett Cannon (Sanchit 
Khurana) in branch 'master':
bpo-21063: Improve module synopsis for distutils (GH-17363)
https://github.com/python/cpython/commit/f8a6316778faff3991144c3aec4fa92d7b30a72b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

[STINNER Victor]

STINNER Victor  added the comment:

Fabio: "I don't really see a problem in breaking the API in major releases (so, 
having access for it in the internal API but without a backwards-compatibility 
guarantee seems like a good fit for me)."

In Python, the internal C API now means that you have to define 
Py_BUILD_CORE_MODULE macro and include a special "internal/pycore_pystate.h" 
header file.

If you are fine with doing that in PyCharm, what is the purpose of this issue?

Do you mean in *internal* API or a *private* API? Private API is basically a 
public API with weaker backward compatibility warranties and with a "_" prefix 
(_Py, _PY prefix).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Steve Dower]

Steve Dower  added the comment:

> Perhaps generalize this as splitext(basename(sys.executable))[0] in
> order to support names other than "python" and "pythonw". It would have
> to handle quoting enough to ignore an initial double quote when parsing
> the executable name out of an args string, or _winapi could wrap the
> shell's CommandLineToArgvW function to handle this.

This is where we've hit the point of complexity that it would have to be added 
as a new API. And since it's now opt-in, we may as well document that 
shutil.which() is the recommended way to resolve a filename against PATH (and 
make that work properly... ISTR it has some inconsistencies with the OS).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38862] IDLE: Include end newlines in whitespace fix.

[Terry J. Reedy]

Change by Terry J. Reedy :


--
pull_requests: +16862
pull_request: https://github.com/python/cpython/pull/17380

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38862] IDLE: Include end newlines in whitespace fix.

[Terry J. Reedy]

Change by Terry J. Reedy :


--
pull_requests: +16861
pull_request: https://github.com/python/cpython/pull/17379

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Eryk Sun]

Eryk Sun  added the comment:

> Possibly we could handle "python[.exe]" literals by substituting 
> sys.executable transparently?

Perhaps generalize this as splitext(basename(sys.executable))[0] in order to 
support names other than "python" and "pythonw". It would have to handle 
quoting enough to ignore an initial double quote when parsing the executable 
name out of an args string, or _winapi could wrap the shell's 
CommandLineToArgvW function to handle this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38890] A subprocess.Popen created with creationFlags=DETACHED_PROCESS on Windows should not emit a ResourceWarning

[Eryk Sun]

Eryk Sun  added the comment:

> Didn't we clean up this warning completely on Windows recently? It
> apparently matters on POSIX to join() the child process, but on
> Windows you aren't going to leak any resources by just forgetting
> about it, so I thought we stopped tracking them entirely.

There's only a PID in POSIX, so the OS keeps a zombie process until the parent 
waits on it. Windows uses a handle for the process, which gets automatically 
closed when the Popen instance is finalized, so we don't have to explicitly 
wait on the process, and doing so has no bearing on the lifetime of the kernel 
process object. 

Recently, a change was made to set _active to None in Windows, so _cleanup is 
no longer called unnecessarily. But Victor wanted to keep the resource warning 
for its educational value, given some programmers might not be aware when 
they've left a background process running. So Popen.__del__ still issues a 
resource warning if returncode is None. The OP's workaround to manually set 
returncode works, but it's undocumented and feels like a kludge to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Steve Dower]

Steve Dower  added the comment:

> `shutil.which` could be a direction, but still not enough since it’d break 
> `flit install --python=py` because that’s give you the location of py.exe, 
> not the actual interperter.

This would be fine if you still run the process to get its sys.executable.

Your specific example would never have worked, FWIW, as it always would have 
picked up pythonA rather than the application one or the base one, unless you 
were relying on python3/python3.7 not being available on Windows (which is no 
longer true - they are included in the Store package now).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Tzu-ping Chung]

Tzu-ping Chung  added the comment:

To provide concrete context, the problem I’m facing is with how Flit resolves 
`flit install --python`:

https://github.com/takluyver/flit/blob/7e65ffc7a540d76b96de0df473d3edff6f97c26c/flit/__init__.py#L18-L28

Generally the setup is to install Flit into a globally available location 
(let’s name it env A), so it’s usable for every project and environment. For a 
project foo you’d have a virtual environment (env X) that’s created from a base 
interpreter (env B, which may or may not be the same as env A).

So the comment workflow would look like this:

> pythonB -m venv project-env
> project-env\Scripts\activate.bat
(project-env) > pythonA -m flit install --python=pythonX

This results in the following subprocess call:

subprocess.check_output(
["pythonX", "-c", "import sys; print(sys.executable)"],
universal_newlines=True,
).strip()

And ideally (pre-3.7.2 Windows, or current POSIX behaviour) this would give you 
the absolute path to pythonX. But right now on Windows the result is pythonB.

So if this is to be determined as acceptable behaviour, we’d need to come up 
with a suggestion how this code can be rewritten. `shutil.which` could be a 
direction, but still not enough since it’d break `flit install --python=py` 
because that’s give you the location of py.exe, not the actual interperter. 
What else?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38890] A subprocess.Popen created with creationFlags=DETACHED_PROCESS on Windows should not emit a ResourceWarning

[Steve Dower]

Steve Dower  added the comment:

Didn't we clean up this warning completely on Windows recently?

It apparently matters on POSIX to join() the child process, but on Windows you 
aren't going to leak any resources by just forgetting about it, so I thought we 
stopped tracking them entirely.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38906] copy2 doesn't copy metadata on Windows and MacOS

[Steve Dower]

Steve Dower  added the comment:

As a general statement (I haven't taken the time yet to understand all the 
intricacies of this particular issue), I would prefer to see CopyFile used 
everywhere on Windows for the performance and reliability benefits. Perfect 
POSIX emulation here doesn't seem helpful.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Steve Dower]

Steve Dower  added the comment:

Yeah, this definitely relates to how Windows handles unqualified argv[0] in 
CreateProcess. Though I thought we checked that out in another issue and 
decided that "most" people are correctly using sys.executable here? (Or decided 
that it was documented well enough that they should, and using a "python" 
literal was relying on OS behaviour.)

I'm not a fan of trying to override the OS handling in subprocess, though that 
would be the fix. Possibly we could handle "python[.exe]" literals by 
substituting sys.executable transparently?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38901] Add a CLI flag to venv to use the pwd basename as the prompt

[Brett Cannon]

Brett Cannon  added the comment:

@Vinay you got exactly what I mean. I had to actually teach a ton of people 
that shell trick with basename and pwd on Twitter, that trick doesn't seem to 
be something people know about, and yet a ton of people seem to be creating 
virtual environments with prompts manually specified like that.

Plus that trick varies from shell to shell and we already have enough issues 
just explaining to people how to activate between Windows and UNIX and then 
between even shells on the same OS.

So this is entirely for convenience.

@Julien default directory names is an entirely separate topic as people can't 
agree on the default name (e.g. I prefer `.venv` as my default).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38890] A subprocess.Popen created with creationFlags=DETACHED_PROCESS on Windows should not emit a ResourceWarning

[Eryk Sun]

Change by Eryk Sun :


--
components: +Windows
nosy: +paul.moore, steve.dower, tim.golden, zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38890] A subprocess.Popen created with creationFlags=DETACHED_PROCESS on Windows should not emit a ResourceWarning

[Eryk Sun]

Eryk Sun  added the comment:

For a console application, normally the system connects to either the console 
that's inherited from the parent or, if no console is inherited, a newly 
allocated console. The creation flag DETACHED_PROCESS sets the initial console 
handle in the child to a special value that tells the system to skip connecting 
to a console. If the child process doesn't subsequently call AllocConsole, 
AttachConsole, or create a visible top-level window, then it will considered to 
be background process. 

Note that 'detached' in this case doesn't mean detached from the parent 
process. There's no such attachment to begin with because Windows does not 
maintain a process tree. It uses a nested job-object hierarchy as the 
equivalent of a Unix process tree. The creation flag CREATE_BREAKAWAY_FROM_JOB 
breaks away from jobs that allow it.

The Popen warning is a tool to help programmers become aware of leaked child 
processes. I do think there should be a documented way to disable this warning 
for a child process. However, this is unrelated to the DETACHED_PROCESS flag.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38895] performance degradation creating a mock object (by factor 7-8)

[Marcel Zięba]

Marcel Zięba  added the comment:

"It seems creating the signature of NonCallableMock.__init__ per mock creation 
is expensive and since it doesn't change can we just create the signature once 
and set it as a module level attribute? There might still be room for some more 
optimisations here to reduce the impact."

This is already done in master branch ;)


"This could be moved to _get_child_mock so that the Mock creation itself for 
all other mocks and common use case is faster. Creating child mocks will have 
the iscoroutine function check performed where maybe we can populate the 
_spec_async list and use it for subsequent calls."

This seems like a reasonable solution.
I've tested it and it improves mock creation speed 2x.

Do you mind if I create PR for it? I would like to start contributing to 
CPython ;)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36375] PEP 499 implementation: "python -m foo" binds the main module as both __main__ and foo in sys.modules

[Eric Snow]

Eric Snow  added the comment:

FWIW, I have some feedback on the PEP.  (See msg357448.)  Can we discuss here 
or should I open a mailing list thread?

--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37292] _xxsubinterpreters: Can't unpickle objects defined in __main__

[Eric Snow]

Eric Snow  added the comment:

In the meantime that leaves the workarounds that @crusaderky originally 
identified.  You could also:

* manually run __main__ in the subinterpreter first (sort of like 
multiprocessing does automatically); this works because the namespace of 
__main__ is not reset for each run_string() call
* (for -m) update __module__ of the relevant objects to be the actual module 
name rather than "__main__"

In the case of that second point, it relates to PEP 499 (which will ensure that 
the module is added to sys.modules in the -m case).  However, that PEP doesn't 
say anything about updating __module__ for objects.  I'll bring that up there.  
With that solution the problem in this issue would go away.

Note that it won't help for objects in the __main__ of subinterpreters, since 
they do not correspond to executed modules.  Hmm, maybe it could still work...

Regardless, I'll open issues over on 
https://github.com/ericsnowcurrently/multi-core-python to track these possible 
future enhancements.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37292] _xxsubinterpreters: Can't unpickle objects defined in __main__

[Eric Snow]

Eric Snow  added the comment:

Yeah, the case of pickle + __main__ is an awkward one. [1]  However, the 
approach taken by multiprocessing isn't the right one for subinterpreters.

Multiprocessing operates under 2 design points that do not apply to 
subinterpreters:

* every process is running in the same __main__ module (sans the "script" part)
* pickle is a critical part of data-passing

For spawn, multiprocessing automatically runs the original __main__ module in 
each newly spawned process. [2]  Note that it runs it there with __name__ set 
to __mp_main__ (rather than __main__), to keep the "script" part from running.  
Subinterpreters could be made to work like this [3] but in reality they are 
more like processes created using the subprocess module.

I do not expect that to change.  However, there is room for add opt-in support 
for rerunning __main__ in a subinterpreter, or helpers to accomplish the same.  
We can address such opt-in support or helpers in a separate issue later.  For 
now we are focusing on the fundamentals (at least in the context of PEP 554).


[1] Note that the problem is only with the __main__ module.  For other modules 
pickle does the right thing.
[2] 
https://github.com/python/cpython/blob/master/Lib/multiprocessing/spawn.py#L287
[3] I expect we will see subinterpreters supported in the multiprocessing 
module just like threads are supported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38861] zipfile: Corrupts filenames containing non-UTF8 characters

[Jon Nalley]

Jon Nalley  added the comment:

Please see a detailed explanation of the behavior here:
https://gist.github.com/jnalley/cec21bca2d865758bc5e23654df28bd5

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38907] Add IPv6 Dual-Stack control for http.server

[JIanqiu Tao]

Change by JIanqiu Tao :


--
keywords: +patch
pull_requests: +16860
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17378

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38909] module name 'stat.py'

[Zachary Ware]

Zachary Ware  added the comment:

It's not clear what you're reporting here, but it looks like the classic issue 
that naming your module the same as one in the standard library will cause 
things to break.  There are numerous other issues about that; please find the 
active one if you have anything to add to it.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38909] module name 'stat.py'

[da-dada]

New submission from da-dada :

well, call your module 'stat.py' and python takes over full ownership: it's 
somewhat all public..

--
components: Windows
messages: 357444
nosy: da-dada, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: module name 'stat.py'
type: compile error
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20928] xml.etree.ElementInclude does not include nested xincludes

[Stefan Behnel]

Stefan Behnel  added the comment:

I think setting "xml:base" from ElementInclude is worth another ticket.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38576] CVE-2019-18348: CRLF injection via the host part of the url passed to urlopen()

[Riccardo Schirone]

Riccardo Schirone  added the comment:

The glibc issue mentioned in the first comment is CVE-2016-10739 .

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20928] xml.etree.ElementInclude does not include nested xincludes

[Stefan Behnel]

Stefan Behnel  added the comment:


New changeset c6a7bdb356835c9d7513b1ea6846683d446fe6c3 by Stefan Behnel in 
branch 'master':
bpo-20928: support base-URL and recursive includes in etree.ElementInclude 
(#5723)
https://github.com/python/cpython/commit/c6a7bdb356835c9d7513b1ea6846683d446fe6c3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38870] Expose ast.unparse in the ast module

[Batuhan]

Change by Batuhan :


--
pull_requests: +16859
pull_request: https://github.com/python/cpython/pull/17377

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Eryk Sun]

Eryk Sun  added the comment:

> "running an unqualified python and expecting a PATH search to pick up
> the same executable as the parent shell would is not supported and may
> produce unexpected results"

CreateProcessW finds "python.exe" in __APPDIR__ before it even searches PATH. I 
expect that some scripts depend on this when python.exe isn't in PATH, or when 
a different version is in PATH. If subprocess implements its own search, it can 
continue to prioritize the *effective* application directory, from 
dirname(sys.executable).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Paul Moore]

Paul Moore  added the comment:

I presume there's also the option of setting up the environment (or however 
it's done now - I know the details changed as the feature was developed) so 
that the "base" python.exe pretends to be the venv one, exactly as the wrapper 
does.

However, that may well have other difficult-to-fix implications, not least that 
calling the base Python using an explicit full path should act like the base 
Python, and *not* like the venv one.

IMO, the key thing here is that either the various limitations/quirks of 
redirecting to the base Python should either be treated as bugs, or they should 
be documented (even if only in the form of explicitly saying not to rely on any 
specific behaviour - e.g. "running an unqualified python and expecting a PATH 
search to pick up the same executable as the parent shell would is not 
supported and may produce unexpected results").

Virtual environments are a key part of most Python development workflows, and 
virtualenv is in the process of switching to use the core venv module 
internally. When that happens, there will be a lot more visibility for 
unexpected behaviours like this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38870] Expose ast.unparse in the ast module

[miss-islington]

miss-islington  added the comment:


New changeset dedfbc33011dd39b7b1c86a5adfacc4943f3 by Miss Islington (bot) 
(Pablo Galindo) in branch 'master':
bpo-38870: Remove dependency on contextlib to avoid performance regression on 
import (GH-17376)
https://github.com/python/cpython/commit/dedfbc33011dd39b7b1c86a5adfacc4943f3


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38870] Expose ast.unparse in the ast module

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +16858
pull_request: https://github.com/python/cpython/pull/17376

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Eryk Sun]

Eryk Sun  added the comment:

> whens searching for executables, Windows gives "exes that are in the
> same directory as the currently executing code" priority over PATH.

That subprocess lets CreateProcessW use a platform-dependent search that 
prioritizes the application directory has come up in previous issues. To avoid 
this, we'd have to implement our own search for the given or parsed executable 
name. Then pass the fully-qualified executable path as the lpApplicationName 
name of CreateProcessW. This is how CMD works, since it has its own search 
routine that incorporates the PATHEXT environment variable.

Because the application directory is searched before the working directory (if 
the working directory is searched at all, depending on context), this issue 
also affects searching for executable paths that contain a path separator. In 
Unix a relative path that contains a path separator is always relative to the 
working directory, but Windows CreateProcessW uses a normal search for a 
relative name unless it explicitly references the working directory as "." 
(e.g. ".\Scripts\pip.exe" instead of "Scripts\pip.exe").

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38881] unexpected behaviour of random.choices with zero weights

[Iza Romanowska]

Iza Romanowska  added the comment:

Many thanks for patching it! 
Much appreciated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38895] performance degradation creating a mock object (by factor 7-8)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Another point is that _spec_asyncs is a list of attributes that pass 
asyncio.iscoroutinefunction which could be also little expensive [0]. The check 
is made for the attribute to be async only when the child mock is created to 
return an AsyncMock [1] during creation. This could be moved to _get_child_mock 
so that the Mock creation itself for all other mocks and common use case is 
faster. Creating child mocks will have the iscoroutine function check performed 
where maybe we can populate the _spec_async list and use it for subsequent 
calls.

# Baseline 3.7

$ python3.7 -m timeit -s 'from unittest.mock import Mock' 'Mock()'
2 loops, best of 5: 17.6 usec per loop

# Move NonCallableMock.__init__ signature to module level attribute. (Python 
3.8 branch HEAD)

$ ./python.exe -m timeit -s 'from unittest.mock import Mock' 'Mock()'
5000 loops, best of 5: 62.1 usec per loop

# Move the iscoroutinefunction check to the child mock creation. I didn't do 
the child mock creation benchmark yet and populating _spec_async as the 
attribute is found to be async would resolve doing iscoroutinefunction check 
everytime. (Python 3.8 branch HEAD)

$ ./python.exe -m timeit -s 'from unittest.mock import Mock' 'Mock()'
1 loops, best of 5: 28.3 usec per loop

[0] 
https://github.com/python/cpython/blob/27fc3b6f3fc49a36d3f962caac5c5495696d12ed/Lib/unittest/mock.py#L488-L492
[1] 
https://github.com/python/cpython/blob/27fc3b6f3fc49a36d3f962caac5c5495696d12ed/Lib/unittest/mock.py#L987


diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py
index 488ab1c23d..7ff99407ab 100644
--- a/Lib/unittest/mock.py
+++ b/Lib/unittest/mock.py
@@ -403,7 +403,6 @@ class NonCallableMock(Base):
 bases = (cls,)
 if not issubclass(cls, AsyncMock):
 # Check if spec is an async object or function
-sig = inspect.signature(NonCallableMock.__init__)
 bound_args = sig.bind_partial(cls, *args, **kw).arguments
 spec_arg = [
 arg for arg in bound_args.keys()
@@ -491,11 +490,6 @@ class NonCallableMock(Base):
_eat_self=False):
 _spec_class = None
 _spec_signature = None
-_spec_asyncs = []
-
-for attr in dir(spec):
-if asyncio.iscoroutinefunction(getattr(spec, attr, None)):
-_spec_asyncs.append(attr)
 
 if spec is not None and not _is_list(spec):
 if isinstance(spec, type):
@@ -513,7 +507,6 @@ class NonCallableMock(Base):
 __dict__['_spec_set'] = spec_set
 __dict__['_spec_signature'] = _spec_signature
 __dict__['_mock_methods'] = spec
-__dict__['_spec_asyncs'] = _spec_asyncs
 
 def __get_return_value(self):
 ret = self._mock_return_value
@@ -989,7 +982,8 @@ class NonCallableMock(Base):
 For non-callable mocks the callable variant will be used (rather than
 any custom subclass)."""
 _new_name = kw.get("_new_name")
-if _new_name in self.__dict__['_spec_asyncs']:
+attribute = getattr(self.__dict__['_spec_class'], _new_name, None)
+if asyncio.iscoroutinefunction(attribute):
 return AsyncMock(**kw)
 
 _type = type(self)
@@ -1032,6 +1026,8 @@ class NonCallableMock(Base):
 return f"\n{prefix}: {safe_repr(self.mock_calls)}."
 
 
+sig = inspect.signature(NonCallableMock.__init__)
+
 
 def _try_iter(obj):
 if obj is None:

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38893] broken container/selinux integration

[Leif Middelschulte]

Leif Middelschulte  added the comment:

For the sake of completeness, the content of `/tmp/test.py`:

```
#!/usr/bin/env python3

from shutil import copy2

copy2('/tmp/some_file', '/relabel_bug/failure')
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38893] broken container/selinux integration

[Leif Middelschulte]

Leif Middelschulte  added the comment:

> Could you please provide name and value of the setxattr() call? I bet it's 
> trying to setxattr 'security.selinux' extended file attribute.

(Pdb) bt full
  /usr/lib64/python3.7/pdb.py(1701)main()
-> pdb._runscript(mainpyfile)
  /usr/lib64/python3.7/pdb.py(1570)_runscript()
-> self.run(statement)
  /usr/lib64/python3.7/bdb.py(585)run()
-> exec(cmd, globals, locals)
  (1)()->None
  /tmp/test.py(6)()->None
-> copy2('/tmp/some_file', '/relabel_bug/failure')
  /usr/lib64/python3.7/shutil.py(267)copy2()
-> copystat(src, dst, follow_symlinks=follow_symlinks)
  /usr/lib64/python3.7/shutil.py(209)copystat()
-> _copyxattr(src, dst, follow_symlinks=follow)
> /usr/lib64/python3.7/shutil.py(165)_copyxattr()
-> os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
(Pdb) p dst
'/relabel_bug/failure'
(Pdb) p name
'security.selinux'
(Pdb) p value
b'system_u:object_r:fusefs_t:s0\x00'
(Pdb)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38854] Decorator breaks inspect.getsource

[PCManticore]

PCManticore  added the comment:

Thanks for the report!

Turns out this was a bug in ``inspect.BlockFinder`` which is responsible for 
extracting out the source code from a given block. This class was considering 
")" inside a decorator call as closing the decorator itself, even when the 
decorator call was accepting additional arguments that needed parentheses, such 
as function calls or tuples.

Just submitted a PR https://github.com/python/cpython/pull/17374 to address 
this issue.

--
nosy: +Claudiu.Popa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38854] Decorator breaks inspect.getsource

[PCManticore]

Change by PCManticore :


--
keywords: +patch
pull_requests: +16857
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17374

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38895] performance degradation creating a mock object (by factor 7-8)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Thanks Marcel for the pointer. I can confirm the performance impact. This 
occurs in the common case where not being an AsyncMock the signature of 
NonCallableMock.__init__ is created every time and then bind_partial is used to 
detect the spec being supplied to be async. It seems creating the signature of 
NonCallableMock.__init__ per mock creation is expensive and since it doesn't 
change can we just create the signature once and set it as a module level 
attribute? There might still be room for some more optimisations here to reduce 
the impact.

$ python3.7 -m timeit -s 'from unittest.mock import Mock' 'Mock()'
2 loops, best of 5: 17.6 usec per loop

# Creating signature object per run (Python 3.8.0)

$ ./python.exe -m timeit -s 'from unittest.mock import Mock' 'Mock()'
2000 loops, best of 5: 109 usec per loop

# Set the signature object of NonCallableMock.__init__ as a private module 
level attribute (Python 3.8.0)

./python.exe -m timeit -s 'from unittest.mock import Mock' 'Mock()'
5000 loops, best of 5: 66.4 usec per loop

--
nosy: +cjw296, lisroach, mariocj89, michael.foord

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Paul Moore]

Paul Moore  added the comment:

Yes, it does.

I think we'd need input from Steve Dower here, as these changes were made (I 
believe) in support of the Windows Store build of Python, so any changes would 
need to be considered in the light of how they would affect that. I do, 
however, consider this to be a regression that should be fixed.

BTW, just for completeness,

>>> subprocess.check_output([sys.executable, '-c', 'import sys; 
>>> print(sys.executable)'])

works as I'd expect, and that's the idiom that is often used. So relying on a 
path search to find the correct Python can be considered an unusual case (but 
nevertheless one I'd expect to be fixed).

I assume that the issue here is that the code is being run by the python.dll in 
the base environment, and whens searching for executables, Windows gives "exes 
that are in the same directory as the currently executing code" priority over 
PATH. See 
https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw,
 specifically

"""
If the file name does not contain a directory path, the system searches for the 
executable file in the following sequence:

1. The directory from which the application loaded.
2. The current directory for the parent process.
3. The 32-bit Windows system directory. Use the GetSystemDirectory function to 
get the path of this directory.
4. The 16-bit Windows system directory. There is no function that obtains the 
path of this directory, but it is searched. The name of this directory is 
System.
5. The Windows directory. Use the GetWindowsDirectory function to get the path 
of this directory.
6. The directories that are listed in the PATH environment variable. Note that 
this function does not search the per-application path specified by the App 
Paths registry key. To include this per-application path in the search 
sequence, use the ShellExecute function.
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Tzu-ping Chung]

Tzu-ping Chung  added the comment:

I tested the following in various versions (all 64-bit) in a VM. All 
installations are 64-bit per-user.

> py -m venv testenv
> testenv\Scripts\python.exe -c "import subprocess; 
> print(subprocess.check_output(['python', '-c', 'import sys; 
> print(sys.executable)']))"

3.8.0: Incorrect
3.7.5: Incorrect
3.7.4: Incorrect
3.7.3: Incorrect
3.7.2: Correct
3.6.8: Correct
3.7.1: Correct
3.7.0: Correct

So the change seems to have happened somewhere between 3.7.2 and 3.7.3. Does 
this timeline line up with the venv redirector change?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38905] venv python reports wrong sys.executable in a subprocess on Windows

[Paul Moore]

Paul Moore  added the comment:

The behaviour in this area is different between 3.7.0, 3.7.2, and 3.7.3 (at 
least). I have reproduced the issue with 3.7.3. Steve Dower made changes to the 
way the python executable works in venvs in the point releases of 3.7 - see 
https://github.com/pypa/virtualenv/issues/1380 and 
https://github.com/pypa/virtualenv/issues/1339 for some discussion of how this 
affected virtualenv.

I suspect this issue is related - from 3.7.2 onwards, the python.exe in a venv 
is a redirector which runs the "base" python.exe, but with sys.executable 
showing the redirector rather than the actual running exe.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com