[issue40124] Clearer assertion error

[SilentGhost]

Change by SilentGhost :


--
versions:  -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40148] Add PurePath.with_stem()

[SilentGhost]

Change by SilentGhost :


--
nosy: +pitrou
type:  -> enhancement
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28029] Replace and empty strings

[Glenn Linderman]

Glenn Linderman  added the comment:

Nope:

I guess for   x.replace( a, b, c )  the workaround would be

x.replace( a, b, c ) if x else x.replace( a, b )

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28029] Replace and empty strings

[Glenn Linderman]

Glenn Linderman  added the comment:

Thanks Stèphańe and Serhiy, I just discovered this strange behavior in 3.8, and 
wondered how my logic was wrong, until I pinpointed the inconsistent behaviour 
of str.replace with an empty first parameter and replace count of 1.

Glad to see it is fixed in 3.9.

I guess for   x.replace( a, b, c )  the workaround would be

x.replace( a, b, c ) if a else x.replace( a, b )

At least for recent versions of Python 3.

--
nosy: +v+python

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40083] No run option available in python idle in version 3.8.2

[Rajesh R Naik]

Rajesh R Naik  added the comment:

i thought this issue is with windows home edition 
please confirm and resolve

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40144] resource: RLIMIT_xxx and RUSAGE_xxx constants are missing on macOS 10.15.4

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40124] Clearer assertion error

[Kyle Stanley]

Kyle Stanley  added the comment:

> Do we really want this to be just an assert, or do we want to raise another 
> type of exception?

IMO, we should look into converting this into an exception. Mistakenly having a 
task await StreamWriter.drain() at the same time as another is calling 
StreamWriter.write() does seem like a reasonable programming error that should 
preferably have an informative error message. Optimally, assertions shouldn't 
occur from normal programming errors in production.

The tricky part is figuring out how to implement it properly. I'm not 100% 
certain that we can make any guarantees that when the _drain_waiter future 
hasn't been cancelled and not set to None that someone is mistakenly doing the 
above. It could potentially trigger from other errors.

Either way though, I think just adding a message to the assert could end up 
being misleading if someone else encounters this in production for another 
reason. Instead, I think we could leave a comment there for now and in the long 
term figure out how to properly implement the exception or warning. We also 
need a reliable way to reproduce it, mainly for the purpose of writing a new 
test to ensure the exception is correctly triggered when someone makes the 
above programming error.

--
nosy: +aeros

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40151] _overlapped room for improvement

[Alexander Riccio]

Change by Alexander Riccio :


--
keywords: +patch
pull_requests: +18658
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19298

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40145] Pyshellext room for binary size improvement

[Alexander Riccio]

Change by Alexander Riccio :


--
pull_requests: +18659
pull_request: https://github.com/python/cpython/pull/19298

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40151] _overlapped room for improvement

[Alexander Riccio]

New submission from Alexander Riccio :

Similarly to bpo-40145, I've tweaked build options to reduce the size of the
binary.

This patch turns on (for release builds) Whole Program Optimization, MinSpace
optimization, /Ob2 AnySuitable function inlining, /Zo (so that people can still
debug it when lots of code has been optimized out), /OPT:REF dead code
elimination, /OPT:ICF COMDAT folding, Link Time Code Generation, and DebugFull
debugging information creation.

For debug builds, it enables all of that, except instead of the MinSpace
optimization, it's only /Ob2 with custom optimization. My intent is to not
optimize any asserts or similar checks, while still getting the benefit of
dead and duplicate code elimination.


_overlapped.pyd   32 bit unimproved release size: 31KB
_overlapped.pyd   32 bit   improved release size: 29KB
size reduction:   -3KB
 %  size reduction:10%

_overlapped_d.pyd 32 bit unimproved release size: 55KB
_overlapped_d.pyd 32 bit   improved release size: 34KB
size reduction:  -21KB
 %  size reduction:38%




_overlapped.pyd   64 bit unimproved release size: 39KB
_overlapped.pyd   64 bit   improved release size: 36KB
size reduction:   -3KB
 %  size reduction: 8%


_overlapped_d.pyd 64 bit unimproved release size: 74KB
_overlapped_d.pyd 64 bit   improved release size: 41KB
size reduction:  -33KB
 %  size reduction:45%


If this patch is merged, and all 7 million (estimated) Python developers update 
their installation, I calculate that I just saved the PSF 21GB worth of 
bandwidth costs :)

--
components: Windows
messages: 365567
nosy: Alexander Riccio, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: _overlapped room for improvement
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40149] test_threading leaked [38, 38, 38] references, sum=114

[Dong-hee Na]

Dong-hee Na  added the comment:

FYI,
first gc collect 772
secondary gc collect 4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40150] (minor) mismatched argument in overlapped_RegisterWaitWithQueue call to RegisterWaitForSingleObject

[Alexander Riccio]

New submission from Alexander Riccio :

This popped out at me while looking for something else. It's probably not much 
of an actual problem, since the wrong datatype is larger than the correct one, 
but it's worth fixing.

The problem is in overlapped_RegisterWaitWithQueue, at overlapped.c:297 (in 
_overlapped):

if (!RegisterWaitForSingleObject(
, Object, (WAITORTIMERCALLBACK)PostToQueueCallback,
pdata, Milliseconds,
WT_EXECUTEINWAITTHREAD | WT_EXECUTEONLYONCE))


...it stood out to me immediately while I was paging past, since function 
pointer casts of this sort are almost always wrong, and I've seen it too many 
times.

WAITORTIMERCALLBACK is a typedef of WAITORTIMERCALLBACKFUNC:

typedef VOID (NTAPI * WAITORTIMERCALLBACKFUNC) (PVOID, BOOLEAN );

...and PostToQueueCallback is defined as:

static VOID CALLBACK
PostToQueueCallback(PVOID lpParameter, BOOL TimerOrWaitFired)

...where BOOL is an int, and BOOLEAN is an unsigned char. I guess there could 
be some kind of issue down the line if the generated code tries to read the 
BOOLEAN/int from the register/memory location where there's actually an 
unsigned char, but after about an hour of debugging, I can't see it. The 
documentation also states explicitly that this should be either TRUE or FALSE. 
Also, it doesn't matter what we actually pass to PostQueuedCompletionStatus: 
https://devblogs.microsoft.com/oldnewthing/20070525-00/?p=26693

By changing the (incorrect) BOOL declaration to BOOLEAN, then we don't need the 
cast.

--
components: IO
messages: 365565
nosy: Alexander Riccio
priority: normal
severity: normal
status: open
title: (minor) mismatched argument in overlapped_RegisterWaitWithQueue call to 
RegisterWaitForSingleObject
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39812] Avoid daemon threads in concurrent.futures

[Kyle Stanley]

Kyle Stanley  added the comment:

> Thanks Kyle for the test_asyncio fix!

No problem! Thanks for the review. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39812] Avoid daemon threads in concurrent.futures

[STINNER Victor]

STINNER Victor  added the comment:

Kyle fixed bpo-40115, I close again this issue. Thanks Kyle for the 
test_asyncio fix!

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40115] test_asyncio leaked [3, 3, 3] references, sum=9

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40115] test_asyncio leaked [3, 3, 3] references, sum=9

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8ec7cb53f0705509bac20c96d19fdbd9baec0cb2 by Kyle Stanley in 
branch 'master':
bpo-40115: Fix refleak in test_asyncio.test_run_in_executor_cancel() (GH-19282)
https://github.com/python/cpython/commit/8ec7cb53f0705509bac20c96d19fdbd9baec0cb2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39837] Remove Azure Pipelines from GitHub PRs

[STINNER Victor]

STINNER Victor  added the comment:

> About the "Install Dependencies" issue, I created bpo-40146: "Upgrade Azure 
> Pipelines to OpenSSL 1.1.1f".

I fixed this issue.

There is another issue: sometimes, the Ubuntu VM fails to download .deb from 
azure.archive.ubuntu.com.

Example:
https://github.com/python/cpython/pull/19282/checks?check_run_id=553744077

---
Err:1 http://azure.archive.ubuntu.com/ubuntu bionic/universe amd64 lcov all 
1.13-3
  Could not connect to azure.archive.ubuntu.com:80 (52.177.174.250), connection 
timed out
---

This error is followed by many similar errors.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40149] test_threading leaked [38, 38, 38] references, sum=114

[STINNER Victor]

STINNER Victor  added the comment:

I'm not sure why, but trigger explicitly a second GC collection fix the issue.

diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index 2d5cb0ff78..d20ae01238 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -1295,6 +1295,7 @@ finalize_interp_clear(PyThreadState *tstate)
 /* Trigger a GC collection on subinterpreters*/
 if (!is_main_interp) {
 _PyGC_CollectNoFail();
+_PyGC_CollectNoFail();
 }
 
 finalize_interp_types(tstate, is_main_interp);

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40149] test_threading leaked [38, 38, 38] references, sum=114

[STINNER Victor]

STINNER Victor  added the comment:

Attached test_leak.py is enough to reproduce the leak. It's related to 
subinterpreters and the _abc module.

--
Added file: https://bugs.python.org/file49024/test_leak.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36541] Make lib2to3 grammar better match Python, support the := walrus

[fireattack]

Change by fireattack :


--
nosy: +fireattack

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40077] Convert static types to PyType_FromSpec()

[STINNER Victor]

STINNER Victor  added the comment:

> bpo-40077: Convert _abc module to use PyType_FromSpec() (GH-19202)

This change introduced a reference leak: bpo-40149 "test_threading leaked [38, 
38, 38] references, sum=114".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40149] test_threading leaked [38, 38, 38] references, sum=114

[STINNER Victor]

New submission from STINNER Victor :

New changeset 53e4c91725083975598350877e2ed8e2d0194114 by Dong-hee Na in branch 
'master':
bpo-40077: Convert _abc module to use PyType_FromSpec() (GH-19202)
https://github.com/python/cpython/commit/53e4c91725083975598350877e2ed8e2d0194114

This change introduced a reference leak:

$ ./python -m test -R 3:3 test_threading -m 
test.test_threading.SubinterpThreadingTests.test_threads_join_2
0:00:00 load avg: 1.52 Run tests sequentially
0:00:00 load avg: 1.52 [1/1] test_threading
beginning 6 repetitions
123456
..
test_threading leaked [19, 19, 19] references, sum=57
test_threading leaked [12, 12, 12] memory blocks, sum=36
test_threading failed

== Tests result: FAILURE ==

1 test failed:
test_threading

Total duration: 768 ms
Tests result: FAILURE

--
components: Tests
messages: 365557
nosy: corona10, vstinner
priority: normal
severity: normal
status: open
title: test_threading leaked [38, 38, 38] references, sum=114
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

STINNER Victor  added the comment:


New changeset e7c98f08e228e9f6e139d61e3e5d0a5018a38f0b by Victor Stinner in 
branch 'master':
bpo-40094: Fix which.py script exit code (GH-19286)
https://github.com/python/cpython/commit/e7c98f08e228e9f6e139d61e3e5d0a5018a38f0b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9a679a0e47d58aa73b7747d4e16140048c10baf5 by Victor Stinner in 
branch 'master':
bpo-40094: CGIHTTPRequestHandler logs exit code (GH-19285)
https://github.com/python/cpython/commit/9a679a0e47d58aa73b7747d4e16140048c10baf5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[STINNER Victor]

STINNER Victor  added the comment:

Ok, the issue should now be fixed.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[miss-islington]

Change by miss-islington :


--
pull_requests: +18656
pull_request: https://github.com/python/cpython/pull/19297

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[miss-islington]

Change by miss-islington :


--
pull_requests: +18655
pull_request: https://github.com/python/cpython/pull/19296

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37207] Use PEP 590 vectorcall to speed up calls to range(), list() and dict()

[STINNER Victor]

STINNER Victor  added the comment:

> When I designed the FASTCALL calling convention, I experimented a new 
> tp_fastcall slot to PyTypeObject to optimize __call__() method: bpo-29259.

Ah, by the way, I also made an attempt to use the FASTCALL calling convention 
for tp_new and tp_init: bpo-29358. Again, the speedup wasn't obvious and the 
implementation was quite complicated with many corner cases. So I gave up on 
this one. It didn't seem to be really worth it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40148] Add PurePath.with_stem()

[Tim Hoffmann]

Change by Tim Hoffmann :


--
keywords: +patch
pull_requests: +18654
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19295

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[miss-islington]

miss-islington  added the comment:


New changeset 40fff1ff04aa5bc2cf1b965d573b87c48e4da8cc by Miss Islington (bot) 
in branch '3.8':
bpo-40146: Update OpenSSL to 1.1.1f in Azure Pipelines (GH-19288)
https://github.com/python/cpython/commit/40fff1ff04aa5bc2cf1b965d573b87c48e4da8cc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[miss-islington]

miss-islington  added the comment:


New changeset 8e069fc2ee19f40ce97e61e880bb409ed415d98c by Miss Islington (bot) 
in branch '3.7':
bpo-40146: Update OpenSSL to 1.1.1f in Azure Pipelines (GH-19288)
https://github.com/python/cpython/commit/8e069fc2ee19f40ce97e61e880bb409ed415d98c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[STINNER Victor]

STINNER Victor  added the comment:

> But I think that the fix should be backported to 3.7 and 3.8 as well.

Alright, it's required. Azure Pipelines now fails on 3.7 as well which prevents 
me to merge PR 19292 security fix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40125] update OpenSSL 1.1.1 in multissltests.py to 1.1.1f

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-40146: the OpenSSL version is also hardcoded in 
.azure-pipelines/ci.yml and .azure-pipelines/pr.yml. I updated it as well.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[miss-islington]

Change by miss-islington :


--
pull_requests: +18653
pull_request: https://github.com/python/cpython/pull/19294

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 2.0 -> 3.0
pull_requests: +18652
pull_request: https://github.com/python/cpython/pull/19293

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[STINNER Victor]

STINNER Victor  added the comment:

Hum, right now 3.7 and 3.8 still work because they use a cache:

"Cache restored from key: Linux-multissl-openssl-1.1.1d"

But I think that the fix should be backported to 3.7 and 3.8 as well.

--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40144] resource: RLIMIT_xxx and RUSAGE_xxx constants are missing on macOS 10.15.4

[STINNER Victor]

STINNER Victor  added the comment:

I upgraded my macbook to macOS 10.15.4:

$ ./python.exe -m test.pythoninfo|grep -E 'uname|platform'

os.uname: posix.uname_result(sysname='Darwin', nodename='macbook', 
release='19.4.0', version='Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 
PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64', machine='x86_64')

platform.platform: macOS-10.15.4-x86_64-i386-64bit


I get resource attributes:

$ ./python.exe
Python 3.9.0a5+ (heads/master:e27916b, Apr  2 2020, 02:58:34) 
[Clang 11.0.3 (clang-1103.0.32.29)] on darwin
>>> import resource
>>> resource.RLIMIT_CORE
4
>>> resource.RLIMIT_NOFILE
8
>>> resource.RUSAGE_SELF
0


I'm not sure what's wrong with the macOS job. It doesn't seem to use a 
"framework" build, but:

./configure --with-pydebug --with-openssl=/usr/local/opt/openssl 
--prefix=/opt/python-dev
make -j4

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37207] Use PEP 590 vectorcall to speed up calls to range(), list() and dict()

[STINNER Victor]

STINNER Victor  added the comment:

Can we now close this issue? Or does someone plan to push further 
optimizations. Maybe new issues can be opened for next optimizations?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37207] Use PEP 590 vectorcall to speed up calls to range(), list() and dict()

[STINNER Victor]

STINNER Victor  added the comment:


New changeset e27916b1fc0364e3627438df48550c16f0b80b82 by Dong-hee Na in branch 
'master':
bpo-37207: Use PEP 590 vectorcall to speed up dict() (GH-19280)
https://github.com/python/cpython/commit/e27916b1fc0364e3627438df48550c16f0b80b82


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 224e1c34d677ef42fe665ac008a000d4dcec1398 by Victor Stinner in 
branch 'master':
bpo-40146: Update OpenSSL to 1.1.1f in Azure Pipelines (GH-19288)
https://github.com/python/cpython/commit/224e1c34d677ef42fe665ac008a000d4dcec1398


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 0b297d4ff1c0e4480ad33acae793fbaf4bf015b4 by Victor Stinner in 
branch 'master':
bpo-39503: CVE-2020-8492: Fix AbstractBasicAuthHandler (GH-18284)
https://github.com/python/cpython/commit/0b297d4ff1c0e4480ad33acae793fbaf4bf015b4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 5.0 -> 6.0
pull_requests: +18650
pull_request: https://github.com/python/cpython/pull/19291

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40145] Pyshellext room for binary size improvement

[Alexander Riccio]

Alexander Riccio  added the comment:

If this patch is merged, and all 7 million (estimated) Python developers update 
their installation, I calculate that I just saved the PSF 119GB worth of 
bandwidth costs :)

I'll take my 10 cents in the mail please :D

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[miss-islington]

Change by miss-islington :


--
pull_requests: +18651
pull_request: https://github.com/python/cpython/pull/19292

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40131] Zipapp example has parameters in the wrong order

[Zackery Spytz]

Change by Zackery Spytz :


--
keywords: +patch
nosy: +ZackerySpytz
nosy_count: 2.0 -> 3.0
pull_requests: +18649
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19290

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[Ben Caller]

Change by Ben Caller :


Removed file: https://bugs.python.org/file49022/bench_parser2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[Ben Caller]

Change by Ben Caller :


Added file: https://bugs.python.org/file49023/bench_parser2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[STINNER Victor]

STINNER Victor  added the comment:

Ooooh, I see. I didn't measure the performance of the right header. I re-run a 
benchmark using the HTTP header (repeat=15):

header = 'Basic ' + ', ' * 15 + 'A'

Now I see a major performance difference. Comparison between master ("ref") and 
PR 18284 ("fix"):

Mean +- std dev: [ref] 88.9 ms +- 2.4 ms -> [fix] 17.5 us +- 0.7 us: 5083.23x 
faster (-100%)

So the worst case is now way faster: more than 5000x faster!

It's even possible to go up to repeat=10**6 characters, it still takes less 
than 1 seconds: 412 ms +- 19 ms.

On the master branch, repeat=20 already takes around 3 seconds... The slowdown 
is exponential with repeat increase.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40148] Add PurePath.with_stem()

[Tim Hoffmann]

New submission from Tim Hoffmann :

Similar to PurePath.with_name() and PurePath.with_suffix() there should be a 
PurePath.with_stem().

A common use case would be appending something before the suffix: 
path.with_stem(path.stem + '_v2')

As of now this must be written more cumbersome as:
path.with_name(path.stem + '_v2' + path.suffix)

--
components: Library (Lib)
messages: 365540
nosy: timhoffm
priority: normal
severity: normal
status: open
title: Add PurePath.with_stem()

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40120] Undefined C behavior going beyond end of struct via a [1] arrays.

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> How is it an undefined C behavior? It works well in practice, no?

Famous last words ;)

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39503] [security][CVE-2020-8492] Denial of service in urllib.request.AbstractBasicAuthHandler

[Ben Caller]

Ben Caller  added the comment:

Instead of

repeat_10_3 = 'Basic ' + ', ' * (10 ** 3) + simple

in the benchmark, try

repeat_10_3 = 'Basic ' + ', ' * (10 ** 3) + 'A'

--
Added file: https://bugs.python.org/file49022/bench_parser2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40147] Move checking for duplicated keywords to the compiler

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
keywords: +patch
pull_requests: +18648
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19289

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40147] Move checking for duplicated keywords to the compiler

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
nosy: +lys.nikolaou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40147] Move checking for duplicated keywords to the compiler

[Pablo Galindo Salgado]

New submission from Pablo Galindo Salgado :

When a keyword is repeated in a call for instance:

'f(1, x=2, *(3, 4), x=5)'

we raise a SyntaxError:

  File "lel.py", line 1
f(1, x=2, *(3, 4), x=5)
   ^
SyntaxError: keyword argument repeated

This error is raised from the AST but there is nothing technically wrong with 
that code from the grammar perspective. Indeed, the grammar must accepts that 
code, but the check must fail later (in the compiler for instance) because the 
code is semantically invalid. When working on the new PEG parser we have 
encountered this situation and changing the parser would remove the check as it 
is right now. For these reasons, the check should be moved from the AST to the 
compiler.

--
messages: 365537
nosy: pablogsal
priority: normal
severity: normal
status: open
title: Move checking for duplicated keywords to the compiler

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[STINNER Victor]

STINNER Victor  added the comment:

Because of this issue, "Azure Pipelines PR" fails on pull requests and so it's 
no longer possible to merge any pull request.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39837] Remove Azure Pipelines from GitHub PRs

[STINNER Victor]

STINNER Victor  added the comment:

About the "Install Dependencies" issue, I created bpo-40146: "Upgrade Azure 
Pipelines to OpenSSL 1.1.1f".

I don't understand how it's possible that the Ubuntu job passed in the GitHub 
action, but failed in Azure Pipelines. Maybe there was a download cache 
somehow, and the cache is now outdated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +18647
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19288

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40146] Upgrade Azure Pipelines to OpenSSL 1.1.1f

[STINNER Victor]

New submission from STINNER Victor :

The "Install Dependencies" step of the Ubuntu PR Tests job of Azure Pipelines 
failed with:
---
*** INFO /home/vsts/work/1/s/multissl/openssl/1.1.1d/bin/openssl
*** INFO Downloading from https://www.openssl.org/source/openssl-1.1.1d.tar.gz
Traceback (most recent call last):
(...)
urllib.error.HTTPError: HTTP Error 404: Not Found
---

The problem is that the tarball of OpenSSL 1.1.1d moved from /source/ to 
/source/old/ directory.

bpo-40125 updated multissl to OpenSSL 1.1.1f.

I propose to use the same OpenSSL version for Azure Pipelines.

By the way, PCbuild/get_externals.bat and Mac/BuildScript/build-installer.py 
still use OpenSSL 1.1.1d (released at 2019-Sep-10). It's maybe time to upgrade 
these as well.

--
components: Tests
messages: 365534
nosy: vstinner
priority: normal
severity: normal
status: open
title: Upgrade Azure Pipelines to OpenSSL 1.1.1f
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40141] Add line and column information for keywords in the AST

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

STINNER Victor  added the comment:


New changeset d57cf557366584539f400db523b555296487e8f5 by Victor Stinner in 
branch 'master':
bpo-40094: mailcap.test() uses waitstatus_to_exitcode() (GH-19287)
https://github.com/python/cpython/commit/d57cf557366584539f400db523b555296487e8f5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

STINNER Victor  added the comment:

> Decide if subprocess should reject WIFSTOPPED() or not.

This code path was added by bpo-29335.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40141] Add line and column information for keywords in the AST

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 168660b547d5f683c5d3c60447cfa8c6d620efc3 by Pablo Galindo in 
branch 'master':
bpo-40141: Add line and column information to ast.keyword nodes (GH-19283)
https://github.com/python/cpython/commit/168660b547d5f683c5d3c60447cfa8c6d620efc3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

STINNER Victor  added the comment:


New changeset c8dd641b6214bdcf794bab469a51da6843feb770 by Miss Islington (bot) 
in branch '3.7':
bpo-40094: Enhance os.WIFEXITED documentation (GH-19244) (GH-19278)
https://github.com/python/cpython/commit/c8dd641b6214bdcf794bab469a51da6843feb770


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18646
pull_request: https://github.com/python/cpython/pull/19287

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 267afc2ab2014e1e3c6b2ff088350a69b691a544 by Miss Islington (bot) 
in branch '3.8':
bpo-40094: Enhance os.WIFEXITED documentation (GH-19244) (GH-19277)
https://github.com/python/cpython/commit/267afc2ab2014e1e3c6b2ff088350a69b691a544


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18645
pull_request: https://github.com/python/cpython/pull/19286

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

STINNER Victor  added the comment:

See also: "Appendix E. Exit Codes With Special Meanings" section of the Bash 
documentation
https://tldp.org/LDP/abs/html/exitcodes.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30465] FormattedValue expressions have wrong lineno and col_offset information

[Anthony Sottile]

Change by Anthony Sottile :


--
nosy: +Anthony Sottile
nosy_count: 6.0 -> 7.0
pull_requests: +18644
pull_request: https://github.com/python/cpython/pull/10021

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40094] Add os.waitstatus_to_exitcode() function

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +18643
pull_request: https://github.com/python/cpython/pull/19285

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40144] resource: RLIMIT_xxx and RUSAGE_xxx constants are missing on macOS 10.15.4

[STINNER Victor]

Change by STINNER Victor :


--
title: test.support.SuppressCrashReport fails on macOS 10.15.4 -> resource: 
RLIMIT_xxx and RUSAGE_xxx constants are missing on macOS 10.15.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40145] Pyshellext room for binary size improvement

[Alexander Riccio]

Change by Alexander Riccio :


--
keywords: +patch
pull_requests: +18642
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19284

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40145] Pyshellext room for binary size improvement

[Alexander Riccio]

New submission from Alexander Riccio :

I've tweaked the pcbuild options for pyshellext to reduce the size of the 
binary.
Since this is a very simple component, there really isn't much benefit of
optimizing for speed, likely the slowest part of this component's lifetime is
simply loading it.

This patch turns on Whole Program Optimization, MinSpace optimization, /Ob2
AnySuitable function inlining (to expose more code to elimination, this does
actually help here), /Zo (so that people can still debug it when lots of code
has been optimized out), turns of C++ RTTI (nobody is using it), /OPT:REF dead
code elimination, /OPT:ICF COMDAT folding, Link Time Code Generation, and
DebugFull debugging information creation.

/Gw global data optimization seemed to do nothing, which makes sense since
there isn't much going on in this project, very little in the way of actual
global data. Disabling C++ exceptions both in the project config (i.e. /EH-),
and disabling stdlib exceptions via _HAS_EXCEPTIONS=0, had no effect.
Strangely, with exceptions disabled and _HAS_EXCEPTIONS=0, exception functions
are still emitted in the binary (as seen in IDA). I presume it has something
to do with the fact that its a dll.

Enabling optimizations (even for Debug builds) should have no effect. The debug
builds were not actually using debugging featuers (not even assert), so nothing
should change.

pyshellext.dll   32 bit unimproved release size: 42KB
pyshellext.dll   32 bit   improved release size: 25KB
size reduction: -17KB
 %  size reduction:   40%

pyshellext_d.dll 32 bit unimproved debug size:   85KB
pyshellext_d.dll 32 bit   improved debug size:   32KB
   size reduction:  -53KB
 %  size reduction:   62%


pyshellext.dll   64 bit unimproved release size: 52KB
pyshellext.dll   64 bit   improved release size: 30KB
   size reduction:  -22KB
 %  size reduction:   42%

pyshellext_d.dll 64 bit unimproved debug size:  105KB
pyshellext_d.dll 64 bit   improved debug size:   38KB
   size reduction:  -67KB
 %  size reduction:   63%

--
components: Windows
messages: 365527
nosy: Alexander Riccio, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Pyshellext room for binary size improvement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40144] test.support.SuppressCrashReport fails on macOS 10.15.4

[STINNER Victor]

STINNER Victor  added the comment:

Yesterday, macOS job was fine with test.pythoninfo:

os.uname: posix.uname_result(sysname='Darwin', nodename='Mac-1303.local', 
release='19.3.0', version='Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 
PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64', machine='x86_64')

platform.platform: macOS-10.15.3-x86_64-i386-64bit


So something changed in header between macOS 10.15.3 and macOS 
10.15.4.

--
components: +macOS
nosy: +ned.deily, ronaldoussoren
title: test.support.SuppressCrashReport fails on macOS -> 
test.support.SuppressCrashReport fails on macOS 10.15.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40144] test.support.SuppressCrashReport fails on macOS

[STINNER Victor]

STINNER Victor  added the comment:

Other errors on the same build:

13 tests failed:
test_capi test_cmd_line test_exceptions test_faulthandler test_gc
test_io test_repl test_resource test_selectors test_subprocess
test_sys test_tempfile test_tracemalloc

test_subprocess: AttributeError: module 'resource' has no attribute 
'RLIMIT_NOFILE'

test_selectors.test_above_fd_setsize(): AttributeError: module 'resource' has 
no attribute 'RLIMIT_NOFILE'

test_resource.test_getrusage(): AttributeError: module 'resource' has no 
attribute 'RUSAGE_SELF'

Many tests failed because of SuppressCrashReport issue (missing 
resource.RLIMIT_CORE).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40141] Add line and column information for keywords in the AST

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
nosy: +BTaskaya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40110] multiprocessing.Pool.imap() should be lazy

[Nick Guenther]

Nick Guenther  added the comment:

Thank you for taking the time to consider my points! Yes, I think you 
understood exactly what I was getting at.

I slept on it and thought about what I'd posted the day after and realized most 
of the points you raise, especially that serialized next() would mean 
serialized processing. So the naive approach is out.

I am motivated by trying to introduce backpressure to my pipelines. The example 
you gave has potentially infinite memory usage; if I simply slow it down with 
sleep() I get a memory leak and the main python proc pinning my CPU, even 
though it "isn't" doing anything:

with multiprocessing.Pool(4) as pool:
for i, v in enumerate(pool.imap(worker, itertools.count(1)), 1):
time.sleep(7)
print(f"At {i}: {v}, memory usage is {sys.getallocatedblocks()}")

At 1->1, memory usage is 230617
At 2->8, memory usage is 411053
At 3->27, memory usage is 581439
At 4->64, memory usage is 748584
At 5->125, memory usage is 909694
At 6->216, memory usage is 1074304
At 7->343, memory usage is 1238106
At 8->512, memory usage is 1389162
At 9->729, memory usage is 1537830
At 10->1000, memory usage is 1648502
At 11->1331, memory usage is 1759864
At 12->1728, memory usage is 1909807
At 13->2197, memory usage is 2005700
At 14->2744, memory usage is 2067407
At 15->3375, memory usage is 2156479
At 16->4096, memory usage is 2240936
At 17->4913, memory usage is 2328123
At 18->5832, memory usage is 2456865
At 19->6859, memory usage is 2614602
At 20->8000, memory usage is 2803736
At 21->9261, memory usage is 2999129

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND 
11314 kousu 20   0  303308  40996   6340 S  91.4  2.1   0:21.68 python3.8   
11317 kousu 20   0   54208  10264   4352 S  16.2  0.5   0:03.76 python3.8   
11315 kousu 20   0   54208  10260   4352 S  15.8  0.5   0:03.74 python3.8   
11316 kousu 20   0   54208  10260   4352 S  15.8  0.5   0:03.74 python3.8   
11318 kousu 20   0   54208  10264   4352 S  15.5  0.5   0:03.72 python3.8 

It seems to me like any usage of Pool.imap() either has the same issue lurking 
or is run on a small finite data set where you are better off using Pool.map().

I like generators because they keep constant-time constant-memory work, which 
seems like a natural fit for stream processing situations. Unix pipelines have 
backpressure built-in, because write() blocks when the pipe buffer is full.

I did come up with one possibility after sleeping on it again: run the final 
iteration in parallel, perhaps by a special plist() method that makes as many 
parallel next() calls as possible. There's definitely details to work out but I 
plan to prototype when I have spare time in the next couple weeks.

You're entirely right that it's a risky change to suggest, so maybe it would be 
best expressed as a package if I get it working. Can I keep this issue open to 
see if it draws in insights from anyone else in the meantime?

Thanks again for taking a look! That's really cool of you!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40144] test.support.SuppressCrashReport fails on macOS

[STINNER Victor]

STINNER Victor  added the comment:

> https://github.com/python/cpython/pull/19252/checks?check_run_id=552502971

test.pythoninfo says:

os.uname: posix.uname_result(sysname='Darwin', nodename='Mac-1307.local', 
release='19.4.0', version='Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 
PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64', machine='x86_64')

platform.platform: macOS-10.15.4-x86_64-i386-64bit

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40024] Add PyModule_AddType helper function

[Kyle Stanley]

Kyle Stanley  added the comment:

(disregard the above, the PR was mistakenly linked from a GitHub bug)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40024] Add PyModule_AddType helper function

[Kyle Stanley]

Change by Kyle Stanley :


--
pull_requests:  -18641

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40144] test.support.SuppressCrashReport fails on macOS

[STINNER Victor]

New submission from STINNER Victor :

Failure seen on the macOS job on a PR:

https://github.com/python/cpython/pull/19252/checks?check_run_id=552502971
of https://github.com/python/cpython/pull/19252/files

Reformatted error:

AssertionError: Regex didn't match:

"Fatal Python error: _enter_buffered_busy: could not acquire lock for 
<(_io\\.)?BufferedWriter name=''> at interpreter shutdown, possibly due 
to daemon threads"

not found in

'Traceback (most recent call last):
  File "", line 15, in 
  File 
"/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/support/__init__.py",
 line 2913, in __enter__
self.old_value = resource.getrlimit(resource.RLIMIT_CORE)
AttributeError: module \'resource\' has no attribute \'RLIMIT_CORE\'
'

It seems like resource.RLIMIT_CORE is not available on this macOS machine.

--
components: Tests
messages: 365521
nosy: vstinner
priority: normal
severity: normal
status: open
title: test.support.SuppressCrashReport fails on macOS
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40143] shutil.rmtree will frequently fail on Windows under heavy load due to racy deletion

[Alexander Riccio]

New submission from Alexander Riccio :

The "obvious" way to delete a directory tree on Windows is wrong. It's 
inherently racy, since deleting a file on Windows *doesn't actually delete it*, 
instead it marks the file for deletion. The system will eventually get around 
to deleting it, but under heavy load, this might be sometime after an attempt 
is made to delete the parent directory. I've seen this (windows error 145, 
directory is not empty) many times when running the testsuite, and it causes 
all kinds of intermittent failures.

The best way to do things correctly is kinda nuts, and is explained well by 
Niall Douglass here: https://www.youtube.com/watch?v=uhRWMGBjlO8=8m54s

In short, the correct way to do it involves moving each file to a randomly 
named file in %TEMP%, then deleting that, and then doing the same with each 
newly-empty directory.

--
components: Windows
messages: 365520
nosy: Alexander Riccio, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: shutil.rmtree will frequently fail on Windows under heavy load due to 
racy deletion
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39837] Remove Azure Pipelines from GitHub PRs

[STINNER Victor]

STINNER Victor  added the comment:

> Yes, but I don't want to to do that as we have had equivalent flakiness 
> issues with Travis which is why it isn't required ATM.

I'm not aware of Travis CI current issue. There were issues in the past, as 
with any CI, right ;-) Travis CI looks quite reliable these days.

Whereas the Docs and Ubuntu GH Action job of Azure failed to install 
dependencies :-(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40024] Add PyModule_AddType helper function

[Kyle Stanley]

Change by Kyle Stanley :


--
nosy: +aeros
nosy_count: 3.0 -> 4.0
pull_requests: +18641
pull_request: https://github.com/python/cpython/pull/19282

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33493] dataclasses: allow keyword-only arguments

[Chris Barker]

Chris Barker  added the comment:

This does not actually appear to be a Duplicate of #33129.

That one was asking to add **kwargs (I think) to the __init__. And was 
discussed and I think rejected on gitHub:

https://github.com/python/cpython/pull/19206

But this calls for having keyword-only parameters, which I think would still be 
a good idea.

--
nosy: +ChrisBarker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31160] Enhance support.reap_children()

[STINNER Victor]

STINNER Victor  added the comment:

> Pablo Galindo opened bpo-40140, let's use this one.

Note: Oops, Batuhan created it, Pablo only commented.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40140] test_builtin crashes when runned in parallel mode on solaris

[STINNER Victor]

STINNER Victor  added the comment:

> 0:00:01 load avg: 1.70 [1/1/1] test_builtin crashed (Exit code -1)

Exit code -1 looks like a process killed by SIGHUP. Which commit did you try?

Can you please check that you tested with my commit 
16d75675d2ad2454f6dfbf333c94e6237df36018?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40142] Modify _PyObject_GC_TRACK() to ensure that newly tracked object is valid

[STINNER Victor]

New submission from STINNER Victor :

In bpo-38392, I modified PyObject_GC_Track() to ensure that the object newly 
tracked is valid: call its traverse function.
=> commit 1b1845569539db5c1a6948a5d32daea381f1e35f

I propose to now also attempt to implement the same check in 
_PyObject_GC_TRACK() which is part of the internal C API.

PyType_GenericAlloc() allocates memory for a type allocated on the heap... and 
then immediately track it in the GC. Problem: this type is not initialized yet, 
all fields are set to zero. Calling type_traverse() at this point fails with an 
assertion error:
---
Objects/typeobject.c:3570: type_traverse: Assertion failed: type_traverse() 
called on non-heap type '(null)'
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x840860
object refcount : 1
object type : 0x7e0900
object type name: type
object repr : 
---

By the way, Python crash in _PyObject_Dump() on PyObject_Repr() call: 
type_repr() crash when accessing type->tp_name which is NULL.

type_call() should only track the newly created type when it's fully 
initialized.


Try attached track.patch to reproduce the crash.

--
components: Interpreter Core
files: track.patch
keywords: patch
messages: 365515
nosy: vstinner
priority: normal
severity: normal
status: open
title: Modify _PyObject_GC_TRACK() to ensure that newly tracked object is valid
type: enhancement
versions: Python 3.9
Added file: https://bugs.python.org/file49021/track.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40140] test_builtin crashes when runned in parallel mode on solaris

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

I am understanding "crashing" as "segfaulting"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40140] test_builtin crashes when runned in parallel mode on solaris

[STINNER Victor]

STINNER Victor  added the comment:

I modified recently the test:

(1) commit 278c1e159c970da6cd6683d18c6211f5118674cc

-os.waitpid(pid, 0)
+support.wait_process(pid, exitcode=0)

(2) commit 16d75675d2ad2454f6dfbf333c94e6237df36018

Close the fd *after* calling support.wait_process() to prevent sending SIGHUP 
to the child process, which made support.wait_process(pid, exitcode=0) to fail 
since exitcode=-1 (-SIGHUP) != 0.

--

test_builtin.test_input_no_stdout_fileno() also hangs on AIX:
https://bugs.python.org/issue31160#msg365478

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31160] Enhance support.reap_children()

[STINNER Victor]

STINNER Victor  added the comment:

> Please open a new issue. This one is closed.

Pablo Galindo opened bpo-40140, let's use this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40140] test_builtin crashes when runned in parallel mode on solaris

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Try making bigger the stack size (with ulimit -s ... or similar)

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40141] Add line and column information for keywords in the AST

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

I am preparing more PRs for other nodes that are missing the meta-information 
as well but will open them in a separate issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40141] Add line and column information for keywords in the AST

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
keywords: +patch
pull_requests: +18640
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19283

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40141] Add line and column information for keywords in the AST

[Pablo Galindo Salgado]

New submission from Pablo Galindo Salgado :

When inspecting keyword parameters in a function call, the keyword is stored as 
a string and not as a AST node:

>>> import ast
>>> r = "f(a,   xxa  =  34, y=23)"
>>> node = ast.parse(r)
>>> ll = node.body[0].value.keywords[0].arg
>>> node.body[0].value.keywords[0].arg
'xxa'

this makes impossible to locate the keyword in the code using the AST as it 
lacks AST attributes (lineno, col_offset, end_lineno, end_col_offset).

On the other hand, this does not happen with args, that has the 
meta-information:

>>> node.body[0].value.args[0].id
'a'
>>> dir(node.body[0].value.args[0])


So one can locate the arg string using:

>>> r[arg.col_offset:arg.end_col_offset]
'a'

For this reason, we should add the same information to keyword nodes.

--
components: Interpreter Core
messages: 365509
nosy: pablogsal
priority: normal
severity: normal
status: open
title: Add line and column information for keywords in the AST
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39812] Avoid daemon threads in concurrent.futures

[Kyle Stanley]

Kyle Stanley  added the comment:

I attached a PR to bpo-40115 to address the refleak in test_asyncio: PR-19282.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40115] test_asyncio leaked [3, 3, 3] references, sum=9

[Kyle Stanley]

Change by Kyle Stanley :


--
keywords: +patch
pull_requests: +18639
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19282

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15140] PEP 384 inconsistent with implementation

[Guido van Rossum]

Change by Guido van Rossum :


--
resolution:  -> fixed
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15140] PEP 384 inconsistent with implementation

[Furkan Onder]

Furkan Onder  added the comment:

It fixed.

--
nosy: +furkanonder

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40115] test_asyncio leaked [3, 3, 3] references, sum=9

[Kyle Stanley]

Kyle Stanley  added the comment:

I was able to find a fix! Specifically, I think the issue was a subtle problem 
with test_run_in_executor_cancel: it never properly shuts down the executor 
prior to closing the event loop. 

We do this in asyncio.run() (with the loop.shutdown_default_executor() that I 
added last year), but it should be called explicitly when using loop.close() 
directly. Otherwise, the resources of the executor may not be cleaned up 
properly. I think the issue was being covered up because of the usage of daemon 
threads in the ThreadPoolExecutor, but revealed itself when the workers were 
converted to non-daemon threads since they require proper cleanup.

The change is very minimal:

```
def test_run_in_executor_cancel(self):
called = False

def patched_call_soon(*args):
nonlocal called
called = True

def run():
time.sleep(0.05)

f2 = self.loop.run_in_executor(None, run)
f2.cancel()
+  self.loop.run_until_complete(
+  self.loop.shutdown_default_executor())
self.loop.close()
self.loop.call_soon = patched_call_soon
self.loop.call_soon_threadsafe = patched_call_soon
time.sleep(0.4)
self.assertFalse(called)
```

Before change:
```
[aeros:~/repos/aeros-cpython]$ ./python -m test --fail-env-changed -R 3:3 
test_asyncio -m 
test.test_asyncio.test_events.EPollEventLoopTests.test_run_in_executor_cancel
0:00:00 load avg: 0.63 Run tests sequentially
0:00:00 load avg: 0.63 [1/1] test_asyncio
beginning 6 repetitions
123456
..
test_asyncio leaked [1, 1, 1] references, sum=3
test_asyncio leaked [2, 1, 1] memory blocks, sum=4
test_asyncio failed

== Tests result: FAILURE ==

1 test failed:
test_asyncio

Total duration: 3.2 sec
Tests result: FAILURE
```

After change:
```
[aeros:~/repos/aeros-cpython]$ ./python -m test --fail-env-changed -R 3:3 
test_asyncio -m 
test.test_asyncio.test_events.EPollEventLoopTests.test_run_in_executor_cancel
0:00:00 load avg: 0.24 Run tests sequentially
0:00:00 load avg: 0.24 [1/1] test_asyncio
beginning 6 repetitions
123456
..

== Tests result: SUCCESS ==

1 test OK.

Total duration: 3.5 sec
Tests result: SUCCESS
```

I'll also test the PR using the `test-with-buildbots` label to double check, it 
should be attached to this issue within the next couple of hours.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40140] test_builtin crashes when runned in parallel mode on solaris

[Batuhan Taskaya]

New submission from Batuhan Taskaya :

test_builting works on serial run

0:00:00 load avg: 2.38 Run tests sequentially
0:00:00 load avg: 2.38 [1/1] test_builtin

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1.3 sec
Tests result: SUCCESS

but with more then one processes, it crashes 

0:00:00 load avg: 1.71 Run tests in parallel using 2 child processes
0:00:01 load avg: 1.70 [1/1/1] test_builtin crashed (Exit code -1)
test_abs (test.test_builtin.BuiltinTest) ... ok
test_all (test.test_builtin.BuiltinTest) ... ok
test_any (test.test_builtin.BuiltinTest) ... ok
test_ascii (test.test_builtin.BuiltinTest) ... ok
test_bin (test.test_builtin.BuiltinTest) ... ok
test_bug_27936 (test.test_builtin.BuiltinTest) ... ok
test_bytearray_extend_error (test.test_builtin.BuiltinTest) ... ok
test_bytearray_translate (test.test_builtin.BuiltinTest) ... ok
test_callable (test.test_builtin.BuiltinTest) ... ok
test_chr (test.test_builtin.BuiltinTest) ... ok
test_cmp (test.test_builtin.BuiltinTest) ... ok
test_compile (test.test_builtin.BuiltinTest) ... ok
test_compile_async_generator (test.test_builtin.BuiltinTest)
With the PyCF_ALLOW_TOP_LEVEL_AWAIT flag added in 3.8, we want to ... ok
test_compile_top_level_await (test.test_builtin.BuiltinTest)
Test whether code some top level await can be compiled. ... ok
test_compile_top_level_await_invalid_cases (test.test_builtin.BuiltinTest) ... 
ok
test_construct_singletons (test.test_builtin.BuiltinTest) ... ok
test_delattr (test.test_builtin.BuiltinTest) ... ok
test_dir (test.test_builtin.BuiltinTest) ... ok
test_divmod (test.test_builtin.BuiltinTest) ... ok
test_eval (test.test_builtin.BuiltinTest) ... ok
test_exec (test.test_builtin.BuiltinTest) ... ok
test_exec_globals (test.test_builtin.BuiltinTest) ... ok
test_exec_redirected (test.test_builtin.BuiltinTest) ... ok
test_filter (test.test_builtin.BuiltinTest) ... ok
test_filter_pickle (test.test_builtin.BuiltinTest) ... ok
test_format (test.test_builtin.BuiltinTest) ... ok
test_general_eval (test.test_builtin.BuiltinTest) ... ok
test_getattr (test.test_builtin.BuiltinTest) ... ok
test_hasattr (test.test_builtin.BuiltinTest) ... ok
test_hash (test.test_builtin.BuiltinTest) ... ok
test_hex (test.test_builtin.BuiltinTest) ... ok
test_id (test.test_builtin.BuiltinTest) ... ok
test_import (test.test_builtin.BuiltinTest) ... ok
test_input (test.test_builtin.BuiltinTest) ... ok
test_isinstance (test.test_builtin.BuiltinTest) ... ok
test_issubclass (test.test_builtin.BuiltinTest) ... ok
test_iter (test.test_builtin.BuiltinTest) ... ok
test_len (test.test_builtin.BuiltinTest) ... ok
test_map (test.test_builtin.BuiltinTest) ... ok
test_map_pickle (test.test_builtin.BuiltinTest) ... ok
test_max (test.test_builtin.BuiltinTest) ... ok
test_min (test.test_builtin.BuiltinTest) ... ok
test_neg (test.test_builtin.BuiltinTest) ... ok
test_next (test.test_builtin.BuiltinTest) ... ok
test_oct (test.test_builtin.BuiltinTest) ... ok
test_open (test.test_builtin.BuiltinTest) ... ok
test_open_default_encoding (test.test_builtin.BuiltinTest) ... ok
test_open_non_inheritable (test.test_builtin.BuiltinTest) ... ok
test_ord (test.test_builtin.BuiltinTest) ... ok
test_pow (test.test_builtin.BuiltinTest) ... ok
test_repr (test.test_builtin.BuiltinTest) ... ok
test_round (test.test_builtin.BuiltinTest) ... ok
test_round_large (test.test_builtin.BuiltinTest) ... ok
test_setattr (test.test_builtin.BuiltinTest) ... ok
test_sum (test.test_builtin.BuiltinTest) ... ok
test_type (test.test_builtin.BuiltinTest) ... ok
test_vars (test.test_builtin.BuiltinTest) ... ok
test_warning_notimplemented (test.test_builtin.BuiltinTest) ... ok
test_zip (test.test_builtin.BuiltinTest) ... ok
test_zip_bad_iterable (test.test_builtin.BuiltinTest) ... ok
test_zip_pickle (test.test_builtin.BuiltinTest) ... ok
test_input_no_stdout_fileno (test.test_builtin.PtyTests) ...

== Tests result: FAILURE ==

1 test failed:
test_builtin

Total duration: 1.4 sec
Tests result: FAILURE

System: SunOS gcc-solaris11 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise
Tested under both gcc (5.5.0) and solaris studio (12)

--
components: Tests
messages: 365505
nosy: BTaskaya
priority: normal
severity: normal
status: open
title: test_builtin crashes when runned in parallel mode on solaris
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com