[issue41135] Suggested change to http.server.HTTPServer to prevent socket reuse in Windows

[Michael Rich]

New submission from Michael Rich :

Hi, a web server can be incorrectly bound to an already in-use socket when 
binding a HTTPServer on windows.  The issue is discussed here:  
https://stackoverflow.com/questions/51090637/running-a-python-web-server-twice-on-the-same-port-on-windows-no-port-already

This only happens on Windows.  In *nix the socketserver will throw an error, on 
Windows it will not.  However the most recently bound server will not receive 
the requests.

I suggest the following code (taken from stackoverflow) at the start of the 
server_bind method:

if hasattr(socket, 'SO_EXCLUSIVEADDRUSE'):
self.socket.setsockopt(socket.SOL_SOCKET, 
socket.SO_EXCLUSIVEADDRUSE, 1)
# Also need to change the value of allow_reuse_address (defined in 
http.server.HTTPServer)
HTTPServer.allow_reuse_address = 0


I have tested this and it will throw an error upon reuse in Windows and does 
not change *nix behavior.

Thanks,

Mike

--
components: Library (Lib)
messages: 372451
nosy: Michael Rich
priority: normal
severity: normal
status: open
title: Suggested change to http.server.HTTPServer to prevent socket reuse in 
Windows
type: behavior
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41131] Augment random.choices() with the alias method

[Raymond Hettinger]

Change by Raymond Hettinger :


Added file: https://bugs.python.org/file49268/choices_binomial.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36656] Add race-free os.link and os.symlink wrapper / helper

[Tom Hale]

Tom Hale  added the comment:

Related:

bpo-41134 distutils.dir_util.copy_tree FileExistsError when updating symlinks


WIP update:

I am just about to ask for feedback on my proposed solution at 
core-mentors...@python.org

--
title: Please add race-free os.link and os.symlink wrapper / helper -> Add 
race-free os.link and os.symlink wrapper / helper

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41134] distutils.dir_util.copy_tree FileExistsError when updating symlinks

[Tom Hale]

New submission from Tom Hale :

Here is a minimal test case:

==
#!/bin/bash

cd /tmp || exit 1

dir=test-copy_tree
src=$dir/src
dst=$dir/dst

mkdir -p "$src"
touch "$src"/file
ln -s file "$src/symlink"

python -c "from distutils.dir_util import copy_tree;
copy_tree('$src', '$dst', preserve_symlinks=1, update=1);
copy_tree('$src', '$dst', preserve_symlinks=1, update=1);"

rm -r "$dir"

==

Traceback (most recent call last):
  File "", line 3, in 
  File "/usr/lib/python3.8/distutils/dir_util.py", line 152, in copy_tree
os.symlink(link_dest, dst_name)
FileExistsError: [Errno 17] File exists: 'file' -> 'test-copy_tree/dst/symlink'

==


Related:
=

This issue will likely be resolved via:

bpo-36656 Add race-free os.link and os.symlink wrapper / helper

https://bugs.python.org/issue36656
(WIP under discussion at python-mentor)


Prior art:
===
https://stackoverflow.com/questions/53090360/python-distutils-copy-tree-fails-to-update-if-there-are-symlinks

--
messages: 372449
nosy: Tom Hale
priority: normal
severity: normal
status: open
title: distutils.dir_util.copy_tree FileExistsError when updating symlinks

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41132] F-String parser uses raw allocator

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> Why would pymalloc be preferred?

For such small and temporary string is normally faster because it will likely 
have pools of that size readily available and it will also benefit further 
allocation of the same pool size.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41132] F-String parser uses raw allocator

[Eric V. Smith]

Eric V. Smith  added the comment:

I don't think I put any thought into it. It was probably just in an example I 
was copying from.

I'm not familiar with the difference between those allocators. Why would 
pymalloc be preferred?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41119] Wrong error message for list/tuple followed by a colon

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset d01a3e76ee0519cba3ccdb5de47b25ec2ed7bee4 by Lysandros Nikolaou in 
branch '3.9':
[3.9] bpo-41119: Output correct error message for list/tuple followed by colon 
(GH-21160) (GH-21172)
https://github.com/python/cpython/commit/d01a3e76ee0519cba3ccdb5de47b25ec2ed7bee4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41133] Insufficient description of cyclic garbage collector for C API

[Daniel Barkalow]

New submission from Daniel Barkalow :

Nothing in the documentation gives you enough information to find what you're 
doing wrong if you have the following bug for an object tracked by the GC: (a) 
you store a borrowed reference to an object whose lifetime is always longer 
that the buggy object (and never decrement it); (b) you visit that object in 
your tp_traverse.

Despite the bug, everything works nearly all of the time. However, when it 
fails, the effect makes no sense from a documentation-only understanding of the 
garbage collection (unless Python is built with --with-assertions, perhaps). 
The only sequence in which it fails is:

* All of the objects that will reference the victim get moved into an older 
generation.
* The victim gets exactly as many buggy references from objects in its own 
generation as there are good references, and no good references from objects in 
its own generation.
* Garbage collection runs for only the younger generation.

At this point, the victim is marked as garbage despite the fact that there was 
a good object referencing it at all times.

Reading the Python source, it becomes clear that the garbage collector handles 
references from older generations not by traversing the objects in older 
generations, but by comparing the count of references from other young objects 
to the usual reference count. That is, visiting an object you don't hold a 
reference to may cause that object to get collected, rather than protecting it, 
even if there are other reasons not to collect it.

The best fix would probably be a warning in the documentation for tp_traverse 
that visiting an object you do not hold a strong reference to can cause 
inexplicable effects, because the intuitive guess would be that it could only 
cause memory leaks.

It would probably also be worth mentioning in the documentation of the garbage 
collector something about its algorithm, so people have a hint that references 
from older objects aren't necessarily sufficient to overcome buggy use of the C 
API.

--
assignee: docs@python
components: C API, Documentation
messages: 372445
nosy: docs@python, iabervon
priority: normal
severity: normal
status: open
title: Insufficient description of cyclic garbage collector for C API
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35823] Use vfork() in subprocess on Linux

[Gregory P. Smith]

Gregory P. Smith  added the comment:

No objections, it would be great to see this finished up and land.

I've encountered a minority of users who are using a wrapped vfork-based C/C++ 
library for process spawning as fork+exec isn't fast enough for them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39883] Use BSD0 license for code in docs

[bbayles]

Change by bbayles :


--
nosy: +bbayles

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41131] Augment random.choices() with the alias method

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I also looked at another method using binomial variates but couldn't get it to 
run faster than the alias method:

def choices(population, weights, *, k=1):
r = 1.0
n = k
selections = []
for elem, p in zip(population, weights):
v = binomial_variate(n, p / r)
selections += [elem] * v
n -= v
r -= p
shuffle(selections)
return selections

The shuffle step took as much time as the alias method.  Also, the binomial 
variate was hard to compute quickly and without overflow/underflow issues for 
large inputs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41132] F-String parser uses raw allocator

[Lysandros Nikolaou]

New submission from Lysandros Nikolaou :

The f-string parser uses the raw allocator in various places. We had a very 
very brief discussion about this with Pablo and he suggested to change the new 
parser and the hand-written f-string parser to use the pymalloc allocator.

Eric, is there a specific reason the raw allocator was used?

--
components: Interpreter Core
messages: 372442
nosy: eric.smith, lys.nikolaou, pablogsal
priority: normal
severity: normal
status: open
title: F-String parser uses raw allocator
versions: Python 3.10, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41119] Wrong error message for list/tuple followed by a colon

[Lysandros Nikolaou]

Change by Lysandros Nikolaou :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41084] Signify that a SyntaxError comes from an fstring in the error message

[Lysandros Nikolaou]

Change by Lysandros Nikolaou :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41131] Augment random.choices() with the alias method

[Raymond Hettinger]

New submission from Raymond Hettinger :

For n unequal weights and k selections, sample selection with the inverse-cdf 
method is O(k log₂ n).  Using the alias method, it improves to O(k).  The 
proportionally constants also favor the alias method so that if the set up 
times were the same, the alias method would always win (even when n=2).

However, the set up times are not the same.  For the inverse-cdf method, set up 
is O(1) if cum_weights are given; otherwise, it is O(n) with a fast loop.  The 
setup time for the alias method is also O(n) but is proportionally much slower.

So, there would need to be a method selection heuristic based on the best 
trade-off between setup time and sample selection time.

Both methods make k calls to random().

See: https://en.wikipedia.org/wiki/Alias_method

Notes on the attached draft implementation:

* Needs to add back the error checking code.

* Need a better method selection heuristic.

* The alias table K defaults to the original index
  so that there is always a valid selection even
  if there are small rounding errors.

* The condition for the aliasing loop is designed
  to have an early-out when the remaining blocks
  all have equal weights.  Also, the loop condition
  makes sure that the pops never fail even if there
  are small rounding errors when partitioning
  oversized bins or if the sum of weights isn't
  exactly 1.0.

--
assignee: rhettinger
components: Library (Lib)
files: choices_proposal.py
messages: 372441
nosy: mark.dickinson, rhettinger, tim.peters
priority: normal
severity: normal
status: open
title: Augment random.choices() with the alias method
type: performance
versions: Python 3.10
Added file: https://bugs.python.org/file49267/choices_proposal.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41119] Wrong error message for list/tuple followed by a colon

[Lysandros Nikolaou]

Change by Lysandros Nikolaou :


--
pull_requests: +20330
pull_request: https://github.com/python/cpython/pull/21172

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41093] TCPServer's server_forever() shutdown immediately when calling shutdown()

[Tony]

Tony  added the comment:

poke

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41130] Improve/fix FreeBSD Bluetooth socket support

[myfreeweb]

New submission from myfreeweb :

1) BTPROTO_HCI addresses only expect string identifiers on NetBSD and 
DragonFly: 
https://github.com/python/cpython/blob/2e0a920e9eb540654c0bb2298143b00637dc5961/Modules/socketmodule.c#L1931

But of course this is true on FreeBSD too. (DragonFly inherited the BT stack 
from FreeBSD!)
For example this is how hccontrol creates an address:
https://github.com/freebsd/freebsd/blob/6bb9221a9b865ee432269099f341e4230a6cbcd4/usr.sbin/bluetooth/hccontrol/hccontrol.c#L115-L129

So currently it is not possible to bind an HCI socket (without using FFI to 
directly use the libc bind function) :(

2) BTPROTO_SCO is excluded on FreeBSD: 
https://github.com/python/cpython/blob/2e0a920e9eb540654c0bb2298143b00637dc5961/Modules/socketmodule.c#L1953

But SCO has been supported since 2008: 
https://github.com/freebsd/freebsd/commit/bb4c6de0cf336d006e41521cbbd4706f60a0dfe0

--
components: FreeBSD
messages: 372439
nosy: koobs, myfreeweb
priority: normal
severity: normal
status: open
title: Improve/fix FreeBSD Bluetooth socket support
type: enhancement
versions: Python 3.10, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 
3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41128] Signal handlers should not hang during blocked main thread

[Josef Havránek]

Change by Josef Havránek :


--
type: enhancement -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41129] Python extension modules fail to build on Mac 10.15.1 (Catalina)

[Andrew]

New submission from Andrew :

Quick repoduction steps:
1) Log into a mac with macOS version 10.15.1 (10.15.x may work)
2) For build tools, use Xcode11. A minimal xcode command-tools installation 
also reproduced for me.
3) Download and decompress the latest python 3.8.2 source
4) run "./configure" in the top-level source folder
5) run "make" in the top-level source folder

I believe this error may also occur when using python 2.7.x, 3.7.x, and others. 
Curiously, the errors do not occur when I use the latest 3.6.10 source.

A text file containing the output of "./configure" and "make" is attached.

Main Description:
On Mac 10.15.1 with Xcode11 I encounter compilation errors when building the 
latest Python 3.8.2 source. When the build reaches the "build_ext" section in 
setup.py, none of the extension modules build, with each compilation attempt 
producing an error. For example, when the _struct extension module attempts 
compilation I get:

--- Error Syndrome ---
building '_struct' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv 
-O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter 
-Wno-missing-field-initializers -Wstrict-prototypes 
-Werror=implicit-function-declaration -I./Include/internal -I./Include -I. 
-I/usr/local/include 
-I/System/Volumes/Data/mathworks/devel/sandbox/aflewell/python38_source/Python-3.8.2/Include
 
-I/System/Volumes/Data/mathworks/devel/sandbox/aflewell/python38_source/Python-3.8.2
 -c _struct.c -o build/temp.macosx-10.15-x86_64-3.8/_struct.o
clang: error: no such file or directory: '_struct.c'
clang: error: no input files
--

After many similar errors, we get to the main build report where we see the 
failed extension modules:

--- Snippet of build report ---
Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm _hashlib  _ssl   
ossaudiodev   spwd 
To find the necessary bits, look in setup.py in detect_modules() for the 
module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc  atexitpwd
time   


Failed to build these modules:
_asyncio  _bisect   _blake2
_bz2  _codecs_cn_codecs_hk 
_codecs_iso2022   _codecs_jp_codecs_kr 
_codecs_tw_contextvars  _crypt 
_csv  _ctypes   _ctypes_test   
_curses   _curses_panel _datetime  
_dbm  _decimal  _elementtree   
_heapq_json _lsprof
_lzma _md5  _multibytecodec
_multiprocessing  _opcode   _pickle
_posixshmem   _posixsubprocess  _queue 
_random   _scproxy  _sha1  
_sha256   _sha3 _sha512
_socket   _sqlite3  _statistics
_struct   _testbuffer   _testcapi  
_testimportmultiple   _testinternalcapi _testmultiphase
_tkinter  _uuid _xxsubinterpreters 
_xxtestfuzz   array audioop
binascii  cmath fcntl  
grp   math  mmap   
nis   parserpyexpat
readline  resource  select 
syslogtermios   unicodedata
xxlimited zlib
--

The part "-c _struct.c" in the error syndrome stands out to me because there is 
not a prefix like the other modules that get built successfully before the 
"build_ext" section. In the successful cases, we see a prefix like "-c 
./Modules/faulthandler.c". This can be seen in my attached log.

Also, when I use the Python 3.6.10 source, the build for _struct looks like 
this and goes fine on Mac 10.15.1. This seems to be the only version of python 
that works for me on 10.15.1 that I know of.

---
building '_struct' extension
creating build/temp.macosx-10.15-x86_64-3.6/System
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company
creating build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel
creating 
build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox
creating 
build/temp.macosx-10.15-x86_64-3.6/System/Volumes/Data/my_company/devel/sandbox/my_username
creating 

[issue41022] AST sum types is unidentifiable after ast.Constant became a base class

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> Where they were identifable in such way? It is incorrect way, because user 
> can create a subclass of every AST class.

You are right, I didn't consider that while I was opening the issue. Also from 
3.9>= can be searched "|" in the ASDL signature to identify these types.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37369] Issue with pip in venv on Powershell in Windows

[Miika Mäkelä]

Miika Mäkelä  added the comment:

I tried to reproduce the issue on a another windows installation (Windows build 
10.0.18363.0), but was however unable to do it following the same steps. I fear 
that my original python/pip installation is just corrupted somehow due to 
having many python interpreters around previously.

My username with the machine I am having issues is typical 
C:\Users\miika.makela\

without any nonascii characters and without spaces in the data path. 

With the other machine I tried my username is just C:\Users\Miika

I'll open an issue if I sometime find out what really did go wrong, thanks for 
the reply.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41128] Signal handlers should not hang during blocked main thread

[Josef Havránek]

New submission from Josef Havránek :

python3 When handling signals (via signal module) have delayed execution when 
main thread is blocked/waiting for event

That is sub-optimal(signal "could get lost"). Signals shoud be handled asap... 
Think about scenario when os may be asking python nicely before it sends 
os.kill...so we could be dead on next bytecode instruction of main thread.

In such scenario when main thread is blocked and VM recives signal it should 
"context switch" to handler immediately thus not being dependent on state of 
main thread to be in executable state.

This gotcha should be included in documentation of all 3.x version since there 
should be nothing running(including long-runing c computation like regex 
matching ;) ) and basicly nothing is preventing handler from execution 

more details and test script in file attached

ps excuse my english/typos and uglines of code... this is my 1st bug 
report/enhancment proposal

--
components: Extension Modules
files: testscript.py
messages: 372435
nosy: Josef Havránek
priority: normal
severity: normal
status: open
title: Signal handlers should not hang during blocked main thread
type: enhancement
versions: Python 3.7, Python 3.8
Added file: https://bugs.python.org/file49265/testscript.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

[Steve Dower]

Steve Dower  added the comment:

Those are all good ideas, but using os.stat(d) instead of d.stat() is shorter, 
more reliable, more compatible, and already works.

There's no middle ground where DirEntry can be faster, because it's already 
using that middle ground. All the discussion between Eryk and myself was 
figuring out whether we can use the DirEntry/FindFileData information to tell 
whether the file needs an explicit stat() or not, and we can't.

Most of the performance impact of stat() is just in opening the file (which 
scandir() does not do). As soon as we have to directly access the file, we may 
as well get all the information from it. We're already getting all the "cheap" 
information we can.

--
assignee:  -> docs@python
components: +Documentation
nosy: +docs@python
versions: +Python 3.10, Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

[utkarsh]

utkarsh  added the comment:

@thatiparthy These were the most logical changes, standard error messages, 
which were already there in the existing code, I just edited them as mentioned 
here. What part of your "work" do you think i copied?
Sent this PR to get familiar to the process mostly, i will close it if you feel 
insecure. No need to be rude.
thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38879] Reordered error checking in PyArena_New().

[Zoltán Szatmáry]

Change by Zoltán Szatmáry :


--
hgrepos:  -386

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

[Cezary Wagner]

Cezary Wagner  added the comment:

I think we can assume that NTFS is priority since that is the most used option.

I can not discuss what with FAT32 or FAT since I am not the best in this domain 
(in NTFS I am not the best too now). Whatever I think that system must do 
allocation for open files to avoid conflicts so it can be tracked but how?

Possible solutions is some extra function, argument for Windows - which makes 
cache dirty between calls.

It is very dirty proposal - I need to think if it is good. Even used names is 
ugly I need think more about it. My imagination tells me that it can be good 
direction.

dir_entry.stat(nt_force_cache_refresh=True) - it can be good for specific 
entries.
os.scandir(nt_force_cache_refresh=True) - it is sometimes not need for all 
entries

I am thinking:
dir_entry.stat(nt_force_cache_refresh=True) should be faster than 
os.stat(dir_entry.path) instead dir_entry.stat() which not works fo open files.
os.scandir(nt_force_cache_refresh=True) should be faster than 
dir_entry.stat(nt_force_cache_refresh=True) and dir_entry.stat() will work for 
open files. It is simpler to understand that Windows is different if such extra 
attribute must be added at all.

nt_force_cache_refresh can add to dir_entry some information that .stat() 
should not use cache.

Then best will be to not use nt_force_cache_refresh for open files - maybe you 
will find the way to detect open files in external application. I think Windows 
API allow to check if file is open - as far as remember sysinternals tools can 
do this so there some API for it I think.

See this tool: https://docs.microsoft.com/en-us/sysinternals/downloads/handle - 
maybe there is source code for it or you can learn for it.

Maybe you can check if file is open with use this API before dir_entry.stat()

I do want to force any solution but just share some rough ideas.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

[శ్రీనివాస్ రెడ్డి తాటిపర్తి]

Srinivas  Reddy Thatiparthy(శ్రీనివాస్ రెడ్డి తాటిపర్తి) 
 added the comment:

@utk You could have taken some other easy issue from 
https://bugs.python.org/issue?status=1&@sort=-activity&@columns=id%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Easy%20issues&@startwith=0&@group=priority=6&@action=search&@filter=&@pagesize=50
 instead of copy pasting my work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.

[Steve Dower]

Steve Dower  added the comment:

> We're faced with the choice between either always calling the real lstat, or 
> just documenting that files with hard links will have stale information if 
> the file was updated using another link.

That's an easy choice: we document it.

The os module comes with the assumption that platform-specific behaviour may 
vary, so this is really just a helpful note about a known variation on Windows. 
It is not a warning, just a note.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

[utkarsh]

Change by utkarsh :


--
nosy: +utk
nosy_count: 8.0 -> 9.0
pull_requests: +20329
pull_request: https://github.com/python/cpython/pull/21170

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35823] Use vfork() in subprocess on Linux

[Alexey Izbyshev]

Alexey Izbyshev  added the comment:

I'd really like to get this merged eventually because vfork()-based solution is 
fundamentally more generic than posix_spawn(). Apart from having no issue with 
close_fds=True, it will also continue to allow subprocess to add any process 
context tweaks without waiting for availability of corresponding features in 
posix_spawn().

If there is no objection from Gregory or other devs, I can pick up from where I 
left in two weeks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41127] Executing code in thread or process pools: run_in_executor example

[Austin Raney]

New submission from Austin Raney :

I found an issue with the concurrent.futures.ProcessPoolExecuter() example (#3) 
in the asyncio event loops documentation. The call to asyncio.run(main()) 
should be guarded by `__name__=="__main__":`, as it sits now a RuntimeError is 
thrown.

https://docs.python.org/3/library/asyncio-eventloop.html#executing-code-in-thread-or-process-pools

--
assignee: docs@python
components: Documentation
messages: 372428
nosy: aaraney, docs@python
priority: normal
severity: normal
status: open
title: Executing code in thread or process pools: run_in_executor example
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40275] test.support has way too many imports

[hai shi]

Change by hai shi :


--
pull_requests: +20328
pull_request: https://github.com/python/cpython/pull/21169

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40725] Documentation: Benchmark table in "What's new in Python 3.9" has weird values

[hai shi]

Change by hai shi :


--
nosy: +shihai1991
nosy_count: 5.0 -> 6.0
pull_requests: +20327
pull_request: https://github.com/python/cpython/pull/21169

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40799] Create Lib/_pydatetime.py file to optimize "import datetime" when _datetime is available

[Paul Ganssle]

Paul Ganssle  added the comment:

As for deciding between moving to `datetime/` and moving to `_pydatetime`, I 
think we should send an e-mail to Python-Dev about it to get a wider 
perspective, because the import machinery is a lot of black magic, and I know 
that there are large proprietary code bases out there that pile weird stuff on 
top of it. I'm not sure I can fully appreciate the trade-offs.

The biggest advantage I see to moving `datetime` into its own folder is that it 
gives us a lot more freedom to expand into smaller sub-modules in the future. 
For example, in `zoneinfo`, we have zoneinfo/_common.py 
(https://github.com/python/cpython/blob/2e0a920e9eb540654c0bb2298143b00637dc5961/Lib/zoneinfo/_common.py),
 which is some logic shared between the C and Python implementations; 
`_zoneinfo.c` is able to rely directly on `_common.py` without importing 
`zoneinfo/_zoneinfo.py` (which saves us a bunch of other module imports as 
well).

Right now the C implementation of `datetime` only directly imports `time` and 
`_strptime`, but I could imagine future enhancements that would be stupidly 
inconvenient to implement in C, but where we wouldn't want to implement all of 
_pydatetime just to get a pure-Python implementation. Having a namespace 
available for such packages would be useful.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35823] Use vfork() in subprocess on Linux

[Yonatan Goldschmidt]

Yonatan Goldschmidt  added the comment:

With issue35537 merged, do we have any intention to get on with this? From what 
I can tell, it provides roughly the same performance benefit - but works also 
with close_fds=True, which is the default of Popen, so IMO it's a worthy 
addition.

I tested a rebase on current master, test_subprocess passes on my Linux box and 
there are no more -Wclobbered warnings after Alexey's latest change of the 
do_fork_exec() helper.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40799] Create Lib/_pydatetime.py file to optimize "import datetime" when _datetime is available

[Paul Ganssle]

Paul Ganssle  added the comment:

> bout _strptime, I see that the time.strptime() imports internally the 
> _strptime module.

Ah, sorry, my remark about including `_strptime` was off the cuff — I thought 
it was only used in `datetime`, which is why I said "possibly _strptime". If 
it's used for `time` as well, we should leave it where it is.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41126] Running test suite gives me - python.exe(14198, 0x114352dc0) malloc: can't allocate region

[శ్రీనివాస్ రెడ్డి తాటిపర్తి]

Srinivas  Reddy Thatiparthy(శ్రీనివాస్ రెడ్డి తాటిపర్తి) 
 added the comment:

It was a success. And yes, this is a duplicate of issue 40928.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1635741] Py_Finalize() doesn't clear all Python objects at exit

[Dong-hee Na]

Change by Dong-hee Na :


--
pull_requests: +20326
pull_request: https://github.com/python/cpython/pull/21168

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41126] Running test suite gives me - python.exe(14198, 0x114352dc0) malloc: can't allocate region

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

This is probably related to issue 40928. Does the test succeed or fail when you 
do this?

--
nosy: +remi.lapeyre

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41126] Running test suite gives me - python.exe(14198, 0x114352dc0) malloc: can't allocate region

[Eric V. Smith]

Change by Eric V. Smith :


--
components: +macOS
nosy: +ned.deily, ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40893] tkinter: integrate TkDND support

[E. Paine]

E. Paine  added the comment:

I have now marked the PR ready for review.

Terry:

Could you please take a quick look to double-check that the changes proposed 
are fully backwards-compatible? I have run the IDLE tests and have tried to pay 
particular attention to this problem when developing, but would appreciate a 
second opinion (to ensure we won't break existing tkinter scripts - such as 
IDLE).

Serhiy:

I would really appreciate your opinion on the patch, particularly the change to 
the _bind method. I am also concerned about the getlist_event method as it can 
split dropped text data (for example) but I don't know how to fix it (unless we 
give the user the raw data for them to fix it?).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41084] Signify that a SyntaxError comes from an fstring in the error message

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 2e0a920e9eb540654c0bb2298143b00637dc5961 by Lysandros Nikolaou in 
branch 'master':
bpo-41084: Adjust message when an f-string expression causes a SyntaxError 
(GH-21084)
https://github.com/python/cpython/commit/2e0a920e9eb540654c0bb2298143b00637dc5961


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41126] Running test suite gives me - python.exe(14198, 0x114352dc0) malloc: can't allocate region

[శ్రీనివాస్ రెడ్డి తాటిపర్తి]

Change by Srinivas  Reddy Thatiparthy(శ్రీనివాస్ రెడ్డి తాటిపర్తి) 
:


--
components: +Build
versions: +Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41126] Running test suite gives me - python.exe(14198, 0x114352dc0) malloc: can't allocate region

[శ్రీనివాస్ రెడ్డి తాటిపర్తి]

New submission from Srinivas  Reddy Thatiparthy(శ్రీనివాస్ రెడ్డి తాటిపర్తి) 
:

While running tests with `./python.exe Lib/test`, I see the following text in 
the console.

Is this a bug? 

``
python.exe(14198,0x114352dc0) malloc: can't allocate region
:*** mach_vm_map(size=842105263157895168, flags: 100) failed (error code=3)
python.exe(14198,0x114352dc0) malloc: *** set a breakpoint in 
malloc_error_break to debug
python.exe(14198,0x114352dc0) malloc: can't allocate region
:*** mach_vm_map(size=842105263157895168, flags: 100) failed (error code=3)
python.exe(14198,0x114352dc0) malloc: *** set a breakpoint in 
malloc_error_break to debug
python.exe(14198,0x114352dc0) malloc: can't allocate region
:*** mach_vm_map(size=421052631578947584, flags: 100) failed (error code=3)
python.exe(14198,0x114352dc0) malloc: *** set a breakpoint in 
malloc_error_break to debug
python.exe(14198,0x114352dc0) malloc: can't allocate region
:*** mach_vm_map(size=421052631578947584, flags: 100) failed (error code=3)
python.exe(14198,0x114352dc0) malloc: *** set a breakpoint in 
malloc_error_break to debug
``

My system is macOS Catalina 10.15.5 (19F101).

unman -a

Darwin Srinivass-MacBook-Pro.local 19.5.0 Darwin Kernel Version 19.5.0: Tue May 
26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64

--
messages: 372420
nosy: thatiparthy
priority: normal
severity: normal
status: open
title: Running test suite gives me - python.exe(14198,0x114352dc0) malloc: 
can't allocate region

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26212] Python with ncurses6.0 will not load _curses module on Solaris 10

[Zackery Spytz]

Zackery Spytz  added the comment:

Python 2 is EOL, so I think this issue should be closed.

--
nosy: +ZackerySpytz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41116] build on macOS 11 (beta) does not find system-supplied third-party libraries

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I see, and (as expected) get the same output when I build like that.

I agree that we should teach _osx_support about this. I guess we should add a 
function that returns the SDK root, which can then be used in distutils (and 
setuptools, ...) instead of looking for -isysroot in CFLAGS.  The new function 
can than look for the default SDK when there is no sysroot flag.

For 3.10 we should look into cleaning up configure.ac, it contains some logic 
that is no longer valid or necessary (such as looking for the architectures 
supported by /usr/lib/libSystem.dylib).  We should also avoid adding 
"-isysroot" to the CFLAGS/LDFLAGS to avoid problems when building on user 
machines that might have a different version of Xcode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41120] Possible performance improvement for itertools.product example on Python Docs

[Abbas Taher]

Abbas Taher  added the comment:

Good morning Raymond,   Thank you for this quick response.
You are absolutely right on all your points if you only consider that the 
example given in the documentation as purely a suggestion of "equivalent" 
version to the one implemented in the library.
However, if you look at the examples given in the documentation as an 
illustration of an algorithm that can be customized and changed by the end-user 
if the existing implementation is not suitable then you might consider 
otherwise. 
This is exactly what happened when I wanted to do a Cartesian Product. 
Starting, I first looked at itertools and 
run my routine using itertools.product(). However, it turns out that I needed 
to flatten the result tuples which prompted me toadjust the given example 
accordingly and then add the nested generator.

As for your statement:  "Nested generators are an intermediate level Python 
skill, so using them in the example code makes the docs less accessible.". Let 
me point to the facts, that someone looking at itertools and trying to do a 
Cartesian Product 
is certainly an intermediate programmer and often likes to understand, learn or 
peek inside the algorithm used. Additionally, the suggested improvement is a 
generator similar to the implementation where as the existing example is not.

Thus, I hope I am able to convince you to reconsider your decision especially 
when you look at the given examples from a wider perspective and more so when 
the suggested improvement is minor and easy to understand.
Best Regards,
Abbas

Side note: one of the first tutorial videos I used to learn Python were the 
ones you prepared for Oreilly. Thank you and thank you for all the work you did 
and still doing for free for the Python language and the Python community over 
so many many many years.

On Thursday, June 25, 2020, 9:19:55 PM EDT, Raymond Hettinger 
 wrote:  

Raymond Hettinger  added the comment:

Thanks for the suggestion, but the purpose of the rough equivalent example is 
to give a better idea of what product() does.  It is not about performance — 
that job falls to the actual implementation.

Nested generators are an intermediate level Python skill, so using them in the 
example code makes the docs less accessible.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41116] build on macOS 11 (beta) does not find system-supplied third-party libraries

[Ned Deily]

Ned Deily  added the comment:

I should have specified that the failures I saw were just with the default 
intel-64 arch on the beta.

After thinking about it a bit and reviewing the code, I think the best approach 
is to handle this in _osx_support.py as part of the compiler flags 
customization.  What's been happening over the years is that Apple has been 
deprecating and then removing system header files and system library files on 
running systems and instead forcing the use of an SDK to build with for both. 
That's fine but, to continue to support a range of systems, we should be able 
to handle the various default configurations when the user does not explicitly 
specify a particular SDK to use. The Apple compiler chain does the right thing 
under the covers so that C compilations just work. setip.py's macosx_sdk_root() 
now has initialization code to discover the path to the system header files 
that the compiler is using; it calls the compiler with -v on a dummy program 
and then scans the compiler output for the include paths searched until it 
finds either /usr/include or a path ending in .sdk/usr/include. (It's a bit
  of a hack but a fairly widely-used hack that works with both clang and gcc, 
at least, and for the Apple-supplied compilers, takes into account the dynamic 
specification of SDK via xcrun.)

I think the thing to do is move that sdk path detection into _osx_support and 
then, at the end of other customization steps, if there isn't already a valid 
-isysroot provided by the user (either through ./configure or CFLAGS et al, add 
a -isysroot for the discovered default SDK path (or /) into the appropriate 
customized compiler and linker flags. It would also provide a function to 
return the SDK path for setup.py (the equivalent of the current 
macosx_sdk_root()) and others to use. setup.py and Distutils should then do the 
right things for both header and library file paths on any configuration 
(whether header files are installed or not and whether lib files are installed 
or not, Command Line Tools or Xcode) and, with luck, any other Distutils 
replacements that also use _osx_support will also do the right thing.

I had contemplated removing _osx_support at some point in the future but, now 
with the return of universal files, I'm glad we didn't.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16623] argparse help formatter does not honor non-breaking space

[Cheryl Sabella]

Change by Cheryl Sabella :


--
resolution:  -> out of date
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41116] build on macOS 11 (beta) does not find system-supplied third-party libraries

[Ronald Oussoren]

Ronald Oussoren  added the comment:

These build for me (master branch), because the support for .tbd files in the 
macOS SDK still works.

The ones that don't build for me:

- Bits not found: _gdbm, _hashlib, _ssl, ossaudiodev, spwd
- Failed to build: _ctypes, _decimal, _lzma

This was with a "Universal 2" build (which explains _ctypes) and without 3party 
libraries (which explains _gdbm, _ssl and _lzma).  ossaudiodev and spwd are not 
supported on macOS even on older versions of the OS.

I can get _decimal to build with a trivial patch to 
Modules/_decimal/mpdecimal.py (I'll create a PR later today). The PR will 
obviously be untested w.r.t. actually supporting ARM64.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40058] Running test_datetime twice fails with: module 'datetime' has no attribute '_divide_and_round'

[hai shi]

hai shi  added the comment:

> My PR of bpo-40799 fix this issue.
Thanks victor for your work, I closed my PR ;)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16623] argparse help formatter does not honor non-breaking space

[hai shi]

hai shi  added the comment:

> Python 2 is EOL, so I think this issue should be closed.

+1, if someone need create a PR, we can reopen it again.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40275] test.support has way too many imports

[hai shi]

Change by hai shi :


--
pull_requests: +20325
pull_request: https://github.com/python/cpython/pull/21167

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41125] Display exit-codes for abruptly terminated processes in concurrent.futures

[Laurie Opperman]

Change by Laurie Opperman :


--
keywords: +patch
pull_requests: +20324
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/21166

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41092] Report actual size from 'os.path.getsize'

[Gareth Rees]

Gareth Rees  added the comment:

The proposed change adds a Boolean flag to os.path.getsize() so that it returns:

os.stat(filename).st_blocks * 512

(where the 512 is the file system block size on Linux; some work is needed to 
make this portable to other operating systems).

The Boolean argument here would always be constant in practice -- that is, 
you'd always call it like this:

virtual_size = os.path.getsize(filename, apparent=True)
allocated_size = os.path.getsize(filename, apparent=False)

and never like this:

x_size = os.path.getsize(filename, apparent=x)

where x varies at runtime.

The "no constant bool arguments" design principle [1] suggests that this should 
be added as a new function, something like os.path.getallocatedsize().

  [1] https://mail.python.org/pipermail/python-ideas/2016-May/040181.html

--
nosy: +g...@garethrees.org

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41111] Convert a few stdlib extensions to the limited C API

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41115] Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError

[శ్రీనివాస్ రెడ్డి తాటిపర్తి]

Change by Srinivas  Reddy Thatiparthy(శ్రీనివాస్ రెడ్డి తాటిపర్తి) 
:


--
keywords: +patch
pull_requests: +20323
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/21165

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com