[issue17561] Add socket.bind_socket() convenience function
Change by Giampaolo Rodola' : -- pull_requests: +12658 stage: resolved -> patch review ___ Python tracker <https://bugs.python.org/issue17561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17561] Add socket.bind_socket() convenience function
Giampaolo Rodola' added the comment: Patch committed as of: https://github.com/python/cpython/commit/eb7e29f2a9d075accc1ab3faf3612ac44f5e2183 For posterity, since the API evolved since the original proposal (as per PR review/suggestions), this is the final incarnation: # IPv4 only >>> socket.create_server(addr) # IPv6 only >>> socket.create_server(addr, family=socket.AF_INET6) # IPv4 + IPv6 >>> socket.create_server(addr, family=socket.AF_INET6, dualstack_ipv6=True) # IPv4/6 if possible and don't care about IPv4 mapped addresses, else IPv4 >>> if socket.has_dualstack_ipv6(): ...s = socket.create_server(addr, family=socket.AF_INET6, dualstack_ipv6=True) ... else: ...s = socket.create_server(addr) -- ___ Python tracker <https://bugs.python.org/issue17561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17561] Add socket.bind_socket() convenience function
Change by Giampaolo Rodola' : -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue17561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35934] Add socket.create_server() utility function
Giampaolo Rodola' added the comment: New changeset eb7e29f2a9d075accc1ab3faf3612ac44f5e2183 by Giampaolo Rodola in branch 'master': bpo-35934: Add socket.create_server() utility function (GH-11784) https://github.com/python/cpython/commit/eb7e29f2a9d075accc1ab3faf3612ac44f5e2183 -- ___ Python tracker <https://bugs.python.org/issue35934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36488] os.sendfile() on BSD, macOS don't return bytes sent on EINTR
Giampaolo Rodola' added the comment: sendfile() on BSD/OSX is complicated by the headers/trailers args. You'll have to take that into account in the retry logic, adding unnecessary complexity. Since sendfile() may already return fewer bytes than requested (e.g. non-blocking sockets or big files) it's just easier to return the bytes sent thus far (if any). I can work on a patch once I find some time. > Wasn't the point of PEP475 that all EINTR returns would be explicitly handled > by retrying rather than forcing the user to handle it? >From PEP475: <<[...] to relieve application code from the burden of doing so>> -- ___ Python tracker <https://bugs.python.org/issue36488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32941] mmap should expose madvise()
Change by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue32941> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36488] os.sendfile() on BSD and macOS does not return bytes sent on EINTR
Change by Giampaolo Rodola' : -- title: os.sendfile() on BSD OSs and macOS does not return bytes sent on EINTR -> os.sendfile() on BSD and macOS does not return bytes sent on EINTR ___ Python tracker <https://bugs.python.org/issue36488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36488] BSD/OSX: os.sendfile() does not return bytes sent on EINTR
Change by Giampaolo Rodola' : -- nosy: +rosslagerwall ___ Python tracker <https://bugs.python.org/issue36488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36488] BSD/OSX: os.sendfile() does not return bytes sent on EINTR
New submission from Giampaolo Rodola' : >From "man sendfile" on both OSX and FreeBSD: <<[EINTR] A signal interrupted sendfile() before it could be completed. If specified, the number of bytes successfully sent will be returned in *len.>> Right now we catch EINTR and simply retry. Instead we should only retry is no bytes were sent, else we should return those bytes, similarly to what we do on EAGAIN and EBUSY: https://github.com/python/cpython/blob/2438cdf0e932a341c7613bf4323d06b91ae9f1f1/Modules/posixmodule.c#L8907-L8917 -- components: Library (Lib) messages: 339214 nosy: giampaolo.rodola priority: normal severity: normal stage: needs patch status: open title: BSD/OSX: os.sendfile() does not return bytes sent on EINTR versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue36488> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Giampaolo Rodola' added the comment: Merged. I'm agnostic about backporting IPPROTO_IPV6 (or others) on < 3.8 so I'll leave it up to somebody else to decide/implement. -- status: open -> pending versions: -Python 3.7 ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Giampaolo Rodola' added the comment: New changeset 3eca28c61363a03b81b9fb12775490d6e42d8ecf by Giampaolo Rodola in branch 'master': bpo-29515: add missing socket.IPPROTO_* constants on Windows (GH-12183) https://github.com/python/cpython/commit/3eca28c61363a03b81b9fb12775490d6e42d8ecf -- ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Giampaolo Rodola' added the comment: I'd like to move forward with this in order to fix issue17561. If there are no complaints I'm gonna merge this in ~ a week. -- ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36256] parser module fails on legal input
Change by Giampaolo Rodola' : -- nosy: -giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue36256> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26826] Expose new copy_file_range() syscall in os module.
Giampaolo Rodola' added the comment: Little update about this. According to: http://man7.org/linux/man-pages/man2/copy_file_range.2.html ...it seems glibc 2.27 released on 2018-02-01 includes copy_file_range(). I'm not the best candidate for giving advice on C syscalls definitions/availability, but FWIW this works on my Linux 4.15: #if defined(__linux__) && \ defined(SYS_copy_file_range) && \ defined(__GLIBC_PREREQ) && \ __GLIBC_PREREQ(2, 27) ... #endif I think (but not sure) this is supposed to fix this condition, which appears to be the major blocker here: https://github.com/python/cpython/blob/9a177061cd7190eabf40efd31e8981e0bccd5dc4/Lib/test/test_os.py#L258-L261 -- ___ Python tracker <https://bugs.python.org/issue26826> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4080] unittest: display time used by each test case
Giampaolo Rodola' added the comment: Hello. This is something I needed so I decided to implement it by taking inspiration from pytest's --durations=N argument, which basically does the same (except that it uses 0.XX precision instead of 0.XXX). PR at: https://github.com/python/cpython/pull/12271 -- versions: +Python 3.8 -Python 3.3 ___ Python tracker <https://bugs.python.org/issue4080> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4080] unittest: display time used by each test case
Change by Giampaolo Rodola' : -- pull_requests: +12254 ___ Python tracker <https://bugs.python.org/issue4080> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36208] AsyncIO V4MAPPED addresses with V6ONLY.
Giampaolo Rodola' added the comment: I'm not sure I understand the issue, but setting IPV6_V6ONLY to 0 by default is not an option because asyncio wants to serve IPv4 and IPv6 with 2 distinct sockets on purpose. The reason for that is because when an IPv4 connection occurs we want getpeername/getsockname to return '127.0.0.1' instead of ':::127.0.0.1'. Also, IPV6_V6ONLY = 1 is set by default on all platforms. It is not set on Windows because IPPROTO_IPV6 constant is missing due to issue29515, but that doesn't matter because IPV6_V6ONLY = 1 is already the default on Windows. As for Linux, /proc/sys/net/ipv6/bindv6only shouldn't matter because setting the option in Python is supposed to overwrite what /proc/sys/net/ipv6/bindv6only dictates. With that said, what's your use case exactly? Why does your test script uses AF_INET6 and an IPv4 address? -- ___ Python tracker <https://bugs.python.org/issue36208> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Giampaolo Rodola' added the comment: IPPROTO_TCP/UDP/RAW/ICMP are already there. AFAICT the only useful one here is IPPROTO_IPV6, because it's needed for enabling dual-stack. I don't think IPPROTO_IPV4 is a must (e.g. it's not available on Linux). Same for IPPROTO_ICMPV6. The only reason we may decide to backport one of these is because they are not officially documented (see issue27409). That gives us some flexibility but potentially it may do more harm than good because an app running on 3.7.3 would break on 3.7.2. Note: the original ticket (issue6926) is from 2009, meaning that whoever wanted these missing constants on Windows already hard-coded them in their Python code long ago. -- ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Giampaolo Rodola' added the comment: Done: https://github.com/python/cpython/pull/12183/ This adds quite a bunch of new constants so I would be keen on considering it an enhancement rather than a bugfix and land it in 3.8 only. As for 3.7 and 3.6 (and 2.7?) it may make sense to fix IPPROTO_IPV6 only. Thoughts? -- ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27409] List socket.SO_*, SCM_*, MSG_*, IPPROTO_* symbols
Change by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue27409> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Change by Giampaolo Rodola' : -- keywords: +patch pull_requests: +12179 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Giampaolo Rodola' added the comment: @Eryk do you mind if I make a PR with your code(I will of course author you)? Regardless from issue17561 I think this is an important fix because without IPPROTO_IPV6 is not possible to implement a dual-stack IPv4/6 socket on Windows. -- ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31861] add aiter() and anext() functions to operator module
Change by Giampaolo Rodola' : -- nosy: -giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue31861> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30235] Validate shutil supports path-like objects, update docs accordingly
Giampaolo Rodola' added the comment: One thing to note is the copy_function parameter: if a path-like object is given as input should the copy_function receive the same object instead of a path string? >>> copytree(path_like_src, path_like_dst, copy_function=foo) Because of issue33695 in case of copytree(..., copy_function=...) we currently force to always pass path strings if copy_function is specified: https://github.com/python/cpython/blob/ed1deb0719f0ac1b08a374e30ad26a701d4d51a2/Lib/shutil.py#L450 -- ___ Python tracker <https://bugs.python.org/issue30235> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35652] Add use_srcentry parameter to shutil.copytree() II
Giampaolo Rodola' added the comment: New changeset c606a9cbd48f69d3f4a09204c781dda9864218b7 by Giampaolo Rodola in branch 'master': bpo-35652: shutil.copytree(copy_function=...) erroneously pass DirEntry instead of path str (GH-11997) https://github.com/python/cpython/commit/c606a9cbd48f69d3f4a09204c781dda9864218b7 -- ___ Python tracker <https://bugs.python.org/issue35652> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36103] Increase shutil.COPY_BUFSIZE
Giampaolo Rodola' added the comment: @Inada: having played with this in the past I seem to remember that on Linux the bigger bufsize doesn't make a reasonable difference (but I may be wrong), that's why I suggest to try some benchmarks. In issue33671 I pasted some one-liners you can use (and you should target copyfileobj() instead of copyfile() in order to skip the os.sendfile() path). Also on Linux "echo 3 | sudo tee /proc/sys/vm/drop_caches" is supposed to disable the cache. -- nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue36103> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35802] os.stat / os.lstat always present, but code checks hastattr(os, 'stat') / hasattr(os, 'lstat')
Giampaolo Rodola' added the comment: A BB failed. Looks like it's unrelated with this change: https://buildbot.python.org/all/#/builders/99/builds/2198 -- nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue35802> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36105] Windows: use GetNativeSystemInfo instead of GetSystemInfo
Giampaolo Rodola' added the comment: It appears you guys are right. Sorry for the noise. -- ___ Python tracker <https://bugs.python.org/issue36105> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)
Giampaolo Rodola' added the comment: Yes, it's deliberate, see PR-12016. -- ___ Python tracker <https://bugs.python.org/issue33671> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36105] Windows: use GetNativeSystemInfo instead of GetSystemInfo
New submission from Giampaolo Rodola' : This is what MS doc says about GetSystemInfo: https://docs.microsoft.com/en-us/windows/desktop/api/sysinfoapi/nf-sysinfoapi-getsysteminfo <> $ grep -r GetSystemInfo Modules/_ctypes/malloc_closure.c:GetSystemInfo(); Modules/mmapmodule.c:GetSystemInfo(); Modules/mmapmodule.c:GetSystemInfo(); Modules/posixmodule.c:GetSystemInfo(); -- components: Windows messages: 336512 nosy: eryksun, giampaolo.rodola, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Windows: use GetNativeSystemInfo instead of GetSystemInfo versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue36105> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)
Change by Giampaolo Rodola' : -- pull_requests: -7529 ___ Python tracker <https://bugs.python.org/issue33671> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)
Change by Giampaolo Rodola' : -- pull_requests: -7536 ___ Python tracker <https://bugs.python.org/issue33671> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)
Change by Giampaolo Rodola' : -- pull_requests: -7528 ___ Python tracker <https://bugs.python.org/issue33671> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33671] Efficient zero-copy for shutil.copy* functions (Linux, OSX and Win)
Change by Giampaolo Rodola' : -- pull_requests: +12047 ___ Python tracker <https://bugs.python.org/issue33671> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35648] Add use_srcentry parameter to shutil.copytree()
Giampaolo Rodola' added the comment: When user uses copy_function argument, the callback function should always receive path strings, both for consistency and for not breaking backward compatibility. This line: use_srcentry = copy_function is copy2 or copy_function is copy ...makes sure of that (only passes DirEntry instances around if copy_function is not specified by user). -- ___ Python tracker <https://bugs.python.org/issue35648> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: IMHO given the size of the change and how quickly this evolved I would probably feel safer to mark the API as experimental (no pun intended, this is great work), at least in alpha2. I believe we now have a good explanation in the docs but it still needs to be integrated better into the larger, existing multiprocessing docs (I will try to add some comments on the PR tomorrow). -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: > Side note: I’m the author of the sysv_ipc package you found, as well as the > posix_ipc package. Sorry, I didn't realize it was you. > Not true. A manager started by one process can be connected to by another > process that is not a child. This is covered in the docs here: > https://docs.python.org/3/library/multiprocessing.html#using-a-remote-manager Ah nice! OK then. With this + Philip's explanation on why shmat()/shmdt() are not needed I guess I ran out of API-related complaints now. =) -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35648] Add use_srcentry parameter to shutil.copytree()
Giampaolo Rodola' added the comment: It turns out that's a bug I introduced in PR-7874. I'm providing: https://github.com/python/cpython/pull/11997 ...which fixes it without introducing a new parameter. I also verified that monkey patching shutil.copy2 and shutil.copy doesn't cause any issue. -- nosy: +giampaolo.rodola pull_requests: +12026 stage: resolved -> patch review ___ Python tracker <https://bugs.python.org/issue35648> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35652] Add use_srcentry parameter to shutil.copytree() II
Change by Giampaolo Rodola' : -- pull_requests: +12024 ___ Python tracker <https://bugs.python.org/issue35652> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33695] Have shutil.copytree(), copy() and copystat() use cached scandir() stat()s
Change by Giampaolo Rodola' : -- pull_requests: +12025 ___ Python tracker <https://bugs.python.org/issue33695> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35648] Add use_srcentry parameter to shutil.copytree()
Change by Giampaolo Rodola' : -- assignee: docs@python -> resolution: later -> status: closed -> open ___ Python tracker <https://bugs.python.org/issue35648> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35652] Add use_srcentry parameter to shutil.copytree() II
Giampaolo Rodola' added the comment: Not sure why you created 2 identical issues. Closing this one as a duplicate of issue35648. -- nosy: +giampaolo.rodola resolution: -> duplicate stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue35652> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35652] Add use_srcentry parameter to shutil.copytree() II
Change by Giampaolo Rodola' : -- pull_requests: -12023 ___ Python tracker <https://bugs.python.org/issue35652> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment:
> We are consciously choosing to not support an atomic "create or attach".
> This significantly simplifies the API and avoids the valid concerns raised
> around user confusion relating to that behavior (including the use of
> different specified 'size' values in a race) but does not preclude our
> potentially introducing this as a feature in the future.
I understand that because of *size* we cannot solve the race condition issue
unless the user uses some sort of synchronization mechanism. FWIW I bumped into
this lib:
http://semanchuk.com/philip/sysv_ipc/
...which provides two separate APIs to "create" and "attach":
>>> SharedMemory("name", IPC_CREX)
>>> attach("name")
At this point I'm agnostic about the API, which is probably just a matter of
personal taste (e.g. one may prefer a separate SharedMemory.attach()
classmethod or a *mode* argument accepting "x" and "a"). I see that that lib
use shmat() on attach and shmdt() on detach. I'm not sure if that makes a
difference, just mentioning it because your implementation doesn't do that on
close() and perhaps it should.
> Combined with a SyncManager.Lock, users can already achieve an atomic "create
> or attach" using this simpler API.
That assumes a single app/process which spawns a child (the "worker"). In that
case SyncManager.Lock/others is indeed compatible with SharedMemory and can be
used to implement non-racy "create or attach" and also to synchronize memory
access on read and write. But AFAICT there are two uses cases for this
functionality, and there currently is no mechanism to do any that if you have
two unrelated apps/processes relying on a common shared memory *name* and
*size*. I'm taking a peek at "man smh_overview" which says:
<>
That would translate into a new Semaphore(name=None, create=False) class which
(possibly?) would also provide better performances compared to
SyncManager.Semaphore. Not sure if we can do the same on Windows though.
Extra 1: apparently there are also POSIX msgget(), msgrcv() and msgsnd()
syscalls which could be used to implement a System-V message Queue similar to
SyncManager.Queue later on.
Extra 2: given the 2 distinct use-cases I wonder if the low-level component
(shared_memory.py) really belongs to multiprocessing module. Perhaps this
should be provided as a separate "sharedmemory" module with
multiprocessing.managers.SharedMemoryMemory being the high-level interface.
--
___
Python tracker
<https://bugs.python.org/issue35813>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35652] Add use_srcentry parameter to shutil.copytree() II
Change by Giampaolo Rodola' : -- pull_requests: +12023 ___ Python tracker <https://bugs.python.org/issue35652> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: Also, there is no way to delete/unwrap memory without using an existing SharedMemory instance, which is something we may not have on startup. Perhaps we should have a "shared_memory.unlink(name)" function similar to os.unlink() which simply calls C shm_unlink(). -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: Code looks much better now. I'm still not convinced "SharedMemory(name=None, create=False, size=0)" is the best API. How are you supposed to "create or attach" atomically? You can do that with O_EXCL but as it stands this is not togglable via the API. Also, could you address my comment about size? https://bugs.python.org/issue35813#msg335731 -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36067] subprocess terminate() "invalid handle" error when process is gone
Giampaolo Rodola' added the comment: Interesting. Because both errors/conditions are mapped to ERROR_INVALID_HANDLE we need the creation time. I can work on a patch for that. Potentially I can also include OSX, Linux and BSD* implementations for methods involving os.kill(pid). That would be a broader task though. That also raises the question if there are other methods other than kill()/terminate()/send_signal() that we want to make "safe" from the reused PID scenario. > Also, unrelated but something I noticed. Using _active list in Windows > shouldn't be necessary. Unlike Unix, a process in Windows doesn't have to be > waited on by its parent to avoid a zombie. Keeping the handle open will > actually create a zombie until the next _cleanup() call, which may be never > if Popen() isn't called again. Good catch. Looks like it deserves a ticket. -- ___ Python tracker <https://bugs.python.org/issue36067> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36067] subprocess terminate() "invalid handle" error when process is gone
Giampaolo Rodola' added the comment: On POSIX there is that risk, yes. As for Windows, the termination is based on the process handle instead of the PID, but I am not sure if that makes a difference. The risk of reusing the PID/handle is not related to this issue though. The solution I propose just ignores ERROR_INVALID_HANDLE and makes it an alias for "process is already gone". It does not involve preventing the termination of other process handles/PIDs (FWIW in order to do that in psutil I use PID + process' creation time to identify a process uniquely). -- ___ Python tracker <https://bugs.python.org/issue36067> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36067] subprocess terminate() "invalid handle" error when process is gone
Giampaolo Rodola' added the comment: I think this is somewhat similar to issue14252. The problem I see is that we should either raise ProcessLookupError or ignore the error (better). This concept of suppressing errors if process is gone is currently already established in 2 places: https://github.com/python/cpython/blob/bafa8487f77fa076de3a06755399daf81cb75598/Lib/subprocess.py#L1389 Basically what I propose is to extend the existing logic to also include ERROR_INVALID_HANDLE other than ERROR_ACCESS_DENIED. Not tested: def terminate(self): """Terminates the process.""" # Don't terminate a process that we know has already died. if self.returncode is not None: return try: _winapi.TerminateProcess(self._handle, 1) except WindowsError as err: if err.errno in (ERROR_ACCESS_DENIED, ERROR_INVALID_HANDLE): rc = _winapi.GetExitCodeProcess(self._handle) if rc == _winapi.STILL_ACTIVE: raise self.returncode = rc else: raise -- ___ Python tracker <https://bugs.python.org/issue36067> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36067] subprocess terminate() "invalid handle" error when process is gone
New submission from Giampaolo Rodola' : Happened in psutil: https://ci.appveyor.com/project/giampaolo/psutil/builds/22546914/job/rlp112gffyf2o30i == ERROR: psutil.tests.test_process.TestProcess.test_halfway_terminated_process -- Traceback (most recent call last): File "c:\projects\psutil\psutil\tests\test_process.py", line 85, in tearDown reap_children() File "c:\projects\psutil\psutil\tests\__init__.py", line 493, in reap_children subp.terminate() File "C:\Python35-x64\lib\subprocess.py", line 1092, in terminate _winapi.TerminateProcess(self._handle, 1) OSError: [WinError 6] The handle is invalid During the test case, the process was already gone (no PID). -- components: Library (Lib) messages: 336231 nosy: giampaolo.rodola priority: normal severity: normal stage: needs patch status: open title: subprocess terminate() "invalid handle" error when process is gone type: behavior versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue36067> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Giampaolo Rodola' added the comment: @eryksun I tried your patch (https://bugs.python.org/issue29515#msg287477) on Windows 10 and it looks good. It introduces the following new constants: IPPROTO_IGMP IPPROTO_GGP IPPROTO_PUP IPPROTO_IDP IPPROTO_ND IPPROTO_MAX IPPROTO_HOPOPTS IPPROTO_IPV4 IPPROTO_IPV6 IPPROTO_ROUTING IPPROTO_FRAGMENT IPPROTO_ESP IPPROTO_AH IPPROTO_ICMPV6 IPPROTO_NONE IPPROTO_DSTOPTS IPPROTO_EGP IPPROTO_PIM IPPROTO_SCTP ...plus these Windows-only ones for which I had to add an additional PyModule_AddIntMacro() call: IPPROTO_ICLFXBM IPPROTO_ST IPPROTO_CBT IPPROTO_IGP IPPROTO_RDP IPPROTO_PGM IPPROTO_L2TP Up 'till now only these ones were exposed: IPPROTO_ICMP IPPROTO_TCP IPPROTO_UDP IPPROTO_RAW I think we can proceed with this. Care to provide a patch? -- ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows
Giampaolo Rodola' added the comment: It turns out having this fix would be useful for proceeding with issue17561, which right now cannot support dual-stack IPv4/6 on Windows because IPPROTO_IPV6 is missing, see my comment: https://github.com/python/cpython/pull/11784#issuecomment-465078646 -- dependencies: +Add socket.bind_socket() convenience function nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue29515> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20215] socketserver.TCPServer can not listen IPv6 address
Change by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue20215> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17561] Add socket.bind_socket() convenience function
Change by Giampaolo Rodola' : -- stage: patch review -> commit review ___ Python tracker <https://bugs.python.org/issue17561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33935] shutil.copyfile throws incorrect SameFileError on Google Drive File Stream
Giampaolo Rodola' added the comment: @eryksun > Using the VSN and file index as if they're the same as POSIX st_dev and > st_ino is technically wrong. There is no guarantee that this tuple uniquely > identifies a file in Windows. I agree. FWIW, I faced the same issue on pyftpdlib and ended up not supporting Windows: https://github.com/giampaolo/pyftpdlib/blob/eef8a5650cd611da1df5fce16974ce90f43f4dc0/pyftpdlib/filesystems.py#L596-L605 It seems to me the problem here is os.path.samefile(): https://github.com/python/cpython/blob/0185f34ddcf07b78feb6ac666fbfd4615d26b028/Lib/genericpath.py#L87-L98 @steve.dower > For Windows it would be best (though slower) to pass the paths through > os._getfinalpathname before comparison. It seems this is how samefile() was originally implemented but it was changed: https://github.com/python/cpython/commit/490b32a3976d84eaf1d6ca8cdcb00eac0ce5055b https://bugs.python.org/issue11939 -- ___ Python tracker <https://bugs.python.org/issue33935> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: > It looks like it does skip calling ftruncate() if size is 0. From > posixshmem.c: Let me rephrase: are we forced to specify a value (aka call ftruncate()) on create ? If we are as I think, could size have a reasonable default value instead of 0? Basically I'm wondering if we can relieve the common user from thinking about what size to use, mostly because it's sort of a low level detail. Could it perhaps default to mmap.PAGESIZE? -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: As for *flags* argument "man shm_open" mentions 5 possible flags: - O_RDONLY, OR_RDWR: these can be served by existing *read_only* arg - O_CREAT, O_EXCL: dictates whether to raise error if file exists; could be served via *attach_if_exists* arg, if added, or internally if not added - O_TRUNC: related to *size* argument (it seems it should be set when size=0) As such I think "flags" should not be exposed. As for Windows, *read_only* is currently exposed but ignored, and no other flags are contemplated. It seems you can implement *read_only* by passing FILE_MAP_READ or FILE_MAP_WRITE to OpenFileMappingW() here: https://github.com/python/cpython/blob/1e5341eb171d7d2b90020cfcc0a64021326d/Lib/multiprocessing/shared_memory.py#L98 Not sure how to reliably check if a file mapping already exists, but possibly POSIX and Windows should behave the same in this regard (and some test cases to exercise all the possible combinations would be good to have). -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment:
> The unlink() method is available on the SharedMemory class. No manager is
> required. This is also captured in the docs.
I missed that, sorry.
>> 3) it seems "size" kwarg cannot be zero (current default)
> From the docs: When attaching to an existing shared memory block, set to 0
Mmm... this makes the API a bit weird because it's a arg which is optional only
for existing memory object, not new ones. I see that internally *size* is
passed to ftruncate(). Can't you just avoid calling ftruncate() if size is not
passed (None)? Also, what happens if you alter the size of an existing object
with a smaller value? Is the memory region overwritten?
a = shared_memory.SharedMemory(None, size=10)
b = shared_memory.SharedMemory(a.name, size=4)
>> Maybe something like this instead?
>>SharedMemory(name=None, attach_if_exists=False, size=0)
> I think this misses the use case when wanting to ensure we only attach to an
> existing shared memory block and if it does not exist, we should raise an
> exception because we can not continue.
It appears this is already covered:
>>> SharedMemory(name="non-existent")
Traceback (most recent call last):
...
_posixshmem.ExistentialError: No shared memory exists with the specified
name
> I believe the two dominant use cases to address are:
> 1) I want to create a shared memory block (either with or without a
> pre-conceived name).
> 2) I want to attach to an existing shared memory block by its unique name.
Don't you also want to "create if it doesn't exist, else attach" as a single,
atomic operation? It seems to me the only way to achieve that would be this,
but it's racy:
try:
m = shared_memory.SharedMemory("foo", size=10) # create
except ExistentialError:
m = shared_memory.SharedMemory("foo") # attach (race condition)
Note that here I'm entering into an unknown territory, because I'm not sure if
there are or should be sync primitives to "wait for another memory to join me"
etc. Just mentioning it for the sake of mental lucubration. =)
> If we move SharedMemoryManager to live in multiprocessing.managers, we need
> to decide how to handle (and communicate to the user appropriately) its
> potential absence.
How about:
# managers.py
try:
from .shared_memory import SharedMemory, SharedList
except ImportError:
pass
else:
class SharedMemoryManager: ...
class SharedMemoryServer: ...
del SharedMemory, SharedList
>> 4) I have some reservations about SharedMemory's "flags" and "mode" args.
> It sounds like you are agreeing with what I advocated in msg335660 (up
> above). Great!
If you mean drop those 2 args entirely then it probably there's no point in
exposing WindowsNamedSharedMemory and PosixSharedMemory . If you mean to keep
them then it seems "read_only=True" would conflict with "flags=O_RDWR". The
fact that the Windows counterpart uses different constants makes me think this
bitwise stuff should probably be handled internally, and exposed via friendly
named-args like "read_only" / "attach_if_exists" or similar (but it's still not
clear what those named args should be exactly).
--
___
Python tracker
<https://bugs.python.org/issue35813>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: Hopefully my last iteration: =) 1) As for SharedMemoryManager, I believe it should live in multiprocessing.managers, not shared_memory.py. It's a subclass of Manager and behaves like a manager (start(), shutdown(), get_server(), etc.), so IMO that's its natural place. 2) Same for SharedMemoryServer (which is a subclass of multiprocessing.managers.Server). SharedMemoryTracker appears to be just a support class for it, so IMO it should either be private or not documented. 3) ShareableList name is kinda inconsistent with other classes (they all have a "Shared" prefix). I'd call it SharedList instead. 4) Ideally the (public) APIs I have in mind are: multiprocessing.managers.SharedMemoryManager multiprocessing.managers._SharedMemoryTracker multiprocessing.managers.SharedMemoryServer (not documented) multiprocessing.shared_memory.SharedMemory multiprocessing.shared_memory.SharedList multiprocessing.shared_memory.WindowsNamedSharedMemory (maybe) multiprocessing.shared_memory.PosixSharedMemory (maybe) AFAIU there are 2 distinct use-cases at play: - two separate apps agreeing on a specific fixed memory *name* to attach to, which will use SharedMemory/List directly - one single app using a manager and a worker, which will use SharedMemoryManager, and get SharedMemory/List directly from it (see https://github.com/python/cpython/pull/11816/files#r257458137) IMO the 2 different namespaces would reflect and enforce this separation of use cases. 4) I have some reservations about SharedMemory's "flags" and "mode" args. * flags (O_CREAT, O_EXCL, O_CREX, O_TRUNC, O_RDONLY): it seems it may conflict with "read_only" arg. I wonder if we could achieve the same goal with more high-level named args instead (e.g. "create" / "attach_if_exists"). If in doubt, I would recommend to simply drop it for now. * mode: same considerations as above. Doc says "Its specification is not enforceable on all platforms" which makes me think it's probably better to drop it (also not sure why it defaults to 384). Hope this helps. -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: > Because shared memory blocks are not "owned" by a single process... > [...] > I propose a simpler, consistent-across-platforms API like: > SharedMemory(name=None, create=False, size=0) Maybe something like this instead? SharedMemory(name=None, attach_if_exists=False, size=0) The use case I'm thinking about is 2 distinct processes/apps which agree on a common fixed name. -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment:
> Because shared memory blocks are not "owned" by a single process, they are
> not destroyed/freed when a process exits.
1) it seems SharedMemory.close() does not destroy the memory region (I'm able
to re-attach to it via name). If I'm not mistaken only the manager can do that.
As such it probably makes sense to have an unlink() or destroy() methods. Maybe
SharedMemory should support the with statement.
2) it seems SharedMemory is only supposed to handle `memoryview`s. I suggest to
turn SharedMemory.buf in a read-onl property:
>>> sm = shared_memory.SharedMemory(None, size=23)
>>> sm.buf = [1,2,3]
>>> sm.close()
File "/home/giampaolo/svn/cpython-shm/Lib/multiprocessing/shared_memory.py",
line 166, in close
self.buf.release()
AttributeError: 'list' object has no attribute 'release'
3) it seems "size" kwarg cannot be zero (current default). Suggestion: either
choose default != 0 or make it a mandatory arg:
>>> shared_memory.SharedMemory(name=None)
ValueError: cannot mmap an empty file
>>> shared_memory.SharedMemory(name=None, size=0)
ValueError: cannot mmap an empty file
>>> shared_memory.SharedMemory(name=None, size=1)
PosixSharedMemory('psm_32161_20838', size=1)
>>>
4) I wonder if we should have multiprocessing.active_memory_children() or
something, similar to multiprocessing.active_children() so that one can do:
@atexit.register
def cleanup():
for x in multiprocessing.active_memory_children():
x.close() # or unlink/destroy
I guess that would be useful for people not using a manager. Also, I'm not sure
if active_memory_children() can return meaningful results with a brand new
process (I suppose it can't).
--
___
Python tracker
<https://bugs.python.org/issue35813>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36003] set better defaults for TCPServer options
Change by Giampaolo Rodola' : -- keywords: +patch pull_requests: +11908 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36003] set better defaults for TCPServer options
Giampaolo Rodola' added the comment: Update: because "request_queue_size" is passed to server_activate() method which can be subclassed, a better default for not breaking backward compatibility is 0 (not None). -- ___ Python tracker <https://bugs.python.org/issue36003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36003] set better defaults for TCPServer options
New submission from Giampaolo Rodola' : socketserver.TCPServer provides the following defaults: allow_reuse_address = False request_queue_size = 5 Proposal is to: * have "allow_reuse_address = True" on POSIX in order to immediately reuse previous sockets which were bound on the same address and remained in TIME_WAIT state * have "request_queue_size = None" so that it's up to socket.listen() to choose a default reasonable value (usually 128) -- components: Library (Lib) keywords: easy messages: 335612 nosy: asvetlov, giampaolo.rodola, neologix, yselivanov priority: normal severity: normal status: open title: set better defaults for TCPServer options type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36003> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12317] inspect.getabsfile() is not documented
Change by Giampaolo Rodola' : -- nosy: -giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue12317> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35934] Add socket.create_server() utility function
Giampaolo Rodola' added the comment: Correct. Sorry for the typo. -- ___ Python tracker <https://bugs.python.org/issue35934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35982] Create unit-tests for os.renames()
Giampaolo Rodola' added the comment: Assigning this to me. FWIW I created a post on python-dev in order to add a test.support.rmpath() utility function and hopefully cover other os.* functions which may also be untested or not properly tested: https://mail.python.org/pipermail/python-dev/2019-February/156267.html -- assignee: -> giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue35982> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35951] os.renames() creates directories if original name doesn't exist
Giampaolo Rodola' added the comment: The proposed change makes the problem a lot less likely to occur, but technically it doesn't fix it because if the src file/dir disappears between "os.path.exists(src)" and os.rename(src, dst)" calls you'll end up with a race condition. We may still want to do it, but can't make promises about full reliability. Also, it seems to me this behavior should be expected, because the doc explains the whole thing basically happens as a 3-step process (create dst dirs, rename, remove old src path). As such the cleanup in case of failure should not be expected, nor is promised. Also, FileNotFoundError is just one. os.rename() may fail for other reasons (and still leave the dst directory tree behind). If there is something we can do here is probably make the doc more clear (it talks about "lack of permissions" when instead it should have talked on "any error". Extra (unrelated): as I commented on the PR, there are no unit-tests for os.renames(). -- nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue35951> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35934] Add socket.bind_socket() utility function
Giampaolo Rodola' added the comment: Closing this out as duplicate. Will continue in issue35934. -- resolution: -> duplicate stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue35934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17561] Add socket.bind_socket() convenience function
Giampaolo Rodola' added the comment: After iterating over this over the last few days I realized it makes more sense to implement and discuss the whole thing in here and as a single PR, so sorry about previously splitting this in a separate ticket/PR. Relevant PR is now this one and is ready for review: https://github.com/python/cpython/pull/11784 Relevant changes which probably require attention/discussion are: 1) since it was sort of ambiguous I renamed "has_dual_stack()" to "supports_hybrid_ipv46()". There could be space for some bikeshed as we already have "socket.has_ipv6" so this may be "has_hybrid_ipv46()" called instead 2) if family is unspecified and determined from *host* (e.g. "localhost" or "") the function sorts getaddrinfo() results preferring AF_INET over AF_INET6 3) doc includes link to my http://code.activestate.com/recipes/578504 recipe for platforms not supporting hybrid_ipv46 natively 4) it may be worthwhile (or maybe not?) to have another complementary bind_sockets() (plural) function returning all items from getaddrinfo(). That would be useful for non-blocking apps/frameworks and could be reused by asyncio. Also, I'm CC-ing people from issue20215 as it contains relevant comments. -- keywords: -easy nosy: +Carlos.Ralli, Paul Marks, andreasr, berker.peksag, dazhaoyu, gregory.p.smith, jleedev, jpokorny, martin.panter, nirs, r.david.murray pull_requests: +11856 title: Add socket.create_server_sock() convenience function -> Add socket.bind_socket() convenience function ___ Python tracker <https://bugs.python.org/issue17561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35918] multiprocessing's SyncManager.dict.has_key() method is broken
Giampaolo Rodola' added the comment: New changeset 58f05ce059cc5207320fef27a9fbc0ffdc2b1d1a by Giampaolo Rodola (Miss Islington (bot)) in branch '3.7': bpo-35918: Remove broken has_key method and add test (GH-11819) (#11824) https://github.com/python/cpython/commit/58f05ce059cc5207320fef27a9fbc0ffdc2b1d1a -- ___ Python tracker <https://bugs.python.org/issue35918> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35918] multiprocessing's SyncManager.dict.has_key() method is broken
Giampaolo Rodola' added the comment: New changeset a31f4cc881992e84d351957bd9ac1a92f882fa39 by Giampaolo Rodola (Rémi Lapeyre) in branch 'master': bpo-35918: Remove broken has_key method and add test (#11819) https://github.com/python/cpython/commit/a31f4cc881992e84d351957bd9ac1a92f882fa39 -- ___ Python tracker <https://bugs.python.org/issue35918> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: I submitted an initial review / comments in the PR. I think this is good for a first iteration in order to understand what APIs to expose publicly (also, keep in mind I may not have a full picture of how this is intended to be used precisely). As for the doc, I find it a bit too verbose. If the namespace change is accepted consider doing the following: * document `SharedMemoryManager` right after `SyncManager`. That would let you reuse the description for `Semaphore`, `Barrier`, `Lock`, etc. * within `SharedMemoryManager` doc link/point to a different section of the doc where you explain the whole thing in more details, and also document the remaining APIs Just an idea. Writing good doc is not easy and it requires actually putting hands on it. Hope this helps. -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35940] multiprocessing manager tests fail in the Refleaks buildbots
Giampaolo Rodola' added the comment: Pablo thanks a lot for taking care of this. -- ___ Python tracker <https://bugs.python.org/issue35940> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17561] Add socket.create_server_sock() convenience function
Giampaolo Rodola' added the comment: After careful thinking I realize I'm not completely sure about how to expose the IPv4/6 functionality yet. I prefer to defer it for later and start landing a bind_socket() utility function which serves as a base for this functionality. See: issue17561. -- ___ Python tracker <https://bugs.python.org/issue17561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35934] Add socket.bind_socket() utility function
Change by Giampaolo Rodola' : -- keywords: +patch pull_requests: +11769 ___ Python tracker <https://bugs.python.org/issue35934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35934] Add socket.bind_socket() utility function
Change by Giampaolo Rodola' : -- keywords: +patch, patch, patch pull_requests: +11769, 11770, 11771 ___ Python tracker <https://bugs.python.org/issue35934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35934] Add socket.bind_socket() utility function
Change by Giampaolo Rodola' : -- keywords: +patch, patch pull_requests: +11769, 11770 ___ Python tracker <https://bugs.python.org/issue35934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35934] Add socket.bind_socket() utility function
New submission from Giampaolo Rodola' : The main point of this patch is to automatize all the necessary tasks which are usually involved when creating a server socket, amongst which: * determining the right family based on address, similarly to socket.create_connection() * whether to use SO_REUSEADDR depending on the platform * set AI_PASSIVE flag for getaddrinfo() * set a more optimal default backlog This is somewhat related to issue17561 which I prefer to leave pending for now (need to think about it more carefully). issue17561 is complementary to this one so it appears it can be integrated later (or never) without altering the base functionality implemented in here. -- components: Library (Lib) messages: 335034 nosy: asvetlov, cheryl.sabella, giampaolo.rodola, jaraco, josiah.carlson, loewis, neologix, pitrou priority: normal severity: normal stage: patch review status: open title: Add socket.bind_socket() utility function versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue35934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35913] asyncore: allow handling of half closed connections
Giampaolo Rodola' added the comment: I agree. The problem I have with this is that it introduces a new method (handle_eof), which ends up in the "new functionality" bucket (even though it's not backward incompatible per-se, as it defaults on calling handle_close() anyway, but still it is technically a new API). Point with asyncore/chat is that every time you try to fix them you end up messing with the public API one way or another. I know from experience (pyftpdlib) that all asyncore/chat users already subclass/overwrite the base classes quite massively, so if they really want to rely on this new functionality they probably already have implemented it themselves. -- ___ Python tracker <https://bugs.python.org/issue35913> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35913] asyncore: allow handling of half closed connections
Giampaolo Rodola' added the comment: > When recv() return 0 we may still have data to send. It seems recv() returning b"" is an alias for "connection lost". E.g. in Twisted: https://github.com/twisted/twisted/blob/06c891502be9f6389451fcc959cad5485f55d653/src/twisted/internet/tcp.py#L227-L248 -- ___ Python tracker <https://bugs.python.org/issue35913> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: Also, for completeness (since discussion is getting split), please see my proposal to move SharedMemoryManager and SharedMemoryServer into multiprocessing.managers namespace and rename shared_memory.py to _shared_memory.py: https://mail.python.org/pipermail/python-dev/2019-February/156235.html Also, it appears ShareableList should be register()ed against SharedMemoryManager. -- ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35919] multiprocessing: shared manager Pool fails with AttributeError
New submission from Giampaolo Rodola' : import multiprocessing import multiprocessing.managers def f(n): return n * n def worker(pool): with pool: pool.apply_async(f, (10, )) manager = multiprocessing.managers.SyncManager() manager.start() pool = manager.Pool(processes=4) proc = multiprocessing.Process(target=worker, args=(pool, )) proc.start() proc.join() This is related to BPO-35917 and it fails with: Process Process-2: Traceback (most recent call last): File "/home/giampaolo/cpython/Lib/multiprocessing/process.py", line 302, in _bootstrap self.run() File "/home/giampaolo/cpython/Lib/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "foo.py", line 54, in worker pool.apply_async(f, (10, )) File "", line 2, in apply_async File "/home/giampaolo/cpython/Lib/multiprocessing/managers.py", line 802, in _callmethod proxytype = self._manager._registry[token.typeid][-1] AttributeError: 'NoneType' object has no attribute '_registry' -- messages: 334962 nosy: davin, giampaolo.rodola, pitrou priority: normal severity: normal status: open title: multiprocessing: shared manager Pool fails with AttributeError ___ Python tracker <https://bugs.python.org/issue35919> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35913] asyncore: allow handling of half closed connections
Giampaolo Rodola' added the comment: Assigning this to me but am not sure 1) when I'll be able to look at this 2) whether it's worth it as asyncore is deprecated in favor of asyncio. -- assignee: -> giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue35913> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35918] multiprocessing's SyncManager.dict.has_key() method is broken
New submission from Giampaolo Rodola' : Related to BPO-35917: $ ./python Python 3.8.0a1+ (heads/master:cd90f6a369, Feb 6 2019, 17:16:10) [GCC 7.3.0] on linux >>> import multiprocessing.managers >>> m = multiprocessing.managers.SyncManager() >>> m.start() >>> d = m.dict() >>> 'has_key' in dir(d) True >>> d.has_key(1) Traceback (most recent call last): File "/home/giampaolo/cpython/Lib/multiprocessing/managers.py", line 271, in serve_client fallback_func = self.fallback_mapping[methodname] KeyError: 'has_key' -- messages: 334959 nosy: davin, giampaolo.rodola, jnoller, pitrou, sbt priority: normal severity: normal stage: needs patch status: open title: multiprocessing's SyncManager.dict.has_key() method is broken versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue35918> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35813] shared memory construct to avoid need for serialization between processes
Giampaolo Rodola' added the comment: Unit-tests at https://bugs.python.org/issue35917. -- nosy: +giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue35813> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35917] multiprocessing: provide unit-tests for manager classes and shareable types
Change by Giampaolo Rodola' : -- keywords: +patch, patch, patch pull_requests: +11740, 11741, 11742 ___ Python tracker <https://bugs.python.org/issue35917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35917] multiprocessing: provide unit-tests for manager classes and shareable types
Change by Giampaolo Rodola' : -- keywords: +patch, patch pull_requests: +11740, 11741 ___ Python tracker <https://bugs.python.org/issue35917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35917] multiprocessing: provide unit-tests for manager classes and shareable types
Change by Giampaolo Rodola' : -- keywords: +patch pull_requests: +11740 ___ Python tracker <https://bugs.python.org/issue35917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35917] multiprocessing: provide unit-tests for manager classes and shareable types
New submission from Giampaolo Rodola' : This is a follow up of BPO-35813 and PR-11664 and it provides unit tests for SyncManager and SharedMemoryManager classes + all the shareable types which are supposed to be supported by them. Also, see relevant python-dev discussion at: https://mail.python.org/pipermail/python-dev/2019-February/156235.html. In doing so I discovered a couple of issues which I will treat in a separate BPO ticket (multiprocessing.managers's dict.has_key() and Pool() appear to be broken). -- components: Tests messages: 334952 nosy: giampaolo.rodola priority: normal severity: normal stage: patch review status: open title: multiprocessing: provide unit-tests for manager classes and shareable types type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue35917> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35537] use os.posix_spawn in subprocess
Change by Giampaolo Rodola' : -- pull_requests: +11528 ___ Python tracker <https://bugs.python.org/issue35537> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20849] add exist_ok to shutil.copytree
Change by Giampaolo Rodola' : -- assignee: -> giampaolo.rodola resolution: -> fixed stage: patch review -> commit review status: open -> closed ___ Python tracker <https://bugs.python.org/issue20849> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20849] add exist_ok to shutil.copytree
Giampaolo Rodola' added the comment: New changeset 9e00d9e88fbf943987e4771c753f5ca8f794103e by Giampaolo Rodola (jab) in branch 'master': bpo-20849: add dirs_exist_ok arg to shutil.copytree (patch by Josh Bronson) https://github.com/python/cpython/commit/9e00d9e88fbf943987e4771c753f5ca8f794103e -- ___ Python tracker <https://bugs.python.org/issue20849> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17561] Add socket.create_server_sock() convenience function
Change by Giampaolo Rodola' : -- pull_requests: +10452 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue17561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35517] selector.EpollSelector: add new parameter to support EPOLLEXCLUSIVE
Giampaolo Rodola' added the comment: I took a look at your PR. As the PR currently stands it only works with epoll() selector. For the other selectors this is just an extra argument which does nothing, so it complicates the API of 2 methods for no real gain. Also, a single argument is not compatible with kqueue() because kqueue() requires two distinct `KQ_FILTER_*` and `KQ_EV_*` constants which cannot be ORed together. Instead, I think simply turning the underlying selector instance into a public property is more flexible and less complex. On Linux you'll do: >>> s = selectors.EpollSelector() >>> s.register(fd, EVENT_READ) >>> s.selector.modify(fd, select.EPOLLEXCLUSIVE) ``` ...and on BSD: >>> s = selectors.KqueueSelector() >>> s.register(fd, EVENT_READ) >>> k = s.selector.kevent(fd, select.KQ_FILTER_READ, select.KQ_EV_CLEAR) >>> s.selector.control([k], 0, 0) ``` Alternatively we could just keep `DefaultSelector._selector` private and document it. Personally I'd also be OK with that even though I'm not sure if there if there are precedents of documenting private APIs. -- nosy: +asvetlov, gvanrossum ___ Python tracker <https://bugs.python.org/issue35517> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35517] selector.EpollSelector: add new parameter to support EPOLLEXCLUSIVE
Giampaolo Rodola' added the comment: I see. Then I would say it's a matter of deciding what's the best API to provide. Another possibility is to promote the underlying epoll() instance as a public property, so one can do: >>> s = selectors.EpollSelector() >>> s.register(fd, EVENT_READ) >>> s.selector.modify(fd, select.EPOLLEXCLUSIVE) That raises the question whether all selector classes should have a public "selector" attribute. poll() and devpoll() related classes may need it for POLLPRI, POLLRDHUP, POLLWRBAND or others (whatever their use case is). kqueue() also has it's own specific constants (KQ_FILTER_* and KQ_EV_*). The only one where a public "selector" property would be useless is SelectSelector. -- stage: patch review -> ___ Python tracker <https://bugs.python.org/issue35517> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35517] selector.EpollSelector: add new parameter to support EPOLLEXCLUSIVE
Giampaolo Rodola' added the comment: I'm not sure I understand what EPOLLEXCLUSIVE is about. Could you provide a use case? Also, there are other constants which may also be useful such as EPOLLWAKEUP and EPOLLONESHOT: http://man7.org/linux/man-pages/man2/epoll_ctl.2.html So perhaps it makes more sense to expose a lower-level "extra_events" argument instead, which is more flexible and also saves us from the hassle of describing what the new argument is about (which really is a Linux kernel thing) in the doc: >>> s.register(fd, EVENT_READ, extra_events=select.EPOLLEXCLUSIVE | >>> select.EPOLLONESHOT) That raises the question whether we should also have an "extra_events" argument for modify() method (probably yes). But again, I think it makes sense to understand what's the use case of these constants first, because if they are rarely used maybe who needs them can simply look at the source and use s._selector.modify() directly. -- nosy: +neologix, yselivanov ___ Python tracker <https://bugs.python.org/issue35517> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17561] Add socket.create_server_sock() convenience function
Giampaolo Rodola' added the comment: Interesting. Yes, I agree this proposal is still desirable. We can reuse socket.create_server_sock() in smtpd, ftplib, socketserver (issue20215) and http.server (issue24209) modules which will inherit dual-stack IPv4/6 capabilities for free. I should be able to provide a PR sometime during this month. -- assignee: -> giampaolo.rodola ___ Python tracker <https://bugs.python.org/issue17561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35434] Wrong bpo linked in What's New in 3.8
Giampaolo Rodola' added the comment: My bad. Thanks for fixing it. -- ___ Python tracker <https://bugs.python.org/issue35434> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
