[issue39492] reference cycle affecting Pickler instances (Python3.8+)

Change by Pierre Glaser : -- keywords: +patch pull_requests: +17643 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18266 ___ Python tracker <https://bugs.python.org/issu

[issue39492] reference cycle affecting Pickler instances (Python3.8+)

New submission from Pierre Glaser : The new Pickler reducer_override mechanism introduced in `Python3.8` generates a reference cycle: for optimization purposes, a the pickler.reducer_override bound method is referenced into the reducer_override attribute of the Pickler's struct. Thus, until

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

Change by Pierre Glaser : -- keywords: +patch pull_requests: +15524 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15883 ___ Python tracker <https://bugs.python.org/issu

[issue38082] datetime.time object incorrectly shows associated date in strftime() output

Change by Pierre Glaser : -- pull_requests: +15523 pull_request: https://github.com/python/cpython/pull/15882 ___ Python tracker <https://bugs.python.org/issue38

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

Pierre Glaser added the comment: > Dropping this into Lib/multiprocessing/spawn.py should cause a repro: if WINSERVICE: _python_exe = os.path.join(sys.exec_prefix, 'python.exe') else: _python_exe = getattr(sys, '_base_executable', sys.executable) In this c

[issue38092] environment variables not passed correctly using new virtualenv launching in windows and python3.7+

New submission from Pierre Glaser : If I am not mistaken, when creating a new process on Python3.7 and later on Windows, if using a virtualenv, Python now uses a launcher. The launcher is being notified that it must create a virtual-environment Python (and not a system Python) program using

[issue28053] parameterize what serialization is used in multiprocessing

Change by Pierre Glaser : -- pull_requests: +14807 pull_request: https://github.com/python/cpython/pull/15058 ___ Python tracker <https://bugs.python.org/issue28

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Pierre Glaser added the comment: Sure, although I won't be able to merge it. Make sure you ping a core-dev such as pitrou or davin :-) -- ___ Python tracker <https://bugs.python.org/issue37

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Pierre Glaser added the comment: The root of the error is that struct.pack_into starts by memsetting the underlying memory area with NULL bytes before filling the data with memcpy. If ShareableList._get_packing_format is called between the two operations (through a concurrent __getitem__

[issue37209] Add what's new entries for pickle enhancements

Change by Pierre Glaser : -- keywords: +patch pull_requests: +14319 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/14503 ___ Python tracker <https://bugs.python.org/issu

[issue37244] test_multiprocessing_forkserver: test_resource_tracker() failed on x86 Gentoo Refleaks 3.8

Change by Pierre Glaser : -- keywords: +patch pull_requests: +14110 stage: -> patch review pull_request: https://github.com/python/cpython/pull/14288 ___ Python tracker <https://bugs.python.org/issu

[issue36888] Create a way to check that the parent process is alive for deamonized processes

Change by Pierre Glaser : -- pull_requests: +14108 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/14286 ___ Python tracker <https://bugs.python.org/issu

[issue37185] use os.memfd_create in multiprocessing.shared_memory?

New submission from Pierre Glaser : Hi, Following https://bugs.python.org/issue26836, I started thinking about using memfd_create instead of shm_open for creating shared-memory segments in multiprocessing.shared_memory. The main advantage of memfd_create over shm_open is that the generated

[issue26836] Add memfd_create to os module

Pierre Glaser added the comment: >From a quick skim at the man page of memfd_create, this looks promising. -- ___ Python tracker <https://bugs.python.org/issu

[issue36977] SharedMemoryManager should relase its resources when its parent process dies

Change by Pierre Glaser : -- keywords: +patch pull_requests: +13360 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36977> ___ ___ Py

[issue36977] SharedMemoryManager should relase its resources when its parent process dies

New submission from Pierre Glaser : The new multiprocessing.managers.SharedMemoryManager spawns a server that delivers memory segments to a parent Python process. If the parent process terminates unexpectedly, we should now make the manager process notice this termination it using the recent

[issue36950] test.support: add an helper to wait for an event with a timeout

Pierre Glaser added the comment: Just did so. -- ___ Python tracker <https://bugs.python.org/issue36950> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36950] test.support: add an helper to wait for an event with a timeout

Change by Pierre Glaser : -- keywords: +patch pull_requests: +13318 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36950> ___ ___ Py

[issue36950] test.support: add an helper to wait for an event with a timeout

Pierre Glaser added the comment: Lib/test/test_asyncio/utils.py defines a similar helper: def run_until(loop, pred, timeout=30): deadline = time.monotonic() + timeout while not pred(): if timeout is not None: timeout = deadline - time.monotonic

[issue36950] test.support: add an helper to wait for an event with a timeout

Change by Pierre Glaser : -- nosy: +pierreglaser ___ Python tracker <https://bugs.python.org/issue36950> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers

Change by Pierre Glaser : -- nosy: +pierreglaser ___ Python tracker <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Change by Pierre Glaser : -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue35933> ___ ___ Pyth

[issue36368] server process of shared_memory shuts down if KeyboardInterrupt

Change by Pierre Glaser : -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36368> ___ ___ Pyth

[issue36894] test_multiprocessing_spawn regression on Windows

Pierre Glaser added the comment: Thanks for the fix Antoine. -- ___ Python tracker <https://bugs.python.org/issue36894> ___ ___ Python-bugs-list mailin

[issue36867] Make semaphore_tracker track other system resources

Change by Pierre Glaser : -- pull_requests: +13203 ___ Python tracker <https://bugs.python.org/issue36867> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36867] Make semaphore_tracker track other system resources

Pierre Glaser added the comment: We can do that, or maybe we can try to wait on the `resource_tracker's` pid? -- ___ Python tracker <https://bugs.python.org/issue36

[issue36867] Make semaphore_tracker track other system resources

Change by Pierre Glaser : -- pull_requests: +13182 ___ Python tracker <https://bugs.python.org/issue36867> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36867] Make semaphore_tracker track other system resources

Pierre Glaser added the comment: Actually, I was properly unlinking the shared_memory segments. The warning messages are due to bad interactions between the ResourceTracker and the SharedMemoryManager object. In this particular case, it's easy to change a little bit the problematic test

[issue36867] Make semaphore_tracker track other system resources

Pierre Glaser added the comment: Yes, one test I wrote in an unrelated commit does not unlink a memory segment. Now the ResourceTracker complains. Fixing it now. -- ___ Python tracker <https://bugs.python.org/issue36

[issue36867] Make semaphore_tracker track other system resources

Pierre Glaser added the comment: Shared memory segments are now tracked by the brand new resource_tracker! Thanks Antoine for the review. Does anyone have an opinion on introducing a public API for users to make the resource_tracker track resources of their choice? What We have in mind

[issue36338] urlparse of urllib returns wrong hostname

Change by Pierre Glaser : -- pull_requests: +13146 ___ Python tracker <https://bugs.python.org/issue36338> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36867] Make semaphore_tracker track other system resources

Change by Pierre Glaser : -- keywords: +patch pull_requests: +13132 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36867> ___ ___ Py

[issue36867] Make semaphore_tracker track other system resources

New submission from Pierre Glaser : Hi all, Olivier Grisel, Thomas Moreau and myself are currently working on increasing the range of action of the semaphore_tracker in Python. multiprocessing.semaphore_tracker is a little known module, that launches a server process used to track the life

[issue35900] Add pickler hook for the user to customize the serialization of user defined functions and types.

Change by Pierre Glaser : -- pull_requests: +12530 ___ Python tracker <https://bugs.python.org/issue35900> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36338] urlparse of urllib returns wrong hostname

Change by Pierre Glaser : -- pull_requests: +12526 ___ Python tracker <https://bugs.python.org/issue36338> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36338] urlparse of urllib returns wrong hostname

Change by Pierre Glaser : -- pull_requests: +12525 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36338> ___ ___ Python-bugs-list mai

[issue35900] Add pickler hook for the user to customize the serialization of user defined functions and types.

Change by Pierre Glaser : -- pull_requests: +12449 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35900> ___ ___ Python-bugs-list mai

[issue36368] server process of shared_memory shuts down if KeyboardInterrupt

Pierre Glaser added the comment: Done. -- ___ Python tracker <https://bugs.python.org/issue36368> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36368] server process of shared_memory shuts down if KeyboardInterrupt

Change by Pierre Glaser : -- pull_requests: +12436 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36368> ___ ___ Python-bugs-list mai

[issue36338] urlparse of urllib returns wrong hostname

Change by Pierre Glaser : -- keywords: +patch pull_requests: +12435 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36338> ___ ___ Py

[issue36368] server process of shared_memory shuts down if KeyboardInterrupt

New submission from Pierre Glaser : When starting a SharedMemoryManager in an interactive session, any KeyboardInterrupt event will be transmitted to the (sub)process running the shared memory server, which causes the Manager to be unusable thereafter: >>> from multiprocessing

[issue36364] errors in multiprocessing.shared_memory examples

Change by Pierre Glaser : -- pull_requests: +12394 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36364> ___ ___ Python-bugs-list mai

[issue36364] errors in multiprocessing.shared_memory examples

New submission from Pierre Glaser : The examples of the new shared_memory module using SharedMemoryManager try to import the class from multiprocessing.shared_memory instead of multiprocessing.managers, making them fail. -- assignee: docs@python components: Documentation files: 0001

[issue35900] Add pickler hook for the user to customize the serialization of user defined functions and types.

Pierre Glaser added the comment: Update: Instead of changing permission on some attributes of function objects (__globals__ and __closure__), we added an optional argument called state_setter to save_reduce. This expects a callable that will be saved inside the object's pickle string

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Pierre Glaser added the comment: I added a PR with a small patch to document this behavior and reconcile _pickle.c and pickle.py Some explanations on why I am pushing this forward: Pickling instances of classes/subclasses with slots is done natively for pickle protocol >= 2. Mention

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Change by Pierre Glaser : -- keywords: +patch pull_requests: +11981 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35933> ___ ___ Py

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Change by Pierre Glaser : Added file: https://bugs.python.org/file48114/test_slots.py ___ Python tracker <https://bugs.python.org/issue35933> ___ ___ Python-bugs-list m

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Change by Pierre Glaser : Removed file: https://bugs.python.org/file48113/test_slots.py ___ Python tracker <https://bugs.python.org/issue35933> ___ ___ Python-bugs-list m

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Pierre Glaser added the comment: Thanks Antoine and Raymond for the feedback. Indeed, a subclass of a slotted class can have a dict: I enriched the script, pickling_depickling instances of such subclasses, with the length-2 tuple __getstate__ method, and made sure their attributes were

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Change by Pierre Glaser : Added file: https://bugs.python.org/file48113/test_slots.py ___ Python tracker <https://bugs.python.org/issue35933> ___ ___ Python-bugs-list m

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Change by Pierre Glaser : Removed file: https://bugs.python.org/file48112/test_slots.py ___ Python tracker <https://bugs.python.org/issue35933> ___ ___ Python-bugs-list m

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Change by Pierre Glaser : Removed file: https://bugs.python.org/file48111/test_slots.py ___ Python tracker <https://bugs.python.org/issue35933> ___ ___ Python-bugs-list m

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Change by Pierre Glaser : Added file: https://bugs.python.org/file48112/test_slots.py ___ Python tracker <https://bugs.python.org/issue35933> ___ ___ Python-bugs-list m

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

Pierre Glaser added the comment: It turns out that both pickle and _pickle implement this feature, but the behavior is inconsistent. - As a reminder, instances of slotted classes do not have a dict attribute (1) - On the other side, when pickling slotted class instances, __getstate__ can

[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)

New submission from Pierre Glaser : Hello all, This 16-year old commit (*) allows an object's state to be updated using its slots instead of its __dict__ at unpickling time. To use this functionality, the state keyword-argument of Pickler.save_reduce (which maps to the third item of the tuple

[issue35911] add a cell construtor, and expose the cell type in Lib/types.py

Change by Pierre Glaser : -- pull_requests: +11738 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35911> ___ ___ Python-bugs-list mai

[issue35911] add a cell construtor, and expose the cell type in Lib/types.py

Change by Pierre Glaser : -- pull_requests: +11738, 11739 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35911> ___ ___ Python-

[issue35911] add a cell construtor, and expose the cell type in Lib/types.py

Change by Pierre Glaser : Added file: https://bugs.python.org/file48106/test_cell.py ___ Python tracker <https://bugs.python.org/issue35911> ___ ___ Python-bugs-list m

[issue35911] add a cell construtor, and expose the cell type in Lib/types.py

New submission from Pierre Glaser : cell objects are containers for the free variables of functions defined in a local scope. They are located in a function's __closure__ attribute (when it is not None). A cell is a very simple object, with a single (optional, e.g the cell can be empty

[issue35900] Add pickler hoor for the user to customize the serialization of user defined functions and types.

Pierre Glaser added the comment: Pickler objects provide a dispatch_table attribute, where the user can specify custom saving functions depending on the object-to-be-saved type. However, for performance purposes, this table is predated (in the C implementation only) by a hardcoded switch

[issue35900] Add pickler hoor for the user to customize the serialization of user defined functions and types.

New submission from Pierre Glaser : Pickler objects provide a dispatch_table attribute, where the user can specify custom saving functions depending on the object-to-be-saved type. However, for performance purposes, this table is predated (in the C implementation only) by a hardcoded switch