[issue35657] multiprocessing.Process.join() ignores timeout if child process use os.exec*()

2019-01-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: I'm in favor of #1 *and* not documenting it either. I don't think it's reasonable for the documentation to enumerate all the kinds of situations where executing arbitrary code in a child process might lead to dysfunction. Realistically,

[issue35629] hang and/or leaked processes with multiprocessing.Pool(...).imap(...)

2019-01-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: Indeed, looks like a duplicate. -- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> multiprocessing.Pool.imaps iterators do not maintain alive the multiprocessing.

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

2019-01-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: I couldn't reproduce on Ubuntu either. I tried the "fork", "forkserver" and "spawn" methods (all with 3.7.2). Terry, if you are on Windows, can you try the script? Be sure to enclose the test() call in

[issue35608] python3 multiprocessing queue deadlock when use thread and process at same time

2018-12-29 Thread Antoine Pitrou
Antoine Pitrou added the comment: Your input_thread puts StopIteration once input the queue. But there are several worker processes popping from that queue, and only one of them will see the StopIteration. So I'm not surprised other worker processes would be stuck waiting in their

[issue35559] Optimize base64.b16decode to use compiled regex

2018-12-22 Thread Antoine Pitrou
Antoine Pitrou added the comment: I don't think performance of base16 encoding is important enough to increase import times. I would recommend rejection. -- ___ Python tracker <https://bugs.python.org/is

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

2018-12-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: By the way you could open an issue so that subprocess uses posix_spawn() where possible. (or you could ask to reopen issue31814, which is basically that request but for a different reason than yours

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

2018-12-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: At any rate, given the constraints you're working with (thousands of child processes, memory conservation issues), I suggest you abandon the idea of using multiprocessing and write your own subprocess-server instead. I would suggest doing so using as

[issue35238] Alleviate memory reservation of fork_exec in subprocess.Popen via forkserver

2018-12-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: I'm not sure I understand the proposed solution. Do you mean you would replace this: Parent -> forkserver -> fork child then exec with: Parent -> forkserver -> posix_spawn child? -- nosy: +pitrou _

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

2018-12-14 Thread Antoine Pitrou
Antoine Pitrou added the comment: > I have to investigate how Process.sentinel can be used here. Look how concurrent.futures uses it: https://github.com/python/cpython/blob/master/Lib/concurrent/futures/process.py#L348 This also means: 1) we could redirect people to ProcessPoolExecu

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

2018-12-14 Thread Antoine Pitrou
Antoine Pitrou added the comment: Using asyncio internally would be an interesting long-term goal, at least for the process pool version. Perhaps a first step is to find out how to await a multiprocessing Connection or Queue, or make async versions of these classes

[issue35493] multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit

2018-12-14 Thread Antoine Pitrou
Antoine Pitrou added the comment: How do you use SIGCHLD on Windows? There is actually a portable (and robust) solution: use Process.sentinel https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.sentinel There is another issue: Pool is currently subclassed by

[issue25430] speed up ipaddress __contain__ method

2018-12-12 Thread Antoine Pitrou
Change by Antoine Pitrou : -- versions: +Python 3.8 -Python 3.7 ___ Python tracker <https://bugs.python.org/issue25430> ___ ___ Python-bugs-list mailin

[issue35378] multiprocessing.Pool.imaps iterators do not maintain alive the multiprocessing.Pool objects

2018-12-06 Thread Antoine Pitrou
Antoine Pitrou added the comment: I agree about making the multiprocessing API safer to use, but please let's have this discussion on a dedicated bug entry. -- ___ Python tracker <https://bugs.python.org/is

[issue35378] multiprocessing.Pool.imaps iterators do not maintain alive the multiprocessing.Pool objects

2018-12-06 Thread Antoine Pitrou
Antoine Pitrou added the comment: I think a weakref is fine. You don't have to *pass* a weakref: just pass a normal ref and let the Result object take a weakref (and a reference to the cache perhaps). -- ___ Python tracker &

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-12-06 Thread Antoine Pitrou
Antoine Pitrou added the comment: I agree that reverting in bugfix branches was the right thing to do. I think the fix should have remained in master, though. -- stage: patch review -> versions: -Python 2.7, Python 3.6, Python 3.7 ___ Pyt

[issue35378] multiprocessing.Pool.imaps iterators do not maintain alive the multiprocessing.Pool objects

2018-12-06 Thread Antoine Pitrou
Antoine Pitrou added the comment: Let's step back a bit here. This kind of code has never been supported. As Victor says, we should be careful not to add any potential sources of reference cycles. The reason the code originally "worked" is that it actually leaked the Po

[issue28608] Support creating hardlink using `pathlib`

2018-11-30 Thread Antoine Pitrou
Antoine Pitrou added the comment: Since Path already supports symlinking, it would be reasonable to add hardlinking (e.g. as `Path.link_to`). But as you said one may as well call `os.link(path1, path2)`. -- ___ Python tracker <ht

[issue35283] "threading._DummyThread" redefines "is_alive" but forgets "isAlive"

2018-11-22 Thread Antoine Pitrou
Antoine Pitrou added the comment: If it's not already deprecated, I'd say deprecate it first. -- ___ Python tracker <https://bugs.python.org/issue35283> ___ __

[issue35242] multiprocessing.Queue in an inconsistent state and a traceback silently suppressed if put an unpickable object and process's target function is finished

2018-11-21 Thread Antoine Pitrou
Antoine Pitrou added the comment: Thanks for reporting this. This sounds like an extreme corner case and I'm not sure how easy it will be to fix. For the record, nowadays concurrent.futures should have more graceful behaviour in the face of errors. Though there probably always wi

[issue33725] Python crashes on macOS after fork with no exec

2018-11-14 Thread Antoine Pitrou
Antoine Pitrou added the comment: Legacy code is easy to migrate as long as it uses Python 3. Just call mp.set_start_method('forkserver') at the top of your code and you're done. Some use cases may fail (if sharing non-picklable types), but they're prob

[issue32485] Multiprocessing dict sharing between forked processes

2018-11-09 Thread Antoine Pitrou
Antoine Pitrou added the comment: Gus, dict don't have a has_key() method in Python 3. multiprocessing objects are generally not fork-safe: multiprocessing has its own mechanisms to duplicate objects between processes, but you cannot assume that after calling fork() yourself, objects

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

2018-11-06 Thread Antoine Pitrou
Antoine Pitrou added the comment: You should start from master. Bugfixes can backported afterwards if appropriate. Thanks! -- ___ Python tracker <https://bugs.python.org/issue22

[issue17560] problem using multiprocessing with really big objects?

2018-11-06 Thread Antoine Pitrou
Change by Antoine Pitrou : -- assignee: davin -> resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 3.8 -Python 3.7 ___ Python tracker <https://bugs.python

[issue17560] problem using multiprocessing with really big objects?

2018-11-06 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset bccacd19fa7b56dcf2fbfab15992b6b94abb by Antoine Pitrou (Alexander Buchkovsky) in branch 'master': bpo-17560: Too small type for struct.pack/unpack in mutliprocessing.Connection (GH-10305) https://github.com/python/cpyt

[issue35095] Implement pathlib.Path.append_bytes and pathlib.Path.append_text

2018-10-29 Thread Antoine Pitrou
Antoine Pitrou added the comment: First, it is quite uncommon to open a file in append mode. Second, when you open in append mode, often you will gradually append (such as for a logfile), so a single method call isn't effective. So I think this does not solve an important use

[issue33015] Fix function cast warning in thread_pthread.h

2018-10-25 Thread Antoine Pitrou
Antoine Pitrou added the comment: > Shall we introduce a new thread-starting API that takes a function with the > "correct" pthread signature? Do we need it? I don't think saving a single memory allocation is worth the bother. -- _

[issue33015] Fix function cast warning in thread_pthread.h

2018-10-23 Thread Antoine Pitrou
Antoine Pitrou added the comment: Right, so one PR is a real fix, the other PR is a workaround (avoids the warning without fixing the underlying problem). The underlying problem is: if a platform has incompatible ABIs for the two function types, casting one function type to another may

[issue34996] Add name to process and thread pool

2018-10-21 Thread Antoine Pitrou
Antoine Pitrou added the comment: Thanks for posting this. I think this is a good idea, will take a look at the PR later. -- ___ Python tracker <https://bugs.python.org/issue34

[issue34996] Add name to process and thread pool

2018-10-18 Thread Antoine Pitrou
Change by Antoine Pitrou : -- versions: +Python 3.8 -Python 2.7, Python 3.7 ___ Python tracker <https://bugs.python.org/issue34996> ___ ___ Python-bugs-list m

[issue34926] Allow querying a Path's mime-type

2018-10-12 Thread Antoine Pitrou
Antoine Pitrou added the comment: No, I simply forgot to close it. Thank you! -- ___ Python tracker <https://bugs.python.org/issue34926> ___ ___ Python-bug

[issue34926] Allow querying a Path's mime-type

2018-10-10 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset 7e18deef652a9d413d5dbd19d61073ba7eb5460e by Antoine Pitrou (Mayank Asthana) in branch 'master': bpo-34926: Make mimetypes.guess_type accept os.PathLike objects (GH-9777) https://github.com/python/cpyt

[issue34926] Allow querying a Path's mime-type

2018-10-09 Thread Antoine Pitrou
Change by Antoine Pitrou : -- title: Adding "mime_type" method to pathlib.Path -> Allow querying a Path's mime-type ___ Python tracker <https://bugs.

[issue34926] Adding "mime_type" method to pathlib.Path

2018-10-09 Thread Antoine Pitrou
Antoine Pitrou added the comment: I would rather have the mimetypes module improved to accept Path-like objects, so that the following works: >>> p = Path('LICENSE.txt') >>> mimetypes.guess_type(p) ('text/plain', None) It should be quite simple to imp

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-03 Thread Antoine Pitrou
Change by Antoine Pitrou : -- status: open -> closed ___ Python tracker <https://bugs.python.org/issue34172> ___ ___ Python-bugs-list mailing list Unsubscrib

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-03 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset 4a7dd30f5810e8861a3834159a222ab32d5c97d0 by Antoine Pitrou (tzickel) in branch '2.7': [2.7] bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-9686) https://github.com/python/cpyt

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-03 Thread Antoine Pitrou
Change by Antoine Pitrou : -- versions: +Python 2.7 ___ Python tracker <https://bugs.python.org/issue34172> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-03 Thread Antoine Pitrou
Change by Antoine Pitrou : -- status: closed -> open ___ Python tracker <https://bugs.python.org/issue34172> ___ ___ Python-bugs-list mailing list Unsubscrib

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-02 Thread Antoine Pitrou
Antoine Pitrou added the comment: (tzickel, sorry for mistyping your handle :-/) -- ___ Python tracker <https://bugs.python.org/issue34172> ___ ___ Python-bug

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-02 Thread Antoine Pitrou
Antoine Pitrou added the comment: Thanks tzickler for the report and pull request, and sorry for the delay. This is now fixed in all 3.x branches. I will close this now as multiprocessing in 2.7 diverges quite a bit from 3.x. If you want to fix the issue in 2.7 as well, please say so and

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-02 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset 07b96a95db78eff3557d1bfed1df9ebecc40815b by Antoine Pitrou (Miss Islington (bot)) in branch '3.6': bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) (GH-9677) https://github.com/python/cpyt

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-02 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset 97f998a4dfd6db6d867f446daa62445d0782bf39 by Antoine Pitrou (Miss Islington (bot)) in branch '3.7': bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) (GH-9676) https://github.com/python/cpyt

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-10-02 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset 97bfe8d3ebb0a54c8798f57555cb4152f9b2e1d0 by Antoine Pitrou (tzickel) in branch 'master': bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) https://github.com/python/cpyt

[issue34609] Importing certain modules while debugging raises an exception

2018-09-29 Thread Antoine Pitrou
Antoine Pitrou added the comment: I think pperry nailed it above: > Pdb fails because it is attempting to import the readline module every time > its `trace_dispatch` is called, and the import implementation is not > reentrant in that way. More precisely, _ModuleLock.acquire()

[issue34819] Executor.map and as_completed timeouts are able to deviate

2018-09-27 Thread Antoine Pitrou
Change by Antoine Pitrou : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.or

[issue34814] makesetup: must link C extensions to libpython when compiled in shared mode

2018-09-27 Thread Antoine Pitrou
Antoine Pitrou added the comment: But as you said, we need consistency: either *never* link to libpython, or *always* link to libpython. You are proposing to always link, I'm arguing we should never link. -- ___ Python tracker &

[issue34819] Executor.map and as_completed timeouts are able to deviate

2018-09-27 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset a94ee12c26aa8dd7dce01373779df8055aff765b by Antoine Pitrou (orlnub123) in branch 'master': bpo-34819: Use a monotonic clock to compute timeouts in concurrent.futures (GH-9599) https://github.com/python/cpyt

[issue34814] makesetup: must link C extensions to libpython when compiled in shared mode

2018-09-27 Thread Antoine Pitrou
Antoine Pitrou added the comment: > Is it a real use case? Why would anyone use a RHEL binary on Debian? Debian > already provides the full standard library. I'm not talking about the standard library obviously. I don't remember my original use case exactly, but I must hav

[issue34814] makesetup: must link C extensions to libpython when compiled in shared mode

2018-09-27 Thread Antoine Pitrou
Antoine Pitrou added the comment: Le 27/09/2018 à 12:49, STINNER Victor a écrit : > > I search if C extensions of the Python standard libraries are always linked > or not to libpython... it's complicated. I tested _ctypes, _hashlib and > _struct modules: > > * Debian

[issue34813] child process disappears when removing a print statement after its creation

2018-09-26 Thread Antoine Pitrou
Antoine Pitrou added the comment: Indeed! -- ___ Python tracker <https://bugs.python.org/issue34813> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34814] makesetup: must link C extensions to libpython when compiled in shared mode

2018-09-26 Thread Antoine Pitrou
Antoine Pitrou added the comment: Of course, one workaround to satisfy everyone would be to build a (empty) libpython.so even on static Python builds. But I'm not sure Debian/Ubuntu would package it. -- nosy: +doko ___ Python tracker &

[issue34814] makesetup: must link C extensions to libpython when compiled in shared mode

2018-09-26 Thread Antoine Pitrou
Antoine Pitrou added the comment: Why do you call this a bug? For me it's the reverse: it's linking to libpython.so which is a bug. It means a C extension compiled with a shared-library Python cannot be imported on a monolithic Python (which doesn't have libpython.so). It&#

[issue34813] child process disappears when removing a print statement after its creation

2018-09-26 Thread Antoine Pitrou
Antoine Pitrou added the comment: Those remarks apply to except-out.py, as well. -- ___ Python tracker <https://bugs.python.org/issue34813> ___ ___ Python-bug

[issue34813] child process disappears when removing a print statement after its creation

2018-09-26 Thread Antoine Pitrou
Antoine Pitrou added the comment: I don't think hang.py is correct. If you launch a process using subprocess, the subprocess owns the child process. You should not call waitpid() or os.kill() separately. Also, since you don't keep a reference to the subprocess.Popen o

[issue34781] infinite waiting in multiprocessing.Pool

2018-09-26 Thread Antoine Pitrou
Antoine Pitrou added the comment: @calimeroteknik, this doesn't seem to have anything to do with the issue at hand. Please open a separate issue with your scripts. -- ___ Python tracker <https://bugs.python.org/is

[issue34781] infinite waiting in multiprocessing.Pool

2018-09-26 Thread Antoine Pitrou
Change by Antoine Pitrou : Removed file: https://bugs.python.org/file47827/hang.py ___ Python tracker <https://bugs.python.org/issue34781> ___ ___ Python-bugs-list mailin

[issue34781] infinite waiting in multiprocessing.Pool

2018-09-26 Thread Antoine Pitrou
Change by Antoine Pitrou : Removed file: https://bugs.python.org/file47828/except-out.py ___ Python tracker <https://bugs.python.org/issue34781> ___ ___ Python-bug

[issue34781] infinite waiting in multiprocessing.Pool

2018-09-26 Thread Antoine Pitrou
Antoine Pitrou added the comment: If I understand correctly you have two cases: - the standalone script hangs with 3.6 on OS X - a much more involved use case hangs with 2.7 on RedHat It's possible you are hitting two different bugs. The 2.7 issue may be due to third-party packages or

[issue34781] infinite waiting in multiprocessing.Pool

2018-09-25 Thread Antoine Pitrou
Antoine Pitrou added the comment: I couldn't reproduce with Python 3.6.5 on Ubuntu 18.04. Does it happen if you reduce logging? Or if you replace f() with: def f(i): os.write(1, "{}\n".format(i).encode()) -- ___ Python

[issue34605] Avoid master/slave terminology

2018-09-10 Thread Antoine Pitrou
Antoine Pitrou added the comment: Indeed, "master" in itself is used in a wide range of contexts and can have quite positive connotations (I master Python programming). The word "slave" I agree with removing if used gratuitously. If its use reflects an established conven

[issue34614] Builtin `abs(Path)` should return `Path.absolute()`.

2018-09-09 Thread Antoine Pitrou
Change by Antoine Pitrou : -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.or

[issue34614] Builtin `abs(Path)` should return `Path.absolute()`.

2018-09-09 Thread Antoine Pitrou
Antoine Pitrou added the comment: Hmm, thanks for taking the time of proposing this and submitting a PR, but no. The documentation for `abs()` states: """ Return the absolute value of a number. The argument may be an integer or a floating point number. If the argument is a

[issue34614] Builtin `abs(Path)` should return `Path.absolute()`.

2018-09-09 Thread Antoine Pitrou
Change by Antoine Pitrou : -- title: Builtin `abs(Path)` returns `Path.absolute()`. -> Builtin `abs(Path)` should return `Path.absolute()`. ___ Python tracker <https://bugs.python.org/issu

[issue33613] test_multiprocessing_fork: test_semaphore_tracker_sigint() fails with -W error

2018-09-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: I don't want to spend hours arguing. This issue is sufficiently rare and unlikely in normal conditions that I don't think we should risk regressions by trying to fix it. -- ___ Python track

[issue33613] test_multiprocessing_fork: test_semaphore_tracker_sigint() fails with -W error

2018-09-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: Le 04/09/2018 à 11:16, STINNER Victor a écrit : > >> They work fine otherwise. > > The race condition impacts everyone. It's just less likely if your computer > is fast enough. The bug is much more likely if you "force" the

[issue33613] test_multiprocessing_fork: test_semaphore_tracker_sigint() fails with -W error

2018-09-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: Please only skip those tests on buildbots. They work fine otherwise. -- ___ Python tracker <https://bugs.python.org/issue33

[issue33613] test_multiprocessing_fork: test_semaphore_tracker_sigint() fails with -W error

2018-09-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: As I wrote on Github: no user-visible bug is fixed here, and we shouldn't risk introducing regressions by backporting those changes. If buildbots hurt, I suggest skipping the tests on the buildbots. -- versions: -Python 3.6, Pytho

[issue33613] test_multiprocessing_fork: test_semaphore_tracker_sigint() fails with -W error

2018-09-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset ec74d187f50a8a48f94eb37023300583fbd644cc by Antoine Pitrou (Pablo Galindo) in branch 'master': bpo-33613, test_semaphore_tracker_sigint: fix race condition (#7850) https://github.com/python/cpyt

[issue30563] [Cygwin] multiprocessing module with pool object issue

2018-08-13 Thread Antoine Pitrou
Antoine Pitrou added the comment: > In fact, the Python package should probably be removed from Cygwin? We don't maintain Cygwin nor Python on Cygwin here. You should contact those people. -- ___ Python tracker <https://bugs

[issue24076] sum() several times slower on Python 3 64-bit

2018-08-12 Thread Antoine Pitrou
Change by Antoine Pitrou : -- nosy: -pitrou ___ Python tracker <https://bugs.python.org/issue24076> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-08-04 Thread Antoine Pitrou
Antoine Pitrou added the comment: Thanks a lot tzickle, I'll take a look. -- ___ Python tracker <https://bugs.python.org/issue34172> ___ ___ Python-bugs-l

[issue1230540] sys.excepthook doesn't work in threads

2018-08-01 Thread Antoine Pitrou
Antoine Pitrou added the comment: No, I think this is a bug that deserves fixing, at least in 3.8. -- versions: +Python 3.8 -Python 2.7, Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <https://bugs.python.org/issue1230

[issue34204] Bump the default pickle protocol in shelve

2018-07-24 Thread Antoine Pitrou
Change by Antoine Pitrou : -- nosy: +rhettinger ___ Python tracker <https://bugs.python.org/issue34204> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-07-23 Thread Antoine Pitrou
Antoine Pitrou added the comment: > What other object in the standard lib, leaks resources when deleted in > CPython ? Threads come to mind, for example: >>> import time, threading, weakref >>> t = threading.Thread(target=time.sleep, args=(10,)) >>>

[issue34172] multiprocessing.Pool and ThreadPool leak resources after being deleted

2018-07-23 Thread Antoine Pitrou
Antoine Pitrou added the comment: Indeed, I think this simply needs a documentation fix. -- assignee: -> docs@python components: +Documentation nosy: +docs@python versions: +Python 3.6, Python 3.8 ___ Python tracker <https://bugs.pyth

[issue34128] Release GIL periodically in _pickle module

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: Interesting, which kind of computer / system / compiler are you on? -- ___ Python tracker <https://bugs.python.org/issue34

[issue34140] Possible multiprocessing deadlock when placing too many objects in Queue()

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: Closing as not a bug. -- resolution: -> not a bug stage: -> resolved status: open -> closed type: performance -> behavior ___ Python tracker <https://bugs.python

[issue34140] Possible multiprocessing deadlock when placing too many objects in Queue()

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: The problem is you're joining the child processes before draining the queue in the parent. Generally, instead of building your own kind of synchronization like this, I would recommend you use the higher-level abstractions provided by multiprocessing

[issue34128] Release GIL periodically in _pickle module

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: (as the demo script shows, there is no detectable slowdown) -- ___ Python tracker <https://bugs.python.org/issue34128> ___ ___

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: (see issue34128 for context) -- ___ Python tracker <https://bugs.python.org/issue25150> ___ ___ Python-bugs-list mailin

[issue34128] Release GIL periodically in _pickle module

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: Attaching demonstration script. -- Added file: https://bugs.python.org/file47701/pickle_gil.py ___ Python tracker <https://bugs.python.org/issue34

[issue34128] Release GIL periodically in _pickle module

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: Attaching proof-of-concept patch. -- keywords: +patch Added file: https://bugs.python.org/file47700/pickle_gil.patch ___ Python tracker <https://bugs.python.org/issue34

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: Case in point: if I want to include "internal/pystate.h" from _pickle.c, I get the following error: """ In file included from ./Include/internal/pystate.h:12:0, from /home/antoine/cpython/default/Modules/_pickle

[issue25150] 3.5: Include/pyatomic.h is incompatible with OpenMP (compilation of the third-party yt module fails on Python 3.5)

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: I think this should be reconsidered. I understand the desire to compile Python with OpenMP. But the resolution here is hiding _Py_atomic symbols all the time, even when OpenMP isn't involved, and even when building a standard extension m

[issue34128] Release GIL periodically in _pickle module

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: The right way to do this is not to pass a timeout parameter but to check for GIL interrupts as done in the main bytecode evaluation loop. -- ___ Python tracker <https://bugs.python.org/issue34

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: It will definitely break *some* user code. Also, everyone not using tkinter (which I think is the majority of users) isn't affected by this. -- ___ Python tracker <https://bugs.python.org/is

[issue34128] Do not block threads when pickle/unpickle

2018-07-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: This is about releasing the GIL periodically to allow other threads to run, as Python already does in its main interpreter loop. -- nosy: +pitrou versions: +Python 3.8 -Python 3.6 ___ Python tracker <ht

[issue32430] Simplify Modules/Setup{,.dist,.local}

2018-07-16 Thread Antoine Pitrou
Antoine Pitrou added the comment: With PR #8229 pushed the distinction between Setup.dist and Setup has disappeared: you have a Modules/Setup file in the source tree and that's where you do any changes, which you are responsible to track yourself (e.g. using a patch file or a git

[issue32430] Simplify Modules/Setup{,.dist,.local}

2018-07-16 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset 961d54c5c1916c09883ebcf7191babc969e5a5cf by Antoine Pitrou in branch 'master': bpo-32430: Rename Modules/Setup.dist to Modules/Setup (GH-8229) https://github.com/python/cpython/commit/961d54c5c1916c09883ebcf7191bab

[issue33111] Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)

2018-07-16 Thread Antoine Pitrou
Antoine Pitrou added the comment: macOS users, feel free to propose a doc PR for multiprocessing. -- assignee: -> docs@python components: +Documentation nosy: +docs@python versions: +Python 3.7, Python 3.8 -Python 3.5 ___ Python tracker <

[issue34101] PyBuffer_GetPointer() not documented

2018-07-12 Thread Antoine Pitrou
Change by Antoine Pitrou : -- assignee: docs@python components: Documentation nosy: docs@python, pitrou, skrah priority: normal severity: normal status: open title: PyBuffer_GetPointer() not documented type: behavior versions: Python 3.5, Python 3.6, Python 3.7

[issue32430] Simplify Modules/Setup{,.dist,.local}

2018-07-10 Thread Antoine Pitrou
Antoine Pitrou added the comment: I've created a PR that renames Setup.dist to Setup and removes extraneous logic. -- ___ Python tracker <https://bugs.python.org/is

[issue32430] Simplify Modules/Setup{,.dist,.local}

2018-07-10 Thread Antoine Pitrou
Change by Antoine Pitrou : -- versions: +Python 3.8 -Python 3.7 ___ Python tracker <https://bugs.python.org/issue32430> ___ ___ Python-bugs-list mailin

[issue32430] Simplify Modules/Setup{,.dist,.local}

2018-07-10 Thread Antoine Pitrou
Change by Antoine Pitrou : -- pull_requests: +7767 ___ Python tracker <https://bugs.python.org/issue32430> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34051] Update multiprocessing example

2018-07-07 Thread Antoine Pitrou
Antoine Pitrou added the comment: To be frank, I don't think that matters much. The user should understand what a lock is already, if they want to make use of multiprocessing fruitfully. The example showcases how to create a lock and how to pass it to child processes (by giving it

[issue34051] Update multiprocessing example

2018-07-07 Thread Antoine Pitrou
Antoine Pitrou added the comment: The example feels a bit artificial indeed, but I don't think adding a sleep() call would make it realistic. Why would you protect sleep() with a lock? -- nosy: +pitrou ___ Python tracker <https://bugs.py

[issue34034] Python 3.7.0 multiprocessing forkserver ForkingPickler behaviour change

2018-07-06 Thread Antoine Pitrou
Antoine Pitrou added the comment: No problem! I'm going to close this issue as I don't think there's anything we can do here. -- resolution: -> not a bug stage: -> resolved status: open -> closed ___ Python tracker

[issue34042] Reference loss for local classes

2018-07-05 Thread Antoine Pitrou
Antoine Pitrou added the comment: Agreed, if it was a real reference bug, the interpreter should crash before the total reference count gets negative ;-) -- ___ Python tracker <https://bugs.python.org/issue34

[issue34042] Reference loss for local classes

2018-07-05 Thread Antoine Pitrou
Antoine Pitrou added the comment: I've bisected myself and the culprit seems to be b0a7a037b8fde56b62f886d5188bced7776777b4 ("""bpo-31179: Make dict.copy() up to 5.5 times faster."""). -- nosy: +yselivanov ___

[issue24596] Script globals in a GC cycle not finalized when exiting with SystemExit

2018-07-03 Thread Antoine Pitrou
Change by Antoine Pitrou : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.or

[issue24596] Script globals in a GC cycle not finalized when exiting with SystemExit

2018-07-03 Thread Antoine Pitrou
Antoine Pitrou added the comment: New changeset e1ebf51f76037b7e02711aaeb9bf66c9147f4c74 by Antoine Pitrou (Miss Islington (bot)) in branch '3.6': bpo-24596: Decref module in PyRun_SimpleFileExFlags() on SystemExit (GH-7918) (GH-8069) https://github.com/python/cpyt

<    1   2   3   4   5   6   7   8   9   10   >