Antoine Pitrou added the comment:
I'm in favor of #1 *and* not documenting it either. I don't think it's
reasonable for the documentation to enumerate all the kinds of situations where
executing arbitrary code in a child process might lead to dysfunction.
Realistically,
Antoine Pitrou added the comment:
Indeed, looks like a duplicate.
--
resolution: -> duplicate
stage: -> resolved
status: open -> closed
superseder: -> multiprocessing.Pool.imaps iterators do not maintain alive the
multiprocessing.
Antoine Pitrou added the comment:
I couldn't reproduce on Ubuntu either. I tried the "fork", "forkserver" and
"spawn" methods (all with 3.7.2).
Terry, if you are on Windows, can you try the script? Be sure to enclose the
test() call in
Antoine Pitrou added the comment:
Your input_thread puts StopIteration once input the queue. But there are
several worker processes popping from that queue, and only one of them will see
the StopIteration. So I'm not surprised other worker processes would be stuck
waiting in their
Antoine Pitrou added the comment:
I don't think performance of base16 encoding is important enough to increase
import times. I would recommend rejection.
--
___
Python tracker
<https://bugs.python.org/is
Antoine Pitrou added the comment:
By the way you could open an issue so that subprocess uses posix_spawn() where
possible.
(or you could ask to reopen issue31814, which is basically that request but for
a different reason than yours
Antoine Pitrou added the comment:
At any rate, given the constraints you're working with (thousands of child
processes, memory conservation issues), I suggest you abandon the idea of using
multiprocessing and write your own subprocess-server instead.
I would suggest doing so using as
Antoine Pitrou added the comment:
I'm not sure I understand the proposed solution. Do you mean you would replace
this:
Parent -> forkserver -> fork child then exec
with:
Parent -> forkserver -> posix_spawn child?
--
nosy: +pitrou
_
Antoine Pitrou added the comment:
> I have to investigate how Process.sentinel can be used here.
Look how concurrent.futures uses it:
https://github.com/python/cpython/blob/master/Lib/concurrent/futures/process.py#L348
This also means:
1) we could redirect people to ProcessPoolExecu
Antoine Pitrou added the comment:
Using asyncio internally would be an interesting long-term goal, at least for
the process pool version.
Perhaps a first step is to find out how to await a multiprocessing Connection
or Queue, or make async versions of these classes
Antoine Pitrou added the comment:
How do you use SIGCHLD on Windows?
There is actually a portable (and robust) solution: use Process.sentinel
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.sentinel
There is another issue: Pool is currently subclassed by
Change by Antoine Pitrou :
--
versions: +Python 3.8 -Python 3.7
___
Python tracker
<https://bugs.python.org/issue25430>
___
___
Python-bugs-list mailin
Antoine Pitrou added the comment:
I agree about making the multiprocessing API safer to use, but please let's
have this discussion on a dedicated bug entry.
--
___
Python tracker
<https://bugs.python.org/is
Antoine Pitrou added the comment:
I think a weakref is fine. You don't have to *pass* a weakref: just pass a
normal ref and let the Result object take a weakref (and a reference to the
cache perhaps).
--
___
Python tracker
&
Antoine Pitrou added the comment:
I agree that reverting in bugfix branches was the right thing to do. I think
the fix should have remained in master, though.
--
stage: patch review ->
versions: -Python 2.7, Python 3.6, Python 3.7
___
Pyt
Antoine Pitrou added the comment:
Let's step back a bit here. This kind of code has never been supported. As
Victor says, we should be careful not to add any potential sources of reference
cycles.
The reason the code originally "worked" is that it actually leaked the Po
Antoine Pitrou added the comment:
Since Path already supports symlinking, it would be reasonable to add
hardlinking (e.g. as `Path.link_to`). But as you said one may as well call
`os.link(path1, path2)`.
--
___
Python tracker
<ht
Antoine Pitrou added the comment:
If it's not already deprecated, I'd say deprecate it first.
--
___
Python tracker
<https://bugs.python.org/issue35283>
___
__
Antoine Pitrou added the comment:
Thanks for reporting this. This sounds like an extreme corner case and I'm not
sure how easy it will be to fix.
For the record, nowadays concurrent.futures should have more graceful behaviour
in the face of errors. Though there probably always wi
Antoine Pitrou added the comment:
Legacy code is easy to migrate as long as it uses Python 3. Just call
mp.set_start_method('forkserver')
at the top of your code and you're done. Some use cases may fail (if sharing
non-picklable types), but they're prob
Antoine Pitrou added the comment:
Gus, dict don't have a has_key() method in Python 3.
multiprocessing objects are generally not fork-safe: multiprocessing has its
own mechanisms to duplicate objects between processes, but you cannot assume
that after calling fork() yourself, objects
Antoine Pitrou added the comment:
You should start from master. Bugfixes can backported afterwards if
appropriate. Thanks!
--
___
Python tracker
<https://bugs.python.org/issue22
Change by Antoine Pitrou :
--
assignee: davin ->
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.8 -Python 3.7
___
Python tracker
<https://bugs.python
Antoine Pitrou added the comment:
New changeset bccacd19fa7b56dcf2fbfab15992b6b94abb by Antoine Pitrou
(Alexander Buchkovsky) in branch 'master':
bpo-17560: Too small type for struct.pack/unpack in mutliprocessing.Connection
(GH-10305)
https://github.com/python/cpyt
Antoine Pitrou added the comment:
First, it is quite uncommon to open a file in append mode. Second, when you
open in append mode, often you will gradually append (such as for a logfile),
so a single method call isn't effective.
So I think this does not solve an important use
Antoine Pitrou added the comment:
> Shall we introduce a new thread-starting API that takes a function with the
> "correct" pthread signature?
Do we need it? I don't think saving a single memory allocation is worth the
bother.
--
_
Antoine Pitrou added the comment:
Right, so one PR is a real fix, the other PR is a workaround (avoids the
warning without fixing the underlying problem).
The underlying problem is: if a platform has incompatible ABIs for the two
function types, casting one function type to another may
Antoine Pitrou added the comment:
Thanks for posting this. I think this is a good idea, will take a look at the
PR later.
--
___
Python tracker
<https://bugs.python.org/issue34
Change by Antoine Pitrou :
--
versions: +Python 3.8 -Python 2.7, Python 3.7
___
Python tracker
<https://bugs.python.org/issue34996>
___
___
Python-bugs-list m
Antoine Pitrou added the comment:
No, I simply forgot to close it. Thank you!
--
___
Python tracker
<https://bugs.python.org/issue34926>
___
___
Python-bug
Antoine Pitrou added the comment:
New changeset 7e18deef652a9d413d5dbd19d61073ba7eb5460e by Antoine Pitrou
(Mayank Asthana) in branch 'master':
bpo-34926: Make mimetypes.guess_type accept os.PathLike objects (GH-9777)
https://github.com/python/cpyt
Change by Antoine Pitrou :
--
title: Adding "mime_type" method to pathlib.Path -> Allow querying a Path's
mime-type
___
Python tracker
<https://bugs.
Antoine Pitrou added the comment:
I would rather have the mimetypes module improved to accept Path-like objects,
so that the following works:
>>> p = Path('LICENSE.txt')
>>> mimetypes.guess_type(p)
('text/plain', None)
It should be quite simple to imp
Change by Antoine Pitrou :
--
status: open -> closed
___
Python tracker
<https://bugs.python.org/issue34172>
___
___
Python-bugs-list mailing list
Unsubscrib
Antoine Pitrou added the comment:
New changeset 4a7dd30f5810e8861a3834159a222ab32d5c97d0 by Antoine Pitrou
(tzickel) in branch '2.7':
[2.7] bpo-34172: multiprocessing.Pool leaks resources after being deleted
(GH-9686)
https://github.com/python/cpyt
Change by Antoine Pitrou :
--
versions: +Python 2.7
___
Python tracker
<https://bugs.python.org/issue34172>
___
___
Python-bugs-list mailing list
Unsubscribe:
Change by Antoine Pitrou :
--
status: closed -> open
___
Python tracker
<https://bugs.python.org/issue34172>
___
___
Python-bugs-list mailing list
Unsubscrib
Antoine Pitrou added the comment:
(tzickel, sorry for mistyping your handle :-/)
--
___
Python tracker
<https://bugs.python.org/issue34172>
___
___
Python-bug
Antoine Pitrou added the comment:
Thanks tzickler for the report and pull request, and sorry for the delay.
This is now fixed in all 3.x branches. I will close this now as
multiprocessing in 2.7 diverges quite a bit from 3.x. If you want to fix the
issue in 2.7 as well, please say so and
Antoine Pitrou added the comment:
New changeset 07b96a95db78eff3557d1bfed1df9ebecc40815b by Antoine Pitrou (Miss
Islington (bot)) in branch '3.6':
bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450)
(GH-9677)
https://github.com/python/cpyt
Antoine Pitrou added the comment:
New changeset 97f998a4dfd6db6d867f446daa62445d0782bf39 by Antoine Pitrou (Miss
Islington (bot)) in branch '3.7':
bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450)
(GH-9676)
https://github.com/python/cpyt
Antoine Pitrou added the comment:
New changeset 97bfe8d3ebb0a54c8798f57555cb4152f9b2e1d0 by Antoine Pitrou
(tzickel) in branch 'master':
bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450)
https://github.com/python/cpyt
Antoine Pitrou added the comment:
I think pperry nailed it above:
> Pdb fails because it is attempting to import the readline module every time
> its `trace_dispatch` is called, and the import implementation is not
> reentrant in that way.
More precisely, _ModuleLock.acquire()
Change by Antoine Pitrou :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Antoine Pitrou added the comment:
But as you said, we need consistency: either *never* link to libpython, or
*always* link to libpython. You are proposing to always link, I'm arguing we
should never link.
--
___
Python tracker
&
Antoine Pitrou added the comment:
New changeset a94ee12c26aa8dd7dce01373779df8055aff765b by Antoine Pitrou
(orlnub123) in branch 'master':
bpo-34819: Use a monotonic clock to compute timeouts in concurrent.futures
(GH-9599)
https://github.com/python/cpyt
Antoine Pitrou added the comment:
> Is it a real use case? Why would anyone use a RHEL binary on Debian? Debian
> already provides the full standard library.
I'm not talking about the standard library obviously. I don't remember my
original use case exactly, but I must hav
Antoine Pitrou added the comment:
Le 27/09/2018 à 12:49, STINNER Victor a écrit :
>
> I search if C extensions of the Python standard libraries are always linked
> or not to libpython... it's complicated. I tested _ctypes, _hashlib and
> _struct modules:
>
> * Debian
Antoine Pitrou added the comment:
Indeed!
--
___
Python tracker
<https://bugs.python.org/issue34813>
___
___
Python-bugs-list mailing list
Unsubscribe:
Antoine Pitrou added the comment:
Of course, one workaround to satisfy everyone would be to build a (empty)
libpython.so even on static Python builds. But I'm not sure Debian/Ubuntu
would package it.
--
nosy: +doko
___
Python tracker
&
Antoine Pitrou added the comment:
Why do you call this a bug?
For me it's the reverse: it's linking to libpython.so which is a bug. It means
a C extension compiled with a shared-library Python cannot be imported on a
monolithic Python (which doesn't have libpython.so). It
Antoine Pitrou added the comment:
Those remarks apply to except-out.py, as well.
--
___
Python tracker
<https://bugs.python.org/issue34813>
___
___
Python-bug
Antoine Pitrou added the comment:
I don't think hang.py is correct. If you launch a process using subprocess,
the subprocess owns the child process. You should not call waitpid() or
os.kill() separately. Also, since you don't keep a reference to the
subprocess.Popen o
Antoine Pitrou added the comment:
@calimeroteknik, this doesn't seem to have anything to do with the issue at
hand. Please open a separate issue with your scripts.
--
___
Python tracker
<https://bugs.python.org/is
Change by Antoine Pitrou :
Removed file: https://bugs.python.org/file47827/hang.py
___
Python tracker
<https://bugs.python.org/issue34781>
___
___
Python-bugs-list mailin
Change by Antoine Pitrou :
Removed file: https://bugs.python.org/file47828/except-out.py
___
Python tracker
<https://bugs.python.org/issue34781>
___
___
Python-bug
Antoine Pitrou added the comment:
If I understand correctly you have two cases:
- the standalone script hangs with 3.6 on OS X
- a much more involved use case hangs with 2.7 on RedHat
It's possible you are hitting two different bugs. The 2.7 issue may be due to
third-party packages or
Antoine Pitrou added the comment:
I couldn't reproduce with Python 3.6.5 on Ubuntu 18.04.
Does it happen if you reduce logging? Or if you replace f() with:
def f(i):
os.write(1, "{}\n".format(i).encode())
--
___
Python
Antoine Pitrou added the comment:
Indeed, "master" in itself is used in a wide range of contexts and can have
quite positive connotations (I master Python programming).
The word "slave" I agree with removing if used gratuitously. If its use
reflects an established conven
Change by Antoine Pitrou :
--
resolution: -> rejected
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Antoine Pitrou added the comment:
Hmm, thanks for taking the time of proposing this and submitting a PR, but no.
The documentation for `abs()` states:
"""
Return the absolute value of a number. The argument may be an integer or a
floating point number. If the argument is a
Change by Antoine Pitrou :
--
title: Builtin `abs(Path)` returns `Path.absolute()`. -> Builtin `abs(Path)`
should return `Path.absolute()`.
___
Python tracker
<https://bugs.python.org/issu
Antoine Pitrou added the comment:
I don't want to spend hours arguing. This issue is sufficiently rare and
unlikely in normal conditions that I don't think we should risk regressions by
trying to fix it.
--
___
Python track
Antoine Pitrou added the comment:
Le 04/09/2018 à 11:16, STINNER Victor a écrit :
>
>> They work fine otherwise.
>
> The race condition impacts everyone. It's just less likely if your computer
> is fast enough. The bug is much more likely if you "force" the
Antoine Pitrou added the comment:
Please only skip those tests on buildbots. They work fine otherwise.
--
___
Python tracker
<https://bugs.python.org/issue33
Antoine Pitrou added the comment:
As I wrote on Github: no user-visible bug is fixed here, and we shouldn't risk
introducing regressions by backporting those changes.
If buildbots hurt, I suggest skipping the tests on the buildbots.
--
versions: -Python 3.6, Pytho
Antoine Pitrou added the comment:
New changeset ec74d187f50a8a48f94eb37023300583fbd644cc by Antoine Pitrou (Pablo
Galindo) in branch 'master':
bpo-33613, test_semaphore_tracker_sigint: fix race condition (#7850)
https://github.com/python/cpyt
Antoine Pitrou added the comment:
> In fact, the Python package should probably be removed from Cygwin?
We don't maintain Cygwin nor Python on Cygwin here. You should contact those
people.
--
___
Python tracker
<https://bugs
Change by Antoine Pitrou :
--
nosy: -pitrou
___
Python tracker
<https://bugs.python.org/issue24076>
___
___
Python-bugs-list mailing list
Unsubscribe:
Antoine Pitrou added the comment:
Thanks a lot tzickle, I'll take a look.
--
___
Python tracker
<https://bugs.python.org/issue34172>
___
___
Python-bugs-l
Antoine Pitrou added the comment:
No, I think this is a bug that deserves fixing, at least in 3.8.
--
versions: +Python 3.8 -Python 2.7, Python 3.5, Python 3.6, Python 3.7
___
Python tracker
<https://bugs.python.org/issue1230
Change by Antoine Pitrou :
--
nosy: +rhettinger
___
Python tracker
<https://bugs.python.org/issue34204>
___
___
Python-bugs-list mailing list
Unsubscribe:
Antoine Pitrou added the comment:
> What other object in the standard lib, leaks resources when deleted in
> CPython ?
Threads come to mind, for example:
>>> import time, threading, weakref
>>> t = threading.Thread(target=time.sleep, args=(10,))
>>>
Antoine Pitrou added the comment:
Indeed, I think this simply needs a documentation fix.
--
assignee: -> docs@python
components: +Documentation
nosy: +docs@python
versions: +Python 3.6, Python 3.8
___
Python tracker
<https://bugs.pyth
Antoine Pitrou added the comment:
Interesting, which kind of computer / system / compiler are you on?
--
___
Python tracker
<https://bugs.python.org/issue34
Antoine Pitrou added the comment:
Closing as not a bug.
--
resolution: -> not a bug
stage: -> resolved
status: open -> closed
type: performance -> behavior
___
Python tracker
<https://bugs.python
Antoine Pitrou added the comment:
The problem is you're joining the child processes before draining the queue in
the parent.
Generally, instead of building your own kind of synchronization like this, I
would recommend you use the higher-level abstractions provided by
multiprocessing
Antoine Pitrou added the comment:
(as the demo script shows, there is no detectable slowdown)
--
___
Python tracker
<https://bugs.python.org/issue34128>
___
___
Antoine Pitrou added the comment:
(see issue34128 for context)
--
___
Python tracker
<https://bugs.python.org/issue25150>
___
___
Python-bugs-list mailin
Antoine Pitrou added the comment:
Attaching demonstration script.
--
Added file: https://bugs.python.org/file47701/pickle_gil.py
___
Python tracker
<https://bugs.python.org/issue34
Antoine Pitrou added the comment:
Attaching proof-of-concept patch.
--
keywords: +patch
Added file: https://bugs.python.org/file47700/pickle_gil.patch
___
Python tracker
<https://bugs.python.org/issue34
Antoine Pitrou added the comment:
Case in point: if I want to include "internal/pystate.h" from _pickle.c, I get
the following error:
"""
In file included from ./Include/internal/pystate.h:12:0,
from /home/antoine/cpython/default/Modules/_pickle
Antoine Pitrou added the comment:
I think this should be reconsidered. I understand the desire to compile Python
with OpenMP. But the resolution here is hiding _Py_atomic symbols all the
time, even when OpenMP isn't involved, and even when building a standard
extension m
Antoine Pitrou added the comment:
The right way to do this is not to pass a timeout parameter but to check for
GIL interrupts as done in the main bytecode evaluation loop.
--
___
Python tracker
<https://bugs.python.org/issue34
Antoine Pitrou added the comment:
It will definitely break *some* user code. Also, everyone not using tkinter
(which I think is the majority of users) isn't affected by this.
--
___
Python tracker
<https://bugs.python.org/is
Antoine Pitrou added the comment:
This is about releasing the GIL periodically to allow other threads to run, as
Python already does in its main interpreter loop.
--
nosy: +pitrou
versions: +Python 3.8 -Python 3.6
___
Python tracker
<ht
Antoine Pitrou added the comment:
With PR #8229 pushed the distinction between Setup.dist and Setup has
disappeared: you have a Modules/Setup file in the source tree and that's where
you do any changes, which you are responsible to track yourself (e.g. using a
patch file or a git
Antoine Pitrou added the comment:
New changeset 961d54c5c1916c09883ebcf7191babc969e5a5cf by Antoine Pitrou in
branch 'master':
bpo-32430: Rename Modules/Setup.dist to Modules/Setup (GH-8229)
https://github.com/python/cpython/commit/961d54c5c1916c09883ebcf7191bab
Antoine Pitrou added the comment:
macOS users, feel free to propose a doc PR for multiprocessing.
--
assignee: -> docs@python
components: +Documentation
nosy: +docs@python
versions: +Python 3.7, Python 3.8 -Python 3.5
___
Python tracker
<
Change by Antoine Pitrou :
--
assignee: docs@python
components: Documentation
nosy: docs@python, pitrou, skrah
priority: normal
severity: normal
status: open
title: PyBuffer_GetPointer() not documented
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7
Antoine Pitrou added the comment:
I've created a PR that renames Setup.dist to Setup and removes extraneous logic.
--
___
Python tracker
<https://bugs.python.org/is
Change by Antoine Pitrou :
--
versions: +Python 3.8 -Python 3.7
___
Python tracker
<https://bugs.python.org/issue32430>
___
___
Python-bugs-list mailin
Change by Antoine Pitrou :
--
pull_requests: +7767
___
Python tracker
<https://bugs.python.org/issue32430>
___
___
Python-bugs-list mailing list
Unsubscribe:
Antoine Pitrou added the comment:
To be frank, I don't think that matters much. The user should understand what
a lock is already, if they want to make use of multiprocessing fruitfully. The
example showcases how to create a lock and how to pass it to child processes
(by giving it
Antoine Pitrou added the comment:
The example feels a bit artificial indeed, but I don't think adding a sleep()
call would make it realistic. Why would you protect sleep() with a lock?
--
nosy: +pitrou
___
Python tracker
<https://bugs.py
Antoine Pitrou added the comment:
No problem! I'm going to close this issue as I don't think there's anything we
can do here.
--
resolution: -> not a bug
stage: -> resolved
status: open -> closed
___
Python tracker
Antoine Pitrou added the comment:
Agreed, if it was a real reference bug, the interpreter should crash before the
total reference count gets negative ;-)
--
___
Python tracker
<https://bugs.python.org/issue34
Antoine Pitrou added the comment:
I've bisected myself and the culprit seems to be
b0a7a037b8fde56b62f886d5188bced7776777b4 ("""bpo-31179: Make dict.copy() up to
5.5 times faster.""").
--
nosy: +yselivanov
___
Change by Antoine Pitrou :
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<https://bugs.python.or
Antoine Pitrou added the comment:
New changeset e1ebf51f76037b7e02711aaeb9bf66c9147f4c74 by Antoine Pitrou (Miss
Islington (bot)) in branch '3.6':
bpo-24596: Decref module in PyRun_SimpleFileExFlags() on SystemExit (GH-7918)
(GH-8069)
https://github.com/python/cpyt
501 - 600 of 16792 matches
Mail list logo