[Python-Dev] [RELEASE] Python 3.7.2rc1 and 3.6.8rc1 now available for testing

2018-12-11 Thread Ned Deily
https://blog.python.org/2018/12/python-372rc1-and-368rc1-now-available.html Python 3.7.2rc1 and 3.6.8rc1 are now available. 3.7.2rc1 is the release preview of the next maintenance release of Python 3.7, the latest feature release of Python. 3.6.8rc1 is the release preview of the next and last

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Antoine Pitrou
On Tue, 11 Dec 2018 11:48:24 -0800 Nathaniel Smith wrote: > > I know this question is rhetorical, but it does actually have a principled > answer. Memory is a special resource, because the GC has a complete picture > of memory use in the program, and if there's a danger of running out of >

Re: [Python-Dev] 3.7.2rc1 and 3.6.8rc1 cutoffs ahead, last 3.6.x bugfix release!

2018-12-11 Thread Ned Deily
https://discuss.python.org/t/3-7-2rc1-and-3-6-8rc1-cutoffs-ahead-last-3-6-x-bugfix-release/510/3?u=nad OK, thanks to a lot of hard work by many of you, I think we are ready to start the manufacturing of **3.7.2rc1** and **3.6.8rc1**. For the **3.7 branch**, as usual feel free to continue to

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Nathaniel Smith
On Tue, Dec 11, 2018, 07:13 Antoine Pitrou > What you are proposing here starts to smell like an anti-pattern to > me. Python _is_ a garbage-collected language, so by definition, there > _are_ going to be resources that are automatically collected when an > object disappears. If I'm allocating

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Pablo Galindo Salgado
> > Your original solution would have added a strong reference back to the > pool from the iterator. At first glance, that seems like a reasonable > solution to me. Victor is worried about the "risk of new reference > cycles". But reference cycles are not a problem - we have the cyclic > GC

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Paul Moore
On Tue, 11 Dec 2018 at 17:50, Pablo Galindo Salgado wrote: > I agree that misusage of the pool should not be encouraged but in this > situation the fact that > this code hangs: > > import multiprocessing > > for x in multiprocessing.Pool().imap(int, ["4", "3"]): > print(x) > > > is a bit

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Pablo Galindo Salgado
> > Pablo's issue35378 evolved to add a weak reference in iterators to try > > to detect when the Pool is destroyed: raise an exception from the > > iterator, if possible. > That's an ok fix for me. I am playing with weakreferences inside the iterator and result objects, but this may not be

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Antoine Pitrou
On Tue, 11 Dec 2018 17:22:49 +0100 Victor Stinner wrote: > Le mar. 11 déc. 2018 à 17:06, Antoine Pitrou a écrit : > > > We are not talking about simple strings, but processes and threads. > > > > Right, but do those have an impact on the program's correctness, or > > simply on its performance

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Victor Stinner
Le mar. 11 déc. 2018 à 17:06, Antoine Pitrou a écrit : > > We are not talking about simple strings, but processes and threads. > > Right, but do those have an impact on the program's correctness, or > simply on its performance (or memory consumption)? Performance. I made a similar change in the

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Antoine Pitrou
On Tue, 11 Dec 2018 16:33:54 +0100 Victor Stinner wrote: > Le mar. 11 déc. 2018 à 16:14, Antoine Pitrou a écrit : > > What you are proposing here starts to smell like an anti-pattern to > > me. Python _is_ a garbage-collected language, so by definition, there > > _are_ going to be resources

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Paul Moore
On Tue, 11 Dec 2018 at 15:13, Antoine Pitrou wrote: > On Tue, 11 Dec 2018 15:21:31 +0100 > Victor Stinner wrote: > > > > Pablo's issue35378 evolved to add a weak reference in iterators to try > > to detect when the Pool is destroyed: raise an exception from the > > iterator, if possible. > >

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Victor Stinner
Le mar. 11 déc. 2018 à 16:14, Antoine Pitrou a écrit : > What you are proposing here starts to smell like an anti-pattern to > me. Python _is_ a garbage-collected language, so by definition, there > _are_ going to be resources that are automatically collected when an > object disappears. If I'm

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Antoine Pitrou
Hi, On Tue, 11 Dec 2018 15:21:31 +0100 Victor Stinner wrote: > > Pablo's issue35378 evolved to add a weak reference in iterators to try > to detect when the Pool is destroyed: raise an exception from the > iterator, if possible. That's an ok fix for me. > By the way, I'm surprised that

[Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Victor Stinner
Hi, tzickel reported a reference cycle bug in multiprocessing which keeps threads and processes alive: https://bugs.python.org/issue34172 He wrote a fix which has been merged in 3.6, 3.7 and master branches. But Pablo Galindo noticed that the fix breaks the following code (he added "I found