Re: [Python-ideas] Deterministic iterator cleanup

On Tuesday, October 25, 2016 at 6:26:17 PM UTC-4, Nathaniel Smith wrote: > > On Sat, Oct 22, 2016 at 9:02 AM, Nick Coghlan > wrote: > > On 20 October 2016 at 07:02, Nathaniel Smith > wrote: > >> The first change is to replace the outer for loop with a

Re: [Python-ideas] Deterministic iterator cleanup

On 26 October 2016 at 08:48, Nathaniel Smith wrote: > If it takes a strong reference, then suddenly we're pinning all > iterators in memory until the end of the enclosing function, which > will often look like a memory leak. I think this would break a *lot* > more existing code

Re: [Python-ideas] Deterministic iterator cleanup

On 26 October 2016 at 08:25, Nathaniel Smith wrote: > On Sat, Oct 22, 2016 at 9:02 AM, Nick Coghlan wrote: >> At this point your code is starting to look a whole lot like the code >> in contextlib.ExitStack.__exit__ :) > > One of the versions I tried but

Re: [Python-ideas] Deterministic iterator cleanup

...Doh. I spent all that time evaluating the function-scoped-cleanup proposal from the high-level design perspective, and then immediately after hitting send, I suddenly realized that I'd missed a much more straightforward technical problem. One thing that 'with' blocks / for-scoped-iterclose do

Re: [Python-ideas] Deterministic iterator cleanup

On Sat, Oct 22, 2016 at 9:17 AM, Nick Coghlan wrote: > This is actually a case where style guidelines would ideally differ > between between scripting use cases ... and > library(/framework/application) development use cases > Hmm -- interesting idea -- and I recall Guido

Re: [Python-ideas] Deterministic iterator cleanup

On 23 October 2016 at 02:17, Nick Coghlan wrote: > On 22 October 2016 at 06:59, Chris Barker wrote: >> And then context managers were introduced. And it seems to be there is a >> consensus in the Python community that we all should be using them when >>

Re: [Python-ideas] Deterministic iterator cleanup

On 22 October 2016 at 06:59, Chris Barker wrote: > And then context managers were introduced. And it seems to be there is a > consensus in the Python community that we all should be using them when > working on files, and I myself have finally started routinely using them,

Re: [Python-ideas] Deterministic iterator cleanup

On Fri, Oct 21, 2016 at 3:29 AM, Steven D'Aprano wrote: > As for the amount of good, this proposal originally came from PyPy. Just to be clear, I'm not a PyPy dev, and the PyPy devs' contribution here was mostly to look over a draft I circulated and to agree that it seemed

Re: [Python-ideas] Deterministic iterator cleanup

On Fri, Oct 21, 2016 at 3:48 PM, Amit Green wrote: > NOTE: This is my first post to this mailing list, I'm not really sure > how to post a message, so I'm attempting a reply-all. > > I like Nathaniel's idea for __iterclose__. > > I suggest the following changes to deal

Re: [Python-ideas] Deterministic iterator cleanup

On 10/21/2016 03:48 PM, Amit Green wrote: NOTE: This is my first post to this mailing list, I'm not really sure how to post a message, so I'm attempting a reply-all. Seems to have worked! :) I like Nathaniel's idea for __iterclose__. I suggest the following changes to deal with a

Re: [Python-ideas] Deterministic iterator cleanup

NOTE: This is my first post to this mailing list, I'm not really sure how to post a message, so I'm attempting a reply-all. I like Nathaniel's idea for __iterclose__. I suggest the following changes to deal with a few of the complex issues he discussed. 1. Missing __iterclose__, or a

Re: [Python-ideas] Deterministic iterator cleanup

On 21 October 2016 at 21:59, Chris Barker wrote: >> So (it seems to >> me) that you're talking about changing the behaviour of for-loops to >> suit only a small proportion of cases: maybe 10% of 10%. > > > I don't see what the big overhead is here. for loops would get a new

Re: [Python-ideas] Deterministic iterator cleanup

On Fri, Oct 21, 2016 at 12:12 AM, Steven D'Aprano wrote: > Portability across Pythons... if all Pythons performed exactly the same, > why would we need multiple implementations? The way I see it, > non-deterministic cleanup is the cost you pay for a non-reference > counting

Re: [Python-ideas] Deterministic iterator cleanup

On 2016-10-21 11:19 AM, Gustavo Carneiro wrote: Personally, I hadn't realised we had this problem in asyncio until now. Does this problem happen in asyncio at all? Or does asyncio somehow work around it by making sure to always explicitly destroy the frames of all coroutine objects, as long

Re: [Python-ideas] Deterministic iterator cleanup

Le 21/10/16 à 14:35, Paul Moore a écrit : [1] As I understand it. CPython's refcounting GC makes this a non-issue, correct? Wrong. Any guarantee that you think the CPython GC provides goes out of the window as soon as you have a reference cycle. Refcounting does not actually make GC

Re: [Python-ideas] Deterministic iterator cleanup

On 21 October 2016 at 12:23, Steven D'Aprano wrote: > On Fri, Oct 21, 2016 at 11:03:51AM +0100, Paul Moore wrote: > >> At the moment, the take home message for such users feels like it's >> "you might need to scatter preserve() around your code, to avoid the >> behaviour

Re: [Python-ideas] Deterministic iterator cleanup

On Fri, Oct 21, 2016 at 11:07:46AM +0100, Paul Moore wrote: > On 21 October 2016 at 10:53, Steven D'Aprano wrote: > > On Wed, Oct 19, 2016 at 12:33:57PM -0700, Nathaniel Smith wrote: > > > >> I should also say, regarding your specific example, I guess it's an > >> open

Re: [Python-ideas] Deterministic iterator cleanup

On Fri, Oct 21, 2016 at 11:03:51AM +0100, Paul Moore wrote: > At the moment, the take home message for such users feels like it's > "you might need to scatter preserve() around your code, to avoid the > behaviour change described above, which you glazed over because it > talked about all that

Re: [Python-ideas] Deterministic iterator cleanup

On 21 October 2016 at 10:53, Steven D'Aprano wrote: > On Wed, Oct 19, 2016 at 12:33:57PM -0700, Nathaniel Smith wrote: > >> I should also say, regarding your specific example, I guess it's an >> open question whether we would want list_iterator.__iterclose__ to >> actually do

Re: [Python-ideas] Deterministic iterator cleanup

You know, I'm actually starting to lean towards this proposal and away from my earlier objections... On Wed, Oct 19, 2016 at 12:33:57PM -0700, Nathaniel Smith wrote: > I should also say, regarding your specific example, I guess it's an > open question whether we would want

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 7:07 PM, Terry Reedy wrote: > On 10/19/2016 12:38 AM, Nathaniel Smith wrote: > >> I'd like to propose that Python's iterator protocol be enhanced to add >> a first-class notion of completion / cleanup. > > > With respect the the standard iterator

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 3:07 PM, Paul Moore wrote: > On 19 October 2016 at 20:21, Nathaniel Smith wrote: >> On Wed, Oct 19, 2016 at 11:38 AM, Paul Moore wrote: >>> On 19 October 2016 at 19:13, Chris Angelico wrote:

Re: [Python-ideas] Deterministic iterator cleanup

On 10/19/2016 12:38 AM, Nathaniel Smith wrote: I'd like to propose that Python's iterator protocol be enhanced to add a first-class notion of completion / cleanup. With respect the the standard iterator protocol, a very solid -1 from me. (I leave commenting specifically on __aiterclose__ to

Re: [Python-ideas] Deterministic iterator cleanup

On 2016-10-19 6:07 PM, Paul Moore wrote: I missed that you propose phasing this in, but it doesn't really alter much, I think the current behaviour is valuable and common, and I'm -1 on breaking it. It's just too much of a fundamental change to how loops and iterators interact for me to be

Re: [Python-ideas] Deterministic iterator cleanup

Hey Nathaniel - I like the intent here, but I think perhaps it would be better if the problem is approached differently. Seems to me that making *generators* have a special 'you are done now' interface is special casing, which usually makes things harder to learn and predict; and that more the

Re: [Python-ideas] Deterministic iterator cleanup

On 19 October 2016 at 20:21, Nathaniel Smith wrote: > On Wed, Oct 19, 2016 at 11:38 AM, Paul Moore wrote: >> On 19 October 2016 at 19:13, Chris Angelico wrote: >>> Now it *won't* correctly call the end-of-iteration function, because >>>

Re: [Python-ideas] Deterministic iterator cleanup

Nathaniel, On 2016-10-19 5:02 PM, Nathaniel Smith wrote: Hi Yury, Thanks for the detailed comments! Replies inline below. NP! On Wed, Oct 19, 2016 at 8:51 AM, Yury Selivanov wrote: I'm -1 on the idea. Here's why: 1. Python is a very dynamic language with GC

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 1:33 PM, Yury Selivanov wrote: > On 2016-10-19 3:33 PM, Nathaniel Smith wrote: > >>> lst = [1,2,3,4] >>> >it = iter(lst) >>> >for i in it: >>... if i == 2: break >> >>> >>> >for i in it:

Re: [Python-ideas] Deterministic iterator cleanup

Hi Yury, Thanks for the detailed comments! Replies inline below. On Wed, Oct 19, 2016 at 8:51 AM, Yury Selivanov wrote: > I'm -1 on the idea. Here's why: > > > 1. Python is a very dynamic language with GC and that is one of its > fundamental properties. This proposal

Re: [Python-ideas] Deterministic iterator cleanup

On 2016-10-19 3:33 PM, Nathaniel Smith wrote: lst = [1,2,3,4] >it = iter(lst) >for i in it: >>... if i == 2: break >> >for i in it: >>... print(i) >>3 >>4 > >> >>With the proposed behaviour, if I understand it, "it" would be closed >>after the first loop, so resuming

Re: [Python-ideas] Deterministic iterator cleanup

On 2016-10-19 12:21, Nathaniel Smith wrote: >Also, unless I'm misunderstanding the proposal, there's a fairly major >compatibility break. At present we have: > lst = [1,2,3,4] it = iter(lst) for i in it: >... if i == 2: break > for i in it: >... print(i) >3 >4 >

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 12:21 PM, Nathaniel Smith wrote: > On Wed, Oct 19, 2016 at 11:38 AM, Paul Moore wrote: >> On 19 October 2016 at 19:13, Chris Angelico wrote: >>> Now it *won't* correctly call the end-of-iteration function, because

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 11:38 AM, Paul Moore wrote: > On 19 October 2016 at 19:13, Chris Angelico wrote: >> Now it *won't* correctly call the end-of-iteration function, because >> there's no 'for' loop. This is going to either (a) require that EVERY >>

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 2:38 PM, Paul Moore wrote: > On 19 October 2016 at 19:13, Chris Angelico wrote: > > Now it *won't* correctly call the end-of-iteration function, because > > there's no 'for' loop. This is going to either (a) require that EVERY > >

Re: [Python-ideas] Deterministic iterator cleanup

On 19 October 2016 at 19:13, Chris Angelico wrote: > Now it *won't* correctly call the end-of-iteration function, because > there's no 'for' loop. This is going to either (a) require that EVERY > consumer of an iterator follow this new protocol, or (b) introduce a > ton of edge

Re: [Python-ideas] Deterministic iterator cleanup

On Thu, Oct 20, 2016 at 3:38 AM, Random832 wrote: > On Wed, Oct 19, 2016, at 11:51, Yury Selivanov wrote: >> I'm -1 on the idea. Here's why: >> >> >> 1. Python is a very dynamic language with GC and that is one of its >> fundamental properties. This proposal might make

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 10:08 AM, Neil Girdhar wrote: > > > On Wed, Oct 19, 2016 at 11:08 AM Todd wrote: >> >> On Wed, Oct 19, 2016 at 3:38 AM, Neil Girdhar >> wrote: >>> >>> This is a very interesting proposal. I just wanted to

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 11:08 AM Todd wrote: > On Wed, Oct 19, 2016 at 3:38 AM, Neil Girdhar > wrote: > > This is a very interesting proposal. I just wanted to share something I > found in my quick search: > > >

Re: [Python-ideas] Deterministic iterator cleanup

On 2016-10-19 12:38 PM, Random832 wrote: On Wed, Oct 19, 2016, at 11:51, Yury Selivanov wrote: I'm -1 on the idea. Here's why: 1. Python is a very dynamic language with GC and that is one of its fundamental properties. This proposal might make GC of iterators more deterministic, but that is

Re: [Python-ideas] Deterministic iterator cleanup

I'm -1 on the idea. Here's why: 1. Python is a very dynamic language with GC and that is one of its fundamental properties. This proposal might make GC of iterators more deterministic, but that is only one case. For instance, in some places in asyncio source code we have statements like

Re: [Python-ideas] Deterministic iterator cleanup

On Wed, Oct 19, 2016 at 3:38 AM, Neil Girdhar wrote: > This is a very interesting proposal. I just wanted to share something I > found in my quick search: > > http://stackoverflow.com/questions/14797930/python- > custom-iterator-close-a-file-on-stopiteration > > Could you

Re: [Python-ideas] Deterministic iterator cleanup

This is a very interesting proposal. I just wanted to share something I found in my quick search: http://stackoverflow.com/questions/14797930/python-custom-iterator-close-a-file-on-stopiteration Could you explain why the accepted answer there doesn't address this issue? class Parse(object):

Re: [Python-ideas] Deterministic iterator cleanup

On 19 October 2016 at 12:33, Oscar Benjamin wrote: > >> New convenience functions >> - >> >> The ``itertools`` module gains a new iterator wrapper that can be used >> to selectively disable the new ``__iterclose__`` behavior:: >> >> # XX

Re: [Python-ideas] Deterministic iterator cleanup

On 17 October 2016 at 09:08, Nathaniel Smith wrote: > Hi all, Hi Nathaniel. I'm just reposting what I wrote on pypy-dev (as requested) but under the assumption that you didn't substantially alter your draft - I apologise if some of the quoted text below has already been edited.

Re: [Python-ideas] Deterministic iterator cleanup

Thanks Nathaniel for this great proposal. As I went through your mail, I realized all the comments I wanted to make were already covered in later paragraphs. And I don't think there's a single point I disagree with. I don't have a strong opinion about the synchronous part of the proposal. I