Re: [Python-ideas] Deterministic iterator cleanup

2016-10-29 Thread Neil Girdhar
On Tuesday, October 25, 2016 at 6:26:17 PM UTC-4, Nathaniel Smith wrote: > > On Sat, Oct 22, 2016 at 9:02 AM, Nick Coghlan > wrote: > > On 20 October 2016 at 07:02, Nathaniel Smith > wrote: > >> The first change is to replace the outer for loop with a

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-26 Thread Nick Coghlan
On 26 October 2016 at 08:48, Nathaniel Smith wrote: > If it takes a strong reference, then suddenly we're pinning all > iterators in memory until the end of the enclosing function, which > will often look like a memory leak. I think this would break a *lot* > more existing code

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-26 Thread Nick Coghlan
On 26 October 2016 at 08:25, Nathaniel Smith wrote: > On Sat, Oct 22, 2016 at 9:02 AM, Nick Coghlan wrote: >> At this point your code is starting to look a whole lot like the code >> in contextlib.ExitStack.__exit__ :) > > One of the versions I tried but

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-26 Thread Nick Coghlan
On 26 October 2016 at 01:59, Yury Selivanov wrote: > But how would it help with a partial iteration over generators > with a "with" statement inside? > >def it(): >with open(file) as f: >for line in f: >yield line > > Nathaniel

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-25 Thread Nathaniel Smith
...Doh. I spent all that time evaluating the function-scoped-cleanup proposal from the high-level design perspective, and then immediately after hitting send, I suddenly realized that I'd missed a much more straightforward technical problem. One thing that 'with' blocks / for-scoped-iterclose do

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-25 Thread Nick Coghlan
On 25 October 2016 at 11:59, Stephen J. Turnbull wrote: > On the semantic side, > constructs like closures and generators (which they may be cargo- > culting!) mean that it's harder to link resource management to > (syntactic) function calls than a new

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-25 Thread Nick Coghlan
On 25 October 2016 at 03:32, Chris Barker wrote: > On Sat, Oct 22, 2016 at 8:22 PM, Nick Coghlan wrote: >> The big refactoring benefit that this feature would offer over with >> statements is that it doesn't require a structural change to the code >> -

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-25 Thread Nick Coghlan
On 25 October 2016 at 03:16, Chris Barker wrote: > On Sat, Oct 22, 2016 at 9:17 AM, Nick Coghlan wrote: > >> >> This is actually a case where style guidelines would ideally differ >> between between scripting use cases ... and >>

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-24 Thread Stephen J. Turnbull
Chris Barker wrote: > Nick Coghlan wrote: >> Chris, would you be open to trying a thought experiment with some of >> your students looking at ways to introduce function-scoped >> deterministic resource management *before* introducing with >> statements? I'm with Chris, I think: this seems

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-24 Thread Chris Barker
On Sat, Oct 22, 2016 at 9:17 AM, Nick Coghlan wrote: > This is actually a case where style guidelines would ideally differ > between between scripting use cases ... and > library(/framework/application) development use cases > Hmm -- interesting idea -- and I recall Guido

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-22 Thread Nick Coghlan
On 23 October 2016 at 02:17, Nick Coghlan wrote: > On 22 October 2016 at 06:59, Chris Barker wrote: >> And then context managers were introduced. And it seems to be there is a >> consensus in the Python community that we all should be using them when >>

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-22 Thread Nick Coghlan
On 22 October 2016 at 06:59, Chris Barker wrote: > And then context managers were introduced. And it seems to be there is a > consensus in the Python community that we all should be using them when > working on files, and I myself have finally started routinely using them,

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-22 Thread Nick Coghlan
On 20 October 2016 at 07:02, Nathaniel Smith wrote: > The first change is to replace the outer for loop with a while/pop > loop, so that if an exception occurs we'll know which iterables remain > to be processed: > > def chain(*iterables): > try: > while iterables: >

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Nathaniel Smith
On Fri, Oct 21, 2016 at 3:29 AM, Steven D'Aprano wrote: > As for the amount of good, this proposal originally came from PyPy. Just to be clear, I'm not a PyPy dev, and the PyPy devs' contribution here was mostly to look over a draft I circulated and to agree that it seemed

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Nathaniel Smith
On Fri, Oct 21, 2016 at 3:48 PM, Amit Green wrote: > NOTE: This is my first post to this mailing list, I'm not really sure > how to post a message, so I'm attempting a reply-all. > > I like Nathaniel's idea for __iterclose__. > > I suggest the following changes to deal

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Ethan Furman
On 10/21/2016 03:48 PM, Amit Green wrote: NOTE: This is my first post to this mailing list, I'm not really sure how to post a message, so I'm attempting a reply-all. Seems to have worked! :) I like Nathaniel's idea for __iterclose__. I suggest the following changes to deal with a

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Amit Green
NOTE: This is my first post to this mailing list, I'm not really sure how to post a message, so I'm attempting a reply-all. I like Nathaniel's idea for __iterclose__. I suggest the following changes to deal with a few of the complex issues he discussed. 1. Missing __iterclose__, or a

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Paul Moore
On 21 October 2016 at 21:59, Chris Barker wrote: >> So (it seems to >> me) that you're talking about changing the behaviour of for-loops to >> suit only a small proportion of cases: maybe 10% of 10%. > > > I don't see what the big overhead is here. for loops would get a new

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Chris Barker
On Fri, Oct 21, 2016 at 12:12 AM, Steven D'Aprano wrote: > Portability across Pythons... if all Pythons performed exactly the same, > why would we need multiple implementations? The way I see it, > non-deterministic cleanup is the cost you pay for a non-reference > counting

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Yury Selivanov
On 2016-10-21 11:19 AM, Gustavo Carneiro wrote: Personally, I hadn't realised we had this problem in asyncio until now. Does this problem happen in asyncio at all? Or does asyncio somehow work around it by making sure to always explicitly destroy the frames of all coroutine objects, as long

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Ronan Lamy
Le 21/10/16 à 14:35, Paul Moore a écrit : [1] As I understand it. CPython's refcounting GC makes this a non-issue, correct? Wrong. Any guarantee that you think the CPython GC provides goes out of the window as soon as you have a reference cycle. Refcounting does not actually make GC

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Paul Moore
On 21 October 2016 at 12:23, Steven D'Aprano wrote: > On Fri, Oct 21, 2016 at 11:03:51AM +0100, Paul Moore wrote: > >> At the moment, the take home message for such users feels like it's >> "you might need to scatter preserve() around your code, to avoid the >> behaviour

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Steven D'Aprano
On Fri, Oct 21, 2016 at 11:07:46AM +0100, Paul Moore wrote: > On 21 October 2016 at 10:53, Steven D'Aprano wrote: > > On Wed, Oct 19, 2016 at 12:33:57PM -0700, Nathaniel Smith wrote: > > > >> I should also say, regarding your specific example, I guess it's an > >> open

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Steven D'Aprano
On Fri, Oct 21, 2016 at 11:03:51AM +0100, Paul Moore wrote: > At the moment, the take home message for such users feels like it's > "you might need to scatter preserve() around your code, to avoid the > behaviour change described above, which you glazed over because it > talked about all that

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Steven D'Aprano
On Wed, Oct 19, 2016 at 05:52:34PM -0400, Yury Selivanov wrote: > IOW I'm not convinced that if we implement your proposal we'll fix 90% > (or even 30%) of cases where non-deterministic and postponed cleanup is > harmful. Just because something doesn't solve ALL problems doesn't mean it isn't

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Paul Moore
On 21 October 2016 at 10:53, Steven D'Aprano wrote: > On Wed, Oct 19, 2016 at 12:33:57PM -0700, Nathaniel Smith wrote: > >> I should also say, regarding your specific example, I guess it's an >> open question whether we would want list_iterator.__iterclose__ to >> actually do

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Paul Moore
On 21 October 2016 at 07:03, Nathaniel Smith wrote: > Oh, goodness, no -- like Yury said, the use cases here are not > specific to async at all. I mean, none of the examples are async even > :-). [...] Ah I follow now. Sorry for the misunderstanding, I'd skimmed a bit more than I

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Steven D'Aprano
You know, I'm actually starting to lean towards this proposal and away from my earlier objections... On Wed, Oct 19, 2016 at 12:33:57PM -0700, Nathaniel Smith wrote: > I should also say, regarding your specific example, I guess it's an > open question whether we would want

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Steven D'Aprano
On Thu, Oct 20, 2016 at 11:03:11PM -0700, Nathaniel Smith wrote: > The motivation here is that prompt (non-GC-dependent) cleanup is a > good thing for a variety of reasons: determinism, portability across > Python implementations, proper exception propagation, etc. async does > add yet another

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Nathaniel Smith
On Wed, Oct 19, 2016 at 7:07 PM, Terry Reedy wrote: > On 10/19/2016 12:38 AM, Nathaniel Smith wrote: > >> I'd like to propose that Python's iterator protocol be enhanced to add >> a first-class notion of completion / cleanup. > > > With respect the the standard iterator

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-21 Thread Nathaniel Smith
On Wed, Oct 19, 2016 at 3:07 PM, Paul Moore wrote: > On 19 October 2016 at 20:21, Nathaniel Smith wrote: >> On Wed, Oct 19, 2016 at 11:38 AM, Paul Moore wrote: >>> On 19 October 2016 at 19:13, Chris Angelico wrote:

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Terry Reedy
On 10/19/2016 12:38 AM, Nathaniel Smith wrote: I'd like to propose that Python's iterator protocol be enhanced to add a first-class notion of completion / cleanup. With respect the the standard iterator protocol, a very solid -1 from me. (I leave commenting specifically on __aiterclose__ to

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Yury Selivanov
On 2016-10-19 6:07 PM, Paul Moore wrote: I missed that you propose phasing this in, but it doesn't really alter much, I think the current behaviour is valuable and common, and I'm -1 on breaking it. It's just too much of a fundamental change to how loops and iterators interact for me to be

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Robert Collins
Hey Nathaniel - I like the intent here, but I think perhaps it would be better if the problem is approached differently. Seems to me that making *generators* have a special 'you are done now' interface is special casing, which usually makes things harder to learn and predict; and that more the

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Paul Moore
On 19 October 2016 at 20:21, Nathaniel Smith wrote: > On Wed, Oct 19, 2016 at 11:38 AM, Paul Moore wrote: >> On 19 October 2016 at 19:13, Chris Angelico wrote: >>> Now it *won't* correctly call the end-of-iteration function, because >>>

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Nathaniel Smith
On Wed, Oct 19, 2016 at 11:13 AM, Chris Angelico wrote: > On Thu, Oct 20, 2016 at 3:38 AM, Random832 wrote: >> On Wed, Oct 19, 2016, at 11:51, Yury Selivanov wrote: >>> I'm -1 on the idea. Here's why: >>> >>> >>> 1. Python is a very dynamic language

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Yury Selivanov
Nathaniel, On 2016-10-19 5:02 PM, Nathaniel Smith wrote: Hi Yury, Thanks for the detailed comments! Replies inline below. NP! On Wed, Oct 19, 2016 at 8:51 AM, Yury Selivanov wrote: I'm -1 on the idea. Here's why: 1. Python is a very dynamic language with GC

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Nathaniel Smith
On Wed, Oct 19, 2016 at 1:33 PM, Yury Selivanov wrote: > On 2016-10-19 3:33 PM, Nathaniel Smith wrote: > >>> lst = [1,2,3,4] >>> >it = iter(lst) >>> >for i in it: >>... if i == 2: break >> >>> >>> >for i in it:

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Nathaniel Smith
Hi Yury, Thanks for the detailed comments! Replies inline below. On Wed, Oct 19, 2016 at 8:51 AM, Yury Selivanov wrote: > I'm -1 on the idea. Here's why: > > > 1. Python is a very dynamic language with GC and that is one of its > fundamental properties. This proposal

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Yury Selivanov
On 2016-10-19 3:33 PM, Nathaniel Smith wrote: lst = [1,2,3,4] >it = iter(lst) >for i in it: >>... if i == 2: break >> >for i in it: >>... print(i) >>3 >>4 > >> >>With the proposed behaviour, if I understand it, "it" would be closed >>after the first loop, so resuming

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Chris Angelico
On Thu, Oct 20, 2016 at 7:14 AM, Neil Girdhar wrote: > class AddIterclose: > > def __init__(self, iterable, iterclose): > self.iterable = iterable > self.iterclose = iterclose > > def __iter__(self): > try: > for x in

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Brendan Barnwell
On 2016-10-19 12:21, Nathaniel Smith wrote: >Also, unless I'm misunderstanding the proposal, there's a fairly major >compatibility break. At present we have: > lst = [1,2,3,4] it = iter(lst) for i in it: >... if i == 2: break > for i in it: >... print(i) >3 >4 >

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Nathaniel Smith
On Wed, Oct 19, 2016 at 12:21 PM, Nathaniel Smith wrote: > On Wed, Oct 19, 2016 at 11:38 AM, Paul Moore wrote: >> On 19 October 2016 at 19:13, Chris Angelico wrote: >>> Now it *won't* correctly call the end-of-iteration function, because

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Nathaniel Smith
On Wed, Oct 19, 2016 at 11:38 AM, Paul Moore wrote: > On 19 October 2016 at 19:13, Chris Angelico wrote: >> Now it *won't* correctly call the end-of-iteration function, because >> there's no 'for' loop. This is going to either (a) require that EVERY >>

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Todd
On Wed, Oct 19, 2016 at 2:38 PM, Paul Moore wrote: > On 19 October 2016 at 19:13, Chris Angelico wrote: > > Now it *won't* correctly call the end-of-iteration function, because > > there's no 'for' loop. This is going to either (a) require that EVERY > >

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Paul Moore
On 19 October 2016 at 19:13, Chris Angelico wrote: > Now it *won't* correctly call the end-of-iteration function, because > there's no 'for' loop. This is going to either (a) require that EVERY > consumer of an iterator follow this new protocol, or (b) introduce a > ton of edge

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Chris Angelico
On Thu, Oct 20, 2016 at 3:38 AM, Random832 wrote: > On Wed, Oct 19, 2016, at 11:51, Yury Selivanov wrote: >> I'm -1 on the idea. Here's why: >> >> >> 1. Python is a very dynamic language with GC and that is one of its >> fundamental properties. This proposal might make

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Nathaniel Smith
On Wed, Oct 19, 2016 at 10:08 AM, Neil Girdhar wrote: > > > On Wed, Oct 19, 2016 at 11:08 AM Todd wrote: >> >> On Wed, Oct 19, 2016 at 3:38 AM, Neil Girdhar >> wrote: >>> >>> This is a very interesting proposal. I just wanted to

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Neil Girdhar
On Wed, Oct 19, 2016 at 11:08 AM Todd wrote: > On Wed, Oct 19, 2016 at 3:38 AM, Neil Girdhar > wrote: > > This is a very interesting proposal. I just wanted to share something I > found in my quick search: > > >

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Yury Selivanov
On 2016-10-19 12:38 PM, Random832 wrote: On Wed, Oct 19, 2016, at 11:51, Yury Selivanov wrote: I'm -1 on the idea. Here's why: 1. Python is a very dynamic language with GC and that is one of its fundamental properties. This proposal might make GC of iterators more deterministic, but that is

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Random832
On Wed, Oct 19, 2016, at 11:51, Yury Selivanov wrote: > I'm -1 on the idea. Here's why: > > > 1. Python is a very dynamic language with GC and that is one of its > fundamental properties. This proposal might make GC of iterators more > deterministic, but that is only one case. There is a

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Yury Selivanov
I'm -1 on the idea. Here's why: 1. Python is a very dynamic language with GC and that is one of its fundamental properties. This proposal might make GC of iterators more deterministic, but that is only one case. For instance, in some places in asyncio source code we have statements like

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Todd
On Wed, Oct 19, 2016 at 3:38 AM, Neil Girdhar wrote: > This is a very interesting proposal. I just wanted to share something I > found in my quick search: > > http://stackoverflow.com/questions/14797930/python- > custom-iterator-close-a-file-on-stopiteration > > Could you

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Neil Girdhar
This is a very interesting proposal. I just wanted to share something I found in my quick search: http://stackoverflow.com/questions/14797930/python-custom-iterator-close-a-file-on-stopiteration Could you explain why the accepted answer there doesn't address this issue? class Parse(object):

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Oscar Benjamin
On 19 October 2016 at 12:33, Oscar Benjamin wrote: > >> New convenience functions >> - >> >> The ``itertools`` module gains a new iterator wrapper that can be used >> to selectively disable the new ``__iterclose__`` behavior:: >> >> # XX

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Oscar Benjamin
On 17 October 2016 at 09:08, Nathaniel Smith wrote: > Hi all, Hi Nathaniel. I'm just reposting what I wrote on pypy-dev (as requested) but under the assumption that you didn't substantially alter your draft - I apologise if some of the quoted text below has already been edited.

Re: [Python-ideas] Deterministic iterator cleanup

2016-10-19 Thread Vincent Michel
Thanks Nathaniel for this great proposal. As I went through your mail, I realized all the comments I wanted to make were already covered in later paragraphs. And I don't think there's a single point I disagree with. I don't have a strong opinion about the synchronous part of the proposal. I

[Python-ideas] Deterministic iterator cleanup

2016-10-18 Thread Nathaniel Smith
Hi all, I'd like to propose that Python's iterator protocol be enhanced to add a first-class notion of completion / cleanup. This is mostly motivated by thinking about the issues around async generators and cleanup. Unfortunately even though PEP 525 was accepted I found myself unable to stop