Re: generator/coroutine terminology
In article 551e2cfd$0$11123$c3e8...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Wednesday 01 April 2015 00:18, Albert van der Horst wrote: In article 55062bda$0$12998$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: The biggest difference is syntactic. Here's an iterator which returns a never-ending sequence of squared numbers 1, 4, 9, 16, ... class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self You should give an example of usage. As a newby I'm not up to figuring out the specification from source for something built of the mysterious __ internal thingies. (I did experiment with Squares interactively. But I didn't get further than creating a Squares object.) Ah, sorry about that! Usage is: it = Squares() # create an iterator print(next(it)) # print the first value x = next(it) # extract the second while x 100: print(x) x = next(it) Beware of doing this: for x in Squares(): print(x) since Squares is an *infinite* generator, it will continue for ever if you let it. Fortunately you can hit Ctrl-C to interrupt the for loop at any point. In Python 2, you will need to rename __next__ to just next without the double-leading-and-trailing underscores. Here's the same thing written as a generator: def squares(): i = 1 while True: yield i**2 i += 1 And for this one: it = squares() # create the iterator print(next(it)) # print the first value x = next(it) # extract the second while x 100: print(x) x = next(it) Usage is pretty much exactly the same. Thanks, I get it now. next and yield are more or less switching between coroutines. -- Steve -- Albert van der Horst, UTRECHT,THE NETHERLANDS Economic growth -- being exponential -- ultimately falters. albert@spearc.xs4all.nl =n http://home.hccnet.nl/a.w.m.van.der.horst -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Friday, March 13, 2015 at 5:36:35 PM UTC+8, Marko Rauhamaa wrote: Rustom Mody rustompm...@gmail.com: Nice demo of the same confusing terminology we are talking about. Why don't you just stick with the terminology of the language specification? I think your students are going to be more confused if you try to come up with something better. When I say expect generators to close I mean 'generator-instances' ie at runtime When you say expect both to close with return you are talking of program-texts ie the 'factory' [Or I am guilty of the same I am accusing python of] Your 'factory' is a: generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html Your 'generator-instance' is an: iterator An object representing a stream of data. URL: https://docs.python.org/3/glossary.html I don't think you should read much into what str(obj) returns. It's not much more than a random printable some CPython coder has cooked up in the heat of the moment. Marko Well, the functional that returns a function instance which can be used as an interator is also a function. Well, some might use terms in generics programming as described in the book by Gangs of 4. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Wednesday 01 April 2015 00:18, Albert van der Horst wrote: In article 55062bda$0$12998$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: The biggest difference is syntactic. Here's an iterator which returns a never-ending sequence of squared numbers 1, 4, 9, 16, ... class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self You should give an example of usage. As a newby I'm not up to figuring out the specification from source for something built of the mysterious __ internal thingies. (I did experiment with Squares interactively. But I didn't get further than creating a Squares object.) Ah, sorry about that! Usage is: it = Squares() # create an iterator print(next(it)) # print the first value x = next(it) # extract the second while x 100: print(x) x = next(it) Beware of doing this: for x in Squares(): print(x) since Squares is an *infinite* generator, it will continue for ever if you let it. Fortunately you can hit Ctrl-C to interrupt the for loop at any point. In Python 2, you will need to rename __next__ to just next without the double-leading-and-trailing underscores. Here's the same thing written as a generator: def squares(): i = 1 while True: yield i**2 i += 1 And for this one: it = squares() # create the iterator print(next(it)) # print the first value x = next(it) # extract the second while x 100: print(x) x = next(it) Usage is pretty much exactly the same. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
alb...@spenarnc.xs4all.nl (Albert van der Horst) writes: You should give an example of usage. As a newby I'm not up to figuring out the specification from source for something built of the mysterious __ internal thingies. In reality because of generator expressions, the yield statement, and some useful built-in generators in the itertools module, you rarely have to use those special methods. I'd advise reading through the itertools module documentation from beginning to end, trying to understand what everything does. Even if some are not that useful, it will help convey the mode of thinking that went into these features. At a somewhat deeper level you might like the SICP book: http://mitpress.mit.edu/sicp/ It's somewhat old now and it's about Scheme rather than Python, but it builds up the relevant concepts quite nicely. Regarding the squares example, consider this even simpler generator: def count(n): while True: yield n n += 1 so count(1) yields 1, 2, 3, 4 ... This is a very useful generator but you don't need to write it since it's included in the itertools module as itertools.count. Its initial value defaults to 0. So the squares generator can be written: def squares(): return (i*i for i in itertools.count(0)) Now if you want all the squares less than 100 (i.e. 0, 1, 4, 9, ..., 81): wanted = itertools.takewhile(lambda x: x100, squares()) You can print that out as a list: print(list(wanted)) The built-in sum function consumes an iterator, so you can add up the squares less than 100: print(sum(itertools.takewhile(lambda x: x100, squares( this prints 205 which is 1+4+9+16+25+36+49+64+81. These features fit together quite elegantly and code like this flows off the fingertips naturally once you've used to it. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
In article 83d579c1-ab61-4a3d-a834-e65d28eac...@googlegroups.com, Rustom Mody rustompm...@gmail.com wrote: On Saturday, March 14, 2015 at 8:59:22 PM UTC+5:30, Rustom Mody wrote: On Saturday, March 14, 2015 at 11:34:27 AM UTC+5:30, Steven D'Aprano wrote: A generator (function) may be a function which returns an iterator,... I find generator-function misleading in the same way that pineapple misleadingly suggests apple that grows on pines A builtin function is a function in the builtin (or builtins -- can never remember) module A pure function is function that does not assign or mutate non-locals A Steven-function is a function that presumably Steven wrote However a generator function is a weird sort of function (at best). Not regarding it as a function is IMO more reasonable. Another analogy somewhat closer home than pineapples. In Pascal there are procedures and functions. Even in the venerable Fortran there are subroutine-subprogram and (sub)function-subprogram. The Algol 68 designers considered it a defect in the design. They created a situation like in Python, where a def-thingy need not return a value. C took the stupid approach of just throwing out one of these. A decade of troubles was enough to convince people that it was needed and the mysterious 'void-returning' function was introduced to simulate procedures The mistake this was intended to fix, was the rule that by default a function returns int in C. Groetjes Albert -- Albert van der Horst, UTRECHT,THE NETHERLANDS Economic growth -- being exponential -- ultimately falters. albert@spearc.xs4all.nl =n http://home.hccnet.nl/a.w.m.van.der.horst -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
In article mailman.372.1427809109.10327.python-l...@python.org, Dave Angel da...@davea.name wrote: On 03/31/2015 09:18 AM, Albert van der Horst wrote: In article 55062bda$0$12998$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: The biggest difference is syntactic. Here's an iterator which returns a never-ending sequence of squared numbers 1, 4, 9, 16, ... class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self You should give an example of usage. As a newby I'm not up to figuring out the specification from source for something built of the mysterious __ internal thingies. (I did experiment with Squares interactively. But I didn't get further than creating a Squares object.) He did say it was an iterator. So for a first try, write a for loop: class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self for i in Squares(): print(i) if i 50: break This is what I get: / -- albert@cherry:/tmp$ more aap.py class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self albert@cherry:/tmp$ python Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48) [GCC 4.4.5] on linux2 Type help, copyright, credits or license for more information. from aap import * for i in Squares(): ... print i ... if i50: break ... Traceback (most recent call last): File stdin, line 1, in module TypeError: instance has no next() method / -- Probably not what is intended. Last minute note: renaming __next__() into next() did the job. -- DaveA Groetjes Albert -- Albert van der Horst, UTRECHT,THE NETHERLANDS Economic growth -- being exponential -- ultimately falters. albert@spearc.xs4all.nl =n http://home.hccnet.nl/a.w.m.van.der.horst -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Wed, Apr 1, 2015 at 2:03 AM, Albert van der Horst alb...@spenarnc.xs4all.nl wrote: class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self albert@cherry:/tmp$ python Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48) [GCC 4.4.5] on linux2 Type help, copyright, credits or license for more information. from aap import * for i in Squares(): ... print i ... if i50: break ... Traceback (most recent call last): File stdin, line 1, in module TypeError: instance has no next() method / -- Probably not what is intended. Last minute note: renaming __next__() into next() did the job. That class was written for Python 3, not Python 2. In Py2, you need to rename __next__ to next, as you noted, and you probably also want to explicitly subclass object. Or just run it under Python 3. :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
In article 55062bda$0$12998$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Marko Rauhamaa wrote: Chris Angelico ros...@gmail.com: On Sun, Mar 15, 2015 at 9:15 AM, Marko Rauhamaa ma...@pacujo.net wrote: Is it necessary/useful for a Python application programmer to be conscious of the different types of iterator? What mistaken usage could arise if the application just treated all iterators as, well, iterators? If you treat them all as iterators, then you're safe, because that's the lowest common denominator. But there are a number of other iterators that have more features, including files, generators, etc. What features do generator iterators provide on top of generic iterators? The biggest difference is syntactic. Here's an iterator which returns a never-ending sequence of squared numbers 1, 4, 9, 16, ... class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self You should give an example of usage. As a newby I'm not up to figuring out the specification from source for something built of the mysterious __ internal thingies. (I did experiment with Squares interactively. But I didn't get further than creating a Squares object.) Here's the same thing written as a generator: def squares(): i = 1 while True: yield i**2 i += 1 Four lines, versus eight. The iterator version has a lot of boilerplate (although some of it, the two-line __iter__ method, could be eliminated if there was a standard Iterator builtin to inherit from). Here's a more complicated example: class Randoms: def __init__(self): self.finished = False def __next__(self): x = random.random() if x 0.5: self.finished = True if self.finished: raise StopIteration else: return x def __iter__(self): return self def randoms(): x = random.random() while x 0.5: yield x x = random.random() Generators, as a rule, are significantly easier to write, understand, and debug. There's nothing they can do that can't be done with an iterator class, but the fiddly, unexciting bits related to halting and resuming and saving state are all handled for you, allowing you to concentrate on the behaviour you want, not the boilerplate. This is illuminating. Thanks. -- Steven Groetjes Albert -- Albert van der Horst, UTRECHT,THE NETHERLANDS Economic growth -- being exponential -- ultimately falters. albert@spearc.xs4all.nl =n http://home.hccnet.nl/a.w.m.van.der.horst -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 03/31/2015 09:18 AM, Albert van der Horst wrote: In article 55062bda$0$12998$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: The biggest difference is syntactic. Here's an iterator which returns a never-ending sequence of squared numbers 1, 4, 9, 16, ... class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self You should give an example of usage. As a newby I'm not up to figuring out the specification from source for something built of the mysterious __ internal thingies. (I did experiment with Squares interactively. But I didn't get further than creating a Squares object.) He did say it was an iterator. So for a first try, write a for loop: class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self for i in Squares(): print(i) if i 50: break print(done) Here's the same thing written as a generator: def squares(): i = 1 while True: yield i**2 i += 1 Four lines, versus eight. The iterator version has a lot of boilerplate (although some of it, the two-line __iter__ method, could be eliminated if there was a standard Iterator builtin to inherit from). -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 17/03/2015 03:33, Rustom Mody wrote: On Tuesday, March 17, 2015 at 8:55:27 AM UTC+5:30, Mark Lawrence wrote: On 17/03/2015 03:18, Rustom Mody wrote: On Tuesday, March 17, 2015 at 8:37:25 AM UTC+5:30, Mark Lawrence wrote: Ok Let me throw out a suggestion: - potato is a generator - tomato is a cursor. Acceptable? No. In Python potato is a generator function, tomato is a generator. Why complicate something that is so simple? I couldn't care less what they are called in any other language. Ok so lets see... https://docs.python.org/3.4/tutorial/classes.html#generators https://docs.python.org/3.4/glossary.html#term-generator Are these consistent with (your notion of) python? Maybe they are any other language? I'll already suggested you write the patches and put them on the bug tracker. If you can't be bothered please have the courtesy to stop bleating about it. Here are two of your posts (within a couple of hours) 1. So the docs are confused and inconsistent but in an open source community it's not *MY* responsibility to deal with it, somebody else can. Making mountains out of mole hills is all I see in this entire thread. 2. No. In Python potato is a generator function, tomato is a generator. Why complicate something that is so simple? I couldn't care less what they are called in any other language. I understand the first as saying: Since something is wrong correct it; Dont bleat! The first is saying that *YOU* are complaining that the docs are inconsistent but *YOU* won't do anything about it. The second is saying Nothing is wrong! The second is saying that those are *MY* definitions that *I'M* perfectly happy with. Please decide which side you belong The side which hopes you give up this now very tedious thread. Mario's response a few minutes back summed things up perfectly. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 17/03/2015 02:52, Rustom Mody wrote: On Monday, March 16, 2015 at 11:50:33 PM UTC+5:30, Mark Lawrence wrote: On 16/03/2015 14:37, Rustom Mody wrote: On Monday, March 16, 2015 at 7:57:08 PM UTC+5:30, Mark Lawrence wrote: On 16/03/2015 14:19, Rustom Mody wrote: == Anyways... Yes 15 years are past. I dont expect the def can be revoked now. [As far as I am concerned its a minor wart] But the mess around the docs can certainly be cleaned up. So write the patches to correct the docs, then raise the issue on the bug tracker to get the patches accepted. IIRC for doc patches you don't even have to provide diff files against the rst files, plain text will do, the core devs will do the rest for you. I would gladly do that if it was a minor correction here and there. But the problem is a bit deeper even though it can be kept mostly¹ in the docs and not modify any syntax/semantics of python. In particular for: def potato(x): yield x+1 tomato = potato(3) what shall we call potato and tomato. I believe this thread clearly shows that the docs are confused and inconsistent. Yet I dont see any consensus on what/how to classify tomato/potato. Function -- trouble on one side Generator -- trouble on another Iterator -- trouble on third etc ¹ Grey areas excepted eg output of help(); inspect module etc So the docs are confused and inconsistent but in an open source community it's not *MY* responsibility to deal with it, somebody else can. Making mountains out of mole hills is all I see in this entire thread. Ok... Lets see... What if any agreement is there on this. To start with basic terminology. Given def potato(x): yield x+1 tomato = potato(3) What shall we call potato and tomato? Steven suggested that potato can be called 'factory' Not ideal, but way better than 'generator-function'. Oscar does not like it. No better/other suggestions (that we see here). What next? Ok Let me throw out a suggestion: - potato is a generator - tomato is a cursor. Acceptable? No. In Python potato is a generator function, tomato is a generator. Why complicate something that is so simple? I couldn't care less what they are called in any other language. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Tuesday, March 17, 2015 at 8:55:27 AM UTC+5:30, Mark Lawrence wrote: On 17/03/2015 03:18, Rustom Mody wrote: On Tuesday, March 17, 2015 at 8:37:25 AM UTC+5:30, Mark Lawrence wrote: Ok Let me throw out a suggestion: - potato is a generator - tomato is a cursor. Acceptable? No. In Python potato is a generator function, tomato is a generator. Why complicate something that is so simple? I couldn't care less what they are called in any other language. Ok so lets see... https://docs.python.org/3.4/tutorial/classes.html#generators https://docs.python.org/3.4/glossary.html#term-generator Are these consistent with (your notion of) python? Maybe they are any other language? I'll already suggested you write the patches and put them on the bug tracker. If you can't be bothered please have the courtesy to stop bleating about it. Here are two of your posts (within a couple of hours) 1. So the docs are confused and inconsistent but in an open source community it's not *MY* responsibility to deal with it, somebody else can. Making mountains out of mole hills is all I see in this entire thread. 2. No. In Python potato is a generator function, tomato is a generator. Why complicate something that is so simple? I couldn't care less what they are called in any other language. I understand the first as saying: Since something is wrong correct it; Dont bleat! The second is saying Nothing is wrong! Please decide which side you belong -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, 16 Mar 2015 19:52:59 -0700 (PDT), Rustom Mody rustompm...@gmail.com wrote: When we go from 'simple-generators' to coroutine-generators there seem to be bigger conceptual problems and implementation-gaffes. Quite frankly I hardly understand this part. There's a line after which terminology just becomes pedantic. And if you cross it once more, you get into the realm of obfuscation. Congratulations, I suppose! You managed to have reached that edge. Seriously, you started this thread well. I even nodded to my screen when many posts back you mentioned correct terminology was an important aspect of any official spec. But you just couldn't stop, could you? What you aren't realizing is that your quest for the perfect set of terms is exactly what is leading you into a confusing state, producing the opposite effect of making things harder to understand, instead of easier. Continue this thread if you must, but please never write a spec in your life. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 17/03/2015 03:18, Rustom Mody wrote: On Tuesday, March 17, 2015 at 8:37:25 AM UTC+5:30, Mark Lawrence wrote: Ok Let me throw out a suggestion: - potato is a generator - tomato is a cursor. Acceptable? No. In Python potato is a generator function, tomato is a generator. Why complicate something that is so simple? I couldn't care less what they are called in any other language. Ok so lets see... https://docs.python.org/3.4/tutorial/classes.html#generators https://docs.python.org/3.4/glossary.html#term-generator Are these consistent with (your notion of) python? Maybe they are any other language? I'll already suggested you write the patches and put them on the bug tracker. If you can't be bothered please have the courtesy to stop bleating about it. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Monday, March 16, 2015 at 11:50:33 PM UTC+5:30, Mark Lawrence wrote: On 16/03/2015 14:37, Rustom Mody wrote: On Monday, March 16, 2015 at 7:57:08 PM UTC+5:30, Mark Lawrence wrote: On 16/03/2015 14:19, Rustom Mody wrote: == Anyways... Yes 15 years are past. I dont expect the def can be revoked now. [As far as I am concerned its a minor wart] But the mess around the docs can certainly be cleaned up. So write the patches to correct the docs, then raise the issue on the bug tracker to get the patches accepted. IIRC for doc patches you don't even have to provide diff files against the rst files, plain text will do, the core devs will do the rest for you. I would gladly do that if it was a minor correction here and there. But the problem is a bit deeper even though it can be kept mostly¹ in the docs and not modify any syntax/semantics of python. In particular for: def potato(x): yield x+1 tomato = potato(3) what shall we call potato and tomato. I believe this thread clearly shows that the docs are confused and inconsistent. Yet I dont see any consensus on what/how to classify tomato/potato. Function -- trouble on one side Generator -- trouble on another Iterator -- trouble on third etc ¹ Grey areas excepted eg output of help(); inspect module etc So the docs are confused and inconsistent but in an open source community it's not *MY* responsibility to deal with it, somebody else can. Making mountains out of mole hills is all I see in this entire thread. Ok... Lets see... What if any agreement is there on this. To start with basic terminology. Given def potato(x): yield x+1 tomato = potato(3) What shall we call potato and tomato? Steven suggested that potato can be called 'factory' Not ideal, but way better than 'generator-function'. Oscar does not like it. No better/other suggestions (that we see here). What next? Ok Let me throw out a suggestion: - potato is a generator - tomato is a cursor. Acceptable? So much for terminological questions. When we go from 'simple-generators' to coroutine-generators there seem to be bigger conceptual problems and implementation-gaffes. Quite frankly I hardly understand this part. Here are some questions. 1. Is the name generator appropriate for coroutine-generator? The wikipedia-link that Steven posted suggests the other-way-round 'is-a' relation: a generator is a (semi)coroutine¹ 2. Can we say yield-statement ≡ generator yield-expression ≡ coroutine ? What of x = yield y x = yield x My (vaguest) impression is that 'yield' stops being the meaningful word for coroutine -- see footnote from Frederik Lundh in the 2001 thread suspend instead of yield, but that's me ¹ Non-trivial terminological headache: Is this 'generator' in the old sense or new sense? -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Tuesday, March 17, 2015 at 8:37:25 AM UTC+5:30, Mark Lawrence wrote: Ok Let me throw out a suggestion: - potato is a generator - tomato is a cursor. Acceptable? No. In Python potato is a generator function, tomato is a generator. Why complicate something that is so simple? I couldn't care less what they are called in any other language. Ok so lets see... https://docs.python.org/3.4/tutorial/classes.html#generators https://docs.python.org/3.4/glossary.html#term-generator Are these consistent with (your notion of) python? Maybe they are any other language? -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Tue, 17 Mar 2015 12:13 am, Marko Rauhamaa wrote: If I get an iterator from a black box source, I don't know if I'm allowed/supposed to call close() on it. In no particular order: #1 if hasattr(some_iterator, 'close'): some_iterator.close() #2 close = getattr(some_iterator, 'close', None) if close is not None: close() #3 try: close = some_iterator.close except AttributeError: pass else: close() #4 (The most pythonic solution of all.) If you want to call close(), for some reason, *make it part of your API* and insist that whatever object is passed to you supports close(), or else it is an error. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Rustom Mody rustompm...@gmail.com: On Monday, March 16, 2015 at 6:02:48 PM UTC+5:30, Marko Rauhamaa wrote: So what we have now is: (1) plain iterators (2) generator iterators (3) coroutine generator iterators (1) and (2) are not unified, which I don't like. Can you expand on that a bit? (2) contains methods, most notably close(), that (1) does not provide. If I get an iterator from a black box source, I don't know if I'm allowed/supposed to call close() on it. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Monday, March 16, 2015 at 7:10:03 PM UTC+5:30, Steven D'Aprano wrote: And of course, from a comp science theoretic perspective, generators are a kind of subroutine, not a kind of type. You just showed Marko a few posts back, that A generator is a special case of an iterator. And you wrote the iterator with 'class'. And from python 2.2(?) classes are types. So I dont know what comp science theoretic perspective you are describing... From mine: The generator def gen(): yield 1 yield 2 is much closer to the list (ie data) [1,2] than to say def foo(): print 1 print 2 The only difference is that the list memoizes the data whereas the generator doesn't. CLU perspective: The iterator for a collection of complex_numbers can be used interchangeably with that for an array of integers from https://en.wikipedia.org/wiki/CLU_%28programming_language%29 [Note: CLU-iterator ≡ python-generator] Haskell perspective: Lists are by default lazy and memoized. IOW in Haskell: List ≡ Lazy list ≡ python generator + memoization Scheme perspective: Programmer can choose between normal and lazy lists. Lazy lists are just normal lists + delay/force where delay x ≡ lambda: x force x ≡ x() == Anyways... Yes 15 years are past. I dont expect the def can be revoked now. [As far as I am concerned its a minor wart] But the mess around the docs can certainly be cleaned up. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, 16 Mar 2015 11:51 pm, Rustom Mody wrote: It may help to read this 15 year old thread starting https://mail.python.org/pipermail/python-dev/2001-June/015478.html to see how many python stalwarts dont like the current state of affairs /s/dont/didn't/ That's a fifteen year old thread, written before generators were introduced. Guido's intuition has been vindicated -- generators have proven to be a great and powerful feature, and the reuse of def for both generator functions and regular functions has turned out to be no more confusing in practice than the use of def for both functions and methods[1]. The argument is that generator-producers (def-with-yield) are not like regular functions because calling the def-with-yield doesn't execute the code inside them. Given three objects: def f(): return 1 def g(): yield 1 class C: ... the argument goes that: * Calling f() executes the code inside f. * Calling g() *doesn't* execute the code inside f, but runs some initialisation code and returns an instance; it's the instance that executes the code inside f, using a special method. * Calling C() also doesn't execute the code inside C, but runs some initialisation code and returns an instance. Therefore def-with-yield is closer to a class than a function and should not reuse the same keyword. Or so the argument goes. But this argument doesn't take into account the way generators are used. They are used like functions, not like instances[2] like this: # No instance = gen() instance.fe() instance.fi() instance.fo() instance.fum() but like functions, like this: # Yes for x in gen(): ... sum(gen()) result = map(func, gen()) Here's a pure-Python version of map from Python 2: def map(func, iterable): accum = [] for item in iterable: accum.append(func(item)) return accum Here's the same thing for the Python 3 iterator version: def map(func, iterable): for item in iterable: yield func(item) The similarities when writing these are undeniable. Drop the accumulator, replace any lines that store values into the accumulator with a yield, and drop the final return. The way we write and use generators is closer to the way we write and use functions than classes. If we rule out introducing a new keyword, and insist on picking either def or class, I think it is obvious that def is the better choice and class would be completely inappropriate. And of course, from a comp science theoretic perspective, generators are a kind of subroutine, not a kind of type. [1] Methods are, of course, actually just functions under the hood. The conversion to MethodType is done at method-lookup time, by the descriptor protocol. [2] I'm aware that they are instances. You know what I mean. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, Mar 16, 2015 at 10:51 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Chris Angelico wrote: Just for the record, it's not just name bindings that make a generator behave as a coroutine. E.g.: py def co(): ... print( (yield 1) + 2 ) ... py a = co() py next(a) 1 py a.send(98) 100 Traceback (most recent call last): File stdin, line 1, in module StopIteration Yep, I agree. It's not the name binding, it's whether the yield expression's result is instantly discarded vs used in some way. It's surprising to me that you can send() into a non-coroutine and have it ignored. I'm not even sure that is documented behaviour or just an accident of implementation. I agree conceptually; but there are two different consistencies here, and they clash, my lords, they clash! One is that the API for a coroutine includes send() but the API for a non-coroutine generator doesn't, and that trying to send() into the latter should raise an error. (Side point: What error? I'd accept AttributeError, but since it's possible to mix sendable and nonsendable yields, it might be better to have it raise at the time of the call - that is, when you send a non-None value into a generator that's just going to discard the value, it raises on arrival. In that case, possibly ValueError? GeneratorStyleError?) And on the other hand, we have a very strong Python convention that any expression can be used as a statement without altering the effect of that expression in any way. Assigning something to a name that you never use, or passing it as an argument to a no-op function, should not suddenly change the behaviour of anything. That said, though, it would be quite reasonable for a *linter* to warn you about sending into a generator that never uses sent values. With a good type inference system (not sure if MyPy is sufficient here), it would be possible to distinguish between those two functions and declare that the first one has the return type sendable_generator and the second one nonsendable_generator, and give a warning if you send into the latter. But I don't think that's the language's job. I do :-) I think it would be hard for a linter to do this. It can't just do a type-check, because the types of generator-iterators and generator-coroutines are the same. It would need to actually analyse the source code, AST, or byte-code. That's doable, but the compiler already does that, and compiles different code for the two cases: py from dis import dis py def g1(): ... yield 1 ... py def g2(): ... x = yield 1 ... py dis(g1) 2 0 LOAD_CONST 1 (1) 3 YIELD_VALUE 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE py dis(g2) 2 0 LOAD_CONST 1 (1) 3 YIELD_VALUE 4 STORE_FAST 0 (x) 7 LOAD_CONST 0 (None) 10 RETURN_VALUE so it should be easy for the compiler to recognise a yield used as if it were a statement versus one used as an expression, and take appropriate steps. But maybe there are subtle corner cases I haven't thought of. That code isn't actually all that different, though. What you have is this: 1) Load up a value 2) Yield, which pops the value off, yields it, and pushes the sent value or None 3a) in g1: Discard the value of the expression that we've just finished 3b) in g2: Store the value of that expression into x 4) Load None, and return it. Steps 1, 3, and 4 would all be identical if you replace the yield 1 with, say, print(1). It'll pop the 1 off, do something, and leave a result on the stack. The difference between assignment and non-assignment comes later. The risk I see here is that a simple optimization might suddenly make a semantic change to something. Imagine this function: def gen(): while True: x = (yield 1) if x == 5: break Okay, so it cares about sent values. Well and good. Then we change it so it doesn't always care: def gen(state): while True: x = (yield 1) if state and x == 5: break Ahh but look! That's a mode-switch parameter. We should break that out into two functions. def gen_stoppable(): while True: x = (yield 1) if x == 5: break def gen_unstoppable(): while True: x = (yield 1) Now, according to the current rules, you can change that last line to just yield 1 without the two functions differing in API. There's no sent value that will stop the second one, but it will accept and ignore sent values, just as the first one does. (Imagine that's some sort of password, and the second generator is for the case where the account is locked and there IS no password.) But according to your rules, changing it to no longer assign the yielded value somewhere would make a semantic and API change to the function. That seems, to me, at least as surprising as the case of
Re: generator/coroutine terminology
On 16.03.2015 13:02, Steven D'Aprano wrote:
(To be honest, I'm not even sure what the use-case for close() on coroutines
is in the first place. If you don't want to send any more items into it,
just don't send any more items into it.)
Just like with file-likes, it is useful to clean up resources. If you
use a coroutine with the lxml.etree.xmlfile interface [1] for instance
(code quoted from the reference):
def writer(out_stream):
with xmlfile(out_stream) as xf:
with xf.element('{http://etherx.jabber.org/streams}stream'):
try:
while True:
el = (yield)
xf.write(el)
xf.flush()
except GeneratorExit:
pass
This allows controlled (i.e. not governed by garbage collection) and
clean shutdown of a coroutine (albeit I’m not sure why they catch the
GeneratorExit here), which will close the opened stream tag.
regards,
jwi
[1]: http://lxml.de/api.html#incremental-xml-generation
signature.asc
Description: OpenPGP digital signature
--
https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Marko Rauhamaa wrote: Anyway, calling close() on an iterator can be a necessary requirement even in ordinary generator usage. Only now they have to know if the underlying iterator was sired by a generator or some other constructor. Can you give an actual example of that? (To be honest, I'm not even sure what the use-case for close() on coroutines is in the first place. If you don't want to send any more items into it, just don't send any more items into it.) -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info: (To be honest, I'm not even sure what the use-case for close() on coroutines is in the first place. If you don't want to send any more items into it, just don't send any more items into it.) Without close(), you might leave resources hanging. See PEP 325: Rejected in favor of PEP 342 which includes substantially all of the requested behavior in a more refined form. And PEP 342: Raymond Hettinger (PEP 288) and Samuele Pedroni (PEP 325) first formally proposed the ideas of communicating values or exceptions into generators, and the ability to close generators. One *might* take the stance that resource allocation is outside the scope of iterators, generator iterators and coroutine generator iterators. As it stands, it is unclear if the application *must* call close() on generator iterators that are left hanging. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Marko Rauhamaa wrote: Chris Angelico ros...@gmail.com: On Mon, Mar 16, 2015 at 6:12 PM, Marko Rauhamaa ma...@pacujo.net wrote: I was actually referring to the offered API. It still seems to me like all iterators could offer close(), send() and throw(). Why? To make all iterators drop-in replacements of each other. [...] That just adds unnecessary overhead to every iterator. Trivial implementations are a normal way to satisfy an API. That's why /dev/null implements a close() method, for example. Absolutely. But you''ll notice it doesn't implement upper() and lower() methods, because it isn't a string, it's a file. Iterators shouldn't implement send() etc. methods because they aren't coroutines. Also, what happens when you throw something into iter([1,2,3]) ? Or send it a value? What happens? I would expect iter([1, 2, 3]) to behave highly analogously to (x for x in [1, 2, 3]). I believe that it is an unfortunate mistake that generators which aren't coroutines support send. At the very least it is an ugly design wart, at worst it is an outright bug. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Chris Angelico wrote: On Mon, Mar 16, 2015 at 7:36 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: I expect that the interpreter can tell the difference between yield spam and x = yield spam and only allow send(), close() and throw() on generators which include the second form. If it can't, then that's a limitation, not a feature to be emulated. Hmm. That would imply that an expression is different if it's used somewhere. In Python, there's no difference between these two function calls: x = func(1,2,3) func(1,2,3) except for the actual name binding. That's a pretty big difference though :-) Just for the record, it's not just name bindings that make a generator behave as a coroutine. E.g.: py def co(): ... print( (yield 1) + 2 ) ... py a = co() py next(a) 1 py a.send(98) 100 Traceback (most recent call last): File stdin, line 1, in module StopIteration Not the most *useful* example, I grant, but it demonstrates the idea. However, for simplicity let's ignore that and pretend it's only name binding we care about. If a yield expression enabled send() if and only if its return value were used, then it would be extremely surprising: It's surprising to me that you can send() into a non-coroutine and have it ignored. I'm not even sure that is documented behaviour or just an accident of implementation. def use_and_discard(): while True: _ = (yield 1) def just_yield(): while True: yield 1 So... I can send into the first but not into the second? That's the idea. The second one is not a coroutine, and cannot be used as a coroutine in any useful way. I maintain that it is a mistake that you can send() into it at all, and if somebody does, it is invariably an error. That would be like send()ing into None, or a str. We know what the Zen says about errors... That said, though, it would be quite reasonable for a *linter* to warn you about sending into a generator that never uses sent values. With a good type inference system (not sure if MyPy is sufficient here), it would be possible to distinguish between those two functions and declare that the first one has the return type sendable_generator and the second one nonsendable_generator, and give a warning if you send into the latter. But I don't think that's the language's job. I do :-) I think it would be hard for a linter to do this. It can't just do a type-check, because the types of generator-iterators and generator-coroutines are the same. It would need to actually analyse the source code, AST, or byte-code. That's doable, but the compiler already does that, and compiles different code for the two cases: py from dis import dis py def g1(): ... yield 1 ... py def g2(): ... x = yield 1 ... py dis(g1) 2 0 LOAD_CONST 1 (1) 3 YIELD_VALUE 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE py dis(g2) 2 0 LOAD_CONST 1 (1) 3 YIELD_VALUE 4 STORE_FAST 0 (x) 7 LOAD_CONST 0 (None) 10 RETURN_VALUE so it should be easy for the compiler to recognise a yield used as if it were a statement versus one used as an expression, and take appropriate steps. But maybe there are subtle corner cases I haven't thought of. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info: If I .send() into either of the generators above, its a conceptual mistake and I should get an error. The fact that I don't is an implementation detail, and one which hopefully will be fixed. So what we have now is: (1) plain iterators (2) generator iterators (3) coroutine generator iterators At the moment (2) and (3) are unified, which you don't like. (1) and (2) are not unified, which I don't like. You and I can pretend (2) didn't have send(), throw() and close() and be satisfied. Then, generator iterators are just plain iterators from the point of view of the application programmer. However, it seems to me close() is intended to be used for (2) as well. The way I'm reading PEP 342, it seems to encompass the objectives of the rejected PEP 325 (Resource-Release Support for Generators): Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Monday, March 16, 2015 at 6:02:48 PM UTC+5:30, Marko Rauhamaa wrote: So what we have now is: (1) plain iterators (2) generator iterators (3) coroutine generator iterators (1) and (2) are not unified, which I don't like. Can you expand on that a bit? It may help to read this 15 year old thread starting https://mail.python.org/pipermail/python-dev/2001-June/015478.html to see how many python stalwarts dont like the current state of affairs -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Ian Kelly ian.g.ke...@gmail.com: Or inversely I write a normal utility function that is called from coroutines, then later add a yield from to it, and now I have to go back and revise every place where it's called to make those use yield from as well. That is actually quite awkward. Smooth interlocking of coroutines suggests you should be able to introduce a blocking state anywhere with a yield from statement. To avoid the problem you mention, you'd then have to give up direct function call altogether and yield from everything under the sun: # Use the supercomputer over RPC -- or not! root = yield from my_math.sqrt(x) It could be done, but it won't look like Python code anymore. It probably won't look like a computer program anymore. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Monday, March 16, 2015 at 8:07:22 PM UTC+5:30, Rustom Mody wrote: I would gladly do that if it was a minor correction here and there. But the problem is a bit deeper even though it can be kept mostly¹ in the docs and not modify any syntax/semantics of python. In particular for: def potato(x): yield x+1 tomato = potato(3) what shall we call potato and tomato. I believe this thread clearly shows that the docs are confused and inconsistent. Yet I dont see any consensus on what/how to classify tomato/potato. Function -- trouble on one side Generator -- trouble on another Iterator -- trouble on third etc And portmanteaus like generator-function are really the worst issue of all I gave the example of 'pineapple'. Steven gave another dozen examples that according to him are all ok. Combine them with his earlier: Hopefully it is clear from context what we actually mean. When in doubt, we should be explicit. So if there is no ambiguity pineapple can be apple (or is it pine)? butterfly can be butter And of course mush is much better than mushroom! If anything the last 15 years have shown that portmanteaus will be shortformed; they will be shortformed randomly and inconsistently; even in the official docs and implementation -- not to mention all over the net. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, Mar 16, 2015 at 9:09 AM, Marko Rauhamaa ma...@pacujo.net wrote: Ian Kelly ian.g.ke...@gmail.com: For generators, the descriptive keyword (yield) could be buried *anywhere* in that block. One can glance at a generator function and fail to notice that it is a generator function. This is the one that really bugs me about reuse of def, although you are correct that this is a case where practicality has won over purity. I don't think that's all that big of a deal even though I will readily admit having stumbled on it early on. I removed the last yield statement from a generator expecting it to keep on being a generator. Thus, you have to do things like: def nothing(): if False: yield None The pass and global statements make me think it might be more Pythonic to have separate syntax for the special case: def nothing(): yield not My (limited) experience with asyncio is that it also makes this a bit worse. I write a function intended to be a coroutine invoked from coroutines using yield from, and then I realize that it's not a coroutine because it never uses yield from itself. Or inversely I write a normal utility function that is called from coroutines, then later add a yield from to it, and now I have to go back and revise every place where it's called to make those use yield from as well. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Tue, 17 Mar 2015 01:35 am, Steven D'Aprano wrote: On Tue, 17 Mar 2015 01:19 am, Rustom Mody wrote: On Monday, March 16, 2015 at 7:10:03 PM UTC+5:30, Steven D'Aprano wrote: And of course, from a comp science theoretic perspective, generators are a kind of subroutine, not a kind of type. You just showed Marko a few posts back, that A generator is a special case of an iterator. And you wrote the iterator with 'class'. And from python 2.2(?) classes are types. Yes, but iterators aren't *classes*, they are instances. So I dont know what comp science theoretic perspective you are describing... http://en.wikipedia.org/wiki/Generator_(computer_programming) A generator is very similar to a function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values. However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short, a generator looks like a function but behaves like an iterator. Oops, itchy trigger finger... the next paragraph is even more important: Generators can be implemented in terms of more expressive control flow constructs, such as coroutines or first-class continuations.[2] Generators, also known as semicoroutines,[3] are a special case of (and weaker than) coroutines, in that they always yield control back to the caller (when passing a value back), rather than specifying a coroutine to jump to; see comparison of coroutines with generators. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Tue, 17 Mar 2015 01:19 am, Rustom Mody wrote: On Monday, March 16, 2015 at 7:10:03 PM UTC+5:30, Steven D'Aprano wrote: And of course, from a comp science theoretic perspective, generators are a kind of subroutine, not a kind of type. You just showed Marko a few posts back, that A generator is a special case of an iterator. And you wrote the iterator with 'class'. And from python 2.2(?) classes are types. Yes, but iterators aren't *classes*, they are instances. So I dont know what comp science theoretic perspective you are describing... http://en.wikipedia.org/wiki/Generator_(computer_programming) A generator is very similar to a function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values. However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short, a generator looks like a function but behaves like an iterator. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, Mar 16, 2015 at 8:32 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Tue, 17 Mar 2015 12:13 am, Marko Rauhamaa wrote: If I get an iterator from a black box source, I don't know if I'm allowed/supposed to call close() on it. In no particular order: #1 if hasattr(some_iterator, 'close'): some_iterator.close() #2 close = getattr(some_iterator, 'close', None) if close is not None: close() #3 try: close = some_iterator.close except AttributeError: pass else: close() Note that these fail if some_iterator is a file-like object. Those have close methods, but as part of the file api, not the iterator api, and so the owner of the object is probably not expecting you to close it for them. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Ian Kelly ian.g.ke...@gmail.com: For generators, the descriptive keyword (yield) could be buried *anywhere* in that block. One can glance at a generator function and fail to notice that it is a generator function. This is the one that really bugs me about reuse of def, although you are correct that this is a case where practicality has won over purity. I don't think that's all that big of a deal even though I will readily admit having stumbled on it early on. I removed the last yield statement from a generator expecting it to keep on being a generator. Thus, you have to do things like: def nothing(): if False: yield None The pass and global statements make me think it might be more Pythonic to have separate syntax for the special case: def nothing(): yield not Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, Mar 16, 2015 at 7:39 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Mon, 16 Mar 2015 11:51 pm, Rustom Mody wrote: It may help to read this 15 year old thread starting https://mail.python.org/pipermail/python-dev/2001-June/015478.html to see how many python stalwarts dont like the current state of affairs /s/dont/didn't/ That's a fifteen year old thread, written before generators were introduced. Guido's intuition has been vindicated -- generators have proven to be a great and powerful feature, and the reuse of def for both generator functions and regular functions has turned out to be no more confusing in practice than the use of def for both functions and methods[1]. The argument is that generator-producers (def-with-yield) are not like regular functions because calling the def-with-yield doesn't execute the code inside them. Given three objects: There is another argument, which is that for functions and classes, the descriptive keyword (def or class) is the first thing you see when reading the code. For generators, the descriptive keyword (yield) could be buried *anywhere* in that block. One can glance at a generator function and fail to notice that it is a generator function. This is the one that really bugs me about reuse of def, although you are correct that this is a case where practicality has won over purity. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 16/03/2015 14:19, Rustom Mody wrote: On Monday, March 16, 2015 at 7:10:03 PM UTC+5:30, Steven D'Aprano wrote: And of course, from a comp science theoretic perspective, generators are a kind of subroutine, not a kind of type. You just showed Marko a few posts back, that A generator is a special case of an iterator. And you wrote the iterator with 'class'. And from python 2.2(?) classes are types. So I dont know what comp science theoretic perspective you are describing... From mine: The generator def gen(): yield 1 yield 2 is much closer to the list (ie data) [1,2] than to say def foo(): print 1 print 2 The only difference is that the list memoizes the data whereas the generator doesn't. CLU perspective: The iterator for a collection of complex_numbers can be used interchangeably with that for an array of integers from https://en.wikipedia.org/wiki/CLU_%28programming_language%29 [Note: CLU-iterator ≡ python-generator] Haskell perspective: Lists are by default lazy and memoized. IOW in Haskell: List ≡ Lazy list ≡ python generator + memoization Scheme perspective: Programmer can choose between normal and lazy lists. Lazy lists are just normal lists + delay/force where delay x ≡ lambda: x force x ≡ x() == Anyways... Yes 15 years are past. I dont expect the def can be revoked now. [As far as I am concerned its a minor wart] But the mess around the docs can certainly be cleaned up. So write the patches to correct the docs, then raise the issue on the bug tracker to get the patches accepted. IIRC for doc patches you don't even have to provide diff files against the rst files, plain text will do, the core devs will do the rest for you. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Monday, March 16, 2015 at 7:57:08 PM UTC+5:30, Mark Lawrence wrote: On 16/03/2015 14:19, Rustom Mody wrote: == Anyways... Yes 15 years are past. I dont expect the def can be revoked now. [As far as I am concerned its a minor wart] But the mess around the docs can certainly be cleaned up. So write the patches to correct the docs, then raise the issue on the bug tracker to get the patches accepted. IIRC for doc patches you don't even have to provide diff files against the rst files, plain text will do, the core devs will do the rest for you. I would gladly do that if it was a minor correction here and there. But the problem is a bit deeper even though it can be kept mostly¹ in the docs and not modify any syntax/semantics of python. In particular for: def potato(x): yield x+1 tomato = potato(3) what shall we call potato and tomato. I believe this thread clearly shows that the docs are confused and inconsistent. Yet I dont see any consensus on what/how to classify tomato/potato. Function -- trouble on one side Generator -- trouble on another Iterator -- trouble on third etc ¹ Grey areas excepted eg output of help(); inspect module etc -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, Mar 16, 2015 at 1:12 AM, Marko Rauhamaa ma...@pacujo.net wrote: I was actually referring to the offered API. It still seems to me like all iterators could offer close(), send() and throw(). Why? To make all iterators drop-in replacements of each other. The purpose of close, send and throw is to implement *coroutines*, not iterators. The fact that coroutines in Python happen to be implemented as a type of iterator doesn't mean that all iterators need to have them. You're free to add these methods to non-generator iterators that you write in order to create iterators that pretend to be coroutines, but I'm having a hard time seeing what purpose would be served by this. If on the other hand these methods were to be added to the iterator protocol, it would just create unnecessary implementation overhead for the 99.99% of non-generator iterators that are under no illusions about the fact that they aren't coroutines. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info: Marko Rauhamaa wrote: What features do generator iterators provide on top of generic iterators? The biggest difference is syntactic. Here's an iterator which returns a never-ending sequence of squared numbers 1, 4, 9, 16, ... class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self Here's the same thing written as a generator: def squares(): i = 1 while True: yield i**2 i += 1 I was actually referring to the offered API. It still seems to me like all iterators could offer close(), send() and throw(). Why? To make all iterators drop-in replacements of each other. Then, you could also stop wondering what to call the thingy returned by a generator. Why, it would be an iterator. You wouldn't then have any other use for a generator than the function that returns an iterator. You could then decide if you want to reserve the name generator for functions that contain a yield statement (syntactic notion) or call any function returning an iterator a generator (semantic notion). Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, Mar 16, 2015 at 6:12 PM, Marko Rauhamaa ma...@pacujo.net wrote: I was actually referring to the offered API. It still seems to me like all iterators could offer close(), send() and throw(). Why? To make all iterators drop-in replacements of each other. Then, you could also stop wondering what to call the thingy returned by a generator. Why, it would be an iterator. You wouldn't then have any other use for a generator than the function that returns an iterator. That just adds unnecessary overhead to every iterator. Also, what happens when you throw something into iter([1,2,3]) ? Or send it a value? What happens? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Chris Angelico ros...@gmail.com: On Mon, Mar 16, 2015 at 6:12 PM, Marko Rauhamaa ma...@pacujo.net wrote: I was actually referring to the offered API. It still seems to me like all iterators could offer close(), send() and throw(). Why? To make all iterators drop-in replacements of each other. [...] That just adds unnecessary overhead to every iterator. Trivial implementations are a normal way to satisfy an API. That's why /dev/null implements a close() method, for example. Also, what happens when you throw something into iter([1,2,3]) ? Or send it a value? What happens? I would expect iter([1, 2, 3]) to behave highly analogously to (x for x in [1, 2, 3]). ===begin /tmp/test.py=== #!/usr/bin/env python3 def main(): it = (x for x in [1, 2, 3]) print(next(it)) while True: try: print(it.send(hello)) except StopIteration: break if __name__ == __main__: main() ===end /tmp/test.py= $ python3 /tmp/test.py 1 2 3 but: ===begin /tmp/test.py=== #!/usr/bin/env python3 def main(): it = iter([1, 2, 3]) print(next(it)) while True: try: print(it.send(hello)) except StopIteration: break if __name__ == __main__: main() ===end /tmp/test.py= $ python3 /tmp/test.py 1 Traceback (most recent call last): File /tmp/test.py, line 13, in module main() File /tmp/test.py, line 8, in main print(it.send(hello)) AttributeError: 'list_iterator' object has no attribute 'send' Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Ian Kelly ian.g.ke...@gmail.com: If on the other hand these methods were to be added to the iterator protocol, it would just create unnecessary implementation overhead for the 99.99% of non-generator iterators that are under no illusions about the fact that they aren't coroutines. Why should (x for x in [1, 2, 3]) be a valid coroutine but iter([1, 2, 3]) not? Anyway, calling close() on an iterator can be a necessary requirement even in ordinary generator usage. Only now they have to know if the underlying iterator was sired by a generator or some other constructor. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 16/03/2015 14:37, Rustom Mody wrote: On Monday, March 16, 2015 at 7:57:08 PM UTC+5:30, Mark Lawrence wrote: On 16/03/2015 14:19, Rustom Mody wrote: == Anyways... Yes 15 years are past. I dont expect the def can be revoked now. [As far as I am concerned its a minor wart] But the mess around the docs can certainly be cleaned up. So write the patches to correct the docs, then raise the issue on the bug tracker to get the patches accepted. IIRC for doc patches you don't even have to provide diff files against the rst files, plain text will do, the core devs will do the rest for you. I would gladly do that if it was a minor correction here and there. But the problem is a bit deeper even though it can be kept mostly¹ in the docs and not modify any syntax/semantics of python. In particular for: def potato(x): yield x+1 tomato = potato(3) what shall we call potato and tomato. I believe this thread clearly shows that the docs are confused and inconsistent. Yet I dont see any consensus on what/how to classify tomato/potato. Function -- trouble on one side Generator -- trouble on another Iterator -- trouble on third etc ¹ Grey areas excepted eg output of help(); inspect module etc So the docs are confused and inconsistent but in an open source community it's not *MY* responsibility to deal with it, somebody else can. Making mountains out of mole hills is all I see in this entire thread. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Monday 16 March 2015 18:12, Marko Rauhamaa wrote:
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info:
Marko Rauhamaa wrote:
What features do generator iterators provide on top of generic
iterators?
The biggest difference is syntactic. Here's an iterator which returns a
never-ending sequence of squared numbers 1, 4, 9, 16, ...
class Squares:
def __init__(self):
self.i = 0
def __next__(self):
self.i += 1
return self.i**2
def __iter__(self):
return self
Here's the same thing written as a generator:
def squares():
i = 1
while True:
yield i**2
i += 1
I was actually referring to the offered API. It still seems to me like
all iterators could offer close(), send() and throw(). Why? To make all
iterators drop-in replacements of each other.
How could these two iterators *possibly* be drop in replacements for each
other in any real body of code?
def inorder(tree):
for node in inorder(tree.left):
yield node.payload
yield tree.payload
for node in inorder(tree.right):
yield node.payload
def clean_up(collection, condition, cleaner=None):
if cleaner is None:
def cleaner(obj):
try:
obj.cache = {}
obj.ref = None
except Exception:
return False
return True
for i, item in enumerate(collection):
if condition(item):
flag = cleaner(item)
if flag:
yield Warn(MSG % (i, item))
You can't just substitute any random iterator for another, any more than you
can substitute any random two functions for each other.
Overly pure (in the comp sci theory sense) interfaces can be silly. Just
because the Soyez rocket and my nephew's tricycle are both instances of
Vehicle doesn't mean that I should be able to substitute one for the other.
OOP theory says they should. Reality says No Way.
Why should I, the class designer, be forced to offer send(), throw() and
close() methods on my iterators if I'm never going to use them as
coroutines? If I .send() into either of the generators above, its a
conceptual mistake and I should get an error. The fact that I don't is an
implementation detail, and one which hopefully will be fixed. I expect that
the interpreter can tell the difference between
yield spam
and
x = yield spam
and only allow send(), close() and throw() on generators which include the
second form. If it can't, then that's a limitation, not a feature to be
emulated.
Then, you could also stop wondering what to call the thingy returned by
a generator.
Apart from a few die-hards who are hoping for some sort of Académie
Française to define the One True meaning of words, most of us stopped
wondering what to call the thingy returned by a generator function years
ago. It's a generator, just like introspection of the object tells you.
Why, it would be an iterator. You wouldn't then have any
other use for a generator than the function that returns an iterator.
Lots of things are iterators. Doesn't make them generators.
You could then decide if you want to reserve the name generator for
functions that contain a yield statement (syntactic notion) or call any
function returning an iterator a generator (semantic notion).
Why would you want the second? Do you have special names for {all functions
which return strings} and {all functions which return bools}?
Functions are normally described by their *form* or *purpose*, not their
return result, e.g. inner function, public function, callback function,
reentrant function, filter function.
--
Steve
--
https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Mon, Mar 16, 2015 at 7:36 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: I expect that the interpreter can tell the difference between yield spam and x = yield spam and only allow send(), close() and throw() on generators which include the second form. If it can't, then that's a limitation, not a feature to be emulated. Hmm. That would imply that an expression is different if it's used somewhere. In Python, there's no difference between these two function calls: x = func(1,2,3) func(1,2,3) except for the actual name binding. If a yield expression enabled send() if and only if its return value were used, then it would be extremely surprising: def use_and_discard(): while True: _ = (yield 1) def just_yield(): while True: yield 1 So... I can send into the first but not into the second? That said, though, it would be quite reasonable for a *linter* to warn you about sending into a generator that never uses sent values. With a good type inference system (not sure if MyPy is sufficient here), it would be possible to distinguish between those two functions and declare that the first one has the return type sendable_generator and the second one nonsendable_generator, and give a warning if you send into the latter. But I don't think that's the language's job. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Rustom Mody wrote: On Saturday, March 14, 2015 at 11:34:27 AM UTC+5:30, Steven D'Aprano wrote: A generator (function) may be a function which returns an iterator,... I find generator-function misleading in the same way that pineapple misleadingly suggests apple that grows on pines You would be wrong. There's nothing objectionable or misleading about generator-function meaning a function which returns generators. English is very flexible like that, and compound nouns can be used in many different ways: * a greenhouse is not a house that happens to be green; * a mushroom is not a room where people mush; * a butterfly is not a fly made of butter; * a swimming pool is not a pool which swims; * but a flying squirrel is a squirrel which (almost) flies. Where a compound noun is made of two nouns, the important one can appear either at the beginning or the end: * a printer cartridge is a type of cartridge, not a type of printer; * but an attorney general is a type of attorney, not a type of general. So there is nothing unusual about generator-function being a type of function. A builtin function is a function in the builtin (or builtins -- can never remember) module A pure function is function that does not assign or mutate non-locals A Steven-function is a function that presumably Steven wrote However a generator function is a weird sort of function (at best). Not regarding it as a function is IMO more reasonable. But it is a function. You can regard it as a kind of yoghurt, if you like, but it isn't one. py def g(): ... yield 1 ... py from inspect import isfunction py isfunction(g) True -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Marko Rauhamaa wrote: Chris Angelico ros...@gmail.com: On Sun, Mar 15, 2015 at 9:15 AM, Marko Rauhamaa ma...@pacujo.net wrote: Is it necessary/useful for a Python application programmer to be conscious of the different types of iterator? What mistaken usage could arise if the application just treated all iterators as, well, iterators? If you treat them all as iterators, then you're safe, because that's the lowest common denominator. But there are a number of other iterators that have more features, including files, generators, etc. What features do generator iterators provide on top of generic iterators? The biggest difference is syntactic. Here's an iterator which returns a never-ending sequence of squared numbers 1, 4, 9, 16, ... class Squares: def __init__(self): self.i = 0 def __next__(self): self.i += 1 return self.i**2 def __iter__(self): return self Here's the same thing written as a generator: def squares(): i = 1 while True: yield i**2 i += 1 Four lines, versus eight. The iterator version has a lot of boilerplate (although some of it, the two-line __iter__ method, could be eliminated if there was a standard Iterator builtin to inherit from). Here's a more complicated example: class Randoms: def __init__(self): self.finished = False def __next__(self): x = random.random() if x 0.5: self.finished = True if self.finished: raise StopIteration else: return x def __iter__(self): return self def randoms(): x = random.random() while x 0.5: yield x x = random.random() Generators, as a rule, are significantly easier to write, understand, and debug. There's nothing they can do that can't be done with an iterator class, but the fiddly, unexciting bits related to halting and resuming and saving state are all handled for you, allowing you to concentrate on the behaviour you want, not the boilerplate. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 14/03/2015 20:14, Ian Kelly wrote: Now which should be considered definitive, the language reference or the PEP? This question is not rhetorical; I don't know the answer. Regardless of the answer though, the PEP at least illuminates the design intent of the terminology. The language reference should be the definitive guide. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Oscar Benjamin oscar.j.benja...@gmail.com: A generator is a special type of iterator that results from generator functions and generator expressions. Is it necessary/useful for a Python application programmer to be conscious of the different types of iterator? What mistaken usage could arise if the application just treated all iterators as, well, iterators? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 12 March 2015 at 16:52, Rustom Mody rustompm...@gmail.com wrote: On Thursday, March 12, 2015 at 9:58:07 PM UTC+5:30, Steven D'Aprano wrote: Rustom Mody wrote: Say I have a simple yielding function: def foo(x): yield x+1 yield x+2 And I have g = foo(2) If I look at type, g's type is 'generator' whereas foo is just plain-ol 'function.' Whereas in informal usage we say foo is a generator. Hopefully it is clear from context what we actually mean. When in doubt, we should be explicit. There is a very important 'context' where both have to exist together -- teaching beginners. It is definitely important to draw the distinction between a generator function and a generator in teaching. I would use distinct terminology in any case. It's also important where possible to use the same terminology as is used in the various different sources of documentation that your students will encounter so inventing a new term like generator factory is probably a bad idea. foo's are written to produce g's. g's come from foo-like. Better naming would help clarify -- your 'factory' is the best Ive seen so far. I dislike the term generator factory. To me it suggests something like foo below: def baz(): yield 4 def bar(): yield 6 yield 7 def foo(x): if x 4: return baz() else: return bar() The existing terminology seems fine to me: A generator function is a special kind of function containing a yield statement that always returns a generator. A generator expression is a special kind of expression that evaluates to a generator. A generator is a special type of iterator that results from generator functions and generator expressions. But the docs?!?! Hoo Boy My head spins trying to grok this https://docs.python.org/3/reference/expressions.html#generator-expressions And thats after being familiar with the origins of the idea in scheme/simula/CLU etc. Perhaps the docs can be improved. I wouldn't recommend that particular page to anyone who wasn't already familiar with the subject though. Oscar -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Sun, Mar 15, 2015 at 9:15 AM, Marko Rauhamaa ma...@pacujo.net wrote: Oscar Benjamin oscar.j.benja...@gmail.com: A generator is a special type of iterator that results from generator functions and generator expressions. Is it necessary/useful for a Python application programmer to be conscious of the different types of iterator? What mistaken usage could arise if the application just treated all iterators as, well, iterators? If you treat them all as iterators, then you're safe, because that's the lowest common denominator. But there are a number of other iterators that have more features, including files, generators, etc. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Sun, Mar 15, 2015 at 11:48 AM, Marko Rauhamaa ma...@pacujo.net wrote: Chris Angelico ros...@gmail.com: On Sun, Mar 15, 2015 at 11:15 AM, Marko Rauhamaa ma...@pacujo.net wrote: What features do generator iterators provide on top of generic iterators? You can send values into them, throw exceptions into them, and close them (which is a special case of the latter). Hm. I wonder why the distinction was made. IOW, why is iter([1, 2, 3]) not equivalent with (x for x in [1, 2, 3]) I can close the latter but not the former. You don't need to close the former. After all, yield from [1, 2, 3] works all right. That's using it as an iterable, which is not quite the same as an iterator. (Iterators are themselves iterable; iter(x) is x, for any iterator.) What you're doing there is roughly equivalent to: for x in [1, 2, 3]: yield x but with a bunch of other simplifications and guarantees for edge cases and things. So it's going to call iter() on what you give it. If you treat things as iterators, you can use all sorts of things. You don't need to distinguish the different types. The distinction is important if you want to do more than just iterate over something. def gather(): lst = [] try: while True: lst.append((yield len(lst))) except StopIteration: yield lst n = gather() next(n) 0 n.send(Hello) 1 n.send(World) 2 n.throw(StopIteration) ['Hello', 'World'] You can't do that with a simple iterator. (Not that I'm saying this is good design...) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Chris Angelico ros...@gmail.com: On Sun, Mar 15, 2015 at 11:15 AM, Marko Rauhamaa ma...@pacujo.net wrote: What features do generator iterators provide on top of generic iterators? You can send values into them, throw exceptions into them, and close them (which is a special case of the latter). Hm. I wonder why the distinction was made. IOW, why is iter([1, 2, 3]) not equivalent with (x for x in [1, 2, 3]) I can close the latter but not the former. After all, yield from [1, 2, 3] works all right. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Sun, Mar 15, 2015 at 11:15 AM, Marko Rauhamaa ma...@pacujo.net wrote: Chris Angelico ros...@gmail.com: On Sun, Mar 15, 2015 at 9:15 AM, Marko Rauhamaa ma...@pacujo.net wrote: Is it necessary/useful for a Python application programmer to be conscious of the different types of iterator? What mistaken usage could arise if the application just treated all iterators as, well, iterators? If you treat them all as iterators, then you're safe, because that's the lowest common denominator. But there are a number of other iterators that have more features, including files, generators, etc. What features do generator iterators provide on top of generic iterators? You can send values into them, throw exceptions into them, and close them (which is a special case of the latter). ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Chris Angelico ros...@gmail.com: On Sun, Mar 15, 2015 at 9:15 AM, Marko Rauhamaa ma...@pacujo.net wrote: Is it necessary/useful for a Python application programmer to be conscious of the different types of iterator? What mistaken usage could arise if the application just treated all iterators as, well, iterators? If you treat them all as iterators, then you're safe, because that's the lowest common denominator. But there are a number of other iterators that have more features, including files, generators, etc. What features do generator iterators provide on top of generic iterators? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Sat, Mar 14, 2015 at 1:54 AM, Marko Rauhamaa ma...@pacujo.net wrote: Steven D'Aprano steve+comp.lang.pyt...@pearwood.info: Marko Rauhamaa wrote: Your 'factory' is a: generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html That glossary entry is misleading, or at least incomplete, and it fails to match the way generator is used by actual Python programmers. I am an actual Python programmer (I develop Python programs) and my definitive source for Python is the documentation. If there is an error in the documentation, I would very much like it to be corrected. I think that Steven was basing his statement on the definition you cited. I don't think that he actually went and looked up the definition. If he had, he would have seen that you omitted 90% of it. Here's the full entry: A function which returns an iterator. It looks like a normal function except that it contains yield statements for producing a series of values usable in a for-loop or that can be retrieved one at a time with the next() function. Each yield temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator resumes, it picks-up where it left-off (in contrast to functions which start fresh on every invocation). A generator (function) may be a function which returns an iterator, but not all functions that return iterators are generators, and in some ways returning an iterator is the *least* interesting part of what makes a generator a generator. What distinguishes a generator from a regular function is the use of `yield`. Any definition which fails to mention that fact is useless. As can be seen above, the glossary definition *does* in fact discuss the use of yield. The language reference had better use more precise language. It *is* the definitive source, after all. Okay, but you cited the glossary entry, not the language reference. The language reference says this [1]: When a generator function is called, it returns an iterator known as a generator. Which I think is quite clear, although to be fair the same section also uses generator alone to refer to the function in the preceding paragraph. That usage however is justified by PEP 255 [2]: When a generator function is called, the actual arguments are bound to function-local formal argument names in the usual way, but no code in the body of the function is executed. Instead a generator-iterator object is returned; this conforms to the iterator protocol, so in particular can be used in for-loops in a natural way. Note that when the intent is clear from context, the unqualified name generator may be used to refer either to a generator-function or a generator-iterator. Now which should be considered definitive, the language reference or the PEP? This question is not rhetorical; I don't know the answer. Regardless of the answer though, the PEP at least illuminates the design intent of the terminology. [1] https://docs.python.org/3/reference/expressions.html#yield-expressions [2] https://www.python.org/dev/peps/pep-0255/ -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Sunday, March 15, 2015 at 1:45:37 AM UTC+5:30, Ian wrote: Now which should be considered definitive, the language reference or the PEP? This question is not rhetorical; I don't know the answer. Regardless of the answer though, the PEP at least illuminates the design intent of the terminology. [1] https://docs.python.org/3/reference/expressions.html#yield-expressions [2] https://www.python.org/dev/peps/pep-0255/ Thanks That PEP deserves careful reading, particularly the end... | BDFL Pronouncements | Issue: Introduce another new keyword (say, gen or generator) in | place of def, or otherwise alter the syntax, to distinguish | generator-functions from non-generator functions. | | Con: In practice (how you think about them), generators *are* | functions, but with the twist that they're resumable. The mechanics of | how they're set up is a comparatively minor technical issue, and | introducing a new keyword would unhelpfully overemphasize the | mechanics of how generators get started (a vital but tiny part of a | generator's life). | | Pro: In reality (how you think about them), generator-functions are | actually factory functions that produce generator-iterators as if by | magic. In this respect they're radically different from non-generator | functions, acting more like a constructor than a function, so reusing | def is at best confusing. A yield statement buried in the body is | not enough warning that the semantics are so different. | | BDFL: def it stays. No argument on either side is totally | convincing, so I have consulted my language designer's intuition. It | tells me that the syntax proposed in the PEP is exactly right - not too | hot, not too cold. But, like the Oracle at Delphi in Greek mythology, | it doesn't tell me why, so I don't have a rebuttal for the arguments | against the PEP syntax. The best I can come up with (apart from | agreeing with the rebuttals ... already made) is FUD. If this had | been part of the language from day one, I very much doubt it would have | made Andrew Kuchling's Python Warts page. So while I dont go to the extent of suggesting introducing a different keyword for generators, the fact that the overloading of def is problematic and arbitrary comes from the horse's (ie BDFL's) mouth. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Marko Rauhamaa wrote: Your 'factory' is a: generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html That glossary entry is misleading, or at least incomplete, and it fails to match the way generator is used by actual Python programmers. Here is a function which returns an iterator. According to the docs, it's a generator, but that's simply wrong. In no way, shape or form should it be considered to be a generator: def not_a_generator(): return iter([1, 2, 3]) A generator (function) may be a function which returns an iterator, but not all functions that return iterators are generators, and in some ways returning an iterator is the *least* interesting part of what makes a generator a generator. What distinguishes a generator from a regular function is the use of `yield`. Any definition which fails to mention that fact is useless. Generator is used to describe both functions with the `yield` statement, and the return result of calling such functions. Where it is necessary to distinguish the two, we call the function-with-yield a generator function. This terminology comes straight from the PEP introducing generators: https://www.python.org/dev/peps/pep-0255/ Such functions-with-yield *are* functions (or methods): py def gen(): ... yield 1 ... py type(gen) type 'function' but they're a special kind of function, distinguished by a flag on the __code__ object (also known as func_code in Python 2). The inspect module has a function to check that flag, which looks like this: def isgeneratorfunction(object): return bool((isfunction(object) or ismethod(object)) and object.func_code.co_flags CO_GENERATOR) The result of calling `gen` is an instance of the generator type, that is to say, an instance of a type which considers its own name to be generator: py x = gen() py type(x) type 'generator' Although this type is built-in, it is not available in the builtins namespace, but it is bound to the name GeneratorType in the types module. The inspect module has a function for this too: def isgenerator(object): return isinstance(object, types.GeneratorType) For brevity, I've deleted the docstring, but it is very informative to read it. Run `import inspect; help(inspect.isgenerator)` for more details. Like many English words, we have two meanings for generator: (1) A function containing the `yield` keyword, or generator-function. (2) The result of calling such a function, an instance of types.GeneratorType. PEP 255 calls that a generator-iterator, but that name doesn't appear to have caught on anywhere. If people can cope with the difference between a TV program and a computer program, they can cope with generator having two meanings, especially since we have ways to disambiguate between the two when needed. Your 'generator-instance' is an: iterator An object representing a stream of data. URL: https://docs.python.org/3/glossary.html Generator instances are iterators, but not all iterators are generator instances. Generator instances are special: they are subroutines which can be suspended and resumed, with multiple exit points (each yield is an exit point). Generators are a subset of coroutines, which are a generalization of generators. Coroutines have multiple entry points and exit points (each yield is both an entry point and exit point). CPython uses the same internal mechanism for both, and as far as I know, there is no programmatic way to distinguish a coroutine from a generator. Or at least no obvious way -- there's no `inspect.iscoroutine` function that I know of. I don't think you should read much into what str(obj) returns. It's not much more than a random printable some CPython coder has cooked up in the heat of the moment. I think that is completely wrong. The repr() and str() of generator instances is hardly random, it reflects the consensus of the PEP authors and the Python core developers, in particular the BDFL Guido who approved the PEP, that the type of object it is should be called generator. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 14/03/2015 07:54, Marko Rauhamaa wrote: Steven D'Aprano steve+comp.lang.pyt...@pearwood.info: Marko Rauhamaa wrote: Your 'factory' is a: generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html That glossary entry is misleading, or at least incomplete, and it fails to match the way generator is used by actual Python programmers. I am an actual Python programmer (I develop Python programs) and my definitive source for Python is the documentation. If there is an error in the documentation, I would very much like it to be corrected. Five minutes work for you (singular) on the bug tracker. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info: Marko Rauhamaa wrote: Your 'factory' is a: generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html That glossary entry is misleading, or at least incomplete, and it fails to match the way generator is used by actual Python programmers. I am an actual Python programmer (I develop Python programs) and my definitive source for Python is the documentation. If there is an error in the documentation, I would very much like it to be corrected. A generator (function) may be a function which returns an iterator, but not all functions that return iterators are generators, and in some ways returning an iterator is the *least* interesting part of what makes a generator a generator. What distinguishes a generator from a regular function is the use of `yield`. Any definition which fails to mention that fact is useless. The language reference had better use more precise language. It *is* the definitive source, after all. Like many English words, we have two meanings for generator: (1) A function containing the `yield` keyword, or generator-function. (2) The result of calling such a function, an instance of types.GeneratorType. PEP 255 calls that a generator-iterator, but that name doesn't appear to have caught on anywhere. If people can cope with the difference between a TV program and a computer program, they can cope with generator having two meanings, especially since we have ways to disambiguate between the two when needed. I think that is untenable when you talk about a programming language. I don't think you should read much into what str(obj) returns. It's not much more than a random printable some CPython coder has cooked up in the heat of the moment. I think that is completely wrong. The repr() and str() of generator instances is hardly random, it reflects the consensus of the PEP authors and the Python core developers, in particular the BDFL Guido who approved the PEP, that the type of object it is should be called generator. Be that as it may, URL: https://docs.python.org/3/reference/ and URL: https://docs.python.org/3/library/ are the definitive sources for Python application programmers. Or is there a better reference you would recommend? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Mark Lawrence breamore...@yahoo.co.uk: On 14/03/2015 07:54, Marko Rauhamaa wrote: I am an actual Python programmer (I develop Python programs) and my definitive source for Python is the documentation. If there is an error in the documentation, I would very much like it to be corrected. Five minutes work for you (singular) on the bug tracker. I haven't determined that there is an error. I couldn't because to me, the documentation defines right and wrong. The only sign of discord so far has been between the return value of str() and the language of the documentation. The return values of str() and repr() are not defined very precisely: object.__repr__(self) Called by the repr() built-in function to compute the “official” string representation of an object. If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment). If this is not possible, a string of the form ...some useful description... should be returned. The return value must be a string object. [...] object.__str__(self) Called by str(object) and the built-in functions format() and print() to compute the “informal” or nicely printable string representation of an object. The return value must be a string object. Thus one shouldn't read too much into what str() and repr() return. In particular, there is no requirement that two valid Python implementations ought to return similar-looking strings. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 03/14/2015 12:51 PM, Chris Angelico wrote: On Sun, Mar 15, 2015 at 3:33 AM, Rustom Mody rustompm...@gmail.com wrote: As best as I can see python makes no distinction between such a foo and the more usual function/methods that have no returns. You can I can talk about these and distinguish them Python has no clue about it. But what support is actually needed? Look, I can write functions that return values: def square(x): return x*x four = square(2) and functions that don't: def save_to_file(fn): with open(fn, w) as f: f.write(stuff) f.write(more_stuff) save_to_file(/tmp/asdf) Who cares that the second one actually returns None and then ignores that? It's not returning anything, it's not using the function return value. This is, in effect, a procedure. How is Python worse off for doing things this way rather than having a true function with no return value or procedure concept? And in C (at least the 80x86 implementations that I used) returns a value from a void function as well. The value is an undefined int value, whatever the AX (EAX) register happened to contain. The check that you didn't use such a value is made at compile time. Further, in the original C definition, if you didn't define a return value, it was assumed to be int. And if you didn't define the arguments, they were assumed to be right. In the C calling convention, the caller was responsible for popping off the arguments, since the called function didn't necessarily have a clue how many there were. Then later, for efficiency reasons, they came up with the Pascal calling convention, where the callee popped the arguments. It perhaps was related to the fact that the 80x86 instruction set had the return N instruction which decremented the call stack pointer an additional amount, at the same time as doing the return. I think the Python convention is exactly right for its other characteristics. No declarations, no distinction. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Saturday, March 14, 2015 at 10:22:51 PM UTC+5:30, Chris Angelico wrote: On Sun, Mar 15, 2015 at 3:33 AM, Rustom Mody wrote: As best as I can see python makes no distinction between such a foo and the more usual function/methods that have no returns. You can I can talk about these and distinguish them Python has no clue about it. But what support is actually needed? Look, I can write functions that return values: def square(x): return x*x four = square(2) and functions that don't: def save_to_file(fn): with open(fn, w) as f: f.write(stuff) f.write(more_stuff) save_to_file(/tmp/asdf) Who cares that the second one actually returns None and then ignores that? It's not returning anything, it's not using the function return value. This is, in effect, a procedure. How is Python worse off for doing things this way rather than having a true function with no return value or procedure concept? Well... We can talk about that... maybe in another thread... My more pressing question is that when I try to read the explanations of generators, my tongue gets stuck behind my ears. And I suspect the problem is with the terminology My original question was quite genuine: Thanks to Marko's collections from the docs and also Steven's That glossary entry is misleading, or at least incomplete, and it fails to match the way generator is used by actual Python programmers. I guess I can confirm that the docs are messy the concepts are confused and we (teachers) have to do whatever we will with that state of affairs. [I admit to even having (tried to) teach C++. But I would not try to repeat that feat if I had a choice] -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Saturday, March 14, 2015 at 9:45:10 PM UTC+5:30, Chris Angelico wrote: On Sun, Mar 15, 2015 at 2:59 AM, Rustom Mody wrote: Causing all sorts of unnecessary confusions: An int-function returns int and a char*-functions returns char*. Does a void-function return void?? No a void function doesn't return anything! Ah So a void function does a longjmp? All of which is to say that in retrospect we need (at least in imperative programming) procedures and functions. Best if the language supports them Python has a broad concept of functions/methods that return something interesting and functions/methods that always return None. (The distinction often corresponds to non-mutator and mutator methods, but that's just convention.) With due respect Chris, you are confused: Sure any effective *pythonista* (who writes useful python) will have this concept. Python (as against pythonistas) has no such concept¹ as function that ALWAYS returns None Consider this foo def foo(x): ... if x0: return x-1 ... foo(3) 2 foo(-1) As best as I can see python makes no distinction between such a foo and the more usual function/methods that have no returns. You can I can talk about these and distinguish them Python has no clue about it. Leaving aside latest type-annotation proposal. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Sun, Mar 15, 2015 at 3:33 AM, Rustom Mody rustompm...@gmail.com wrote: As best as I can see python makes no distinction between such a foo and the more usual function/methods that have no returns. You can I can talk about these and distinguish them Python has no clue about it. But what support is actually needed? Look, I can write functions that return values: def square(x): return x*x four = square(2) and functions that don't: def save_to_file(fn): with open(fn, w) as f: f.write(stuff) f.write(more_stuff) save_to_file(/tmp/asdf) Who cares that the second one actually returns None and then ignores that? It's not returning anything, it's not using the function return value. This is, in effect, a procedure. How is Python worse off for doing things this way rather than having a true function with no return value or procedure concept? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 14/03/2015 16:33, Rustom Mody wrote: On Saturday, March 14, 2015 at 9:45:10 PM UTC+5:30, Chris Angelico wrote: On Sun, Mar 15, 2015 at 2:59 AM, Rustom Mody wrote: Causing all sorts of unnecessary confusions: An int-function returns int and a char*-functions returns char*. Does a void-function return void?? No a void function doesn't return anything! Ah So a void function does a longjmp? All of which is to say that in retrospect we need (at least in imperative programming) procedures and functions. Best if the language supports them Python has a broad concept of functions/methods that return something interesting and functions/methods that always return None. (The distinction often corresponds to non-mutator and mutator methods, but that's just convention.) With due respect Chris, you are confused: Sure any effective *pythonista* (who writes useful python) will have this concept. Python (as against pythonistas) has no such concept¹ as function that ALWAYS returns None Consider this foo def foo(x): ... if x0: return x-1 ... foo(3) 2 foo(-1) As best as I can see python makes no distinction between such a foo and the more usual function/methods that have no returns. You can I can talk about these and distinguish them Python has no clue about it. Python *ALWAYS* returns None for any path through a function that doesn't specify a return value. Taking your example. def foo(x): ... if x0: return x-1 ... import dis dis.dis(foo) 2 0 LOAD_FAST0 (x) 3 LOAD_CONST 1 (0) 6 COMPARE_OP 4 () 9 POP_JUMP_IF_FALSE 20 12 LOAD_FAST0 (x) 15 LOAD_CONST 2 (1) 18 BINARY_SUBTRACT 19 RETURN_VALUE 20 LOAD_CONST 0 (None) 23 RETURN_VALUE -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Saturday, March 14, 2015 at 11:34:27 AM UTC+5:30, Steven D'Aprano wrote: A generator (function) may be a function which returns an iterator,... I find generator-function misleading in the same way that pineapple misleadingly suggests apple that grows on pines A builtin function is a function in the builtin (or builtins -- can never remember) module A pure function is function that does not assign or mutate non-locals A Steven-function is a function that presumably Steven wrote However a generator function is a weird sort of function (at best). Not regarding it as a function is IMO more reasonable. -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Sun, Mar 15, 2015 at 2:29 AM, Rustom Mody rustompm...@gmail.com wrote: However a generator function is a weird sort of function (at best). Not regarding it as a function is IMO more reasonable. But it *is* a function. You call it, possibly with positional and/or keyword arguments, and you get back a returned object. How is this different from any other function? (Yes, by this definition a class could be called an object function. But the line is already somewhat blurry; if I've read my history correctly, int used to be a function, but now it's a type. A class is really just a special type of factory function, just as a generator function is.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Saturday, March 14, 2015 at 8:59:22 PM UTC+5:30, Rustom Mody wrote: On Saturday, March 14, 2015 at 11:34:27 AM UTC+5:30, Steven D'Aprano wrote: A generator (function) may be a function which returns an iterator,... I find generator-function misleading in the same way that pineapple misleadingly suggests apple that grows on pines A builtin function is a function in the builtin (or builtins -- can never remember) module A pure function is function that does not assign or mutate non-locals A Steven-function is a function that presumably Steven wrote However a generator function is a weird sort of function (at best). Not regarding it as a function is IMO more reasonable. Another analogy somewhat closer home than pineapples. In Pascal there are procedures and functions. Even in the venerable Fortran there are subroutine-subprogram and (sub)function-subprogram. C took the stupid approach of just throwing out one of these. A decade of troubles was enough to convince people that it was needed and the mysterious 'void-returning' function was introduced to simulate procedures Causing all sorts of unnecessary confusions: An int-function returns int and a char*-functions returns char*. Does a void-function return void?? No a void function doesn't return anything! Ah So a void function does a longjmp? All of which is to say that in retrospect we need (at least in imperative programming) procedures and functions. Best if the language supports them If not, then we need them all the more in the conceptual ontology we use to build programs. Analogously we need the generator 'design-pattern' as well as the function design-pattern. Conflating one as a special case of the other is a recipe for confusion -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Sun, Mar 15, 2015 at 2:59 AM, Rustom Mody rustompm...@gmail.com wrote: Causing all sorts of unnecessary confusions: An int-function returns int and a char*-functions returns char*. Does a void-function return void?? No a void function doesn't return anything! Ah So a void function does a longjmp? All of which is to say that in retrospect we need (at least in imperative programming) procedures and functions. Best if the language supports them Python has a broad concept of functions/methods that return something interesting and functions/methods that always return None. (The distinction often corresponds to non-mutator and mutator methods, but that's just convention.) Pike has void and non-void functions, but you can sometimes trick the system into giving you the return value of a void function. (Spoiler: It'll be the integer 0.) HTTP has some responses which contain no body (eg 304 NOT MODIFIED) as well as many which do. Ditto many other languages and protocols. In each case, there's no _real_ distinction between functions and procedures (Pike follows C in having void functions, but they're still functions), and I don't see any evidence indicating that this has damaged the languages or protocols concerned - can you show otherwise? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Friday, March 13, 2015 at 1:53:50 PM UTC+5:30, Chris Angelico wrote: On Fri, Mar 13, 2015 at 4:28 PM, Rustom Mody wrote: And even there I would expect generators to close with StopIteration Whereas I would expect coroutines to close (on close method) with GeneratorExit [Ive not thought all this through properly so may be off the mark] I expect both of them to close with return. Nice demo of the same confusing terminology we are talking about. When I say expect generators to close I mean 'generator-instances' ie at runtime When you say expect both to close with return you are talking of program-texts ie the 'factory' [Or I am guilty of the same I am accusing python of] -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Rustom Mody rustompm...@gmail.com: Nice demo of the same confusing terminology we are talking about. Why don't you just stick with the terminology of the language specification? I think your students are going to be more confused if you try to come up with something better. When I say expect generators to close I mean 'generator-instances' ie at runtime When you say expect both to close with return you are talking of program-texts ie the 'factory' [Or I am guilty of the same I am accusing python of] Your 'factory' is a: generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html Your 'generator-instance' is an: iterator An object representing a stream of data. URL: https://docs.python.org/3/glossary.html I don't think you should read much into what str(obj) returns. It's not much more than a random printable some CPython coder has cooked up in the heat of the moment. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Fri, Mar 13, 2015 at 4:28 PM, Rustom Mody rustompm...@gmail.com wrote: And even there I would expect generators to close with StopIteration Whereas I would expect coroutines to close (on close method) with GeneratorExit [Ive not thought all this through properly so may be off the mark] I expect both of them to close with return. You can throw GeneratorExit into a generator, but that's an implementation detail more than anything else (same as the throwing of SystemExist is, and for the same reason). When any iterator is exhausted, next() will raise StopIteration rather than returning something; if you're simply iterating over a generator, you can treat StopIteration as an implementation detail too: def squares_up_to(top): for n in range(top): if n*n top: return yield n*n for sq in squares_up_to(50): print(sq) You don't need to be concerned about all those little details of resuming and stack frames and so on. You definitely don't need to worry about strange interactions around a with block inside the generator - you can confidently trust that everything will behave correctly. As it happens, this notion of behaving correctly is largely implemented using exceptions, but if it were implemented with a bunch of special cases in the CPython source code, it wouldn't matter. Write idiomatic source code and don't worry about the details. Learn how things work if you're curious, but it's seldom going to matter. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Fri, Mar 13, 2015 at 8:12 PM, Rustom Mody rustompm...@gmail.com wrote: On Friday, March 13, 2015 at 1:53:50 PM UTC+5:30, Chris Angelico wrote: On Fri, Mar 13, 2015 at 4:28 PM, Rustom Mody wrote: And even there I would expect generators to close with StopIteration Whereas I would expect coroutines to close (on close method) with GeneratorExit [Ive not thought all this through properly so may be off the mark] I expect both of them to close with return. Nice demo of the same confusing terminology we are talking about. When I say expect generators to close I mean 'generator-instances' ie at runtime When you say expect both to close with return you are talking of program-texts ie the 'factory' [Or I am guilty of the same I am accusing python of] Well, if you're going to discuss the specifics of terminology, we'd best figure out what close means :) You spoke of generators closing with StopIteration. Does that mean that, once a generator closes, it raises StopIteration? That's the job of the next() method, in effect. Or do you mean that raising StopIteration will terminate the generator? That won't be true forever, and shouldn't be relied on - just use return. If you use the close() method, Python throws a GeneratorExit into your generator. But unless you have something that catches that, you shouldn't need to worry about it; all you need to know is that you call close() and the function cleanly terminates. Like I said, there are implementation details that don't usually matter. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
generator/coroutine terminology
This is more a question about standard terminology/conventions than about semantics - of course assuming I understand :-) Say I have a simple yielding function: def foo(x): yield x+1 yield x+2 And I have g = foo(2) If I look at type, g's type is 'generator' whereas foo is just plain-ol 'function.' Whereas in informal usage we say foo is a generator. So the question: What should we call foo and what should we call g? Same applies when foo is a 'coroutine' ie something having yield used in an rhs and used with '.send' from outside: What to call foo and what to call foo(x)? -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Fri, Mar 13, 2015 at 12:35 AM, Rustom Mody rustompm...@gmail.com wrote: If I look at type, g's type is 'generator' whereas foo is just plain-ol 'function.' Whereas in informal usage we say foo is a generator. So the question: What should we call foo and what should we call g? g is a generator object; foo is a generator function - a function which returns generator objects. Usually, calling both of them generators isn't confusing, as there's not often any context in which they're ambiguous. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Thursday, March 12, 2015 at 1:35:48 PM UTC, Rustom Mody wrote: This is more a question about standard terminology/conventions than about semantics - of course assuming I understand :-) Say I have a simple yielding function: def foo(x): yield x+1 yield x+2 And I have g = foo(2) If I look at type, g's type is 'generator' whereas foo is just plain-ol 'function.' Whereas in informal usage we say foo is a generator. So the question: What should we call foo and what should we call g? Same applies when foo is a 'coroutine' ie something having yield used in an rhs and used with '.send' from outside: What to call foo and what to call foo(x)? Try the glossary https://docs.python.org/3/glossary.html If this comes out badly please free to shout as I'm on gg :) -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Friday, March 13, 2015 at 9:00:17 AM UTC+5:30, Steven D'Aprano wrote: Rustom Mody wrote: On Thursday, March 12, 2015 at 11:25:32 PM UTC+5:30, Marko Rauhamaa wrote: Rustom Mody : I guess we need 1. A clear ontology of the base concepts (which is a buzzword for nailed-down terminology) According to the documentation, a function whose definition contains a yield statement is a generator: Using a yield expression in a function's body causes that function to be a generator. URL: https://docs.python.org/3/reference/expressions.html#yield-exp ressions generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html#term-generator Apparently, what we have here is a discrepancy between the documentation and the CPython implementation. In a word, it's a bug (somewhere). Throwing out some thought for better terminology. [Yeah needs to be chewed on a bit...] With respect to my example above: def foo(x): yield x+1 yield x+2 g = foo(2) Generator-instance: g Generator-factory: foo Generator: Concept in python (ontology) elaborated in terms of the above I think that is is reasonable. Good that we agree! Where the distinction is clear, or not important, than calling both g and foo generators is acceptable shorthand. I realise that can be confusing to beginners, but jargon often is confusing to beginners until they have learned enough to be able to correctly interpret things in context. I would prefer -- when shortforming is required: generator-instance - instance generator-factory - factory rather than generator-instance - generator generator-factory - generator Yeah that can quarrel with the more usual OO notion of instance, but I find that ambiguity more remote Situation with coroutines is worse [Or I cant think of good terms...] Not really. The difference between a coroutine and a generator is mostly in usage, that is, intent. They both use the same type, they both have send methods, they both are generated the same way. If you are foolish, you can even mix generator-behaviour and coroutine-behaviour in the same function: py def g(): ... yield 1 ... yield 2 ... x = yield 3 ... yield x + 1000 ... py a = g() py next(a) 1 py next(a) 2 py next(a) 3 py a.send(99) 1099 But don't do that. So as far as Python is concerned, a coroutine is just a generator instance which you use in a particular way, not a different kind of object. So I would use similar terminology: A generator function/factory returns a generator instance, which we use as a coroutine, so we can call it a coroutine. I think coroutines are intended to be symmetric in a way that generators are not. The nut-n-bolts of both may involve using yield but conceptually I find them different. And even there I would expect generators to close with StopIteration Whereas I would expect coroutines to close (on close method) with GeneratorExit [Ive not thought all this through properly so may be off the mark] -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Rustom Mody wrote: On Thursday, March 12, 2015 at 11:25:32 PM UTC+5:30, Marko Rauhamaa wrote: Rustom Mody : I guess we need 1. A clear ontology of the base concepts (which is a buzzword for nailed-down terminology) According to the documentation, a function whose definition contains a yield statement is a generator: Using a yield expression in a function's body causes that function to be a generator. URL: https://docs.python.org/3/reference/expressions.html#yield-exp ressions generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html#term-generator Apparently, what we have here is a discrepancy between the documentation and the CPython implementation. In a word, it's a bug (somewhere). Throwing out some thought for better terminology. [Yeah needs to be chewed on a bit...] With respect to my example above: def foo(x): yield x+1 yield x+2 g = foo(2) Generator-instance: g Generator-factory: foo Generator: Concept in python (ontology) elaborated in terms of the above I think that is is reasonable. I would include generator function as a synonym for generator factory. (Factories are more inclusive than functions -- a factory could include a class with a make_generator method, for example.) Where the distinction is clear, or not important, than calling both g and foo generators is acceptable shorthand. I realise that can be confusing to beginners, but jargon often is confusing to beginners until they have learned enough to be able to correctly interpret things in context. Situation with coroutines is worse [Or I cant think of good terms...] Not really. The difference between a coroutine and a generator is mostly in usage, that is, intent. They both use the same type, they both have send methods, they both are generated the same way. If you are foolish, you can even mix generator-behaviour and coroutine-behaviour in the same function: py def g(): ... yield 1 ... yield 2 ... x = yield 3 ... yield x + 1000 ... py a = g() py next(a) 1 py next(a) 2 py next(a) 3 py a.send(99) 1099 But don't do that. So as far as Python is concerned, a coroutine is just a generator instance which you use in a particular way, not a different kind of object. So I would use similar terminology: A generator function/factory returns a generator instance, which we use as a coroutine, so we can call it a coroutine. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On Thursday, March 12, 2015 at 11:25:32 PM UTC+5:30, Marko Rauhamaa wrote: Rustom Mody : I guess we need 1. A clear ontology of the base concepts (which is a buzzword for nailed-down terminology) According to the documentation, a function whose definition contains a yield statement is a generator: Using a yield expression in a function's body causes that function to be a generator. URL: https://docs.python.org/3/reference/expressions.html#yield-exp ressions generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html#term-generator Apparently, what we have here is a discrepancy between the documentation and the CPython implementation. In a word, it's a bug (somewhere). Throwing out some thought for better terminology. [Yeah needs to be chewed on a bit...] With respect to my example above: def foo(x): yield x+1 yield x+2 g = foo(2) Generator-instance: g Generator-factory: foo Generator: Concept in python (ontology) elaborated in terms of the above Situation with coroutines is worse [Or I cant think of good terms...] -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Rustom Mody wrote: This is more a question about standard terminology/conventions than about semantics - of course assuming I understand :-) Say I have a simple yielding function: def foo(x): yield x+1 yield x+2 And I have g = foo(2) If I look at type, g's type is 'generator' whereas foo is just plain-ol 'function.' Whereas in informal usage we say foo is a generator. Hopefully it is clear from context what we actually mean. When in doubt, we should be explicit. So the question: What should we call foo and what should we call g? foo is a generator function, i.e. a function which returns a generator. I'd also allow generator factory, naming it by intent rather than type. The result of calling foo, namely foo(x), which you have named g, is a generator object, which is a kind of iterator. Same applies when foo is a 'coroutine' ie something having yield used in an rhs and used with '.send' from outside: What to call foo and what to call foo(x)? foo is a generator function, and foo(x) is a generator object. In this case we can call foo(x) by usage rather than type, and call it a coroutine. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Guess I should be pleased that I am doing as good as you (and Chris) describe. For some reason or not I am not... On Thursday, March 12, 2015 at 9:58:07 PM UTC+5:30, Steven D'Aprano wrote: Rustom Mody wrote: This is more a question about standard terminology/conventions than about semantics - of course assuming I understand :-) Say I have a simple yielding function: def foo(x): yield x+1 yield x+2 And I have g = foo(2) If I look at type, g's type is 'generator' whereas foo is just plain-ol 'function.' Whereas in informal usage we say foo is a generator. Hopefully it is clear from context what we actually mean. When in doubt, we should be explicit. There is a very important 'context' where both have to exist together -- teaching beginners. foo's are written to produce g's. g's come from foo-like. Better naming would help clarify -- your 'factory' is the best Ive seen so far. But the docs?!?! Hoo Boy My head spins trying to grok this https://docs.python.org/3/reference/expressions.html#generator-expressions And thats after being familiar with the origins of the idea in scheme/simula/CLU etc. Noobs could be forgiven for doing worse dont you think?? I guess we need 1. A clear ontology of the base concepts (which is a buzzword for nailed-down terminology) 2. Some neat pictures would sure help (me!) -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Terry Reedy tjre...@udel.edu: On 3/12/2015 9:35 AM, Rustom Mody wrote: This is more a question about standard terminology/conventions than about semantics - of course assuming I understand :-) Say I have a simple yielding function: def foo(x): yield x+1 yield x+2 This is a generator function Which in the official language specification is an elaborate synonym of a generator. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
On 3/12/2015 9:35 AM, Rustom Mody wrote: This is more a question about standard terminology/conventions than about semantics - of course assuming I understand :-) Say I have a simple yielding function: def foo(x): yield x+1 yield x+2 This is a generator function And I have g = foo(2) If I look at type, g's type is 'generator' whereas foo is just plain-ol 'function.' Whereas in informal usage we say foo is a generator. I do not, because it does cause confusion, in spite of denials by people who use 'generator' ambiguously. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: generator/coroutine terminology
Rustom Mody rustompm...@gmail.com: I guess we need 1. A clear ontology of the base concepts (which is a buzzword for nailed-down terminology) According to the documentation, a function whose definition contains a yield statement is a generator: Using a yield expression in a function’s body causes that function to be a generator. URL: https://docs.python.org/3/reference/expressions.html#yield-exp ressions generator A function which returns an iterator. URL: https://docs.python.org/3/glossary.html#term-generator Apparently, what we have here is a discrepancy between the documentation and the CPython implementation. In a word, it's a bug (somewhere). Marko -- https://mail.python.org/mailman/listinfo/python-list
