Re: [Python-Dev] yield back-and-forth?
Christian Tanzer wrote: How about: def main_generator(): ... yield * sub_generator() Ducking-ly yrs, I like that one, but I'd stick the star to the generator (e.g. yield *sub_generator), the meaning being to unpack the generator into the yield, same as unpacking a sequence passed as function parameters. As an extension, the syntax may even (though I'm not sure it'd be a good idea to do so) work with any iterable, behaving the same way (yielding the successive values of the iterable object) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] yield back-and-forth?
The discussion about PEP 343 reminds me of the following. Bram Cohen pointed out in private email that, before PEP 342, there wasn't a big need for a shortcut to pass control to a sub-generator because the following for-loop works well enough: def main_generator(): ... for value in sub_generator(): yield value but now that yield can return a value, that value might have to be passed into sub_generator(), not to mention of exceptions. I'm sure there's a way to write that (although I haven't found the time to figure it out) but I expect it to be cumbersome and subtle. I don't think a helper function can be created to solve this problem, because the yield syntax is essential. Bram suggested the following syntax: def main_generator(): ... yieldthrough sub_generator() I'm not keen on that particular keyword, but I do believe a syntactic solution is needed, if the problem is important enough to be solved. Thoughts? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yield back-and-forth?
Guido van Rossum [EMAIL PROTECTED] wrote: The discussion about PEP 343 reminds me of the following. Bram Cohen pointed out in private email that, before PEP 342, there wasn't a big need for a shortcut to pass control to a sub-generator because the following for-loop works well enough: def main_generator(): ... for value in sub_generator(): yield value For small values of `well enough`. I sometimes override a generator in a subclass and it sucks to add the loop just because the derived generator wants to add a value at the beginning or end. but now that yield can return a value, that value might have to be passed into sub_generator(), not to mention of exceptions. I'm sure there's a way to write that (although I haven't found the time to figure it out) but I expect it to be cumbersome and subtle. I don't think a helper function can be created to solve this problem, because the yield syntax is essential. Bram suggested the following syntax: def main_generator(): ... yieldthrough sub_generator() I'm not keen on that particular keyword, but I do believe a syntactic solution is needed, if the problem is important enough to be solved. How about: def main_generator(): ... yield * sub_generator() Ducking-ly yrs, -- Christian Tanzerhttp://www.c-tanzer.at/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yield back-and-forth?
At 10:17 AM 01/20/2006 -0800, Guido van Rossum wrote: The discussion about PEP 343 reminds me of the following. Bram Cohen pointed out in private email that, before PEP 342, there wasn't a big need for a shortcut to pass control to a sub-generator because the following for-loop works well enough: def main_generator(): ... for value in sub_generator(): yield value but now that yield can return a value, that value might have to be passed into sub_generator(), not to mention of exceptions. I'm sure there's a way to write that (although I haven't found the time to figure it out) but I expect it to be cumbersome and subtle. I don't think a helper function can be created to solve this problem, because the yield syntax is essential. Bram suggested the following syntax: def main_generator(): ... yieldthrough sub_generator() I'm not keen on that particular keyword, but I do believe a syntactic solution is needed, if the problem is important enough to be solved. What's the use case for this? In the coroutine use case, the PEP 342 sample trampoline takes care of this. If you yield a sub-generator (i.e. 'yield sub_generator()'), the trampoline effectively replaces the parent with the child until the child is complete. So, that leaves only non-coroutine use cases, and I'm having a hard time thinking of any where there would be bidirectional communication. Thoughts? If we have to have a syntax, yield from sub_generator() seems clearer than yieldthrough, and doesn't require a new keyword. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yield back-and-forth?
On 1/20/06, Phillip J. Eby [EMAIL PROTECTED] wrote: At 10:17 AM 01/20/2006 -0800, Guido van Rossum wrote: The discussion about PEP 343 reminds me of the following. Bram Cohen pointed out in private email that, before PEP 342, there wasn't a big need for a shortcut to pass control to a sub-generator because the following for-loop works well enough: def main_generator(): ... for value in sub_generator(): yield value but now that yield can return a value, that value might have to be passed into sub_generator(), not to mention of exceptions. I'm sure there's a way to write that (although I haven't found the time to figure it out) but I expect it to be cumbersome and subtle. I don't think a helper function can be created to solve this problem, because the yield syntax is essential. Bram suggested the following syntax: def main_generator(): ... yieldthrough sub_generator() I'm not keen on that particular keyword, but I do believe a syntactic solution is needed, if the problem is important enough to be solved. What's the use case for this? In the coroutine use case, the PEP 342 sample trampoline takes care of this. If you yield a sub-generator (i.e. 'yield sub_generator()'), the trampoline effectively replaces the parent with the child until the child is complete. That's a rather specialized, subtle and elaborate framework though, and at this time I believe it isn't planned to be part of Python 2.5 (right?). So, that leaves only non-coroutine use cases, and I'm having a hard time thinking of any where there would be bidirectional communication. Any other use of generators where the return value of yield is used; as soon as you do this you may later want to refactor the code to to include sub-iterators. (There *are*other uses besides the trampoline, right? :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yield back-and-forth?
Guido van Rossum [EMAIL PROTECTED] wrote: The discussion about PEP 343 reminds me of the following. Bram Cohen pointed out in private email that, before PEP 342, there wasn't a big need for a shortcut to pass control to a sub-generator because the following for-loop works well enough: def main_generator(): ... for value in sub_generator(): yield value This is an important programming trick. It allows recursive use of generators for walking trees, etc. See the source code for os.walk for an example. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yield back-and-forth?
At 11:19 AM 01/20/2006 -0800, Guido van Rossum wrote: (There *are*other uses besides the trampoline, right? :-) It's easy to come up with use cases where you feed data *into* a generator (parsers and pipelines, for example). I just don't know of any simultaneous bidirectional uses other than in association with a coroutine scheduler, or a dedicated pipelining tool. In both the coroutine and pipelining cases, generator composition is an easy job for whatever coroutine or pipelining library is in use; it just happens external to the generator, perhaps by yielding a special value. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yield back-and-forth?
Phillip J. Eby wrote: Thoughts? If we have to have a syntax, yield from sub_generator() seems clearer than yieldthrough, and doesn't require a new keyword. Andrew Koenig suggested the same phrasing last year [1], and I liked it then. I don't like it any more, though, as I think it is too inflexible, and we have a better option available (specifically, stealing continue with an argument from PEP 340). The following ramblings (try to) explain my reasoning :) Guido does raise an interesting point. The introduction of send and throw means that the current simple loop approach does not easily allow values to be passed down to nested generators, nor does not it correctly terminate nested generators in response to an invocation of throw. Because of the explicit for loop, the nested generator only gets cleaned up in response to GC - it never sees the exception that occurs in the body of the for loop (at the point of the yield expression). The yield from iterable concept could be translated roughly as follows: itr = iter(iterable) try: send_input = itr.send # Can input values be passed down? except AttributeError: send_input = None try: next = itr.next() # Get the first output except StopIteration: pass else: while 1: try: input = yield next # yield and get input except: try: throw_exc = itr.throw # Can exception be passed down? except AttributeError: raise # Nope, reraise else: throw_exc(sys.exc_info()) # Yep, pass it down else: try: if send_input is None: if input is not None: raise TypeError(Cannot send input!) next = itr.next() else: next = send_input(input) # Pass input down except StopIteration: break I'm not particularly happy with this, though, as not only is it horribly implicit and magical, it's trivial to accidentally break the chain - consider what happens if you naively do: yield from (x*x for x in sub_generator()) The chain has been broken - the sub generator no longer sees either passed in values or thrown exceptions, as the generator expression intercepts them without passing them down. Even worse, IMO, is that the syntax is entirely inflexible - we have no easy way to manipulate either the results sent from the generator, or the input values passed to it. However, an idea from Guido's PEP 340 helps with the send part of the story, involving passing an argument to continue: def main_generator(): ... for value in sub_generator(): continue yield value Here, sub_generator's send method would be invoked with the result of the call to yield value. Manipulation in either direction (input or output) is trivial: def main_generator(): ... for value in sub_generator(): input = yield value*value # Square the output values continue input*input# Square the input values, too You could even do odd things like yield each value twice, and then pass down pairs of inputs: def main_generator(): ... for value in sub_generator(): continue (yield value), (yield value) The need to use a continue statement eliminates the temptation to use a generator expression, and makes it far less likely the downwards connection between the main generator and the sub generator will be accidentally broken. Exception propagation is a different story. What do you want to propagate? All exceptions from the body of the for loop? Or just those from the yield statement? Well, isn't factoring out exception processing part of what PEP 343 is for? # Simply make sure the generator is closed promptly def main_generator(): ... with closing(sub_generator()) as gen: for value in gen: continue yield value # Or throw the real exception to the nested generator class throw_to(object): def __init__(self, gen): self.gen = gen def __enter__(self): return self.gen def __exit__(self, exc_type, *exc_details): if exc_type is not None: try: self.gen.throw(exc_type, *exc_details) except StopIteration: pass def main_generator(): ... with throw_to(sub_generator()) as gen: for value in gen: continue yield value # We can even limit the propagated exceptions to those # from the outside world and leave the rest alone def main_generator(): ... gen = sub_generator() for value in gen: with throw_to(gen): input = yield value continue
Re: [Python-Dev] yield back-and-forth?
Nick Coghlan wrote: Exception propagation is a different story. What do you want to propagate? All exceptions from the body of the for loop? Or just those from the yield statement? Well, isn't factoring out exception processing part of what PEP 343 is for? # We can even limit the propagated exceptions to those # from the outside world and leave the rest alone def main_generator(): ... gen = sub_generator() for value in gen: with throw_to(gen): input = yield value continue input A point I should have made here (but didn't, despite using it in a later example) is that the with versions don't allow the nested generator to suppress the exception. A small refinement to gen.throw() that makes the first argument optional (similar to the raise statement itself) would make it straightforward to allow the generator to suppress passed in exceptions: def main_generator(): ... gen = sub_generator() for value in gen: try: input = yield value except: gen.throw() # Default to sys.exc_info(), just like raise continue input Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yield back-and-forth?
The discussion about PEP 343 reminds me of the following. Bram Cohen pointed out in private email that, before PEP 342, there wasn't a big need for a shortcut to pass control to a sub-generator because the following for-loop works well enough: def main_generator(): ... for value in sub_generator(): yield value but now that yield can return a value, that value might have to be passed into sub_generator(), not to mention of exceptions. I'm sure there's a way to write that (although I haven't found the time to figure it out) but I expect it to be cumbersome and subtle. It looks to me like continuations are starting to rear their heads... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] yield back-and-forth?
On Jan 20, 2006, at 1:39 PM, Phillip J. Eby wrote:
At 11:19 AM 01/20/2006 -0800, Guido van Rossum wrote:
(There *are*other uses besides the trampoline,
right? :-)
It's easy to come up with use cases where you feed data *into* a
generator
(parsers and pipelines, for example). I just don't know of any
simultaneous bidirectional uses other than in association with a
coroutine scheduler, or a dedicated pipelining tool. In both the
coroutine
Hmm, what about an iterator of iterators, where the sub-iterators
need to be advanced until some condition is satisfied, then processed
from that point onwards (only those subiterators for which some item
did satisfy the condition). E.g.:
goodfiles = []
for afile in manyfiles:
for aline in afile:
if aline.startswith('*starthere*'):
goodfiles.append(afile)
break
for afile in goodfiles: ...
I'm sure we've all met this kind of issue (indeed, standard library
module fileinput does support this case directly, by wrapping the
double loop into one yielding lines, and still providing ways to get
the current file and skipping to the next file).
It might be nice to wrap the general logic (double nested looping c)
in a generator leaving the specifics (exactly when to consider a
subiterator 'good' and what to do in that case) out of it, sort of
like fileinput does but with more generality than just files and
lines thereof. In Python 2.5 it seems that a .send call with a non-
None argument might be a natural convention for the loop body to tell
the generator abandon this subiterator and yield it, then move to
the next one. E.g.:
goodfiles = []
allinput = inputfrom(manyfiles)
for aline in allinput:
if aline.startswith('*starthere*'):
goodfiles.append(allinput.send(True))
for afile in goodfiles: ...
Perhaps not the greatest of simplifications, but it might be
generalized to (e.g.) recursive walks with the same client-logic as
above, while the original (nested for-loops) is pretty rigid. And
the inputfrom generator doesn't look too complicated either:
def inputfrom(iterators):
for iterator in iterators:
for item in iterator:
if (yield item) is not None:
yield iterator
break
Wouldn't this be considered a reasonable use for the new generator
features?
Alex
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
