Re: revive a generator
On Oct 21, 12:09 pm, Yingjie Lan wrote: > Secondly, it would be nice to automatically revive it. Sure, it's always nice when your expectation of a language feature exactly matches with its capabilities. When it doesn't, you suck it up and code around it. Because at the very least it's a hell of a lot quicker than waiting for the language to change for you. -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On Oct 21, 11:46 am, Yingjie Lan wrote: > I am still not sure why should we enforce that > a generator can not be reused after an explicit > request to revive it? No one is "enforcing" anything, you're simply resisting implementing this yourself. Consider the following generator: import random def randgen(): for _ in xrange(10): yield random.choice(['Hi','Lo']) >>> [x for x in randgen()] ['Hi', 'Hi', 'Lo', 'Hi', 'Lo', 'Lo', 'Lo', 'Lo', 'Hi', 'Hi'] What would you expect when you reset that generator? A newly randomised set of values, or the _same_ set of values? What if the generator is reading from an external source, like temperature values? Once revived, should it return the exact same sequence it originally did, or should it retrieve new values? Now, no matter which decision you made, why is your expectation of behaviour the right one? Why should the generator protocol support your convience in particular? If you need your generator to be re-usable, make a factory that creates it. -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On Thursday, October 20, 2011 6:23:50 AM UTC-7, Yingjie Lan wrote: > Hi, > > it seems a generator expression can be used only once: > > >>> g = (x*x for x in range(3)) > >>> for x in g: print x > 0 > 1 > 4 > >>> for x in g: print x #nothing printed > >>> > > Is there any way to revive g here? Revive is the wrong word for what you want. Once an iterator (be it a generator or some other kind of iterator) is done, it's done. What you are asking for is, given a generator, to create a new generator from the same expression/function that created the original generator. This is not reviving, but recreating. I have two objections to this: a major ideological one and a minor practical one. The practical drawback to allowing generators to be recreated is that it forces all generators to carry around a reference to the code object that created it. if random.random() > 5: g = (x*x for x in xrange(3)) else: g = (x+x for x in xrange(3)) for y in g: print x revive(g) # which generator expression was it? # need to carry around a reference to be able to tell for y in g: print x Carrying a reference to a code object in turn carries around any closures used in the generator expression or function, so it can potentially keep a large amount of data alive. Given that the vast majority of generators would never be recreated, this is quite wasteful. My ideological objection is that it forces the programmer to be wary of the effects of recreation. Right now, if someone writes a generator expression, they can rely on the fact that it can only be iterated through once (per time the generator expression is evaluated). But if you allow a downstream user to recreate the generator at will, then the writer will always have to be wary of adverse side-effects if the generator is iterated through twice. So, although I can see it being occasionally useful, I'm going to opine that it is more trouble than it's worth. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On Fri, 21 Oct 2011 16:25:47 -0400, Terry Reedy wrote: > Here is a class that creates a re-iterable from any callable, such as a > generator function, that returns an iterator when called, + captured > arguments to be given to the function. > > class reiterable(): >def __init__(self, itercall, *args, **kwds): > self.f = itercall # callable that returns an iterator > self.args = args > self.kwds = kwds >def __iter__(self): > return self.f(*self.args, **self.kwds) > > def squares(n): > for i in range(n): > yield i*i > > sq3 = reiterable(squares, 3) We can do that even more simply, using a slightly different interface. >>> from functools import partial >>> sq3 = partial(squares, 3) sq3 is now an iterator factory. You can't iterate over it directly, but it's easy to restart: just call it to return a fresh iterator. >>> list(sq3()) [0, 1, 4] >>> list(sq3()) [0, 1, 4] -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
Here is a class that creates a re-iterable from any callable, such as a generator function, that returns an iterator when called, + captured arguments to be given to the function. class reiterable(): def __init__(self, itercall, *args, **kwds): self.f = itercall # callable that returns an iterator self.args = args self.kwds = kwds def __iter__(self): return self.f(*self.args, **self.kwds) def squares(n): for i in range(n): yield i*i sq3 = reiterable(squares, 3) for i in sq3: print(i) for i in sq3: print(i) >>> 0 1 4 0 1 4 -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On Fri, Oct 21, 2011 at 2:02 AM, Yingjie Lan wrote: > Oops, my former reply has the code indentation messed up > by the mail system. Here is a reformatted one: > > > What if the generator involves a variable from another scope, > and before re-generating, the variable changed its value. > Also, the generator could be passed in as an argument, > so that we don't know its exact expression. In the former case, use a named generator function and call it twice to create two generators. In the latter case, don't pass in the generator as an argument. Pass in a callable that constructs the iterator instead. Modifying your example: vo = 34 def mygen(): for x in range(3): yield vo * x def myfun(g): global vo for i in g(): print(i) vo += 3 for i in g(): print(i) myfun(mygen) Cheers, Ian -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On Thu, 20 Oct 2011 19:09:42 -0700, Yingjie Lan wrote: >> Here's an example of an explicit request to revive the generator: > > > g = (x*x for x in range(3)) > for x in g: print x >> 0 >> 1 >> 4 > g = (x*x for x in range(3)) # revive the generator for x in g: > print x #now this will work >> 0 >> 1 >> 4 >> >> ChrisA > > > What if the generator is passed in as an argument when you are writing a > function? That is, the expression is not available? If the expression is not available, how do you expect to revive it? The expression is gone, it no longer exists. As you said in another post: "What if the generator involves a variable from another scope, and before re-generating, the variable changed its value." Exactly. In general, you *can't* revive general iterators. It simply isn't possible. The variables that defined it might be gone. They might be non-deterministic: random numbers, data from the Internet or a file system that has changed, or user input. Trying to enforce the rule "iterators must support restarting" is foolish: it can't be done. You use an iterator when you want to iterate over something *once*, that is why they exist. If you want to iterate over it twice, don't use an iterator, use a sequence. Forcing all iterators to save their data, on the off-chance that maybe somebody might want to iterate over it twice, defeats the purpose of an iterator. > Secondly, it would be nice to automatically revive it. For example, when > another for-statement or other equivalent is applied to it. Nice? No, it would be horrible. It goes against the basic concept of an iterator: iterators should be memory efficient, generating values lazily. If you want an iterable sequence that you can iterate over multiple times, then use a list, or a custom iterable class. If you want a socket wrench, use a socket wrench. Don't insist that hammers have to have a socket wrench attachment. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On 10/20/2011 10:09 PM, Yingjie Lan wrote: What if the generator is passed in as an argument when you are writing a function? That is, the expression is not available? Secondly, it would be nice to automatically revive it. For example, when another for-statement or other equivalent is applied to it. Yingjie That last point would definitely be incompatible. It's quite useful to be able to peel some values off a generator, then use a for loop to get the remainder. Simplest example I can think of would be to read a file where the first line(s) represent header information, and the body is obtainable with a simple loop. I wouldn't be surprised if the csv module does that. Not arguing whether an explicit revival is useful or not. Although as others have pointed out, not all generators could accomplish it, and in some cases not unambiguously. -- DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
- Original Message -
> From: Chris Angelico
> To: python-list@python.org
> Cc:
> Sent: Friday, October 21, 2011 4:27 PM
> Subject: Re: revive a generator
>
> On Fri, Oct 21, 2011 at 7:02 PM, Yingjie Lan wrote:
>> What if the generator involves a variable from another scope,
>> and before re-generating, the variable changed its value.
>> Also, the generator could be passed in as an argument,
>> so that we don't know its exact expression.
>>
>
> There's actually no way to know that the generator's even
> deterministic. Try this, for instance:
>
>>>> g=(input("Enter value %d or blank to stop: "%n) for n in
> range(1,11))
>>>> for s in g:
> if not s: break
> print("Processing input: "+s)
>
> It may not be particularly useful, but it's certainly legal. And this
> generator cannot viably be restarted.
Depends on what you want. If you want ten more inputs from user,
reviving this generator is certainly a good thing to do.
> The only way is to cast it to
> list first, but that doesn't work when you have to stop reading
> expressions from the generator part way.
>
> What you could perhaps do is wrap the generator in something that
> saves its values:
>
>>>> class restartable(object):
> def __init__(self,gen):
> self.gen=gen
> self.yielded=[]
> self.iter=iter(self.yielded)
> def restart(self):
> self.iter=iter(self.yielded)
> def __iter__(self):
> return self
> def __next__(self): # Remove the underscores for Python 2
> try:
> return self.iter.__next__()
> except StopIteration:
> pass
> ret=self.gen.__next__()
> self.yielded.append(ret)
> return ret
>
>>>> h=restartable(g)
>>>> for i in h:
> if not i: break
> print("Using: ",i)
>>>> h.restart()
>>>> for i in h:
> if not i: break
> print("Using: ",i)
>
> Complicated, but what this does is returns a value from its saved list
> if there is one, otherwise returns a value from the original
> generator. It can be restarted as many times as necessary, and any
> time you read "past the end" of where you've read so far, the
> original
> generator will be called upon.
>
> Actually, this model might be useful for a repeatable random-number
> generator. But those are more efficiently restarted by means of
> reseeding the PRNG.
>
Sure. Or you would like to have the next few random numbers with
the same PRNG.
These two cases seem to be strong use cases for reviving a generator.
Yingjie
--
http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
- Original Message - > From: Paul Rudin > > I'm not really sure whether you intend g to yield the original values > after your "revive" or new values based on the new value of vo. But > still you can make a class that supports the iterator protocol and does > whatever you want (but you can't use the generator expression syntax). > > If you want something along these lines you should probably read up on > the .send() part of the generator protocol. > > As an aside you shouldn't really write code that uses a global in that > way.. it'll end up biting you eventually. > > Anyway... we can speculate endlessly about non-existent language > constructs, but I think we've probably done this one to death. > -- Maybe no new language construct is needed: just define that x.send() revives a generator. Yingjie -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On Fri, Oct 21, 2011 at 7:02 PM, Yingjie Lan wrote:
> What if the generator involves a variable from another scope,
> and before re-generating, the variable changed its value.
> Also, the generator could be passed in as an argument,
> so that we don't know its exact expression.
>
There's actually no way to know that the generator's even
deterministic. Try this, for instance:
>>> g=(input("Enter value %d or blank to stop: "%n) for n in range(1,11))
>>> for s in g:
if not s: break
print("Processing input: "+s)
It may not be particularly useful, but it's certainly legal. And this
generator cannot viably be restarted. The only way is to cast it to
list first, but that doesn't work when you have to stop reading
expressions from the generator part way.
What you could perhaps do is wrap the generator in something that
saves its values:
>>> class restartable(object):
def __init__(self,gen):
self.gen=gen
self.yielded=[]
self.iter=iter(self.yielded)
def restart(self):
self.iter=iter(self.yielded)
def __iter__(self):
return self
def __next__(self): # Remove the underscores for Python 2
try:
return self.iter.__next__()
except StopIteration:
pass
ret=self.gen.__next__()
self.yielded.append(ret)
return ret
>>> h=restartable(g)
>>> for i in h:
if not i: break
print("Using: ",i)
>>> h.restart()
>>> for i in h:
if not i: break
print("Using: ",i)
Complicated, but what this does is returns a value from its saved list
if there is one, otherwise returns a value from the original
generator. It can be restarted as many times as necessary, and any
time you read "past the end" of where you've read so far, the original
generator will be called upon.
Actually, this model might be useful for a repeatable random-number
generator. But those are more efficiently restarted by means of
reseeding the PRNG.
ChrisA
--
http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
Yingjie Lan writes: > > > What if the generator involves a variable from another scope, > and before re-generating, the variable changed its value. > Also, the generator could be passed in as an argument, > so that we don't know its exact expression. > vo = 34 g = (vo*x for x in range(3)) def myfun(g): > for i in g: print(i) > vo += 3 > revive(g) #best if revived automatically > for i in g: print(i) myfun(g) > > I'm not really sure whether you intend g to yield the original values after your "revive" or new values based on the new value of vo. But still you can make a class that supports the iterator protocol and does whatever you want (but you can't use the generator expression syntax). If you want something along these lines you should probably read up on the .send() part of the generator protocol. As an aside you shouldn't really write code that uses a global in that way.. it'll end up biting you eventually. Anyway... we can speculate endlessly about non-existent language constructs, but I think we've probably done this one to death. -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
- Original Message - > From: Paul Rudin > To: python-list@python.org > Cc: > Sent: Friday, October 21, 2011 3:27 PM > Subject: Re: revive a generator > > > The language has no explicit notion of a request to "revive" a > generator. You could use the same syntax to make a new generator that > yeilds the same values as the one you started with if that's what you > want. > > As we've already discussed if you want to iterate several times over the > same values then it probably makes sense to compute them and store them > in e.g. a list (although there are always trade-offs between storage use > and the cost of computing things again). > > What if the generator involves a variable from another scope, and before re-generating, the variable changed its value. Also, the generator could be passed in as an argument, so that we don't know its exact expression. >>> vo = 34 >>> g = (vo*x for x in range(3)) >>> def myfun(g): for i in g: print(i) vo += 3 revive(g) #best if revived automatically for i in g: print(i) myfun(g) Yingjie -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
- Original Message - > From: Paul Rudin > The language has no explicit notion of a request to "revive" a > generator. You could use the same syntax to make a new generator that > yeilds the same values as the one you started with if that's what you > want. > > As we've already discussed if you want to iterate several times over the > same values then it probably makes sense to compute them and store them > in e.g. a list (although there are always trade-offs between storage use > and the cost of computing things again). > Oops, my former reply has the code indentation messed up by the mail system. Here is a reformatted one: What if the generator involves a variable from another scope, and before re-generating, the variable changed its value. Also, the generator could be passed in as an argument, so that we don't know its exact expression. >>> vo = 34 >>> g = (vo*x for x in range(3)) >>> def myfun(g): for i in g: print(i) vo += 3 revive(g) #best if revived automatically for i in g: print(i) >>> myfun(g) Yingjie -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
Yingjie Lan writes: > - Original Message - >> From: Paul Rudin >> Generators are like that - you consume them until they run out of >> values. You could have done [x*x for x in range(3)] and then iterated >> over that list as many times as you wanted. >> >> A generator doesn't have to remember all the values it generates so it >> can be more memory efficient that a list. Also it can, for example, >> generate an infinite sequence. >> >> > Thanks a lot to all who answered my question. > I am still not sure why should we enforce that > a generator can not be reused after an explicit > request to revive it? The language has no explicit notion of a request to "revive" a generator. You could use the same syntax to make a new generator that yeilds the same values as the one you started with if that's what you want. As we've already discussed if you want to iterate several times over the same values then it probably makes sense to compute them and store them in e.g. a list (although there are always trade-offs between storage use and the cost of computing things again). -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
> Here's an example of an explicit request to revive the generator: > g = (x*x for x in range(3)) for x in g: print x > 0 > 1 > 4 g = (x*x for x in range(3)) # revive the generator for x in g: print x #now this will work > 0 > 1 > 4 > > ChrisA What if the generator is passed in as an argument when you are writing a function? That is, the expression is not available? Secondly, it would be nice to automatically revive it. For example, when another for-statement or other equivalent is applied to it. Yingjie -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On Fri, Oct 21, 2011 at 12:46 PM, Yingjie Lan wrote: > > Thanks a lot to all who answered my question. > I am still not sure why should we enforce that > a generator can not be reused after an explicit > request to revive it? Here's an example of an explicit request to revive the generator: >>> g = (x*x for x in range(3)) >>> for x in g: print x 0 1 4 >>> g = (x*x for x in range(3)) # revive the generator >>> for x in g: print x #now this will work 0 1 4 ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
- Original Message - > From: Paul Rudin > To: python-list@python.org > Cc: > Sent: Thursday, October 20, 2011 10:28 PM > Subject: Re: revive a generator > > Yingjie Lan writes: > >> Hi, >> >> it seems a generator expression can be used only once: >> >>>>> g = (x*x for x in range(3)) >>>>> for x in g: print x >> 0 >> 1 >> 4 >>>>> for x in g: print x #nothing printed >>>>> >> >> Is there any way to revive g here? >> > > Generators are like that - you consume them until they run out of > values. You could have done [x*x for x in range(3)] and then iterated > over that list as many times as you wanted. > > A generator doesn't have to remember all the values it generates so it > can be more memory efficient that a list. Also it can, for example, > generate an infinite sequence. > > Thanks a lot to all who answered my question. I am still not sure why should we enforce that a generator can not be reused after an explicit request to revive it? -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On 10/20/2011 9:23 AM, Yingjie Lan wrote: it seems a generator expression can be used only once: Generators are iterators. Once iterators raise StopIteration, they are supposed to continue doing so. A generator expression defines a temporary anonymous generator function that is called once to produce a generator and then deleted. It, like all comprehensions, is purely a convenient abbreviation. g = (x*x for x in range(3)) for x in g: print x 0 1 4 for x in g: print x #nothing printed Define a named generator function (and add a parameter to make it more flexible and useful and reuse it. def g(n): for i in range(n): yield i*i Then, "for x in g(3)", "for x in g(8)", "for x in g(y*x)", etc, as many times as you want. You might call it 'square' or even 'r_square' instead of 'g'. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
Yingjie Lan writes: > Hi, > > it seems a generator expression can be used only once: > g = (x*x for x in range(3)) for x in g: print x > 0 > 1 > 4 for x in g: print x #nothing printed > > Is there any way to revive g here? > Generators are like that - you consume them until they run out of values. You could have done [x*x for x in range(3)] and then iterated over that list as many times as you wanted. A generator doesn't have to remember all the values it generates so it can be more memory efficient that a list. Also it can, for example, generate an infinite sequence. -- http://mail.python.org/mailman/listinfo/python-list
Re: revive a generator
On Fri, Oct 21, 2011 at 12:23 AM, Yingjie Lan wrote: > Hi, > > it seems a generator expression can be used only once: > g = (x*x for x in range(3)) for x in g: print x > 0 > 1 > 4 for x in g: print x #nothing printed > > Is there any way to revive g here? If you're not generating very much, just use a list comprehension instead; you can iterate over the list as many times as you like: >>> g = [x*x for x in range(3)] >>> for x in g: print(x) 0 1 4 >>> for x in g: print(x) 0 1 4 Of course, since this is Python 3, you need the parens on print, but I assume you had something else you were doing with x. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
revive a generator
Hi, it seems a generator expression can be used only once: >>> g = (x*x for x in range(3)) >>> for x in g: print x 0 1 4 >>> for x in g: print x #nothing printed >>> Is there any way to revive g here? Yingjie -- http://mail.python.org/mailman/listinfo/python-list
