Robert Xiao added the comment:

I think this _is_ a bug. Most of the itertools are quite thrifty with memory, 
and exceptions are explicitly documented.

The itertools generally only require memory proportional to the number of 
iterations consumed. However, `itertools.product` requires an enormous up-front 
memory cost even if only a few iterations are performed. There are good reasons 
to extract only a few iterations from a `product` - for example, a search 
procedure where only the first item in the product is infinite:

    for year, day in product(count(2010), range(365)):
        if rare_event(year, day):

Or, in an application I recently tried, to count occurrences of something up to 
a limit:

    prod = product(count(), range(n))
    filtered = ifilter(pred, prod)
    want = list(islice(filtered, 101))
    if len(want) > 100:
        print "Too many matches"

Having `product` greedily consume all of its input iterators is rather 
antithetical to the notion of itertools being lazy generators, and it breaks 
efficient composability with the other itertools.

The fix should be fairly straightforward: like tee, maintain lists of items 
already generated as the input iterators are exhausted, and add a note that 
"product"'s memory usage may slowly grow over time (as opposed to growing to 
maximum size immediately).

nosy: +nneonneo

Python tracker <>
Python-bugs-list mailing list

Reply via email to