[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Koos Zevenhoven]

Koos Zevenhoven  added the comment:

That said, I agree with Antoine that being able to Ctrl-C would be the most 
useful of these fixes. But it seems clear that people are experiencing these 
issues differently (if at all).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Koos Zevenhoven]

Koos Zevenhoven  added the comment:

I'd say there are three different problems related to this:

(1) The inability of infinite iterators/iterables to say anything about their 
length

(2) The inability to interrupt C-level loops

(3) The horrible consequences of things like list(itertools.count()) that fill 
up the memory


The problems overlap only partially. Unfortunately, fixing any single one of 
these problems does not eliminate the two others. For example, (2) and (3) may 
happen just as well without the iterator protocol being involved. And (1) may 
just prevent you from checking if the iterable has enough elements for whatever 
you're doing.

--
nosy: +koos.zevenhoven

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Erik Bray]

Erik Bray  added the comment:

> Thinking about the generator-iterator case a bit more, "False" would be a bad 
> default for that. Allowing "NotImplemented" to indicate the ambiguous "Might 
> be finite, might be infinite" state, and using that as the default for 
> generator-iterators, would probably make sense.

This is why I suggested the converse--something like __finite_iterator__ 
(nevermind bikeshedding over the name or "yet another dunder method).  This is 
something one would use to mark as iterator as "this is definitely expected to 
terminate at some point, assuming it is correctly implemented".  

If __finite_iterator__ == False, which should be the default, it doesn't 
necessarily mean it is infinite either, it just may or may not be finite, so 
there's no guarantee.

I think that __finite_iterator__ == True is more or less equivalent to 
returning a non-zero value from __length_hint__, whereas __finite_iterator__ == 
False is equivalent to raising NotImplemented for __length_hint__.  Either way 
it means adding __length_hint__ to all iterators, and also (as Nick suggested) 
having a decorator for generators to set the appropriate hint as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I think being able to handle Ctrl-C would be the right solution here.  Not 
handling Ctrl-C is a problem also for very long but non-infinite iterators.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Nick Coghlan]

Nick Coghlan  added the comment:

If C level iterator implementations in the standard library natively handled 
Ctrl-C (to cope with naive third party C level iterator consumers), and so did 
C level iterator consumers in the standard library (to cope with with naive 
third party C level iterators), I agree this wouldn't be that useful.

However, if folks are going to actively oppose making it possible for users to 
interrupt tight loops over non-Python iterators that are "bigger or slower than 
convenient" on the grounds of it making the C code more complicated to solve a 
problem they personally consider to be purely theoretical (despite other core 
developers telling them otherwise), then special casing known-infinite 
iterators is better than nothing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

> Infinite is just a special case of "bigger or slower than convenient".

I have the same opinion.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Raymond Hettinger]

Raymond Hettinger  added the comment:

After more thought, I'm no longer convinced this would be useful.  Infinite is 
just a special case of "bigger or slower than convenient".  In a way, 
min(count()) is no more interesting than min(range(1_000_000_000_000)) or 
min(slow_data_source()).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Downside of __infinite_iterator__: another additional attribute to look up, and 
its functionality overlaps with __length_hint__.  I'd rather have a special 
return value for __length_hint__ (e.g. -1, -2, whatever).

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Nick Coghlan]

Nick Coghlan  added the comment:

Thinking about the generator-iterator case a bit more, "False" would be a bad 
default for that. Allowing "NotImplemented" to indicate the ambiguous "Might be 
finite, might be infinite" state, and using that as the default for 
generator-iterators, would probably make sense.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33939] Provide a robust O(1) mechanism to check for infinite iterators

[Nick Coghlan]

Nick Coghlan  added the comment:

(I've updated the issue title to state the design requirement, rather than one 
potential solution to the design requirement)

I like the declarative `__infinite_iterator__` suggestion, as that's:

1. really easy to implement when defining a custom iterator type (whether in 
Python or in an extension module

2. relatively easy to add to generator-iterators as a read-write property that 
defaults to False (which could then be set to True via an 
"itertools.infinite_generator" decorator)

3. relatively easy to support in iterators like map, filter, etc via a property 
that delegates the response to the underlying iterator

4. even more complex iterators like chain, product, and combinations would be 
able to define a property based on "any(itr.__infinite__iterator__ in 
underyling_iterators)"

--
title: Raise OverflowError in __length_hint__ for consistently infinite 
iterators -> Provide a robust O(1) mechanism to check for infinite iterators

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com