[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

STINNER Victor added the comment: The main issue with using length hint is that calling a method is Python is not free. Calling the method may add more overhead than the speedup provided by smarter memory allocations. The worst case for this optimization should be measured on very small

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Nicholas Musolino added the comment: Before seeing this issue and its closed status, I created PR 14432, which adds `__length_hint__()` to the iterator returned by builtin `map`. This PR differs from the original 2017 PR by MSeifert in that the code can distinguish between the cases where a

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Change by Nicholas Musolino : -- pull_requests: +14249 pull_request: https://github.com/python/cpython/pull/14432 ___ Python tracker ___

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Changes by Raymond Hettinger : -- resolution: -> rejected stage: -> resolved status: open -> closed ___ Python tracker ___

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Michael Seifert added the comment: > zip.__length_hint__() must return NotImplemented or raise TypeError if any of > iterators don't implement __length_hint__ or its __length_hint__() returns > NotImplemented or raises TypeError. > And what should return zip(range(3),

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Michael Seifert added the comment: > I would like to mark this tracker item as closed. IMO it is a dead-end. Yes, even some (not very uncommon cases) give incorrect results: it = iter([1,2,3]) zip(it, it, it) has length_hint 3 but will only yield one item. --

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Serhiy Storchaka added the comment: zip.__length_hint__() must return NotImplemented or raise TypeError if any of iterators don't implement __length_hint__ or its __length_hint__() returns NotImplemented or raises TypeError. And what should return zip(range(3),

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Raymond Hettinger added the comment: Also, the ifilter() suggestion is a complete non-starter. There is no way to know in advance whether filter will return all the elements of the input or none of them. In the absence of other knowledge, the best strategy is what list.append() already does

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Raymond Hettinger added the comment: > But isn't that the point of the length_hint? Where it could be used reliably, it was used and used broadly. However, there are circumstances (iterator vs iterable) where it can't know how to make an estimate and is perhaps wildly off. I would like to

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Michael Seifert added the comment: > I explored that notion of iterator length transparency years ago. While I > don't remember all the details, I did record some notes at the top of > Lib/test/test_iterlen.py. But isn't that the point of the length_hint? To provide an *estimate* for the

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Raymond Hettinger added the comment: [STINNER Victor] > Oh, there is no slot for __length_hint__(). > Maybe we should also try to add a new slot for it? +1 -- ___ Python tracker

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Raymond Hettinger added the comment: Length hints were intentionally omitted from the Python 3 map() and filter() which used to be in the itertools module. I explored that notion of iterator length transparency years ago. While I don't remember all the details, I did record some notes at

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Changes by Raymond Hettinger : -- assignee: -> rhettinger ___ Python tracker ___

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Changes by Michael Seifert : -- pull_requests: +1220 ___ Python tracker ___ ___

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Serhiy Storchaka added the comment: map and zip should use the minimum, zip_longest should use the maximum, and filter shouldn't have __length_hint__(). -- ___ Python tracker

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

STINNER Victor added the comment: I checked Python 2 for map(): it gets the length hint of all iterables and use the maximum, with a default of 8. I'm not sure of what you mean by errors, the function to get the length hint already catchs and ignores errors no? --

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Terry J. Reedy added the comment: Victor, are you suggesting the following? "If map has a single iterable, call and return as hint, otherwise give up." Or something else? -- nosy: +terry.reedy ___ Python tracker

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

STINNER Victor added the comment: > But note that map() and zip() take several iterables, and we should call > __length_hint__() for every of them (unless found a one with not implemented > __length_hint__()). This can slow down the execution for short sequences. Oh, there is no slot for

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

Serhiy Storchaka added the comment: See also issue14126. It makes sense to implement map.__length_hint__() and zip.__length_hint__(). But note that map() and zip() take several iterables, and we should call __length_hint__() for every of them (unless found a one with not implemented

[issue26828] Implement __length_hint__() on map() and filter() to optimize list(map) and list(filter)

New submission from STINNER Victor: When I compared the performance of filter() and map() between Python 2.7 and 3.4, I noticed a huge performance drop in Python 3! http://bugs.python.org/issue26814#msg264003 I didn't analyze yet exactly why Python 3 is so much slower (almost 100% slower for