New submission from David Lukeš <[email protected]>:
The docstring for `concurrent.futures.Executor.map` starts by stating that it
is "Equivalent to map(func, *iterables)". In the case of Python 3, I would
argue this is true only superficially: with `map`, the user expects
memory-efficient processing, i.e. that the entire resulting collection will not
be held in memory at once unless s/he requests so e.g. with `list(map(...))`.
(In Python 2, the expectations are different of course.) On the other hand,
while `Executor.map` superficially returns a generator, which seems to align
with this expectation, what happens behind the scenes is that the call blocks
until all results are computed and only then starts yielding them. In other
words, they have to be stored in memory all at once at some point.
The lower-level multiprocessing module also describes
`multiprocessing.pool.Pool.map` as "A parallel equivalent of the map() built-in
function", but at least it immediately goes on to add that "It blocks until the
result is ready.", which is a clear indication that all of the results will
have to be stored somewhere before being yielded.
I can think of several ways the situation could be improved, listed here from
most conservative to most progressive:
1. Add "It blocks until the result is ready." to the docstring of
`Executor.map` as well, preferably somewhere at the beginning.
2. Reword the docstrings of both `Executor.map` and `Pool.map` so that they
don't describe the functions as "equivalent" to built-in `map`, which raises
the wrong expectations. ("Similar to map(...), but blocks until all results are
collected and only then yields them.")
3. I would argue that the function that can be described as semantically
equivalent to `map` is actually `Pool.imap`, which yields results as they're
being computed. It would be really nice if this could be added to the
higher-level `futures` API, along with `Pool.imap_unordered`. `Executor.map`
simply doesn't work for very long streams of data.
4. Maybe instead of adding `imap` and `imap_unordered` methods to `Executor`,
it would be a good idea to change the signature of `Executor.map(func,
*iterables, timeout=None, chunksize=1)` to `Executor.map(func, *iterables,
timeout=None, chunksize=1, block=True, ordered=True)`, in order to keep the API
simple with good defaults while providing flexibility via keyword arguments.
5. I would go so far as to say that for me personally, the `block` argument to
the version of `Executor.map` proposed in #4 above should be `False` by
default, because that would make it behave most like built-in `map`, which is
the least suprising behavior. But I've observed that for smaller work loads,
`imap` tends to be slower than `map`, so I understand it might be a tradeoff
between performance and semantics. Still, in a higher-level API meant for
non-experts, I think semantics should be emphasized.
If the latter options seem much too radical, please consider at least something
along the lines of #1 above, I think it would help people correct their
expectations when they first encounter the API :)
----------
components: Library (Lib)
messages: 308221
nosy: David Lukeš
priority: normal
severity: normal
status: open
title: Clarify map API in concurrent.futures
type: enhancement
versions: Python 3.8
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue32306>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
