New submission from Forest <[email protected]>:
In the multiprocessing Pool methods like map, chunksize determines the
trade-off between computation per task and inter-process communication. Setting
chunksize appropriately has a large effect on efficiency.
However, for users directly interacting with the map methods, the way to find
the appropriate chunksize is by manually checking different sizes and observing
the program behavior.
For library developers, you have to hope that you set an reasonable value that
will work okay across different hardware, operating systems, and task
characteristics.
Generally, users of these methods want maximum throughput. It would be great if
the map-like methods could adapt their chunksize towards that goal.
Something along the lines of this:
n_items = 0
queue = Queue(N)
while True:
chunk = tuple(itertools.islice(iterable, chunk_size))
if chunk:
queue.put(chunk)
n_items += chunk_size
i += 1
if i % 10:
time_delta = max(time.perf_counter() - t0, 0.0001)
current_rate = n_items / time_delta
# chunk_size is always either growing or shrinking, if
# the shrinking led to a faster rate, keep
# shrinking. Same with growing. If the rate decreased,
# reverse directions
if current_rate < last_rate:
multiplier = 1 / multiplier
chunk_size = int(min(max(chunk_size * multiplier, 1),
upper_bound))
last_rate = current_rate
n_items = 0
t0 = time.perf_counter()
Would such a feature be desirable?
----------
components: macOS
messages: 360126
nosy: fgregg, ned.deily, ronaldoussoren
priority: normal
severity: normal
status: open
title: add option to make chunksize adaptive for multiprocessing.pool methods
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue39362>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
