On 08/05/2014 21:44, Ian Kelly wrote: > I don't think it needs to be "messy". Something like this should do > the trick, I think: > > from concurrent.futures import * > from itertools import islice > > def batched_pool_runner(f, iterable, pool, batch_size): > it = iter(iterable) > # Submit the first batch of tasks. > futures = set(pool.submit(f, x) for x in islice(it, batch_size)) > while futures: > done, futures = wait(futures, return_when=FIRST_COMPLETED) > # Replenish submitted tasks up to the number that completed. > futures.update(pool.submit(f, x) for x in islice(it, len(done))) > yield from done >
Thank you, that's very neat. It's just the sort of thing I was looking for. Nice use of itertools.islice and "yield from". I'll try this out in the next few days and report back. - Andrew -- https://mail.python.org/mailman/listinfo/python-list