On 02/25/2018 07:17 PM, Alec Warner wrote: > > > On Sun, Feb 25, 2018 at 8:50 PM, Zac Medico <zmed...@gentoo.org > <mailto:zmed...@gentoo.org>> wrote: > > The iter_completed function is similar to asyncio.as_completed, but > takes an iterator of futures as input, and includes support for > max_jobs and max_load parameters. The default values for max_jobs > and max_load correspond to multiprocessing.cpu_count(). > > Example usage for async_aux_get: > > import portage > from portage.util.futures.iter_completed import iter_completed > > portdb = portage.portdb > future_cpv = {} > > > I'm not sure I grasp the purpose of this dict, can't we just modify the > async aux get to return the cpv from the future?
If we do that then we should probably return all of the other aux_get
inputs too, including mylist, mytree, and myrepo. Pretty soon it feels
like there's a lot of clutter here. If we leave the burden to the
caller, then the API is simpler, and it's not much of a burden to the
caller anyway.
> def future_generator():
> for cpv in portdb.cp_list('sys-apps/portage'):
> future = portdb.async_aux_get(cpv, portage.auxdbkeys)
> future_cpv[id(future)] = cpv
> yield future
>
>
> for cpv in portdb.cp_list('...'):
> yield portdb.async_aux_get(cpv, portage.auxdbkeys)
>
>
> for future in iter_completed(future_generator()):
> cpv = future_cpv.pop(id(future))
> try:
> result = future.result()
> except KeyError as e:
> # aux_get failed
> print('error:', cpv, e)
> else:
> print(cpv, result)
>
>
> for future in iter_completed(future_generator()):
> try:
> cpv, result = future.result()
> except KeyError as e:
> print('error', cpv, e)
>
>
> Or do we expect callers to need other things to key off of in this API?
Yeah it's complicated because of the number of input arguments to
aux_get. You can have the same cpv existing in multiple repos. It's so
much simpler to let the caller manage the mapping from input arguments
to future instance.
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
