date:20180225

Re: [gentoo-portage-dev] [PATCH 2/2] Add iter_completed convenience function (bug 648790)

2018-02-25 Thread Zac Medico

On 02/25/2018 07:17 PM, Alec Warner wrote:
> 
> 
> On Sun, Feb 25, 2018 at 8:50 PM, Zac Medico  > wrote:
> 
> The iter_completed function is similar to asyncio.as_completed, but
> takes an iterator of futures as input, and includes support for
> max_jobs and max_load parameters. The default values for max_jobs
> and max_load correspond to multiprocessing.cpu_count().
> 
> Example usage for async_aux_get:
> 
>   import portage
>   from portage.util.futures.iter_completed import iter_completed
> 
>   portdb = portage.portdb
>   future_cpv = {}
> 
> 
> I'm not sure I grasp the purpose of this dict, can't we just modify the
> async aux get to return the cpv from the future?

If we do that then we should probably return all of the other aux_get
inputs too, including mylist, mytree, and myrepo. Pretty soon it feels
like there's a lot of clutter here. If we leave the burden to the
caller, then the API is simpler, and it's not much of a burden to the
caller anyway.

>   def future_generator():
>     for cpv in portdb.cp_list('sys-apps/portage'):
>       future = portdb.async_aux_get(cpv, portage.auxdbkeys)
>       future_cpv[id(future)] = cpv
>       yield future
> 
> 
> for cpv in portdb.cp_list('...'):
>    yield portdb.async_aux_get(cpv, portage.auxdbkeys)
>  
> 
>   for future in iter_completed(future_generator()):
>     cpv = future_cpv.pop(id(future))
>     try:
>       result = future.result()
>     except KeyError as e:
>       # aux_get failed
>       print('error:', cpv, e)
>     else:
>       print(cpv, result)
> 
> 
> for future in iter_completed(future_generator()):
>   try:
>     cpv, result = future.result() 
>   except KeyError as e:
>     print('error', cpv, e)
>  
> 
> Or do we expect callers to need other things to key off of in this API?

Yeah it's complicated because of the number of input arguments to
aux_get. You can have the same cpv existing in multiple repos. It's so
much simpler to let the caller manage the mapping from input arguments
to future instance.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH 2/2] Add iter_completed convenience function (bug 648790)

2018-02-25 Thread Alec Warner

On Sun, Feb 25, 2018 at 8:50 PM, Zac Medico  wrote:

> The iter_completed function is similar to asyncio.as_completed, but
> takes an iterator of futures as input, and includes support for
> max_jobs and max_load parameters. The default values for max_jobs
> and max_load correspond to multiprocessing.cpu_count().
>
> Example usage for async_aux_get:
>
>   import portage
>   from portage.util.futures.iter_completed import iter_completed
>
>   portdb = portage.portdb
>   future_cpv = {}
>

I'm not sure I grasp the purpose of this dict, can't we just modify the
async aux get to return the cpv from the future?


>
>   def future_generator():
> for cpv in portdb.cp_list('sys-apps/portage'):
>   future = portdb.async_aux_get(cpv, portage.auxdbkeys)
>   future_cpv[id(future)] = cpv
>   yield future
>
>
for cpv in portdb.cp_list('...'):
   yield portdb.async_aux_get(cpv, portage.auxdbkeys)


>   for future in iter_completed(future_generator()):
> cpv = future_cpv.pop(id(future))
> try:
>   result = future.result()
> except KeyError as e:
>   # aux_get failed
>   print('error:', cpv, e)
> else:
>   print(cpv, result)
>

for future in iter_completed(future_generator()):
  try:
cpv, result = future.result()
  except KeyError as e:
print('error', cpv, e)


Or do we expect callers to need other things to key off of in this API?

-A


> See: https://docs.python.org/3/library/asyncio-task.html#
> asyncio.as_completed
> Bug: https://bugs.gentoo.org/648790
> ---
>  .../tests/util/futures/test_iter_completed.py  | 50 
>  pym/portage/util/_async/FuturePollTask.py  | 27 ++
>  pym/portage/util/futures/iter_completed.py | 63 ++
>  pym/portage/util/futures/wait.py   | 95
> ++
>  4 files changed, 235 insertions(+)
>  create mode 100644 pym/portage/tests/util/futures/test_iter_completed.py
>  create mode 100644 pym/portage/util/_async/FuturePollTask.py
>  create mode 100644 pym/portage/util/futures/iter_completed.py
>  create mode 100644 pym/portage/util/futures/wait.py
>
> diff --git a/pym/portage/tests/util/futures/test_iter_completed.py
> b/pym/portage/tests/util/futures/test_iter_completed.py
> new file mode 100644
> index 0..6607d871c
> --- /dev/null
> +++ b/pym/portage/tests/util/futures/test_iter_completed.py
> @@ -0,0 +1,50 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import time
> +from portage.tests import TestCase
> +from portage.util._async.ForkProcess import ForkProcess
> +from portage.util._eventloop.global_event_loop import global_event_loop
> +from portage.util.futures.iter_completed import iter_completed
> +
> +
> +class SleepProcess(ForkProcess):
> +   __slots__ = ('future', 'seconds')
> +   def _start(self):
> +   self.addExitListener(self._future_done)
> +   ForkProcess._start(self)
> +
> +   def _future_done(self, task):
> +   self.future.set_result(self.seconds)
> +
> +   def _run(self):
> +   time.sleep(self.seconds)
> +
> +
> +class IterCompletedTestCase(TestCase):
> +
> +   def testIterCompleted(self):
> +
> +   # Mark this as todo, since we don't want to fail if heavy
> system
> +   # load causes the tasks to finish in an unexpected order.
> +   self.todo = True
> +
> +   loop = global_event_loop()
> +   tasks = [
> +   SleepProcess(seconds=0.200),
> +   SleepProcess(seconds=0.100),
> +   SleepProcess(seconds=0.001),
> +   ]
> +
> +   expected_order = sorted(task.seconds for task in tasks)
> +
> +   def future_generator():
> +   for task in tasks:
> +   task.future = loop.create_future()
> +   task.scheduler = loop
> +   task.start()
> +   yield task.future
> +
> +   for seconds, future in zip(expected_order,
> iter_completed(future_generator(),
> +   max_jobs=None, max_load=None, loop=loop)):
> +   self.assertEqual(seconds, future.result())
> diff --git a/pym/portage/util/_async/FuturePollTask.py
> b/pym/portage/util/_async/FuturePollTask.py
> new file mode 100644
> index 0..6b7cdf7d5
> --- /dev/null
> +++ b/pym/portage/util/_async/FuturePollTask.py
> @@ -0,0 +1,27 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import os
> +import signal
> +
> +from _emerge.AbstractPollTask import AbstractPollTask
> +
> +
> +class FuturePollTask(AbstractPollTask):
> +   """
> +   Wraps a Future in an AsynchronousTask, which is useful for
> +   scheduling with TaskScheduler.
> +   """
> +

[gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

2018-02-25 Thread Zac Medico

Add async_aux_get method that returns a Future and otherwise
behaves identically to aux_get. Use async_aux_get to implement
the synchronous aux_get method.

Bug: https://bugs.gentoo.org/648790
---
 pym/portage/dbapi/porttree.py | 91 +--
 1 file changed, 70 insertions(+), 21 deletions(-)

diff --git a/pym/portage/dbapi/porttree.py b/pym/portage/dbapi/porttree.py
index f5979d2d0..abcd47238 100644
--- a/pym/portage/dbapi/porttree.py
+++ b/pym/portage/dbapi/porttree.py
@@ -45,6 +45,7 @@ import traceback
 import warnings
 import errno
 import collections
+import functools
 
 try:
from urllib.parse import urlparse
@@ -577,11 +578,46 @@ class portdbapi(dbapi):
"stub code for returning auxilliary db information, such as 
SLOT, DEPEND, etc."
'input: "sys-apps/foo-1.0",["SLOT","DEPEND","HOMEPAGE"]'
'return: ["0",">=sys-libs/bar-1.0","http://www.foo.com;] or 
raise PortageKeyError if error'
+   # For external API consumers, self._event_loop returns a new 
event
+   # loop on each access, so a local reference is needed in order
+   # to avoid instantiating more than one.
+   loop = self._event_loop
+   return loop.run_until_complete(
+   self.async_aux_get(mycpv, mylist, mytree=mytree,
+   myrepo=myrepo, loop=loop))
+
+   def async_aux_get(self, mycpv, mylist, mytree=None, myrepo=None, 
loop=None):
+   """
+   Asynchronous form form of aux_get.
+
+   @param mycpv: cpv for an ebuild
+   @type mycpv: str
+   @param mylist: list of metadata keys
+   @type mylist: list
+   @param mytree: The canonical path of the tree in which the 
ebuild
+   is located, or None for automatic lookup
+   @type mytree: str
+   @param myrepo: name of the repo in which the ebuild is located,
+   or None for automatic lookup
+   @type myrepo: str
+   @param loop: event loop (defaults to global event loop)
+   @type loop: EventLoop
+   @return: list of metadata values
+   @rtype: asyncio.Future (or compatible)
+   """
+   # Don't default to self._event_loop here, since that creates a
+   # local event loop for thread safety, and that could easily lead
+   # to simultaneous instantiation of multiple event loops here.
+   # Callers of this method certainly want the same event loop to
+   # be used for all calls.
+   loop = loop or global_event_loop()
+   future = loop.create_future()
cache_me = False
if myrepo is not None:
mytree = self.treemap.get(myrepo)
if mytree is None:
-   raise PortageKeyError(myrepo)
+   future.set_exception(PortageKeyError(myrepo))
+   return future
 
if mytree is not None and len(self.porttrees) == 1 \
and mytree == self.porttrees[0]:
@@ -596,43 +632,56 @@ class portdbapi(dbapi):
mylist).difference(self._aux_cache_keys):
aux_cache = self._aux_cache.get(mycpv)
if aux_cache is not None:
-   return [aux_cache.get(x, "") for x in mylist]
+   future.set_result([aux_cache.get(x, "") for x 
in mylist])
+   return future
cache_me = True
 
try:
cat, pkg = mycpv.split("/", 1)
except ValueError:
# Missing slash. Can't find ebuild so raise 
PortageKeyError.
-   raise PortageKeyError(mycpv)
+   future.set_exception(PortageKeyError(mycpv))
+   return future
 
myebuild, mylocation = self.findname2(mycpv, mytree)
 
if not myebuild:
writemsg("!!! aux_get(): %s\n" % \
_("ebuild not found for '%s'") % mycpv, 
noiselevel=1)
-   raise PortageKeyError(mycpv)
+   future.set_exception(PortageKeyError(mycpv))
+   return future
 
mydata, ebuild_hash = self._pull_valid_cache(mycpv, myebuild, 
mylocation)
-   doregen = mydata is None
-
-   if doregen:
-   if myebuild in self._broken_ebuilds:
-   raise PortageKeyError(mycpv)
-
-   proc = EbuildMetadataPhase(cpv=mycpv,
-   ebuild_hash=ebuild_hash, portdb=self,
-

[gentoo-portage-dev] [PATCH 2/2] Add iter_completed convenience function (bug 648790)

2018-02-25 Thread Zac Medico

The iter_completed function is similar to asyncio.as_completed, but
takes an iterator of futures as input, and includes support for
max_jobs and max_load parameters. The default values for max_jobs
and max_load correspond to multiprocessing.cpu_count().

Example usage for async_aux_get:

  import portage
  from portage.util.futures.iter_completed import iter_completed

  portdb = portage.portdb
  future_cpv = {}

  def future_generator():
for cpv in portdb.cp_list('sys-apps/portage'):
  future = portdb.async_aux_get(cpv, portage.auxdbkeys)
  future_cpv[id(future)] = cpv
  yield future

  for future in iter_completed(future_generator()):
cpv = future_cpv.pop(id(future))
try:
  result = future.result()
except KeyError as e:
  # aux_get failed
  print('error:', cpv, e)
else:
  print(cpv, result)

See: https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed
Bug: https://bugs.gentoo.org/648790
---
 .../tests/util/futures/test_iter_completed.py  | 50 
 pym/portage/util/_async/FuturePollTask.py  | 27 ++
 pym/portage/util/futures/iter_completed.py | 63 ++
 pym/portage/util/futures/wait.py   | 95 ++
 4 files changed, 235 insertions(+)
 create mode 100644 pym/portage/tests/util/futures/test_iter_completed.py
 create mode 100644 pym/portage/util/_async/FuturePollTask.py
 create mode 100644 pym/portage/util/futures/iter_completed.py
 create mode 100644 pym/portage/util/futures/wait.py

diff --git a/pym/portage/tests/util/futures/test_iter_completed.py 
b/pym/portage/tests/util/futures/test_iter_completed.py
new file mode 100644
index 0..6607d871c
--- /dev/null
+++ b/pym/portage/tests/util/futures/test_iter_completed.py
@@ -0,0 +1,50 @@
+# Copyright 2018 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import time
+from portage.tests import TestCase
+from portage.util._async.ForkProcess import ForkProcess
+from portage.util._eventloop.global_event_loop import global_event_loop
+from portage.util.futures.iter_completed import iter_completed
+
+
+class SleepProcess(ForkProcess):
+   __slots__ = ('future', 'seconds')
+   def _start(self):
+   self.addExitListener(self._future_done)
+   ForkProcess._start(self)
+
+   def _future_done(self, task):
+   self.future.set_result(self.seconds)
+
+   def _run(self):
+   time.sleep(self.seconds)
+
+
+class IterCompletedTestCase(TestCase):
+
+   def testIterCompleted(self):
+
+   # Mark this as todo, since we don't want to fail if heavy system
+   # load causes the tasks to finish in an unexpected order.
+   self.todo = True
+
+   loop = global_event_loop()
+   tasks = [
+   SleepProcess(seconds=0.200),
+   SleepProcess(seconds=0.100),
+   SleepProcess(seconds=0.001),
+   ]
+
+   expected_order = sorted(task.seconds for task in tasks)
+
+   def future_generator():
+   for task in tasks:
+   task.future = loop.create_future()
+   task.scheduler = loop
+   task.start()
+   yield task.future
+
+   for seconds, future in zip(expected_order, 
iter_completed(future_generator(),
+   max_jobs=None, max_load=None, loop=loop)):
+   self.assertEqual(seconds, future.result())
diff --git a/pym/portage/util/_async/FuturePollTask.py 
b/pym/portage/util/_async/FuturePollTask.py
new file mode 100644
index 0..6b7cdf7d5
--- /dev/null
+++ b/pym/portage/util/_async/FuturePollTask.py
@@ -0,0 +1,27 @@
+# Copyright 2018 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import os
+import signal
+
+from _emerge.AbstractPollTask import AbstractPollTask
+
+
+class FuturePollTask(AbstractPollTask):
+   """
+   Wraps a Future in an AsynchronousTask, which is useful for
+   scheduling with TaskScheduler.
+   """
+   __slots__ = ('future',)
+   def _start(self):
+   self.future.add_done_callback(self._done_callback)
+
+   def _done_callback(self, future):
+   if future.cancelled():
+   self.cancelled = True
+   self.returncode = -signal.SIGINT
+   elif future.exception() is None:
+   self.returncode = os.EX_OK
+   else:
+   self.returncode = 1
+   self.wait()
diff --git a/pym/portage/util/futures/iter_completed.py 
b/pym/portage/util/futures/iter_completed.py
new file mode 100644
index 0..0540cc986
--- /dev/null
+++ b/pym/portage/util/futures/iter_completed.py
@@ -0,0 +1,63 @@
+# Copyright 2018 Gentoo

Re: [gentoo-portage-dev] [PATCH] Deprecate EAPI 6_pre1

2018-02-25 Thread Zac Medico

On 02/25/2018 11:59 AM, Michał Górny wrote:
> Deprecated the testing variant of EAPI 6.
> ---
>  pym/portage/__init__.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/pym/portage/__init__.py b/pym/portage/__init__.py
> index 99f3f98ac..4773738b2 100644
> --- a/pym/portage/__init__.py
> +++ b/pym/portage/__init__.py
> @@ -1,4 +1,4 @@
> -# Copyright 1998-2014 Gentoo Foundation
> +# Copyright 1998-2018 Gentoo Foundation
>  # Distributed under the terms of the GNU General Public License v2
>  
>  from __future__ import unicode_literals
> @@ -462,8 +462,8 @@ def abssymlink(symlink, target=None):
>  
>  _doebuild_manifest_exempt_depend = 0
>  
> -_testing_eapis = frozenset(["4-python", "4-slot-abi", "5-progress", 
> "5-hdepend", "6_pre1"])
> -_deprecated_eapis = frozenset(["4_pre1", "3_pre2", "3_pre1", "5_pre1", 
> "5_pre2"])
> +_testing_eapis = frozenset(["4-python", "4-slot-abi", "5-progress", 
> "5-hdepend"])
> +_deprecated_eapis = frozenset(["4_pre1", "3_pre2", "3_pre1", "5_pre1", 
> "5_pre2", "6_pre1"])
>  _supported_eapis = frozenset([str(x) for x in range(portage.const.EAPI + 1)] 
> + list(_testing_eapis) + list(_deprecated_eapis))
>  
>  def _eapi_is_deprecated(eapi):
> 

Looks good, please merge.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] Deprecate EAPI 6_pre1

2018-02-25 Thread Michał Górny

Deprecated the testing variant of EAPI 6.
---
 pym/portage/__init__.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/pym/portage/__init__.py b/pym/portage/__init__.py
index 99f3f98ac..4773738b2 100644
--- a/pym/portage/__init__.py
+++ b/pym/portage/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 1998-2014 Gentoo Foundation
+# Copyright 1998-2018 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -462,8 +462,8 @@ def abssymlink(symlink, target=None):
 
 _doebuild_manifest_exempt_depend = 0
 
-_testing_eapis = frozenset(["4-python", "4-slot-abi", "5-progress", 
"5-hdepend", "6_pre1"])
-_deprecated_eapis = frozenset(["4_pre1", "3_pre2", "3_pre1", "5_pre1", 
"5_pre2"])
+_testing_eapis = frozenset(["4-python", "4-slot-abi", "5-progress", 
"5-hdepend"])
+_deprecated_eapis = frozenset(["4_pre1", "3_pre2", "3_pre1", "5_pre1", 
"5_pre2", "6_pre1"])
 _supported_eapis = frozenset([str(x) for x in range(portage.const.EAPI + 1)] + 
list(_testing_eapis) + list(_deprecated_eapis))
 
 def _eapi_is_deprecated(eapi):
-- 
2.16.2

Re: [gentoo-portage-dev] [PATCH 2/2] Add iter_completed convenience function (bug 648790)

Re: [gentoo-portage-dev] [PATCH 2/2] Add iter_completed convenience function (bug 648790)

[gentoo-portage-dev] [PATCH 1/2] portdbapi: add async_aux_get method (bug 648790)

[gentoo-portage-dev] [PATCH 2/2] Add iter_completed convenience function (bug 648790)

Re: [gentoo-portage-dev] [PATCH] Deprecate EAPI 6_pre1

[gentoo-portage-dev] [PATCH] Deprecate EAPI 6_pre1

6 matches

Site Navigation

Mail list logo

Footer information