Re: [gentoo-portage-dev] [PATCH] emerge: use asyncio interfaces for spinner during owner lookup (bug 591760)
On Sun, Mar 26, 2017 at 7:25 PM, Zac Medicowrote: > On Sun, Mar 26, 2017 at 6:58 PM, Brian Dolbec wrote: >> On Sun, 26 Mar 2017 18:01:01 -0700 >> Zac Medico wrote: >> >>> X-Gentoo-bug: 591760 >>> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 >>> --- >>> pym/_emerge/depgraph.py | 13 - >>> pym/portage/dbapi/vartree.py | 22 +- >>> 2 files changed, 17 insertions(+), 18 deletions(-) >>> >> looks fine. > > Thanks, pushed: > > https://gitweb.gentoo.org/proj/portage.git/commit/?id=dfa317d0b8572a6c463ad5e778a8ca4633086f1b > >> Question, will this help make the spinner more consistent with spin? > > Well, that problem was already solved for the case where emerge is > looking up owners of files, which is the case that this code affects. > > I'm sure there are plenty of other places where the spinner does not > update consistently. This occurs when the main (and only) thread goes > for a long time without calling depgraph._spinner_update(). For > example, some of the code called via the > depgraph._process_slot_conflicts method might we consuming a lot of > time without calling the _spinner_update method. When the spinner appears to be stuck, you can send a SIGUSR1 signal to the emerge process, and then use 'bt' to get a backtrace so that we can see what it's doing. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] emerge: use asyncio interfaces for spinner during owner lookup (bug 591760)
On Sun, Mar 26, 2017 at 6:58 PM, Brian Dolbecwrote: > On Sun, 26 Mar 2017 18:01:01 -0700 > Zac Medico wrote: > >> X-Gentoo-bug: 591760 >> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 >> --- >> pym/_emerge/depgraph.py | 13 - >> pym/portage/dbapi/vartree.py | 22 +- >> 2 files changed, 17 insertions(+), 18 deletions(-) >> > looks fine. Thanks, pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=dfa317d0b8572a6c463ad5e778a8ca4633086f1b > Question, will this help make the spinner more consistent with spin? Well, that problem was already solved for the case where emerge is looking up owners of files, which is the case that this code affects. I'm sure there are plenty of other places where the spinner does not update consistently. This occurs when the main (and only) thread goes for a long time without calling depgraph._spinner_update(). For example, some of the code called via the depgraph._process_slot_conflicts method might we consuming a lot of time without calling the _spinner_update method. -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] emerge: use asyncio interfaces for spinner during owner lookup (bug 591760)
On Sun, 26 Mar 2017 18:01:01 -0700 Zac Medicowrote: > X-Gentoo-bug: 591760 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 > --- > pym/_emerge/depgraph.py | 13 - > pym/portage/dbapi/vartree.py | 22 +- > 2 files changed, 17 insertions(+), 18 deletions(-) > > diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py > index 7c9130b..04e724d 100644 > --- a/pym/_emerge/depgraph.py > +++ b/pym/_emerge/depgraph.py > @@ -3668,17 +3668,20 @@ class depgraph(object): > def select_files(self, args): > # Use the global event loop for spinner progress > # indication during file owner lookups (bug #461412). > - spinner_id = None > + def spinner_cb(): > + self._frozen_config.spinner.update() > + spinner_cb.handle = > self._event_loop.call_soon(spinner_cb) + > + spinner_cb.handle = None > try: > spinner = self._frozen_config.spinner > if spinner is not None and \ > spinner.update is not > spinner.update_quiet: > - spinner_id = > self._event_loop.idle_add( > - > self._frozen_config.spinner.update) > + spinner_cb.handle = > self._event_loop.call_soon(spinner_cb) return self._select_files(args) > finally: > - if spinner_id is not None: > - > self._event_loop.source_remove(spinner_id) > + if spinner_cb.handle is not None: > + spinner_cb.handle.cancel() > > def _select_files(self, myfiles): > """Given a list of .tbz2s, .ebuilds sets, and deps, > populate diff --git a/pym/portage/dbapi/vartree.py > b/pym/portage/dbapi/vartree.py index c421dc5..7c8f150 100644 > --- a/pym/portage/dbapi/vartree.py > +++ b/pym/portage/dbapi/vartree.py > @@ -1369,32 +1369,28 @@ class vardbapi(dbapi): > global_event_loop() or > EventLoop(main=False)) root = self._vardb._eroot > > - def search_pkg(cpv): > + def search_pkg(cpv, search_future): > dblnk = self._vardb._dblink(cpv) > + results = [] > for path, name, is_basename in > path_info_list: if is_basename: > for p in > dblnk._contents.keys(): if os.path.basename(p) == name: > - > search_pkg.results.append((dblnk, > + > results.append((dblnk, dblnk._contents.unmap_key( > > p)[len(root):])) > else: > key = > dblnk._match_contents(path) if key is not False: > - > search_pkg.results.append( > + > results.append( (dblnk, key[len(root):])) > - search_pkg.complete = True > - return False > - > - search_pkg.results = [] > + search_future.set_result(results) > > for cpv in self._vardb.cpv_all(): > - del search_pkg.results[:] > - search_pkg.complete = False > - event_loop.idle_add(search_pkg, cpv) > - while not search_pkg.complete: > - event_loop.iteration() > - for result in search_pkg.results: > + search_future = > event_loop.create_future() > + event_loop.call_soon(search_pkg, > cpv, search_future) > + > event_loop.run_until_complete(search_future) > + for result in search_future.result(): > yield result > > class vartree(object): looks fine. Question, will this help make the spinner more consistent with spin? -- Brian Dolbec
[gentoo-portage-dev] [PATCH] emerge: use asyncio interfaces for spinner during owner lookup (bug 591760)
X-Gentoo-bug: 591760 X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 --- pym/_emerge/depgraph.py | 13 - pym/portage/dbapi/vartree.py | 22 +- 2 files changed, 17 insertions(+), 18 deletions(-) diff --git a/pym/_emerge/depgraph.py b/pym/_emerge/depgraph.py index 7c9130b..04e724d 100644 --- a/pym/_emerge/depgraph.py +++ b/pym/_emerge/depgraph.py @@ -3668,17 +3668,20 @@ class depgraph(object): def select_files(self, args): # Use the global event loop for spinner progress # indication during file owner lookups (bug #461412). - spinner_id = None + def spinner_cb(): + self._frozen_config.spinner.update() + spinner_cb.handle = self._event_loop.call_soon(spinner_cb) + + spinner_cb.handle = None try: spinner = self._frozen_config.spinner if spinner is not None and \ spinner.update is not spinner.update_quiet: - spinner_id = self._event_loop.idle_add( - self._frozen_config.spinner.update) + spinner_cb.handle = self._event_loop.call_soon(spinner_cb) return self._select_files(args) finally: - if spinner_id is not None: - self._event_loop.source_remove(spinner_id) + if spinner_cb.handle is not None: + spinner_cb.handle.cancel() def _select_files(self, myfiles): """Given a list of .tbz2s, .ebuilds sets, and deps, populate diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py index c421dc5..7c8f150 100644 --- a/pym/portage/dbapi/vartree.py +++ b/pym/portage/dbapi/vartree.py @@ -1369,32 +1369,28 @@ class vardbapi(dbapi): global_event_loop() or EventLoop(main=False)) root = self._vardb._eroot - def search_pkg(cpv): + def search_pkg(cpv, search_future): dblnk = self._vardb._dblink(cpv) + results = [] for path, name, is_basename in path_info_list: if is_basename: for p in dblnk._contents.keys(): if os.path.basename(p) == name: - search_pkg.results.append((dblnk, + results.append((dblnk, dblnk._contents.unmap_key( p)[len(root):])) else: key = dblnk._match_contents(path) if key is not False: - search_pkg.results.append( + results.append( (dblnk, key[len(root):])) - search_pkg.complete = True - return False - - search_pkg.results = [] + search_future.set_result(results) for cpv in self._vardb.cpv_all(): - del search_pkg.results[:] - search_pkg.complete = False - event_loop.idle_add(search_pkg, cpv) - while not search_pkg.complete: - event_loop.iteration() - for result in search_pkg.results: + search_future = event_loop.create_future() + event_loop.call_soon(search_pkg, cpv, search_future) + event_loop.run_until_complete(search_future) + for result in search_future.result(): yield result class vartree(object): -- 2.10.2
Re: [gentoo-portage-dev] [PATCH] Future: implement add_done_callback for asyncio compat (bug 591760)
On Sun, Mar 26, 2017 at 12:07 PM, Brian Dolbecwrote: > On Sun, 26 Mar 2017 03:13:11 -0700 > Zac Medico wrote: > >> Implement the add_done_callback and remove_done_callback methods, >> since they are required in order to make further progress toward >> asyncio compatibility. >> >> Also implement the AbstractEventLoop create_future method for the >> EventLoop class, so that it returns an instance of _EventLoopFuture. >> EventLoop currently does not implement some of the >> asyncio.AbstractEventLoop methods that asyncio.Future requires for >> its add_done_callback implementation, and the create_future method >> conveniently solves this problem. >> >> X-Gentoo-bug: 591760 >> X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 >> --- >> pym/portage/tests/ebuild/test_ipc_daemon.py| 3 +- >> .../tests/util/eventloop/test_call_soon_fifo.py| 6 +- >> pym/portage/tests/util/futures/__init__.py | 0 >> pym/portage/tests/util/futures/__test__.py | 0 >> .../tests/util/futures/test_done_callback.py | 35 + >> pym/portage/util/_async/SchedulerInterface.py | 3 +- >> pym/portage/util/_eventloop/EventLoop.py | 14 >> pym/portage/util/futures/futures.py| 82 >> -- 8 files changed, 132 insertions(+), 11 >> deletions(-) create mode 100644 >> pym/portage/tests/util/futures/__init__.py create mode 100644 >> pym/portage/tests/util/futures/__test__.py create mode 100644 >> pym/portage/tests/util/futures/test_done_callback.py >> > > looks fine... /me ignoring the lack of parameters descriptions in the > docstrings Yeah, I skipped parameter docstrings because the interfaces are compatible with the python standard libraries. Pushed: https://gitweb.gentoo.org/proj/portage.git/commit/?id=4b12ed04ec6b99f5a948e0eea5778a4fac502740 -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH] Future: implement add_done_callback for asyncio compat (bug 591760)
On Sun, 26 Mar 2017 03:13:11 -0700 Zac Medicowrote: > Implement the add_done_callback and remove_done_callback methods, > since they are required in order to make further progress toward > asyncio compatibility. > > Also implement the AbstractEventLoop create_future method for the > EventLoop class, so that it returns an instance of _EventLoopFuture. > EventLoop currently does not implement some of the > asyncio.AbstractEventLoop methods that asyncio.Future requires for > its add_done_callback implementation, and the create_future method > conveniently solves this problem. > > X-Gentoo-bug: 591760 > X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 > --- > pym/portage/tests/ebuild/test_ipc_daemon.py| 3 +- > .../tests/util/eventloop/test_call_soon_fifo.py| 6 +- > pym/portage/tests/util/futures/__init__.py | 0 > pym/portage/tests/util/futures/__test__.py | 0 > .../tests/util/futures/test_done_callback.py | 35 + > pym/portage/util/_async/SchedulerInterface.py | 3 +- > pym/portage/util/_eventloop/EventLoop.py | 14 > pym/portage/util/futures/futures.py| 82 > -- 8 files changed, 132 insertions(+), 11 > deletions(-) create mode 100644 > pym/portage/tests/util/futures/__init__.py create mode 100644 > pym/portage/tests/util/futures/__test__.py create mode 100644 > pym/portage/tests/util/futures/test_done_callback.py > > diff --git a/pym/portage/tests/ebuild/test_ipc_daemon.py > b/pym/portage/tests/ebuild/test_ipc_daemon.py index 68f139a..fc79165 > 100644 --- a/pym/portage/tests/ebuild/test_ipc_daemon.py > +++ b/pym/portage/tests/ebuild/test_ipc_daemon.py > @@ -16,7 +16,6 @@ from portage.util import ensure_dirs > from portage.util._async.ForkProcess import ForkProcess > from portage.util._async.TaskScheduler import TaskScheduler > from portage.util._eventloop.global_event_loop import > global_event_loop -from portage.util.futures.futures import Future > from _emerge.SpawnProcess import SpawnProcess > from _emerge.EbuildBuildDir import EbuildBuildDir > from _emerge.EbuildIpcDaemon import EbuildIpcDaemon > @@ -150,7 +149,7 @@ class IpcDaemonTestCase(TestCase): > self._run_done.set_result(True) > > def _run(self, event_loop, task_scheduler, timeout): > - self._run_done = Future() > + self._run_done = event_loop.create_future() > timeout_id = event_loop.timeout_add(timeout, > self._timeout_callback, task_scheduler) > task_scheduler.addExitListener(self._exit_callback) > diff --git a/pym/portage/tests/util/eventloop/test_call_soon_fifo.py > b/pym/portage/tests/util/eventloop/test_call_soon_fifo.py index > 5ecc13f..f970c67 100644 --- > a/pym/portage/tests/util/eventloop/test_call_soon_fifo.py +++ > b/pym/portage/tests/util/eventloop/test_call_soon_fifo.py @@ -7,22 > +7,22 @@ import random from portage import os > from portage.tests import TestCase > from portage.util._eventloop.global_event_loop import > global_event_loop -from portage.util.futures.futures import Future > + > > class CallSoonFifoTestCase(TestCase): > > def testCallSoonFifo(self): > > + event_loop = global_event_loop() > inputs = [random.random() for index in range(10)] > outputs = [] > - finished = Future() > + finished = event_loop.create_future() > > def add_output(value): > outputs.append(value) > if len(outputs) == len(inputs): > finished.set_result(True) > > - event_loop = global_event_loop() > for value in inputs: > event_loop.call_soon(functools.partial(add_output, > value)) > diff --git a/pym/portage/tests/util/futures/__init__.py > b/pym/portage/tests/util/futures/__init__.py new file mode 100644 > index 000..e69de29 > diff --git a/pym/portage/tests/util/futures/__test__.py > b/pym/portage/tests/util/futures/__test__.py new file mode 100644 > index 000..e69de29 > diff --git a/pym/portage/tests/util/futures/test_done_callback.py > b/pym/portage/tests/util/futures/test_done_callback.py new file mode > 100644 index 000..76b727b > --- /dev/null > +++ b/pym/portage/tests/util/futures/test_done_callback.py > @@ -0,0 +1,35 @@ > +# Copyright 2017 Gentoo Foundation > +# Distributed under the terms of the GNU General Public License v2 > + > +from portage.tests import TestCase > +from portage.util._eventloop.global_event_loop import > global_event_loop + > + > +class FutureDoneCallbackTestCase(TestCase): > + > + def testFutureDoneCallback(self): > + > + event_loop = global_event_loop() > + > + def done_callback(finished): > + done_callback_called.set_result(True) > + > + done_callback_called =
[gentoo-portage-dev] [PATCH] Future: implement add_done_callback for asyncio compat (bug 591760)
Implement the add_done_callback and remove_done_callback methods, since they are required in order to make further progress toward asyncio compatibility. Also implement the AbstractEventLoop create_future method for the EventLoop class, so that it returns an instance of _EventLoopFuture. EventLoop currently does not implement some of the asyncio.AbstractEventLoop methods that asyncio.Future requires for its add_done_callback implementation, and the create_future method conveniently solves this problem. X-Gentoo-bug: 591760 X-Gentoo-bug-url: https://bugs.gentoo.org/show_bug.cgi?id=591760 --- pym/portage/tests/ebuild/test_ipc_daemon.py| 3 +- .../tests/util/eventloop/test_call_soon_fifo.py| 6 +- pym/portage/tests/util/futures/__init__.py | 0 pym/portage/tests/util/futures/__test__.py | 0 .../tests/util/futures/test_done_callback.py | 35 + pym/portage/util/_async/SchedulerInterface.py | 3 +- pym/portage/util/_eventloop/EventLoop.py | 14 pym/portage/util/futures/futures.py| 82 -- 8 files changed, 132 insertions(+), 11 deletions(-) create mode 100644 pym/portage/tests/util/futures/__init__.py create mode 100644 pym/portage/tests/util/futures/__test__.py create mode 100644 pym/portage/tests/util/futures/test_done_callback.py diff --git a/pym/portage/tests/ebuild/test_ipc_daemon.py b/pym/portage/tests/ebuild/test_ipc_daemon.py index 68f139a..fc79165 100644 --- a/pym/portage/tests/ebuild/test_ipc_daemon.py +++ b/pym/portage/tests/ebuild/test_ipc_daemon.py @@ -16,7 +16,6 @@ from portage.util import ensure_dirs from portage.util._async.ForkProcess import ForkProcess from portage.util._async.TaskScheduler import TaskScheduler from portage.util._eventloop.global_event_loop import global_event_loop -from portage.util.futures.futures import Future from _emerge.SpawnProcess import SpawnProcess from _emerge.EbuildBuildDir import EbuildBuildDir from _emerge.EbuildIpcDaemon import EbuildIpcDaemon @@ -150,7 +149,7 @@ class IpcDaemonTestCase(TestCase): self._run_done.set_result(True) def _run(self, event_loop, task_scheduler, timeout): - self._run_done = Future() + self._run_done = event_loop.create_future() timeout_id = event_loop.timeout_add(timeout, self._timeout_callback, task_scheduler) task_scheduler.addExitListener(self._exit_callback) diff --git a/pym/portage/tests/util/eventloop/test_call_soon_fifo.py b/pym/portage/tests/util/eventloop/test_call_soon_fifo.py index 5ecc13f..f970c67 100644 --- a/pym/portage/tests/util/eventloop/test_call_soon_fifo.py +++ b/pym/portage/tests/util/eventloop/test_call_soon_fifo.py @@ -7,22 +7,22 @@ import random from portage import os from portage.tests import TestCase from portage.util._eventloop.global_event_loop import global_event_loop -from portage.util.futures.futures import Future + class CallSoonFifoTestCase(TestCase): def testCallSoonFifo(self): + event_loop = global_event_loop() inputs = [random.random() for index in range(10)] outputs = [] - finished = Future() + finished = event_loop.create_future() def add_output(value): outputs.append(value) if len(outputs) == len(inputs): finished.set_result(True) - event_loop = global_event_loop() for value in inputs: event_loop.call_soon(functools.partial(add_output, value)) diff --git a/pym/portage/tests/util/futures/__init__.py b/pym/portage/tests/util/futures/__init__.py new file mode 100644 index 000..e69de29 diff --git a/pym/portage/tests/util/futures/__test__.py b/pym/portage/tests/util/futures/__test__.py new file mode 100644 index 000..e69de29 diff --git a/pym/portage/tests/util/futures/test_done_callback.py b/pym/portage/tests/util/futures/test_done_callback.py new file mode 100644 index 000..76b727b --- /dev/null +++ b/pym/portage/tests/util/futures/test_done_callback.py @@ -0,0 +1,35 @@ +# Copyright 2017 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +from portage.tests import TestCase +from portage.util._eventloop.global_event_loop import global_event_loop + + +class FutureDoneCallbackTestCase(TestCase): + + def testFutureDoneCallback(self): + + event_loop = global_event_loop() + + def done_callback(finished): + done_callback_called.set_result(True) + + done_callback_called = event_loop.create_future() + finished = event_loop.create_future() + finished.add_done_callback(done_callback) + event_loop.call_soon(finished.set_result, True) +
Re: [gentoo-portage-dev] [PATCH] phase-helpers.sh: Loop over A rather than SRC_URI in __eapi0_pkg_nofetch.
> On Thu, 23 Mar 2017, Ulrich Mueller wrote: > On Thu, 23 Mar 2017, Zac Medico wrote: >> On Thu, Mar 23, 2017 at 2:55 AM, Ulrich Müllerwrote: >>> Looping over SRC_URI also outputs non-filename elements (e.g, use >>> conditionals) which is avoided when looping over A. >>> >>> Gentoo-Bug: 613132 >>> --- >>> bin/phase-helpers.sh | 8 >>> 1 file changed, 4 insertions(+), 4 deletions(-) >> Looks good. > Presumably we should wait for the ack (or rather the usual consensus > timeout) of the corresponding PMS patch: > https://archives.gentoo.org/gentoo-pms/message/6044537cd2cd36bc23307918a9afced9 Pushed. pgpy3h8Z_85CP.pgp Description: PGP signature
[gentoo-portage-dev] [RFC] eslurp helper
Hi, I'm thinking of creating a new helper in Portage. The draft name is eslurp. It would be used like: eslurp [--dist|--binpkg] ... i.e. it would have two modes: 1. --dist would place all listed files in DISTDIR (respecting any possible future DISTDIR split), 2. --binpkg would place all listed packages in PKGDIR (respecting subdirectories, updating index). If neither of the options were specified, it would guess the directory type by the filename suffix. What do you think? -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] phase-functions: Fix cleaning fake FILESDIR, reported by Arfrever
On nie, 2017-03-26 at 00:38 -0700, Zac Medico wrote: > On Sun, Mar 26, 2017 at 12:30 AM, Michał Górnywrote: > > --- > > bin/phase-functions.sh | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh > > index 50f0fdb9b..dfd8733c8 100644 > > --- a/bin/phase-functions.sh > > +++ b/bin/phase-functions.sh > > @@ -283,6 +283,7 @@ __dyn_clean() { > > > > rm -rf "${PORTAGE_BUILDDIR}/build-info" > > rm -rf "${WORKDIR}" > > + rm -f "${PORTAGE_BUILDDIR}/files" > > fi > > > > if [ -f "${PORTAGE_BUILDDIR}/.unpacked" ]; then > > -- > > 2.12.1 > > > > > > Looks good. Pushed, thanks. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] phase-functions: Fix cleaning fake FILESDIR, reported by Arfrever
On Sun, Mar 26, 2017 at 12:30 AM, Michał Górnywrote: > --- > bin/phase-functions.sh | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh > index 50f0fdb9b..dfd8733c8 100644 > --- a/bin/phase-functions.sh > +++ b/bin/phase-functions.sh > @@ -283,6 +283,7 @@ __dyn_clean() { > > rm -rf "${PORTAGE_BUILDDIR}/build-info" > rm -rf "${WORKDIR}" > + rm -f "${PORTAGE_BUILDDIR}/files" > fi > > if [ -f "${PORTAGE_BUILDDIR}/.unpacked" ]; then > -- > 2.12.1 > > Looks good. -- Thanks, Zac
[gentoo-portage-dev] [PATCH] phase-functions: Fix cleaning fake FILESDIR, reported by Arfrever
--- bin/phase-functions.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh index 50f0fdb9b..dfd8733c8 100644 --- a/bin/phase-functions.sh +++ b/bin/phase-functions.sh @@ -283,6 +283,7 @@ __dyn_clean() { rm -rf "${PORTAGE_BUILDDIR}/build-info" rm -rf "${WORKDIR}" + rm -f "${PORTAGE_BUILDDIR}/files" fi if [ -f "${PORTAGE_BUILDDIR}/.unpacked" ]; then -- 2.12.1