Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
On Sun, Apr 1, 2018 at 2:29 PM, Michał Górny wrote: > W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico > napisał: > > On 04/01/2018 03:57 AM, Michał Górny wrote: > > > W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico > > > napisał: > > > > Since key refresh is prone to failure, retry using exponential > > > > backoff with random jitter. This adds the following sync-openpgp-* > > > > configuration settings: > > > > > > > > sync-openpgp-key-refresh-retry-count = 40 > > > > > > > > Maximum number of times to retry key refresh if it fails. Between > > > > each key refresh attempt, there is an exponential delay with a > > > > constant multiplier and a uniform random multiplier between 0 and > 1. > > > > > > > > sync-openpgp-key-refresh-retry-delay-exp-base = 2 > > > > > > > > The base of the exponential expression. The exponent is the number > > > > of previous refresh attempts. > > > > > > > > sync-openpgp-key-refresh-retry-delay-max = 60 > > > > > > > > Maximum delay between each retry attempt, in units of seconds. > This > > > > places a limit on the length of the exponential delay. > > > > > > > > sync-openpgp-key-refresh-retry-delay-mult = 4 > > > > > > > > Multiplier for the exponential delay. > > > > > > > > sync-openpgp-key-refresh-retry-overall-timeout = 1200 > > > > > > > > Combined time limit for all refresh attempts, in units of seconds. > > > > > > > > Bug: https://bugs.gentoo.org/649276 > > > > > > > > Zac Medico (4): > > > > Add ForkExecutor (bug 649588) > > > > Add ExponentialBackoff and RandomExponentialBackoff > > > > Add retry decorator (API inspired by tenacity) > > > > rsync: add key refresh retry (bug 649276) > > > > > > > > cnf/repos.conf| 5 + > > > > man/portage.5 | 19 +++ > > > > pym/portage/repository/config.py | 22 > > > > pym/portage/sync/modules/rsync/rsync.py | 16 ++- > > > > pym/portage/sync/syncbase.py | 85 +++- > > > > pym/portage/tests/util/futures/test_retry.py | 147 > + > > > > pym/portage/util/_eventloop/EventLoop.py | 45 ++- > > > > pym/portage/util/backoff.py | 48 +++ > > > > pym/portage/util/futures/executor/__init__.py | 0 > > > > pym/portage/util/futures/executor/fork.py | 130 > +++ > > > > pym/portage/util/futures/futures.py | 6 + > > > > pym/portage/util/futures/retry.py | 178 > ++ > > > > 12 files changed, 697 insertions(+), 4 deletions(-) > > > > create mode 100644 pym/portage/tests/util/futures/test_retry.py > > > > create mode 100644 pym/portage/util/backoff.py > > > > create mode 100644 pym/portage/util/futures/executor/__init__.py > > > > create mode 100644 pym/portage/util/futures/executor/fork.py > > > > create mode 100644 pym/portage/util/futures/retry.py > > > > > > > > > > This essentially looks like ~700 lines of code to try to workaround > > > broken networking. I would rather try to do that using 5 lines of code > > > but that's just me, and my programs aren't enterprise quality. I just > > > hope it actually solves as many problems as it's going to introduce. > > > > The vast majority of this code is generic and reusable, and I do intend > > to reuse it. For example, the executor support will be an essential > > piece for the asyncio.AbstractEventLoop for bug 649588. > > Sure it is and sure you will. But tell me: who is going to maintain it > all? Because as far as I can see, we're still dealing with a bus factor > of one and all you're doing is making it worse. More code, more > complexity, more creeping featurism and hacks. > I'll split this reply into two sorts of points. One is a brief point about this patchset. The other is a larger point about the project and its direction. About this patchset: I don't mind this code because its not domain specific and I can read it and understand it. Many apps need async ways to run code, and anyone who has used the futures API will find this code familiar. I reviewed it in about 10 minutes. I'm not heavily invested with the async-wagon that Zac is driving, but I also see how its an area where performance can be improved by doing more stuff at once using an async operations. I think if you have more generic concerns with the async frameworks I'd love to see some discussion about them (particularly around how we can improve perf by using tighter algorithms; often the 'perf' boost is just higher CPU utilization and driving multi-core systems which results in better walltime perf, but not necessarily cpu-perf.) Its fine to disagree here. I think the project has some requirements around minimized dependencies, so the choice is either to pull in some kind of retry-lib and add a dep, or add this code. Its also not immediately clear if the 3rd party deps would still not require some glue c
Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
On 04/01/2018 11:29 AM, Michał Górny wrote: > W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico > napisał: >> On 04/01/2018 03:57 AM, Michał Górny wrote: >>> W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico >>> napisał: Since key refresh is prone to failure, retry using exponential backoff with random jitter. This adds the following sync-openpgp-* configuration settings: sync-openpgp-key-refresh-retry-count = 40 Maximum number of times to retry key refresh if it fails. Between each key refresh attempt, there is an exponential delay with a constant multiplier and a uniform random multiplier between 0 and 1. sync-openpgp-key-refresh-retry-delay-exp-base = 2 The base of the exponential expression. The exponent is the number of previous refresh attempts. sync-openpgp-key-refresh-retry-delay-max = 60 Maximum delay between each retry attempt, in units of seconds. This places a limit on the length of the exponential delay. sync-openpgp-key-refresh-retry-delay-mult = 4 Multiplier for the exponential delay. sync-openpgp-key-refresh-retry-overall-timeout = 1200 Combined time limit for all refresh attempts, in units of seconds. Bug: https://bugs.gentoo.org/649276 Zac Medico (4): Add ForkExecutor (bug 649588) Add ExponentialBackoff and RandomExponentialBackoff Add retry decorator (API inspired by tenacity) rsync: add key refresh retry (bug 649276) cnf/repos.conf| 5 + man/portage.5 | 19 +++ pym/portage/repository/config.py | 22 pym/portage/sync/modules/rsync/rsync.py | 16 ++- pym/portage/sync/syncbase.py | 85 +++- pym/portage/tests/util/futures/test_retry.py | 147 + pym/portage/util/_eventloop/EventLoop.py | 45 ++- pym/portage/util/backoff.py | 48 +++ pym/portage/util/futures/executor/__init__.py | 0 pym/portage/util/futures/executor/fork.py | 130 +++ pym/portage/util/futures/futures.py | 6 + pym/portage/util/futures/retry.py | 178 ++ 12 files changed, 697 insertions(+), 4 deletions(-) create mode 100644 pym/portage/tests/util/futures/test_retry.py create mode 100644 pym/portage/util/backoff.py create mode 100644 pym/portage/util/futures/executor/__init__.py create mode 100644 pym/portage/util/futures/executor/fork.py create mode 100644 pym/portage/util/futures/retry.py >>> >>> This essentially looks like ~700 lines of code to try to workaround >>> broken networking. I would rather try to do that using 5 lines of code >>> but that's just me, and my programs aren't enterprise quality. I just >>> hope it actually solves as many problems as it's going to introduce. >> >> The vast majority of this code is generic and reusable, and I do intend >> to reuse it. For example, the executor support will be an essential >> piece for the asyncio.AbstractEventLoop for bug 649588. > > Sure it is and sure you will. But tell me: who is going to maintain it > all? Because as far as I can see, we're still dealing with a bus factor > of one and all you're doing is making it worse. More code, more > complexity, more creeping featurism and hacks. In the worst case, people can disable the retry feature and all of the associated code can be disabled simply by setting sync-openpgp-key-refresh-retry-count = 0 in repos.conf. In order to reduce the bus factor, I'm modeling the code on the standard library's asyncio framework. This allows use to leverage people's experience with asyncio, and also introduces people to asyncio concepts if they are not yet familar. > Last time you went away, you left us with a horribly unmaintainable > package manager full of complexity, hacks and creeping featurism > and a Portage team whose members had barely any knowledge of the code. > Just when things started moving again, you came back and we're back to > square one. I'd love to help reduce the bus factor, and I'm working to do that, like with the asyncio stuff. > Today Portage once again is a one-developer project, full of more > complexity, more hacks and more creeping featurism. And we once again > have a bus factor of one -- one developer who apparently knows > everything, does everything and tries to be nice to everyone, except he > really ignores others, makes a lot of empty promises and consistently > makes the health of the project go from bad to worse. I disagree with your synopsis. > So, please tell me: what happens when you leave again? How have you used > your time in the project? What have you done to make sure that > the project stays alive
Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
On 01-04-2018 20:29:59 +0200, Michał Górny wrote: > > > This essentially looks like ~700 lines of code to try to workaround > > > broken networking. I would rather try to do that using 5 lines of code > > > but that's just me, and my programs aren't enterprise quality. I just > > > hope it actually solves as many problems as it's going to introduce. > > > > The vast majority of this code is generic and reusable, and I do intend > > to reuse it. For example, the executor support will be an essential > > piece for the asyncio.AbstractEventLoop for bug 649588. > > Sure it is and sure you will. But tell me: who is going to maintain it > all? Because as far as I can see, we're still dealing with a bus factor > of one and all you're doing is making it worse. More code, more > complexity, more creeping featurism and hacks. Well, well, well. This could be said about a lot of code, including your own. > Last time you went away, you left us with a horribly unmaintainable > package manager full of complexity, hacks and creeping featurism > and a Portage team whose members had barely any knowledge of the code. > Just when things started moving again, you came back and we're back to > square one. I don't see why this has to be made personal. In the olde days people just pushed whatever they needed, now it's reviewed and acked, so how can it be back to square one? > Today Portage once again is a one-developer project, full of more > complexity, more hacks and more creeping featurism. And we once again > have a bus factor of one -- one developer who apparently knows > everything, does everything and tries to be nice to everyone, except he > really ignores others, makes a lot of empty promises and consistently > makes the health of the project go from bad to worse. Errr, didn't you recently start your own fork with creeping featurism of its own because mainline was/is too slow? > So, please tell me: what happens when you leave again? How have you used > your time in the project? What have you done to make sure that > the project stays alive without you? Because as far as I can see, adding > few thousand of lines of practically unreviewed code every month does > not help with that. Why is this Zac's problem specifically? Isn't this a general problem of Gentoo? And isn't your attitude contributing in a large factor to the bus-factor getting down from 1 to 0? > I forked Portage because I didn't want to fight with you. When I forked > it, I declared that I will merge mainline changes regularly for > the benefit of my users. But after a week, I really start feeling like > that's going to be a really bad idea. Like it's time to forget about > mainline Portage as a completely dead end, and go our separate ways. So you fork. Like many forks, that is because people feel that it isn't going "their way". Yay. That doesn't make you "right". > I'm seriously worried about the future of Gentoo. I'd really appreciate > if you started focusing on that as well. I get that all this stuff looks > cool on paper but few months or years from now, someone will curse > 'whoever wrote that code' while trying to fix some nasty bug. Or get > things moving forward. Or implement EAPI 8. Perhaps it's time to look into the mirror. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico napisał: > On 04/01/2018 03:57 AM, Michał Górny wrote: > > W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico > > napisał: > > > Since key refresh is prone to failure, retry using exponential > > > backoff with random jitter. This adds the following sync-openpgp-* > > > configuration settings: > > > > > > sync-openpgp-key-refresh-retry-count = 40 > > > > > > Maximum number of times to retry key refresh if it fails. Between > > > each key refresh attempt, there is an exponential delay with a > > > constant multiplier and a uniform random multiplier between 0 and 1. > > > > > > sync-openpgp-key-refresh-retry-delay-exp-base = 2 > > > > > > The base of the exponential expression. The exponent is the number > > > of previous refresh attempts. > > > > > > sync-openpgp-key-refresh-retry-delay-max = 60 > > > > > > Maximum delay between each retry attempt, in units of seconds. This > > > places a limit on the length of the exponential delay. > > > > > > sync-openpgp-key-refresh-retry-delay-mult = 4 > > > > > > Multiplier for the exponential delay. > > > > > > sync-openpgp-key-refresh-retry-overall-timeout = 1200 > > > > > > Combined time limit for all refresh attempts, in units of seconds. > > > > > > Bug: https://bugs.gentoo.org/649276 > > > > > > Zac Medico (4): > > > Add ForkExecutor (bug 649588) > > > Add ExponentialBackoff and RandomExponentialBackoff > > > Add retry decorator (API inspired by tenacity) > > > rsync: add key refresh retry (bug 649276) > > > > > > cnf/repos.conf| 5 + > > > man/portage.5 | 19 +++ > > > pym/portage/repository/config.py | 22 > > > pym/portage/sync/modules/rsync/rsync.py | 16 ++- > > > pym/portage/sync/syncbase.py | 85 +++- > > > pym/portage/tests/util/futures/test_retry.py | 147 + > > > pym/portage/util/_eventloop/EventLoop.py | 45 ++- > > > pym/portage/util/backoff.py | 48 +++ > > > pym/portage/util/futures/executor/__init__.py | 0 > > > pym/portage/util/futures/executor/fork.py | 130 +++ > > > pym/portage/util/futures/futures.py | 6 + > > > pym/portage/util/futures/retry.py | 178 > > > ++ > > > 12 files changed, 697 insertions(+), 4 deletions(-) > > > create mode 100644 pym/portage/tests/util/futures/test_retry.py > > > create mode 100644 pym/portage/util/backoff.py > > > create mode 100644 pym/portage/util/futures/executor/__init__.py > > > create mode 100644 pym/portage/util/futures/executor/fork.py > > > create mode 100644 pym/portage/util/futures/retry.py > > > > > > > This essentially looks like ~700 lines of code to try to workaround > > broken networking. I would rather try to do that using 5 lines of code > > but that's just me, and my programs aren't enterprise quality. I just > > hope it actually solves as many problems as it's going to introduce. > > The vast majority of this code is generic and reusable, and I do intend > to reuse it. For example, the executor support will be an essential > piece for the asyncio.AbstractEventLoop for bug 649588. Sure it is and sure you will. But tell me: who is going to maintain it all? Because as far as I can see, we're still dealing with a bus factor of one and all you're doing is making it worse. More code, more complexity, more creeping featurism and hacks. Last time you went away, you left us with a horribly unmaintainable package manager full of complexity, hacks and creeping featurism and a Portage team whose members had barely any knowledge of the code. Just when things started moving again, you came back and we're back to square one. Today Portage once again is a one-developer project, full of more complexity, more hacks and more creeping featurism. And we once again have a bus factor of one -- one developer who apparently knows everything, does everything and tries to be nice to everyone, except he really ignores others, makes a lot of empty promises and consistently makes the health of the project go from bad to worse. So, please tell me: what happens when you leave again? How have you used your time in the project? What have you done to make sure that the project stays alive without you? Because as far as I can see, adding few thousand of lines of practically unreviewed code every month does not help with that. I forked Portage because I didn't want to fight with you. When I forked it, I declared that I will merge mainline changes regularly for the benefit of my users. But after a week, I really start feeling like that's going to be a really bad idea. Like it's time to forget about mainline Portage as a completely dead end, and go our separate ways. I'm seriously worried about the future of Gentoo. I'd really appreciate if you st
Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
On 04/01/2018 03:57 AM, Michał Górny wrote: > W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico > napisał: >> Since key refresh is prone to failure, retry using exponential >> backoff with random jitter. This adds the following sync-openpgp-* >> configuration settings: >> >> sync-openpgp-key-refresh-retry-count = 40 >> >> Maximum number of times to retry key refresh if it fails. Between >> each key refresh attempt, there is an exponential delay with a >> constant multiplier and a uniform random multiplier between 0 and 1. >> >> sync-openpgp-key-refresh-retry-delay-exp-base = 2 >> >> The base of the exponential expression. The exponent is the number >> of previous refresh attempts. >> >> sync-openpgp-key-refresh-retry-delay-max = 60 >> >> Maximum delay between each retry attempt, in units of seconds. This >> places a limit on the length of the exponential delay. >> >> sync-openpgp-key-refresh-retry-delay-mult = 4 >> >> Multiplier for the exponential delay. >> >> sync-openpgp-key-refresh-retry-overall-timeout = 1200 >> >> Combined time limit for all refresh attempts, in units of seconds. >> >> Bug: https://bugs.gentoo.org/649276 >> >> Zac Medico (4): >> Add ForkExecutor (bug 649588) >> Add ExponentialBackoff and RandomExponentialBackoff >> Add retry decorator (API inspired by tenacity) >> rsync: add key refresh retry (bug 649276) >> >> cnf/repos.conf| 5 + >> man/portage.5 | 19 +++ >> pym/portage/repository/config.py | 22 >> pym/portage/sync/modules/rsync/rsync.py | 16 ++- >> pym/portage/sync/syncbase.py | 85 +++- >> pym/portage/tests/util/futures/test_retry.py | 147 + >> pym/portage/util/_eventloop/EventLoop.py | 45 ++- >> pym/portage/util/backoff.py | 48 +++ >> pym/portage/util/futures/executor/__init__.py | 0 >> pym/portage/util/futures/executor/fork.py | 130 +++ >> pym/portage/util/futures/futures.py | 6 + >> pym/portage/util/futures/retry.py | 178 >> ++ >> 12 files changed, 697 insertions(+), 4 deletions(-) >> create mode 100644 pym/portage/tests/util/futures/test_retry.py >> create mode 100644 pym/portage/util/backoff.py >> create mode 100644 pym/portage/util/futures/executor/__init__.py >> create mode 100644 pym/portage/util/futures/executor/fork.py >> create mode 100644 pym/portage/util/futures/retry.py >> > > This essentially looks like ~700 lines of code to try to workaround > broken networking. I would rather try to do that using 5 lines of code > but that's just me, and my programs aren't enterprise quality. I just > hope it actually solves as many problems as it's going to introduce. The vast majority of this code is generic and reusable, and I do intend to reuse it. For example, the executor support will be an essential piece for the asyncio.AbstractEventLoop for bug 649588. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico napisał: > Since key refresh is prone to failure, retry using exponential > backoff with random jitter. This adds the following sync-openpgp-* > configuration settings: > > sync-openpgp-key-refresh-retry-count = 40 > > Maximum number of times to retry key refresh if it fails. Between > each key refresh attempt, there is an exponential delay with a > constant multiplier and a uniform random multiplier between 0 and 1. > > sync-openpgp-key-refresh-retry-delay-exp-base = 2 > > The base of the exponential expression. The exponent is the number > of previous refresh attempts. > > sync-openpgp-key-refresh-retry-delay-max = 60 > > Maximum delay between each retry attempt, in units of seconds. This > places a limit on the length of the exponential delay. > > sync-openpgp-key-refresh-retry-delay-mult = 4 > > Multiplier for the exponential delay. > > sync-openpgp-key-refresh-retry-overall-timeout = 1200 > > Combined time limit for all refresh attempts, in units of seconds. > > Bug: https://bugs.gentoo.org/649276 > > Zac Medico (4): > Add ForkExecutor (bug 649588) > Add ExponentialBackoff and RandomExponentialBackoff > Add retry decorator (API inspired by tenacity) > rsync: add key refresh retry (bug 649276) > > cnf/repos.conf| 5 + > man/portage.5 | 19 +++ > pym/portage/repository/config.py | 22 > pym/portage/sync/modules/rsync/rsync.py | 16 ++- > pym/portage/sync/syncbase.py | 85 +++- > pym/portage/tests/util/futures/test_retry.py | 147 + > pym/portage/util/_eventloop/EventLoop.py | 45 ++- > pym/portage/util/backoff.py | 48 +++ > pym/portage/util/futures/executor/__init__.py | 0 > pym/portage/util/futures/executor/fork.py | 130 +++ > pym/portage/util/futures/futures.py | 6 + > pym/portage/util/futures/retry.py | 178 > ++ > 12 files changed, 697 insertions(+), 4 deletions(-) > create mode 100644 pym/portage/tests/util/futures/test_retry.py > create mode 100644 pym/portage/util/backoff.py > create mode 100644 pym/portage/util/futures/executor/__init__.py > create mode 100644 pym/portage/util/futures/executor/fork.py > create mode 100644 pym/portage/util/futures/retry.py > This essentially looks like ~700 lines of code to try to workaround broken networking. I would rather try to do that using 5 lines of code but that's just me, and my programs aren't enterprise quality. I just hope it actually solves as many problems as it's going to introduce. -- Best regards, Michał Górny
[gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)
Since key refresh is prone to failure, retry using exponential backoff with random jitter. This adds the following sync-openpgp-* configuration settings: sync-openpgp-key-refresh-retry-count = 40 Maximum number of times to retry key refresh if it fails. Between each key refresh attempt, there is an exponential delay with a constant multiplier and a uniform random multiplier between 0 and 1. sync-openpgp-key-refresh-retry-delay-exp-base = 2 The base of the exponential expression. The exponent is the number of previous refresh attempts. sync-openpgp-key-refresh-retry-delay-max = 60 Maximum delay between each retry attempt, in units of seconds. This places a limit on the length of the exponential delay. sync-openpgp-key-refresh-retry-delay-mult = 4 Multiplier for the exponential delay. sync-openpgp-key-refresh-retry-overall-timeout = 1200 Combined time limit for all refresh attempts, in units of seconds. Bug: https://bugs.gentoo.org/649276 Zac Medico (4): Add ForkExecutor (bug 649588) Add ExponentialBackoff and RandomExponentialBackoff Add retry decorator (API inspired by tenacity) rsync: add key refresh retry (bug 649276) cnf/repos.conf| 5 + man/portage.5 | 19 +++ pym/portage/repository/config.py | 22 pym/portage/sync/modules/rsync/rsync.py | 16 ++- pym/portage/sync/syncbase.py | 85 +++- pym/portage/tests/util/futures/test_retry.py | 147 + pym/portage/util/_eventloop/EventLoop.py | 45 ++- pym/portage/util/backoff.py | 48 +++ pym/portage/util/futures/executor/__init__.py | 0 pym/portage/util/futures/executor/fork.py | 130 +++ pym/portage/util/futures/futures.py | 6 + pym/portage/util/futures/retry.py | 178 ++ 12 files changed, 697 insertions(+), 4 deletions(-) create mode 100644 pym/portage/tests/util/futures/test_retry.py create mode 100644 pym/portage/util/backoff.py create mode 100644 pym/portage/util/futures/executor/__init__.py create mode 100644 pym/portage/util/futures/executor/fork.py create mode 100644 pym/portage/util/futures/retry.py -- 2.13.6
