[gentoo-dev] Re: Monthly x11@ project status for April 2018

2018-04-01 Thread Duncan 
Matt Turner posted on Sun, 01 Apr 2018 20:08:35 -0700 as excerpted:

> My list of to-do items consists of:
> 
> == Fix x11-base/xorg-server suid/systemd situation ==
> https://bugs.gentoo.org/635102
> 
> Under some circumstances (kernel modesetting driver + systemd, I think)
> Xorg should be able to run without root privileges. We were shipping a
> USE=suid option without anyone knowing or understanding its purpose.

FWIW I understood it, but also knew it broke X for me back when I first 
tried it.  However...

> For >=x11-base/xorg-server-1.20 I plan to ship the xserver in a way that
> allows systemd/elogind users with kernel modesetting drivers to run Xorg
> without root privileges. I expect to push version 1.19.99.902 (1.20 RC2)
> into the tree soon with something working for systemd. I would very much
> appreciate an ebuild patch from any elogind user as well as non-systemd
> testing to make sure I haven't broken anything like I did with
> 1.19.99.901.

I noticed the recent no-superuser X changes here (on ~amd64), and decided 
to try it again...

And now (after undoing an old hack I had to manually set SUID here) I 
have X running as my normal user.  Thanks! =:^)

FWIW, systemd with modesetting (amdgpu), as you suspected.  startx (no *dm 
at all merged).  X starts on top of the vt1 login.  xorg-
server-1.19.99.901-r1

> == Update packages to depend on x11-base/xorg-proto ==
> https://bugs.gentoo.org/651286
> 
> The new x11-base/xorg-proto package combines nearly all (28 in fact) of
> the x11-proto/* packages into one, with a very fast Meson build system.
> It installs on my laptop in less time than it takes to ./configure one
> of the individual x11-proto/ packages. I've kept empty versions of the
> x11-proto/ packages to ease the transition.

I noticed that I didn't need many of the protos any longer here too, and 
figured it was a recombining.  Thanks for the confirmation. =:^)

And thanks for the roadmap to what's ahead re X. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Monthly mips@ project status for April 2018

2018-04-01 Thread Matt Turner 

I'd like to start giving ~monthly updates on the status of mips@ in
Gentoo.

Recently I received a Loongson 3A system (quad-core 1.35GHz, 16GB RAM,
AMD graphics) which is significantly faster and more stable than any
other mips system I have.

mips@ is currently assigned or cc'd on 29 bugs. This number is down from
probably >50 at the beginning of March.

My list of to-do items consists of:

== Switch to stable profiles ==
https://github.com/gentoo/gentoo/pull/7672

While keeping mips unstable (KEYWORDS=~mips) I plan to switch the
profiles from exp to stable so that no new breakage is added.

To that end, I've been working to test and keyword packages until we can
flip the switch without adding any breakage ourselves. See the github
pull request for the current status. I'll periodically rebase the branch
as I add more keywords to the main tree.

My plan is to add stable 17.0 mips profiles when the keywording is
sorted out and kill two birds with one stone.

== Loongson 3A support ==

The Loongson 3A system I received requires a number of out of tree
patches to the kernel, gcc, binutils, and glibc. I've seen their
developers working to upstream kernel patches, but it's slow going and
there are a lot of them.

I haven't seen much work to upstream the other patches. I'm not sure how
to reasonably support hardware requiring so many out-of-tree patches.

== n64 (the ABI!) support ==
https://bugs.gentoo.org/show_bug.cgi?id=442548
https://bugs.gentoo.org/show_bug.cgi?id=513042

I want to support the tree main ABIs: Old 32 ("o32"), New 32 ("n32"),
and New 64 ("n64").

n32 is equivalent to amd64's x32 ABI: 64-bit ABI but with 32-bit
pointers. It's the default ABI we use for 64-bit MIPS.

I'd like to offer n64 support as well, but gcc is not quite ready. At
this point I have figured out the gcc build system enough to hack
something together that works. I just need to... upstream something I
guess?

== n64 toolchain in n32 system ==
https://bugs.gentoo.org/show_bug.cgi?id=477956

n32 (and o32) only offer 31-bits of address space (2GB). That's not
sufficient to link large libraries like webkit-gtk. For n32 systems, I'd
like the toolchain binaries to be n64, to avoid these problems.

That's going to be some amount of work on top of regular n64 support,
but hopefully not too much. multilib/ABI_MIPS support already exists and
works.

== stages and installation media ==
https://bugs.gentoo.org/show_bug.cgi?id=150402
https://bugs.gentoo.org/show_bug.cgi?id=348647

With so many subarchitectures, ABIs, and two byte orderings providing
stages and installation media has been a pain point.

I'd like to automate as much of this as possible. I really need Kumba to
build a new SGI CD, but well, that bug's been open since 2006.


signature.asc
Description: Digital signature

[gentoo-dev] Monthly x11@ project status for April 2018

2018-04-01 Thread Matt Turner 

I'd like to start giving ~monthly updates on the status of x11@ packages
in Gentoo. I hope it gives some insight into the status of a rather
important set of packages and maybe encourages others to lend a hand to
a very understaffed project when possible.

I expect future reports to be significantly shorter.

x11@ currently maintains 291 packages. This number is down from 328
after the removal of 37 drivers for very old hardware in bug 606132.

x11@ is currently assigned or cc'd on 222 bugs. This number is down from
more than 412 in February 2015 (I reported this on #gentoo-desktop after
closing out a bunch of bugs that day).

I work on media-libs/mesa professionally for Intel, so I am involved
with upstream for a number of the projects maintained by x11@ and know
the right people to contact for the others. My strategy maintaining x11
packages is to keep a git ebuild in sync with upstream changes. When a
release is made, everything should already be ready on the Gentoo side,
and I'm able to just copy the git ebuild. This has enabled me to provide
new versions of packages very quickly. I believe it has also helped
reduce the time to stabilizing packages, since users often test and
report bugs against the git ebuilds.

My list of to-do items consists of:

== Fix x11-base/xorg-server suid/systemd situation ==
https://bugs.gentoo.org/635102

Under some circumstances (kernel modesetting driver + systemd, I think)
Xorg should be able to run without root privileges. We were shipping a
USE=suid option without anyone knowing or understanding its purpose.

For >=x11-base/xorg-server-1.20 I plan to ship the xserver in a way that
allows systemd/elogind users with kernel modesetting drivers to run Xorg
without root privileges. I expect to push version 1.19.99.902 (1.20 RC2)
into the tree soon with something working for systemd. I would very much
appreciate an ebuild patch from any elogind user as well as non-systemd
testing to make sure I haven't broken anything like I did with
1.19.99.901.

== Update packages to depend on x11-base/xorg-proto ==
https://bugs.gentoo.org/651286

The new x11-base/xorg-proto package combines nearly all (28 in fact) of
the x11-proto/* packages into one, with a very fast Meson build system.
It installs on my laptop in less time than it takes to ./configure one
of the individual x11-proto/ packages. I've kept empty versions of the
x11-proto/ packages to ease the transition.

Once the last two architectures have stabilized it (arm@ and arm64@), we
can begin transitioning to depending on x11-base/xorg-proto directly
instead of the x11-proto/ packages.

If there's a way to have repoman alert developers to deprecated
dependencies in the same way we handle deprecated eclasses, I'd like to
know about it.

== Remove x11-proto/printproto and x11-libs/libXp ==
https://bugs.gentoo.org/649076

Support for the Xprint extension was removed from xorg-server-1.6.0
released 9 years ago, yet a handful of packages still claimed a
dependence on the Xprint client library libXp.

We've nearly removed all traces of it, and are waiting on the last few
bugs to be closed out.

== Convert media-libs/mesa ebuild to build with Meson ==

Mesa has grown a Meson build system over the last few months. I plan to
ship media-libs/mesa-18.1 (Q2 2018) using Meson.

hanetzer in #gentoo-desktop has been working on this.

== Convert media-libs/xorg-server ebuild to build with Meson ==

The Xserver has also grown a Meson build system. I think it will
probably be in reasonable shape by 1.20, but I don't want to delay it
getting into the tree for that. 1.21 will probably be a long way off,
but I expect to ship it with Meson, if not something before.

No work has been done on this.

== Add and require glvnd ==
https://bugs.gentoo.org/606924
https://github.com/NVIDIA/libglvnd

OpenGL has historically been provided by a vendor's libGL.so. That's
caused a number of headaches for distributions, since both
x11-drivers/nvidia-drivers and media-libs/mesa would like to install
/usr/lib/libGL.so.

The GL Vendor-Neutral Dispatch library ("glvnd") defines a new common
library interface, which dispatches to the underlying hardware driver.

Mesa and the NVIDIA drivers both have support, but it needs to be wired
up in Gentoo. Doing this would allow us to get rid of
app-eselect/eselect-opengl and a whole class of build failures.

No work has been done on this. I plan to do it after Mesa is converted
to build with Meson.

== Drop app-eselect/eselect-mesa ==
https://bugs.gentoo.org/576334

While app-eselect/eselect-opengl allows users to switch between OpenGL
implementations (x11-drivers/nvidia-drivers vs media-libs/mesa),
app-eselect/eselect-mesa allows users to switch between Mesa drivers for
the same hardware. This made sense a few years ago, when there were both
Gallium and "classic" DRI drivers for the same hardware, but today only
the 10 year old i915/i945 integrated graphics still falls into this
category. Frankly, both of the driver options

[gentoo-dev] Automated Package Removal and Addition Tracker, for the week ending 2018-04-01 23:59 UTC

2018-04-01 Thread Robin H. Johnson 
The attached list notes all of the packages that were added or removed
from the tree, for the week ending 2018-04-01 23:59 UTC.

Removals:
app-pda/p3nfs  20180330-18:07 dilfridge ea6e2f40a16
app-text/searchmonkey  20180328-21:43 asturm12272bc5a86
dev-tex/qtexengine 20180328-21:43 asturme8057e08c55
dev-util/drone-ui  20180330-16:52 mrueg 1599a85cd68
media-video/abby   20180328-20:48 asturm75d1824f032
net-fs/torus   20180328-21:53 zmedico   ca7746f52fa

Additions:
app-emacs/ghub 20180331-06:22 graaff1eb09752210
app-metrics/prometheus-mysqld_exporter 20180326-22:38 whissi101f87f123c
dev-go/fuzzy   20180328-21:49 mgorny6360f43aa6d
dev-go/godebug-pretty  20180328-21:47 mgornye223e4b86ea
dev-lang/mono-basic20180325-12:42 cynedecb1aff090b1
media-fonts/fira-code  20180331-07:39 graaff1bc69a5e200
media-fonts/iosevka20180326-14:57 mgorny218d847e9f3
net-analyzer/kapacitor 20180326-02:54 williamh  f3ed20bf98a
net-libs/srt   20180322-16:45 lu_zero   aeccfd18102
net-misc/casync20180326-14:45 mrueg eb6b676bea2
sys-apps/smcipmitool   20180329-14:40 mgornyffa696581eb
sys-cluster/pmix   20180401-20:43 jlec  69972533a7a
sys-cluster/ucx20180401-20:36 jlec  978c8e081d9
sys-fs/compsize20180327-19:34 mgorny6b6d7b9052d
sys-libs/rpmatch-standalone20180329-11:28 blueness  1ffea40c4a4
www-apps/hugo  20180330-15:18 mrueg b393e4812cd
x11-misc/xflux 20180327-20:41 mgornyf8368de4a58
x11-misc/xflux-gui 20180329-12:56 mgorny133aec6aa2e

--
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail : robb...@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85
Removed Packages:
app-pda/p3nfs,removed,dilfridge,20180330-18:07,ea6e2f40a16
dev-util/drone-ui,removed,mrueg,20180330-16:52,1599a85cd68
net-fs/torus,removed,zmedico,20180328-21:53,ca7746f52fa
dev-tex/qtexengine,removed,asturm,20180328-21:43,e8057e08c55
app-text/searchmonkey,removed,asturm,20180328-21:43,12272bc5a86
media-video/abby,removed,asturm,20180328-20:48,75d1824f032
Added Packages:
sys-cluster/pmix,added,jlec,20180401-20:43,69972533a7a
sys-cluster/ucx,added,jlec,20180401-20:36,978c8e081d9
media-fonts/fira-code,added,graaff,20180331-07:39,1bc69a5e200
app-emacs/ghub,added,graaff,20180331-06:22,1eb09752210
sys-fs/compsize,added,mgorny,20180327-19:34,6b6d7b9052d
dev-go/fuzzy,added,mgorny,20180328-21:49,6360f43aa6d
dev-go/godebug-pretty,added,mgorny,20180328-21:47,e223e4b86ea
www-apps/hugo,added,mrueg,20180330-15:18,b393e4812cd
sys-apps/smcipmitool,added,mgorny,20180329-14:40,ffa696581eb
x11-misc/xflux-gui,added,mgorny,20180329-12:56,133aec6aa2e
sys-libs/rpmatch-standalone,added,blueness,20180329-11:28,1ffea40c4a4
x11-misc/xflux,added,mgorny,20180327-20:41,f8368de4a58
media-fonts/iosevka,added,mgorny,20180326-14:57,218d847e9f3
app-metrics/prometheus-mysqld_exporter,added,whissi,20180326-22:38,101f87f123c
net-libs/srt,added,lu_zero,20180322-16:45,aeccfd18102
net-misc/casync,added,mrueg,20180326-14:45,eb6b676bea2
dev-lang/mono-basic,added,cynede,20180325-12:42,cb1aff090b1
net-analyzer/kapacitor,added,williamh,20180326-02:54,f3ed20bf98a

Done.

[gentoo-dev] Packages up for grabs: app-office/mdbtools

2018-04-01 Thread Jonas Stein 
Dear all,

The following packages are up for grabs:

app-office/mdbtools

after retirement of the proxied maintainer.
https://packages.gentoo.org/packages/app-office/mdbtools

"Set of libraries and utilities for reading Microsoft Access database
(MDB) files"

Todo:
HOMEPAGE links to SF, but we distribute via SRC_URI from github. It is
unclear, if the project moved, or forked.
See also: https://github.com/brianb/mdbtools/issues/14

CC to the former maintainer in the hope, that he can explain why this
source was chosen.

-- 
Best,
Jonas













































signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Alec Warner 
On Sun, Apr 1, 2018 at 2:29 PM, Michał Górny  wrote:

> W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico
> napisał:
> > On 04/01/2018 03:57 AM, Michał Górny wrote:
> > > W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico
> > > napisał:
> > > > Since key refresh is prone to failure, retry using exponential
> > > > backoff with random jitter. This adds the following sync-openpgp-*
> > > > configuration settings:
> > > >
> > > > sync-openpgp-key-refresh-retry-count = 40
> > > >
> > > >   Maximum number of times to retry key refresh if it fails.  Between
> > > >   each  key  refresh attempt, there is an exponential delay with a
> > > >   constant multiplier and a uniform random multiplier between 0 and
> 1.
> > > >
> > > > sync-openpgp-key-refresh-retry-delay-exp-base = 2
> > > >
> > > >   The base of the exponential expression. The exponent is the number
> > > >   of previous refresh attempts.
> > > >
> > > > sync-openpgp-key-refresh-retry-delay-max = 60
> > > >
> > > >   Maximum  delay between each retry attempt, in units of seconds.
> This
> > > >   places a limit on the length of the exponential delay.
> > > >
> > > > sync-openpgp-key-refresh-retry-delay-mult = 4
> > > >
> > > >   Multiplier for the exponential delay.
> > > >
> > > > sync-openpgp-key-refresh-retry-overall-timeout = 1200
> > > >
> > > >   Combined time limit for all refresh attempts, in units of seconds.
> > > >
> > > > Bug: https://bugs.gentoo.org/649276
> > > >
> > > > Zac Medico (4):
> > > >   Add ForkExecutor (bug 649588)
> > > >   Add ExponentialBackoff and RandomExponentialBackoff
> > > >   Add retry decorator (API inspired by tenacity)
> > > >   rsync: add key refresh retry (bug 649276)
> > > >
> > > >  cnf/repos.conf|   5 +
> > > >  man/portage.5 |  19 +++
> > > >  pym/portage/repository/config.py  |  22 
> > > >  pym/portage/sync/modules/rsync/rsync.py   |  16 ++-
> > > >  pym/portage/sync/syncbase.py  |  85 +++-
> > > >  pym/portage/tests/util/futures/test_retry.py  | 147
> +
> > > >  pym/portage/util/_eventloop/EventLoop.py  |  45 ++-
> > > >  pym/portage/util/backoff.py   |  48 +++
> > > >  pym/portage/util/futures/executor/__init__.py |   0
> > > >  pym/portage/util/futures/executor/fork.py | 130
> +++
> > > >  pym/portage/util/futures/futures.py   |   6 +
> > > >  pym/portage/util/futures/retry.py | 178
> ++
> > > >  12 files changed, 697 insertions(+), 4 deletions(-)
> > > >  create mode 100644 pym/portage/tests/util/futures/test_retry.py
> > > >  create mode 100644 pym/portage/util/backoff.py
> > > >  create mode 100644 pym/portage/util/futures/executor/__init__.py
> > > >  create mode 100644 pym/portage/util/futures/executor/fork.py
> > > >  create mode 100644 pym/portage/util/futures/retry.py
> > > >
> > >
> > > This essentially looks like ~700 lines of code to try to workaround
> > > broken networking. I would rather try to do that using 5 lines of code
> > > but that's just me, and my programs aren't enterprise quality. I just
> > > hope it actually solves as many problems as it's going to introduce.
> >
> > The vast majority of this code is generic and reusable, and I do intend
> > to reuse it. For example, the executor support will be an essential
> > piece for the asyncio.AbstractEventLoop for bug 649588.
>
> Sure it is and sure you will. But tell me: who is going to maintain it
> all? Because as far as I can see, we're still dealing with a bus factor
> of one and all you're doing is making it worse. More code, more
> complexity, more creeping featurism and hacks.
>

I'll split this reply into two sorts of points. One is a brief point about
this patchset. The other is a larger point about the project and its
direction.

About this patchset:

I don't mind this code because its not domain specific and I can read it
and understand it. Many apps need async ways to run code, and anyone who
has used the futures API will find this code familiar. I reviewed it in
about 10 minutes. I'm not heavily invested with the async-wagon that Zac is
driving, but I also see how its an area where performance can be improved
by doing more stuff at once using an async operations. I think if you have
more generic concerns with the async frameworks I'd love to see some
discussion about them (particularly around how we can improve perf by using
tighter algorithms; often the 'perf' boost is just higher CPU utilization
and driving multi-core systems which results in better walltime perf, but
not necessarily cpu-perf.)

Its fine to disagree here. I think the project has some requirements around
minimized dependencies, so the choice is either to pull in some kind of
retry-lib and add a dep, or add this code. Its also not immediately clear
if the 3rd party deps would still not

Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Zac Medico 
On 04/01/2018 11:29 AM, Michał Górny wrote:
> W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico
> napisał:
>> On 04/01/2018 03:57 AM, Michał Górny wrote:
>>> W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico
>>> napisał:
 Since key refresh is prone to failure, retry using exponential
 backoff with random jitter. This adds the following sync-openpgp-*
 configuration settings:

 sync-openpgp-key-refresh-retry-count = 40

   Maximum number of times to retry key refresh if it fails.  Between
   each  key  refresh attempt, there is an exponential delay with a
   constant multiplier and a uniform random multiplier between 0 and 1.

 sync-openpgp-key-refresh-retry-delay-exp-base = 2

   The base of the exponential expression. The exponent is the number
   of previous refresh attempts.

 sync-openpgp-key-refresh-retry-delay-max = 60

   Maximum  delay between each retry attempt, in units of seconds. This
   places a limit on the length of the exponential delay.

 sync-openpgp-key-refresh-retry-delay-mult = 4

   Multiplier for the exponential delay.

 sync-openpgp-key-refresh-retry-overall-timeout = 1200

   Combined time limit for all refresh attempts, in units of seconds.

 Bug: https://bugs.gentoo.org/649276

 Zac Medico (4):
   Add ForkExecutor (bug 649588)
   Add ExponentialBackoff and RandomExponentialBackoff
   Add retry decorator (API inspired by tenacity)
   rsync: add key refresh retry (bug 649276)

  cnf/repos.conf|   5 +
  man/portage.5 |  19 +++
  pym/portage/repository/config.py  |  22 
  pym/portage/sync/modules/rsync/rsync.py   |  16 ++-
  pym/portage/sync/syncbase.py  |  85 +++-
  pym/portage/tests/util/futures/test_retry.py  | 147 +
  pym/portage/util/_eventloop/EventLoop.py  |  45 ++-
  pym/portage/util/backoff.py   |  48 +++
  pym/portage/util/futures/executor/__init__.py |   0
  pym/portage/util/futures/executor/fork.py | 130 +++
  pym/portage/util/futures/futures.py   |   6 +
  pym/portage/util/futures/retry.py | 178 
 ++
  12 files changed, 697 insertions(+), 4 deletions(-)
  create mode 100644 pym/portage/tests/util/futures/test_retry.py
  create mode 100644 pym/portage/util/backoff.py
  create mode 100644 pym/portage/util/futures/executor/__init__.py
  create mode 100644 pym/portage/util/futures/executor/fork.py
  create mode 100644 pym/portage/util/futures/retry.py

>>>
>>> This essentially looks like ~700 lines of code to try to workaround
>>> broken networking. I would rather try to do that using 5 lines of code
>>> but that's just me, and my programs aren't enterprise quality. I just
>>> hope it actually solves as many problems as it's going to introduce.
>>
>> The vast majority of this code is generic and reusable, and I do intend
>> to reuse it. For example, the executor support will be an essential
>> piece for the asyncio.AbstractEventLoop for bug 649588.
> 
> Sure it is and sure you will. But tell me: who is going to maintain it
> all? Because as far as I can see, we're still dealing with a bus factor
> of one and all you're doing is making it worse. More code, more
> complexity, more creeping featurism and hacks.

In the worst case, people can disable the retry feature and all of the
associated code can be disabled simply by setting
sync-openpgp-key-refresh-retry-count = 0 in repos.conf.

In order to reduce the bus factor, I'm modeling the code on the standard
library's asyncio framework. This allows use to leverage people's
experience with asyncio, and also introduces people to asyncio concepts
if they are not yet familar.

> Last time you went away, you left us with a horribly unmaintainable
> package manager full of complexity, hacks and creeping featurism
> and a Portage team whose members had barely any knowledge of the code.
> Just when things started moving again, you came back and we're back to
> square one.

I'd love to help reduce the bus factor, and I'm working to do that, like
with the asyncio stuff.

> Today Portage once again is a one-developer project, full of more
> complexity, more hacks and more creeping featurism. And we once again
> have a bus factor of one -- one developer who apparently knows
> everything, does everything and tries to be nice to everyone, except he
> really ignores others, makes a lot of empty promises and consistently
> makes the health of the project go from bad to worse.

I disagree with your synopsis.

> So, please tell me: what happens when you leave again? How have you used
> your time in the project? What have you done to make sure that
> the project stays

Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Fabian Groffen 
On 01-04-2018 20:29:59 +0200, Michał Górny wrote:
> > > This essentially looks like ~700 lines of code to try to workaround
> > > broken networking. I would rather try to do that using 5 lines of code
> > > but that's just me, and my programs aren't enterprise quality. I just
> > > hope it actually solves as many problems as it's going to introduce.
> > 
> > The vast majority of this code is generic and reusable, and I do intend
> > to reuse it. For example, the executor support will be an essential
> > piece for the asyncio.AbstractEventLoop for bug 649588.
> 
> Sure it is and sure you will. But tell me: who is going to maintain it
> all? Because as far as I can see, we're still dealing with a bus factor
> of one and all you're doing is making it worse. More code, more
> complexity, more creeping featurism and hacks.

Well, well, well.  This could be said about a lot of code, including
your own.

> Last time you went away, you left us with a horribly unmaintainable
> package manager full of complexity, hacks and creeping featurism
> and a Portage team whose members had barely any knowledge of the code.
> Just when things started moving again, you came back and we're back to
> square one.

I don't see why this has to be made personal.  In the olde days people
just pushed whatever they needed, now it's reviewed and acked, so how
can it be back to square one?

> Today Portage once again is a one-developer project, full of more
> complexity, more hacks and more creeping featurism. And we once again
> have a bus factor of one -- one developer who apparently knows
> everything, does everything and tries to be nice to everyone, except he
> really ignores others, makes a lot of empty promises and consistently
> makes the health of the project go from bad to worse.

Errr, didn't you recently start your own fork with creeping featurism of
its own because mainline was/is too slow?

> So, please tell me: what happens when you leave again? How have you used
> your time in the project? What have you done to make sure that
> the project stays alive without you? Because as far as I can see, adding
> few thousand of lines of practically unreviewed code every month does
> not help with that.

Why is this Zac's problem specifically?  Isn't this a general problem of
Gentoo?  And isn't your attitude contributing in a large factor to
the bus-factor getting down from 1 to 0?

> I forked Portage because I didn't want to fight with you. When I forked
> it, I declared that I will merge mainline changes regularly for
> the benefit of my users. But after a week, I really start feeling like
> that's going to be a really bad idea. Like it's time to forget about
> mainline Portage as a completely dead end, and go our separate ways.

So you fork.  Like many forks, that is because people feel that it isn't
going "their way".  Yay.  That doesn't make you "right".

> I'm seriously worried about the future of Gentoo. I'd really appreciate
> if you started focusing on that as well. I get that all this stuff looks
> cool on paper but few months or years from now, someone will curse
> 'whoever wrote that code' while trying to fix some nasty bug. Or get
> things moving forward. Or implement EAPI 8.

Perhaps it's time to look into the mirror.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level


signature.asc
Description: PGP signature

Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Michał Górny 
W dniu nie, 01.04.2018 o godzinie 08∶59 -0700, użytkownik Zac Medico
napisał:
> On 04/01/2018 03:57 AM, Michał Górny wrote:
> > W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico
> > napisał:
> > > Since key refresh is prone to failure, retry using exponential
> > > backoff with random jitter. This adds the following sync-openpgp-*
> > > configuration settings:
> > > 
> > > sync-openpgp-key-refresh-retry-count = 40
> > > 
> > >   Maximum number of times to retry key refresh if it fails.  Between
> > >   each  key  refresh attempt, there is an exponential delay with a
> > >   constant multiplier and a uniform random multiplier between 0 and 1.
> > > 
> > > sync-openpgp-key-refresh-retry-delay-exp-base = 2
> > > 
> > >   The base of the exponential expression. The exponent is the number
> > >   of previous refresh attempts.
> > > 
> > > sync-openpgp-key-refresh-retry-delay-max = 60
> > > 
> > >   Maximum  delay between each retry attempt, in units of seconds. This
> > >   places a limit on the length of the exponential delay.
> > > 
> > > sync-openpgp-key-refresh-retry-delay-mult = 4
> > > 
> > >   Multiplier for the exponential delay.
> > > 
> > > sync-openpgp-key-refresh-retry-overall-timeout = 1200
> > > 
> > >   Combined time limit for all refresh attempts, in units of seconds.
> > > 
> > > Bug: https://bugs.gentoo.org/649276
> > > 
> > > Zac Medico (4):
> > >   Add ForkExecutor (bug 649588)
> > >   Add ExponentialBackoff and RandomExponentialBackoff
> > >   Add retry decorator (API inspired by tenacity)
> > >   rsync: add key refresh retry (bug 649276)
> > > 
> > >  cnf/repos.conf|   5 +
> > >  man/portage.5 |  19 +++
> > >  pym/portage/repository/config.py  |  22 
> > >  pym/portage/sync/modules/rsync/rsync.py   |  16 ++-
> > >  pym/portage/sync/syncbase.py  |  85 +++-
> > >  pym/portage/tests/util/futures/test_retry.py  | 147 +
> > >  pym/portage/util/_eventloop/EventLoop.py  |  45 ++-
> > >  pym/portage/util/backoff.py   |  48 +++
> > >  pym/portage/util/futures/executor/__init__.py |   0
> > >  pym/portage/util/futures/executor/fork.py | 130 +++
> > >  pym/portage/util/futures/futures.py   |   6 +
> > >  pym/portage/util/futures/retry.py | 178 
> > > ++
> > >  12 files changed, 697 insertions(+), 4 deletions(-)
> > >  create mode 100644 pym/portage/tests/util/futures/test_retry.py
> > >  create mode 100644 pym/portage/util/backoff.py
> > >  create mode 100644 pym/portage/util/futures/executor/__init__.py
> > >  create mode 100644 pym/portage/util/futures/executor/fork.py
> > >  create mode 100644 pym/portage/util/futures/retry.py
> > > 
> > 
> > This essentially looks like ~700 lines of code to try to workaround
> > broken networking. I would rather try to do that using 5 lines of code
> > but that's just me, and my programs aren't enterprise quality. I just
> > hope it actually solves as many problems as it's going to introduce.
> 
> The vast majority of this code is generic and reusable, and I do intend
> to reuse it. For example, the executor support will be an essential
> piece for the asyncio.AbstractEventLoop for bug 649588.

Sure it is and sure you will. But tell me: who is going to maintain it
all? Because as far as I can see, we're still dealing with a bus factor
of one and all you're doing is making it worse. More code, more
complexity, more creeping featurism and hacks.

Last time you went away, you left us with a horribly unmaintainable
package manager full of complexity, hacks and creeping featurism
and a Portage team whose members had barely any knowledge of the code.
Just when things started moving again, you came back and we're back to
square one.

Today Portage once again is a one-developer project, full of more
complexity, more hacks and more creeping featurism. And we once again
have a bus factor of one -- one developer who apparently knows
everything, does everything and tries to be nice to everyone, except he
really ignores others, makes a lot of empty promises and consistently
makes the health of the project go from bad to worse.

So, please tell me: what happens when you leave again? How have you used
your time in the project? What have you done to make sure that
the project stays alive without you? Because as far as I can see, adding
few thousand of lines of practically unreviewed code every month does
not help with that.

I forked Portage because I didn't want to fight with you. When I forked
it, I declared that I will merge mainline changes regularly for
the benefit of my users. But after a week, I really start feeling like
that's going to be a really bad idea. Like it's time to forget about
mainline Portage as a completely dead end, and go our separate ways.

I'm seriously worried about the future of Gentoo. I'd really appreciate
if you

Re: [gentoo-portage-dev] [PATCH] INSTALL_MASK: honor install time config for binary packages (bug 651952)

2018-04-01 Thread Zac Medico 
On 04/01/2018 06:54 AM, Michał Górny wrote:
> W dniu czw, 29.03.2018 o godzinie 15∶34 -0700, użytkownik Zac Medico
> napisał:
>> For binary packages, honor the INSTALL_MASK configuration that
>> exists at install time, since it might differ from the build time
>> setting.
>>
>> Fixes: 3416876c0ee7 ("{,PKG_}INSTALL_MASK: python implementation")
>> Bug: https://bugs.gentoo.org/651952
>> ---
>>  bin/misc-functions.sh| 23 +++
>>  bin/phase-functions.sh   | 10 +-
>>  pym/portage/dbapi/vartree.py |  5 +
>>  3 files changed, 29 insertions(+), 9 deletions(-)
>>
>> diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
>> index 26f589915..a6330ee93 100755
>> --- a/bin/misc-functions.sh
>> +++ b/bin/misc-functions.sh
>> @@ -323,6 +323,29 @@ postinst_qa_check() {
>>  done < <(printf "%s\0" "${qa_checks[@]}" | LC_ALL=C sort -u -z)
>>  }
>>  
>> +preinst_mask() {
>> +# Remove man pages, info pages, docs if requested. This is
>> +# implemented in bash in order to respect INSTALL_MASK settings
>> +# from bashrc.
>> +local f x
>> +for f in man info doc; do
>> +if has no${f} ${FEATURES}; then
>> +INSTALL_MASK+=" /usr/share/${f}"
>> +fi
>> +done
>> +
>> +# Store modified variables in build-info.
>> +cd "${PORTAGE_BUILDDIR}"/build-info || die
>> +set -f
>> +
>> +IFS=$' \t\n\r'
>> +for f in INSTALL_MASK; do
> 
> This loop along with the whole indirection is entirely pointless, given
> that you're processing exactly one variable.

I did this for consistency with the related loop in dyn_install, since
we might save values of other variables from the binary package
environment. In fact, I think we should record DOC_SYMLINKS_DIR here,
since it affects CONTENTS, much like INSTALL_MASK.

>> +x=$(echo -n ${!f})
>> +[[ -n ${x} ]] && echo "${x}" > "${f}"
> 
> There's probably no point in this [[ -n ... ]], as that:
> 
> a. requires you to special-handle missing INSTALL_MASK file, while it's
> easier to just ensure that it's there (and I think you requested
> the same thing from me before you rewritten my commit into breakage),
> 
> b. makes it impossible to distinguish packages from before INSTALL_MASK
> storing was added from those where it is empty.

Maybe this only matters within the context of bug 364633 [1], and for
that I think we need to introduce a separate CONTENTS.INSTALL_MASK file
so that we can easily toggle collision-protect behavior to use
CONTENTS.INSTALL_MASK when desired.

[1] https://bugs.gentoo.org/364633
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Zac Medico 
On 04/01/2018 03:57 AM, Michał Górny wrote:
> W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico
> napisał:
>> Since key refresh is prone to failure, retry using exponential
>> backoff with random jitter. This adds the following sync-openpgp-*
>> configuration settings:
>>
>> sync-openpgp-key-refresh-retry-count = 40
>>
>>   Maximum number of times to retry key refresh if it fails.  Between
>>   each  key  refresh attempt, there is an exponential delay with a
>>   constant multiplier and a uniform random multiplier between 0 and 1.
>>
>> sync-openpgp-key-refresh-retry-delay-exp-base = 2
>>
>>   The base of the exponential expression. The exponent is the number
>>   of previous refresh attempts.
>>
>> sync-openpgp-key-refresh-retry-delay-max = 60
>>
>>   Maximum  delay between each retry attempt, in units of seconds. This
>>   places a limit on the length of the exponential delay.
>>
>> sync-openpgp-key-refresh-retry-delay-mult = 4
>>
>>   Multiplier for the exponential delay.
>>
>> sync-openpgp-key-refresh-retry-overall-timeout = 1200
>>
>>   Combined time limit for all refresh attempts, in units of seconds.
>>
>> Bug: https://bugs.gentoo.org/649276
>>
>> Zac Medico (4):
>>   Add ForkExecutor (bug 649588)
>>   Add ExponentialBackoff and RandomExponentialBackoff
>>   Add retry decorator (API inspired by tenacity)
>>   rsync: add key refresh retry (bug 649276)
>>
>>  cnf/repos.conf|   5 +
>>  man/portage.5 |  19 +++
>>  pym/portage/repository/config.py  |  22 
>>  pym/portage/sync/modules/rsync/rsync.py   |  16 ++-
>>  pym/portage/sync/syncbase.py  |  85 +++-
>>  pym/portage/tests/util/futures/test_retry.py  | 147 +
>>  pym/portage/util/_eventloop/EventLoop.py  |  45 ++-
>>  pym/portage/util/backoff.py   |  48 +++
>>  pym/portage/util/futures/executor/__init__.py |   0
>>  pym/portage/util/futures/executor/fork.py | 130 +++
>>  pym/portage/util/futures/futures.py   |   6 +
>>  pym/portage/util/futures/retry.py | 178 
>> ++
>>  12 files changed, 697 insertions(+), 4 deletions(-)
>>  create mode 100644 pym/portage/tests/util/futures/test_retry.py
>>  create mode 100644 pym/portage/util/backoff.py
>>  create mode 100644 pym/portage/util/futures/executor/__init__.py
>>  create mode 100644 pym/portage/util/futures/executor/fork.py
>>  create mode 100644 pym/portage/util/futures/retry.py
>>
> 
> This essentially looks like ~700 lines of code to try to workaround
> broken networking. I would rather try to do that using 5 lines of code
> but that's just me, and my programs aren't enterprise quality. I just
> hope it actually solves as many problems as it's going to introduce.

The vast majority of this code is generic and reusable, and I do intend
to reuse it. For example, the executor support will be an essential
piece for the asyncio.AbstractEventLoop for bug 649588.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-portage-dev] [PATCH 3/4] Add retry decorator (API inspired by tenacity)

2018-04-01 Thread Alec Warner 
also lgtm.

On Sat, Mar 31, 2018 at 10:46 PM, Zac Medico  wrote:

> This decorator will be useful for retrying asynchronous
> operations, such as gpg key refresh (bug 649276). The
> API is inspired by tenacity, but is simpler. Only
> asynchronous functions (like @asyncio.coroutine functions)
> are supported. In order to retry a synchronous function,
> first convert it to an asynchronous function as follows:
>
> asynchronous_func = functools.partial(
> loop.run_in_executor, None, synchronous_func)
>
> Bug: https://bugs.gentoo.org/649276
> See: https://github.com/jd/tenacity
> ---
>  pym/portage/tests/util/futures/test_retry.py | 147 ++
>  pym/portage/util/futures/futures.py  |   6 +
>  pym/portage/util/futures/retry.py| 178
> +++
>  3 files changed, 331 insertions(+)
>  create mode 100644 pym/portage/tests/util/futures/test_retry.py
>  create mode 100644 pym/portage/util/futures/retry.py
>
> diff --git a/pym/portage/tests/util/futures/test_retry.py
> b/pym/portage/tests/util/futures/test_retry.py
> new file mode 100644
> index 0..7641e4e92
> --- /dev/null
> +++ b/pym/portage/tests/util/futures/test_retry.py
> @@ -0,0 +1,147 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import functools
> +
> +try:
> +   import threading
> +except ImportError:
> +   import dummy_threading as threading
> +
> +from portage.tests import TestCase
> +from portage.util._eventloop.global_event_loop import global_event_loop
> +from portage.util.backoff import RandomExponentialBackoff
> +from portage.util.futures.futures import TimeoutError
> +from portage.util.futures.retry import retry
> +from portage.util.futures.wait import wait
> +from portage.util.monotonic import monotonic
> +
> +
> +class SucceedLaterException(Exception):
> +   pass
> +
> +
> +class SucceedLater(object):
> +   """
> +   A callable object that succeeds some duration of time has passed.
> +   """
> +   def __init__(self, duration):
> +   self._succeed_time = monotonic() + duration
> +
> +   def __call__(self):
> +   remaining = self._succeed_time - monotonic()
> +   if remaining > 0:
> +   raise SucceedLaterException('time until success:
> {} seconds'.format(remaining))
> +   return 'success'
> +
> +
> +class SucceedNeverException(Exception):
> +   pass
> +
> +
> +class SucceedNever(object):
> +   """
> +   A callable object that never succeeds.
> +   """
> +   def __call__(self):
> +   raise SucceedNeverException('expected failure')
> +
> +
> +class HangForever(object):
> +   """
> +   A callable object that sleeps forever.
> +   """
> +   def __call__(self):
> +   threading.Event().wait()
> +
> +
> +class RetryTestCase(TestCase):
> +   def testSucceedLater(self):
> +   loop = global_event_loop()
> +   func = SucceedLater(1)
> +   func_coroutine = functools.partial(loop.run_in_executor,
> None, func)
> +   decorator = retry(try_max=,
> +   delay_func=RandomExponentialBackoff(multiplier=0.1,
> base=2))
> +   decorated_func = decorator(func_coroutine)
> +   result = loop.run_until_complete(decorated_func())
> +   self.assertEqual(result, 'success')
> +
> +   def testSucceedNever(self):
> +   loop = global_event_loop()
> +   func = SucceedNever()
> +   func_coroutine = functools.partial(loop.run_in_executor,
> None, func)
> +   decorator = retry(try_max=4, try_timeout=None,
> +   delay_func=RandomExponentialBackoff(multiplier=0.1,
> base=2))
> +   decorated_func = decorator(func_coroutine)
> +   done, pending = loop.run_until_complete(wait([
> decorated_func()]))
> +   self.assertEqual(len(done), 1)
> +   self.assertTrue(isinstance(done[0].exception().__cause__,
> SucceedNeverException))
> +
> +   def testSucceedNeverReraise(self):
> +   loop = global_event_loop()
> +   func = SucceedNever()
> +   func_coroutine = functools.partial(loop.run_in_executor,
> None, func)
> +   decorator = retry(reraise=True, try_max=4,
> try_timeout=None,
> +   delay_func=RandomExponentialBackoff(multiplier=0.1,
> base=2))
> +   decorated_func = decorator(func_coroutine)
> +   done, pending = loop.run_until_complete(wait([
> decorated_func()]))
> +   self.assertEqual(len(done), 1)
> +   self.assertTrue(isinstance(done[0].exception(),
> SucceedNeverException))
> +
> +   def testHangForever(self):
> +   loop = global_event_loop()
> +   func = HangForever()
> +

Re: [gentoo-portage-dev] [PATCH 1/4] Add ForkExecutor (bug 649588)

2018-04-01 Thread Alec Warner 
lgtm.

On Sat, Mar 31, 2018 at 10:46 PM, Zac Medico  wrote:

> This is useful for asynchronous operations that we might
> need to cancel if they take too long, since (concurrent.
> futures.ProcessPoolExecutor tasks are not cancellable).
> This ability to cancel tasks makes this executor useful
> as an alternative to portage.exception.AlarmSignal.
>
> Also add an asyncio-compatible EventLoop.run_in_executor
> method the uses ForkExecutor as the default executor,
> which will later be used to implement the corresponding
> asyncio.AbstractEventLoop run_in_executor method.
>
> Bug: https://bugs.gentoo.org/649588
> ---
>  pym/portage/util/_eventloop/EventLoop.py  |  45 -
>  pym/portage/util/futures/executor/__init__.py |   0
>  pym/portage/util/futures/executor/fork.py | 130
> ++
>  3 files changed, 174 insertions(+), 1 deletion(-)
>  create mode 100644 pym/portage/util/futures/executor/__init__.py
>  create mode 100644 pym/portage/util/futures/executor/fork.py
>
> diff --git a/pym/portage/util/_eventloop/EventLoop.py
> b/pym/portage/util/_eventloop/EventLoop.py
> index f472a3dae..1574a6837 100644
> --- a/pym/portage/util/_eventloop/EventLoop.py
> +++ b/pym/portage/util/_eventloop/EventLoop.py
> @@ -24,6 +24,7 @@ except ImportError:
>  import portage
>  portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.util.futures.futures:_EventLoopFuture',
> +   'portage.util.futures.executor.fork:ForkExecutor',
>  )
>
>  from portage import OrderedDict
> @@ -122,6 +123,7 @@ class EventLoop(object):
> self._idle_callbacks = OrderedDict()
> self._timeout_handlers = {}
> self._timeout_interval = None
> +   self._default_executor = None
>
> self._poll_obj = None
> try:
> @@ -721,6 +723,46 @@ class EventLoop(object):
> return self._handle(self.timeout_add(
> delay * 1000, self._call_soon_callback(callback,
> args)), self)
>
> +   def run_in_executor(self, executor, func, *args):
> +   """
> +   Arrange for a func to be called in the specified executor.
> +
> +   The executor argument should be an Executor instance. The
> default
> +   executor is used if executor is None.
> +
> +   Use functools.partial to pass keywords to the *func*.
> +
> +   @param executor: executor
> +   @type executor: concurrent.futures.Executor or None
> +   @param func: a function to call
> +   @type func: callable
> +   @return: a Future
> +   @rtype: asyncio.Future (or compatible)
> +   """
> +   if executor is None:
> +   executor = self._default_executor
> +   if executor is None:
> +   executor = ForkExecutor(loop=self)
> +   self._default_executor = executor
> +   return executor.submit(func, *args)
> +
> +   def close(self):
> +   """Close the event loop.
> +
> +   This clears the queues and shuts down the executor,
> +   and waits for it to finish.
> +   """
> +   executor = self._default_executor
> +   if executor is not None:
> +   self._default_executor = None
> +   executor.shutdown(wait=True)
> +
> +   if self._poll_obj is not None:
> +   close = getattr(self._poll_obj, 'close')
> +   if close is not None:
> +   close()
> +   self._poll_obj = None
> +
>
>  _can_poll_device = None
>
> @@ -782,10 +824,11 @@ class _epoll_adapter(object):
> that is associated with an epoll instance will close automatically
> when
> it is garbage collected, so it's not necessary to close it
> explicitly.
> """
> -   __slots__ = ('_epoll_obj',)
> +   __slots__ = ('_epoll_obj', 'close')
>
> def __init__(self, epoll_obj):
> self._epoll_obj = epoll_obj
> +   self.close = epoll_obj.close
>
> def register(self, fd, *args):
> self._epoll_obj.register(fd, *args)
> diff --git a/pym/portage/util/futures/executor/__init__.py
> b/pym/portage/util/futures/executor/__init__.py
> new file mode 100644
> index 0..e69de29bb
> diff --git a/pym/portage/util/futures/executor/fork.py
> b/pym/portage/util/futures/executor/fork.py
> new file mode 100644
> index 0..9cd1db2ca
> --- /dev/null
> +++ b/pym/portage/util/futures/executor/fork.py
> @@ -0,0 +1,130 @@
> +# Copyright 2018 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +import collections
> +import functools
> +import multiprocessing
> +import os
> +import sys
> +import traceback
> +
> +from

Re: [gentoo-portage-dev] [PATCH] INSTALL_MASK: honor install time config for binary packages (bug 651952)

2018-04-01 Thread Michał Górny 
W dniu czw, 29.03.2018 o godzinie 15∶34 -0700, użytkownik Zac Medico
napisał:
> For binary packages, honor the INSTALL_MASK configuration that
> exists at install time, since it might differ from the build time
> setting.
> 
> Fixes: 3416876c0ee7 ("{,PKG_}INSTALL_MASK: python implementation")
> Bug: https://bugs.gentoo.org/651952
> ---
>  bin/misc-functions.sh| 23 +++
>  bin/phase-functions.sh   | 10 +-
>  pym/portage/dbapi/vartree.py |  5 +
>  3 files changed, 29 insertions(+), 9 deletions(-)
> 
> diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
> index 26f589915..a6330ee93 100755
> --- a/bin/misc-functions.sh
> +++ b/bin/misc-functions.sh
> @@ -323,6 +323,29 @@ postinst_qa_check() {
>   done < <(printf "%s\0" "${qa_checks[@]}" | LC_ALL=C sort -u -z)
>  }
>  
> +preinst_mask() {
> + # Remove man pages, info pages, docs if requested. This is
> + # implemented in bash in order to respect INSTALL_MASK settings
> + # from bashrc.
> + local f x
> + for f in man info doc; do
> + if has no${f} ${FEATURES}; then
> + INSTALL_MASK+=" /usr/share/${f}"
> + fi
> + done
> +
> + # Store modified variables in build-info.
> + cd "${PORTAGE_BUILDDIR}"/build-info || die
> + set -f
> +
> + IFS=$' \t\n\r'
> + for f in INSTALL_MASK; do

This loop along with the whole indirection is entirely pointless, given
that you're processing exactly one variable.

> + x=$(echo -n ${!f})
> + [[ -n ${x} ]] && echo "${x}" > "${f}"

There's probably no point in this [[ -n ... ]], as that:

a. requires you to special-handle missing INSTALL_MASK file, while it's
easier to just ensure that it's there (and I think you requested
the same thing from me before you rewritten my commit into breakage),

b. makes it impossible to distinguish packages from before INSTALL_MASK
storing was added from those where it is empty.

> + done
> + set +f
> +}
> +
>  preinst_sfperms() {
>   if [ -z "${D}" ]; then
>eerror "${FUNCNAME}: D is unset"
> diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
> index bdae68f79..3de8d01b5 100644
> --- a/bin/phase-functions.sh
> +++ b/bin/phase-functions.sh
> @@ -661,14 +661,6 @@ __dyn_install() {
>   set -f
>   local f x
>  
> - # remove man pages, info pages, docs if requested
> - for f in man info doc; do
> - if has no${f} ${FEATURES} && \
> - ! has "/usr/share/${f}" ${INSTALL_MASK}; then
> - INSTALL_MASK+=" /usr/share/${f}"
> - fi
> - done
> -
>   IFS=$' \t\n\r'
>   for f in CATEGORY DEFINED_PHASES FEATURES INHERITED IUSE \
>   PF PKGUSE SLOT KEYWORDS HOMEPAGE DESCRIPTION \
> @@ -676,7 +668,7 @@ __dyn_install() {
>   CXXFLAGS EXTRA_ECONF EXTRA_EINSTALL EXTRA_MAKE \
>   LDFLAGS LIBCFLAGS LIBCXXFLAGS QA_CONFIGURE_OPTIONS \
>   QA_DESKTOP_FILE QA_PREBUILT PROVIDES_EXCLUDE REQUIRES_EXCLUDE \
> - INSTALL_MASK PKG_INSTALL_MASK; do
> + PKG_INSTALL_MASK; do
>  
>   x=$(echo -n ${!f})
>   [[ -n $x ]] && echo "$x" > $f
> diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> index 378d42dc0..a136c38f1 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -3846,6 +3846,11 @@ class dblink(object):
>   # be useful to avoid collisions in some scenarios.
>   # We cannot detect if this is needed or not here as 
> INSTALL_MASK can be
>   # modified by bashrc files.
> + phase = MiscFunctionsProcess(background=False,
> + commands=["preinst_mask"], phase="preinst",
> + scheduler=self._scheduler, settings=self.settings)
> + phase.start()
> + phase.wait()
>   try:
>   with io.open(_unicode_encode(os.path.join(inforoot, 
> "INSTALL_MASK"),
>   encoding=_encodings['fs'], errors='strict'),

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 0/4] rsync: add key refresh retry (bug 649276)

2018-04-01 Thread Michał Górny 
W dniu sob, 31.03.2018 o godzinie 19∶46 -0700, użytkownik Zac Medico
napisał:
> Since key refresh is prone to failure, retry using exponential
> backoff with random jitter. This adds the following sync-openpgp-*
> configuration settings:
> 
> sync-openpgp-key-refresh-retry-count = 40
> 
>   Maximum number of times to retry key refresh if it fails.  Between
>   each  key  refresh attempt, there is an exponential delay with a
>   constant multiplier and a uniform random multiplier between 0 and 1.
> 
> sync-openpgp-key-refresh-retry-delay-exp-base = 2
> 
>   The base of the exponential expression. The exponent is the number
>   of previous refresh attempts.
> 
> sync-openpgp-key-refresh-retry-delay-max = 60
> 
>   Maximum  delay between each retry attempt, in units of seconds. This
>   places a limit on the length of the exponential delay.
> 
> sync-openpgp-key-refresh-retry-delay-mult = 4
> 
>   Multiplier for the exponential delay.
> 
> sync-openpgp-key-refresh-retry-overall-timeout = 1200
> 
>   Combined time limit for all refresh attempts, in units of seconds.
> 
> Bug: https://bugs.gentoo.org/649276
> 
> Zac Medico (4):
>   Add ForkExecutor (bug 649588)
>   Add ExponentialBackoff and RandomExponentialBackoff
>   Add retry decorator (API inspired by tenacity)
>   rsync: add key refresh retry (bug 649276)
> 
>  cnf/repos.conf|   5 +
>  man/portage.5 |  19 +++
>  pym/portage/repository/config.py  |  22 
>  pym/portage/sync/modules/rsync/rsync.py   |  16 ++-
>  pym/portage/sync/syncbase.py  |  85 +++-
>  pym/portage/tests/util/futures/test_retry.py  | 147 +
>  pym/portage/util/_eventloop/EventLoop.py  |  45 ++-
>  pym/portage/util/backoff.py   |  48 +++
>  pym/portage/util/futures/executor/__init__.py |   0
>  pym/portage/util/futures/executor/fork.py | 130 +++
>  pym/portage/util/futures/futures.py   |   6 +
>  pym/portage/util/futures/retry.py | 178 
> ++
>  12 files changed, 697 insertions(+), 4 deletions(-)
>  create mode 100644 pym/portage/tests/util/futures/test_retry.py
>  create mode 100644 pym/portage/util/backoff.py
>  create mode 100644 pym/portage/util/futures/executor/__init__.py
>  create mode 100644 pym/portage/util/futures/executor/fork.py
>  create mode 100644 pym/portage/util/futures/retry.py
> 

This essentially looks like ~700 lines of code to try to workaround
broken networking. I would rather try to do that using 5 lines of code
but that's just me, and my programs aren't enterprise quality. I just
hope it actually solves as many problems as it's going to introduce.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] rfc: empty directories in ${D}

2018-04-01 Thread Ciaran McCreesh 
On Sat, 31 Mar 2018 20:46:56 +0100
Andrey Utkin  wrote:
> Right, I am not aware why PMS has left this explicitly undefined.

Because when we wrote the spec, Portage's actual behaviour on empty
directories was extremely tricky to define and depended upon things
like whether you were upgrading or reinstalling a package.

-- 
Ciaran McCreesh