[gentoo-dev] FYI: Unexpected outage May 11 04:40 - 20:00 UTC

2021-05-12 Thread Alec Warner
The following nodes were offline:

bittern.gentoo.org (blogs, Bouncer, devmanual)
bobolink.gentoo.org (ns1; dns)
devbox.amd64.dev.gentoo.org
elivepatch.amd64.dev.gentoo.org
motmot.gentoo.org (qa-reports, keys.g.o)
roverlay.dev.gentoo.org
woodpecker.gentoo.org (dev.g.o, smtp.g.o)

These are all VMs; the host running them failed and it took a while to
bring the failed host back up[0]. I believe all services are restored,
sorry for the inconvenience. Mainly I'd expect a delay in email; as
email is a store-and-forward system and our mailserver was down for
half a day.

-A

[0] Infra members were busy in that AM with work meetings and the
remote login documentation was a bit out of date; we are working to
update that playbook.



Re: [gentoo-dev] [PATCH v2 3/3] gnuconfig.eclass: use BDEPEND, BROOT where available (drop support for EAPI <4)

2021-04-09 Thread Alec Warner
On Fri, Apr 9, 2021 at 5:19 PM Sam James  wrote:
>
>
>
> > On 10 Apr 2021, at 01:13, Michael Orlitzky  wrote:
> >
> > On Sat, 2021-04-10 at 00:32 +0100, Sam James wrote:
> >>
> >>
> >> Yes, this is the part I find difficult too. The important
> >> distinction here was *bootstrapping* (which I missed)
> >> but I think at least we should make a list of packages generally considered
> >> critical for bootstrap.
> >>
> >
> > What is a bootstrap package?
> >
> > There is some chicken-and-egg problem to be solved, but I don't think
> > that we should be assuming that e.g. GNU grep is always present just
> > because, during the base case of some recursive process, POSIX grep
> > must be available temporarily.
> >
> > Anyway, https://bugs.gentoo.org/485356 awaits reopening if you make any
> > progress on this.
> >
>
> Oh, I agree completely. CCed myself on the bug and added to the list
> to think about/work on.
>
> I’m pleased a bug existed in the past! I don’t agree with it being closed 
> though:
> documentation issues can exist without a patch existing to fix them yet, 
> right?

I worry a lot about more complex dependencies trees (imagine all of
the exciting cycles in the currently excluded @system depgraph.)
Remember that while in theory it would be great if portage knew about
all of them; computing these nodes and edges isn't free. I question
what we are really buying with the extra complexity.

-A



Re: [gentoo-dev] timezone configuration - why copying, not symlinking /etc/localtime ?

2021-03-20 Thread Alec Warner
On Sat, Mar 20, 2021 at 10:27 PM Ulrich Mueller  wrote:
>
> >>>>> On Sun, 21 Mar 2021, Alec Warner wrote:
>
> >> Which doesn't imply that we deliberately break things.
>
> > Not sure I follow.. how is updating the handbook breaking anything?
>
> Both configurations (regular file and symlink) work just fine, and
> sys-libs/timezone-data supports them. I don't see a good reason why we
> would suddenly declare the regular file to be an invalid option.

https://bugs.gentoo.org/737914 seems to imply for some upstreams, it
being a file is not a valid option anymore?

(I'm ignoring the logic of that decision of course, but this was the
original reason this was raised.)

-A

>
> Ulrich



Re: [gentoo-dev] timezone configuration - why copying, not symlinking /etc/localtime ?

2021-03-20 Thread Alec Warner
On Sat, Mar 20, 2021 at 9:19 PM Ulrich Mueller  wrote:
>
> > On Sat, 20 Mar 2021, William Hubbs wrote:
>
> > /etc/localtime should definitely be a symlink to the proper file in
> > /usr/share/zoneinfo.
>
> > This works fine if /usr is on a separate partition *and* you are using
> > an initramfs. The only time it doesn't work is if /usr is separate
> > without using an initramfs.
>
> > Council decided years ago that we don't support separate /usr without
> > an initramfs, but we haven't completed that transition yet.
>
> Which doesn't imply that we deliberately break things.

Not sure I follow.. how is updating the handbook breaking anything?

-A

>
> Ulrich



Re: [gentoo-dev] [PATCH] gstreamer-meson.eclass: New eclass required for gstreamer-1.18.0+

2021-03-17 Thread Alec Warner
On Tue, Mar 16, 2021 at 6:49 PM Haelwenn (lanodan) Monnier
 wrote:
>
> Gstreamer switched to meson in 1.16.0 and removed autotools support in 1.18.0,
> this eclass is an update of gstreamer.eclass.
>
> One significant change between autotools and meson is that in the latter we
> don't have easily extractable semantics in the buildsystem to get a list
> of plugins with extraneous dependencies that we currently split in other
> packages.
> Hence the rather ugly but currently required GST_PLUGINS_DISABLED block.
>
> Fixes: https://bugs.gentoo.org/690468
>
> Signed-off-by: Haelwenn (lanodan) Monnier 
> ---
>  eclass/gstreamer-meson.eclass | 293 ++
>  1 file changed, 293 insertions(+)
>  create mode 100644 eclass/gstreamer-meson.eclass
>
> diff --git a/eclass/gstreamer-meson.eclass b/eclass/gstreamer-meson.eclass
> new file mode 100644
> index 000..263deeb08aa
> --- /dev/null
> +++ b/eclass/gstreamer-meson.eclass
> @@ -0,0 +1,293 @@
> +# Copyright 1999-2021 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: gstreamer-meson.eclass
> +# @MAINTAINER:
> +# gstrea...@gentoo.org
> +# @AUTHOR:
> +# Michał Górny 
> +# Gilles Dartiguelongue 
> +# Saleem Abdulrasool 
> +# foser 
> +# zaheerm 
> +# Steven Newbury
> +# Haelwenn (lanodan) Monnier 
> +# @SUPPORTED_EAPIS: 5 6
> +# @BLURB: Helps building core & split gstreamer plugins.
> +# @DESCRIPTION:
> +# Eclass to make external gst-plugins emergable on a per-plugin basis
> +# and to solve the problem with gst-plugins generating far too much
> +# unneeded dependencies.
> +#
> +# GStreamer consuming applications should depend on the specific plugins
> +# they need as defined in their source code. Usually you can find that
> +# out by grepping the source tree for 'factory_make'. If it uses playbin
> +# plugin, consider adding media-plugins/gst-plugins-meta dependency, but
> +# also list any packages that provide explicitly requested plugins.
> +
> +inherit eutils multilib meson multilib-minimal toolchain-funcs versionator 
> xdg-utils
> +
> +case "${EAPI:-0}" in
> +   5|6)
> +   ;;
> +   0|1|2|3|4)
> +   die "EAPI=\"${EAPI:-0}\" is not supported anymore"
> +   ;;
> +   *)
> +   die "EAPI=\"${EAPI}\" is not supported yet"
> +   ;;
> +esac
> +
> +# @ECLASS-VARIABLE: GST_PLUGINS_ENABLED
> +# @DESCRIPTION:
> +# Defines the plugins to be built.
> +# May be set by an ebuild and contain more than one indentifier, space
> +# seperated (only src_configure can handle mutiple plugins at this time).
> +: ${GST_PLUGINS_ENABLED:=${PN/gst-plugins-/}}
> +
> +# @ECLASS-VARIABLE: GST_PLUGINS_DISABLED
> +# @DESCRIPTION:
> +# Defines the plugins to not be built, GST_PLUGINS_ENABLED overrides it.
> +# May be set by an ebuild and contain more than one indentifier, space
> +# seperated (only src_configure can handle mutiple plugins at this time).
> +case "${GST_ORG_MODULE}" in
> +   # copied GST_PLUGINS_DISABLED from media-libs/${GST_ORG_MODULE} then 
> added GST_PLUGINS_ENABLED
> +   gst-plugins-bad)
> +   # removed from list: shm ipcpipeline gl
> +   GST_PLUGINS_DISABLED="aom avtp androidmedia applemedia 
> assrender bluez bs2b bz2 chromaprint closedcaption colormanagement curl 
> curl-ssh2 d3dvideosink d3d11 dash dc1394 decklink directfb directsound dtls 
> dts dvb faac faad fbdev fdkaac flite fluidsynth gme gsm iqa kate kms ladspa 
> libde265 libmms lv2 mediafoundation microdns modplug mpeg2enc mplex msdk 
> musepack neon nvcodec ofa openal openexr openh264 openjpeg openmpt openni2 
> opensles opus resindvd rsvg rtmp sbc sctp smoothstreaming sndfile soundtouch 
> spandsp srt srtp svthevcenc teletext tinyalsa transcode ttml uvch264 va 
> voaacenc voamrwbenc vulkan wasapi wasapi2 webp webrtc webrtcdsp wildmidi 
> winks winscreencap x265 zbar zxing wpe magicleap v4l2codecs hls opencv"
> +   GST_PLUGINS_DISABLED="${GST_PLUGINS_DISABLED} accurip 
> adpcmdec adpcmenc aiff asfmux audiobuffersplit audiofxbad audiolatency 
> audiomixmatrix audiovisualizers autoconvert bayer camerabin2 coloreffects deb 
> ugutils dvbsubenc dvbsuboverlay dvdspu faceoverlay festival fieldanalysis 
> freeverb frei0r gaudieffects gdp geometrictransform id3tag inter interlace 
> ivfpars e ivtc jp2kdecimator jpegformat librfb midi mpegdemux mpegpsmux 
> mpegtsdemux mpegtsmux mxf netsim onvif pcapparse pnm proxy rawparse 
> removesilence rist rtmp2 rtp sdp segmentclip siren smooth speed subenc 
> switchbin timecode videofilters videoframe_audiolevel videoparsers 
> videosignal vmnc y4m"
> +   ;;
> +   gst-plugins-base)
> +   GST_PLUGINS_DISABLED="cdparanoia libvisual opus tremor"
> +   GST_PLUGINS_DISABLED="${GST_PLUGINS_DISABLED} adder app 
> audioconvert audiomixer audiorate audioresample audiotestsrc compositor 
> encoding gio gio-typefinder overlaycomposition pbtypes playback 

Re: [gentoo-dev] Fwd: [Python-Dev] Move support of legacy platforms/architectures outside Python

2021-02-22 Thread Alec Warner
So my question is basically, how much does Gentoo care about these
ports? Should we be funding ports of rust to these arches (assuming
that would continue ensure gentoo works on those arches.)

-A


On Sun, Feb 21, 2021 at 5:01 AM Michał Górny  wrote:
>
> Hi,
>
> FYI, a few member of Python upstream are continuing their crusade
> against minor architectures not supported by Rust.  This time, they're
> discussing actively removing support for platforms they don't officially
> support, and requiring people to maintain external patches forever.
>
>
>  Forwarded Message 
> From: Victor Stinner 
> To: Python Dev 
> Subject: [Python-Dev] Move support of legacy platforms/architectures
> outside Python
> Date: Sun, 21 Feb 2021 13:13:45 +0100
>
> > Hi,
> >
> > I propose to actively remove support for *legacy* platforms and
> > architectures which are not supported by Python according to PEP 11
> > rules: hardware no longer sold and end-of-life operating systems. The
> > removal should be discussed on a case by case basis, but I would like
> > to get an agreement on the overall idea first. Hobbyists wanting to
> > support these platforms/archs can continue to support them with
> > patches maintained outside Python. For example, I consider that the
> > 16-bit m68k architecture is legacy, whereas the OpenBSD platform is
> > still actively maintained.
> >
> > I already know that there will be a strike back: "oh no, you must
> > continue to support my architecture" and "their existing code should
> > stay and doesn't cost anything to maintain". Python is maintained by
> > volunteers, the majority is contributing in their free time, so people
> > are free to use their free time as they want. You cannot ask core
> > developers to support your favorite *legacy* platform/architecture if
> > they don't want to.
> >
> > In short, I propose to move maintenance of the legacy platforms/archs
> > outside Python: people are free to continue support them as patches.
> >
> > --
> >
> > Concrete example: Christian Heimes proposed to drop support for 31-bit
> > s390 Linux:
> > https://bugs.python.org/issue43179
> >
> > The lack of clear definition on how a platform is supported or not
> > confuses users who consider that their favorite platform/arch is
> > supported, whereas core developers don't want to support it since it
> > would be too much work.
> >
> > In fact, the PEP 11 has clear and explicit rules:
> > https://www.python.org/dev/peps/pep-0011/#supporting-platforms
> >
> > A platform is only considered as supported if the following two
> > conditions are met:
> >
> > 1) a core developer needs to volunteer to maintain platform-specific code
> > 2) a stable buildbot must be provided
> >
> > Last October, I proposed to drop Solaris support (bpo-42173). Jakub
> > Kulik stepped in and proposed some Solaris patches, so I abandoned my
> > idea. But I still don't see any running Solaris buildbot worker, and
> > there is still no core developer volunteer to maintain Solaris
> > support. It's unclear to me if Python has "best effort" support for
> > Solaris, of if Solaris is "not supported".
> >
> > --
> >
> > Over the years, Python was ported to tons of platforms and CPU
> > architectures. It didn't matter if the platform or the architecture
> > was commonly used or not. 30 years later, Python still has the code
> > for many legacy platforms and architectures. Some hardware is no
> > longer sold but kept alive by hobbyists, especially members of retro
> > computing groups.
> >
> > Some Linux distributions like Gentoo and Debian are trying to support
> > most architectures which are supported by these hobbyist groups,
> > whereas some other distributions like Ubuntu are limited to a few
> > platforms. For example, Ubuntu 20.4.2 LTS server supports 4
> > architectures: x86-64, AArch64 (ARM), POWER and s390x. I guess that
> > the difference between Debian and Ubuntu is that Ubuntu is a Canonical
> > product, Canonical sells professional support and so cannot support
> > too many architectures. Each architecture support requires to build
> > all packages on it, tests the packages, have experts who fix issues
> > specific to this arch, etc.
> >
> > --
> >
> > Python has different kinds of platform and architecture supports. In
> > practice, I would say that we have:
> >
> > * (1) Fully supported. Platform/architecture used by core developers
> > and have at least one working buildbot worker: fully supported. Since
> > core developers want to use Python on their machine, they fix issues
> > as soon as they notice them. Examples: x86-64 on Linux, Windows and
> > macOS.
> >
> > * (2) Best effort. Platform/architecture which has a buildbot worker
> > usually not used by core developers. Regressions (buildbot failures)
> > are reported to bugs.python.org, if someone investigates and provides
> > a fix, the fix is merged. But there is usually no "proactive" work to
> > ensure that Python works "perfectly" on these platforms. 

Re: [gentoo-dev] [RFC] Timeline for Python 3.9 switch and Python 3.7 removal

2021-02-18 Thread Alec Warner
On Thu, Feb 18, 2021 at 3:45 PM Michał Górny  wrote:
>
> Hi,
>
> I'd like to discuss a rough timeline for switching the default
> PYTHON_TARGETS to python3.9.
>
> According to the upstream release schedule [1], the last bugfix release
> is planned for May.  Afterwards, upstream will release only security
> fixes.  At the same time, our rough current schedule [2] suggests that
> we'll start last-riting stuff May 1st, so the 3.7 target would be
> removed in 15-30 days.
>
> Do you think 2021-06-01 would be a reasonable planned switchover date?

So you are saying we will start masking packages may 1st and we will
drop 3.7 targets 30 days later (june 1?)

(I felt like this was a bit ambiguous?)

-A

>
> [1] https://www.python.org/dev/peps/pep-0569/#bugfix-releases
> [2] https://wiki.gentoo.org/wiki/Project:Python/Implementations
>
> --
> Best regards,
> Michał Górny
>
>
>



Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

2021-02-08 Thread Alec Warner
On Mon, Feb 8, 2021 at 9:56 AM Alessandro Barbieri
 wrote:
>
> Il Lun 8 Feb 2021, 12:19 Michał Górny  ha scritto:
>>
>> Hi,
>>
>> FYI the developers of dev-python/cryptography decided that Rust is going
>> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
>> provide LTS support or security fixes for the old versions.
>>
>> Since cryptography is a very important package in the Python ecosystem,
>> and it is an indirect dependency of Portage, this means that we will
>> probably have to entirely drop support for architectures that are not
>> supported by Rust.
>>
>> According to upstream platform support information [1], this probably
>> means (eventually) entirely removing the following architectures:
>> - alpha (stable)
>> - hppa (stable)
>> - ia64 (stable)
>> - m68k (exp)
>> - s390 (except for s390x, exp)
>>
>> Furthermore, the Gentoo Rust packages are missing support
>> for the following platforms, apparently supported upstream:
>> - mips (exp)
>> - ppc (32) (stable)
>> - sparc (stable)
>> - s390x (exp)
>> - riscv (stable)
>>
>> Apparently it's non-trivial to bootstrap Rust on these platforms,
>> so it's unclear when Gentoo is going to start providing Rust on them.
>>
>> I've raised a protest on the cryptography bug tracker [2] but apparently
>> upstream considers Rust's 'memory safety' more important than ability to
>> actually use the package.
>>
>> Honestly, I don't think it likely that Rust will gain support for these
>> platforms.  This involves a lot of work, starting with writing a new
>> LLVM backend and getting it accepted (getting new code into LLVM is very
>> hard unless you're doing that on behalf one of the big companies).  You
>> can imagine how much effort that involves compared to rewriting the new
>> code from Cryptography into C.
>>
>> If we can't convince upstream, I'm afraid we'll either have to drop
>> these architectures entirely or fork Cryptography.
>>
>>
>> [1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
>> [2] https://github.com/pyca/cryptography/issues/5771
>>
>> --
>> Best regards,
>> Michał Górny
>
>
> Should we shed tears for those legacy architectures or move forward? Does 
> anyone really use them in production?

Many users don't use gentoo in 'production' so I'm not sure how that
matters in our decision making. I'm not sure the trade off here
(taking rust as a core dep) is reasonable and it looks like we can
avoid it.

-A



Re: [gentoo-dev] GTK:2 EOL and incoming migration to GTK:3

2021-02-08 Thread Alec Warner
On Mon, Feb 8, 2021 at 6:59 AM Peter Stuge  wrote:
>
> Hanno Böck wrote:
> > > "It does mean, however, that GTK 2 has reached the end of its life.
> > > We will do one final 2.x release in the coming days, and we encourage
> > > everybody to port their GTK 2 applications to GTK 3 or 4."
> >
> > I read that as there will be one more gtk2 release and none after that.
> >
> > This seems to imply:
> > * When there's a security flaw in gtk2 there won't be a fix from
> >   upstream.
> > * When there's an incompatibility with new infrastructure (e.g. new gcc
> >   version / new glibc / changing API of libraries gtk depends on) there
> >   will be no updates from upstream.
> >
> > This means in all those instances maintainers will have to get patches
> > from somewhere. We'll likely end up with some form of
> > gtk-2.x-r[largenumber] with a large patchset at some point.
> > Maintaining that will be an increasing burden.
> >
> > No urgency, but a sign to slowly move off gtk2.
>
> Until there's a relevant flaw that will remain unfixed or there is
> significant incompatibility with infrastructure (recurse my argument)
> no signs actually exist.

So I think as the next post in the thread hints at:

 - I expect gtk2 (the library) to be around for a while. As written it
gets at least one more release.
 - I expect Gentoo to come after gtk2-only leaf packages pretty hard;
either to get upstream to port, or to remove them.
   - This is true even if the packages are fully functional with gtk2,
or don't have other bugs.
   - This is because we will eventually remove gtk2 from the tree
(which will make these packages unbuildable, and cause their removal.)

I'm less clear why we would keep libgtk2 in the tree for years and
years (just to keep nominally unmaintained gtk2 leaf packages
buildable?)

>
> Assuming that there will be a significant maintenance burden which
> affects all uses doesn't seem rational - hence my question.

I think you need to keep gtk2 (the library) for a fair bit (just like
we kept python2.7; the interpreter; for a fair while after its EOL.)

>
> The blog post shouldn't be misunderstood. The intended audience seems
> to be application developers, encouraging them to port applications,
> not so much distributions.
>
> Distributions quite often overlook that they wield much power, and
> thus also have much responsibility.
>
> Of course, GTK maintainers in Gentoo choose what to work on, and have
> made many (only?) excellent choices.
>
> I'm merely pleading for rational choices based on actual problems.
>
>
> //Peter
>



Re: [gentoo-dev] [RFC] Moving subslot-only virtuals to a separate category to reduce confusion

2021-01-30 Thread Alec Warner
On Sat, Jan 30, 2021 at 9:36 AM Michał Górny  wrote:
>
> Hi,
>
> TL;DR: I'd like to move virtual/libjpeg, virtual/libudev and so on to
> another category (e.g. lib-sover/*) to make it clear that they are used
> for := deps and have valid use even with a single provider.
>
>
> Right now we have at least a few packages that provide more than one
> valid consumer-intended library, each with a different SOVERSION that is
> bumped independently.  To list a few examples:
>
> - media-libs/libjpeg-turbo: libturbojpeg.so.0, libjpeg.so.62
> - sys-apps/systemd: libsystemd.so.0, libudev.so.1
> - app-text/poppler: libpoppler.so.106, libpoppler-cpp.so.0, libpoppler-
> glib.so.8, libpoppler-qt5.so.1
>
> The problem is that the current subslot mechanism doesn't really provide
> for that.  Laying aside possible future EAPI extensions (that are
> debatable due to added complexity to an already complex syntax), there
> are three commonly used possibilities here:
>
> a. bumping subslot whenever any of SOVERSIONs change -- meaning possibly
> unnecessary rebuilds,
>
> b. using subslot for one of the SOVERSIONs and assuming the rest stable
> -- this is what we do for poppler, and it's mildly confusing,
>
> c. using additional packages to represent SOVERSION of individual
> libraries -- this is e.g. used for libudev or libjpeg.
>
>
> I'd like to discuss option c. in more detail.  Right now, we're creating
> additional packages in virtual/ category.  This was mostly fine,
> as the libraries in question (libjpeg, libudev) had multiple providers.
> However, now that we've removed the old media-libs/jpeg, people get
> the obvious idea that the virtual is no longer necessary -- except that
> it is!
>
> The rough idea is that the subslot of libjpeg-turbo is supposed to
> represent the ABI version of libjpeg-turbo -- that is actually rarely
> used.  On the other hand, the subslot of libjpeg is represented by
> virtual/jpeg.  So if your package is using libjpeg.so, it should :=-
> depend on the latter and not on the former.
>
> The problem is that this is confusing, and if somebody doesn't know
> the secret, he can easily consider the virtual obsolete and depend
> on the library directly.
>
> To make this SOVERSION-virtual concept more visible and easily
> distinguishable from regular virtuals, I'd like to propose that we start
> moving them into a dedicated category.  For example, 'lib-sover' comes
> to my mind.  While this ofc doesn't guarantee that people will do things
> right, it will at least make it clear that these packages are somewhat
> different from regular virtuals and hopefully avoid making wrong
> assumptions.
>
> WDYT?

I'm not sure I understand the problem with (a). But this is partly my
bias here. I expect to rebuild with Gentoo; often; its literally the
whole point. So when the developer community is like "Well we want
fewer rebuilds" I sort of scoff. "This is Gentoo, rebuilds are a
thing." I wish I was more in sync with the community on this ;(

For (c) I'd rather see something like a linter:
 - If I dep on libjpeg-turbo, raise a linter hint that says "hey if
this dep is for libjpeg.so, if you want that, swap this *DEPEND for
virtual/jpeg: read this wiki page for more information..."
 - Its not meant as a QA warning (the hint will not prevent submission
or block CI.)
 - But its a mechanization of this problem and solution. Instead of
reading some thread on -dev,  there is a tool to detect instances of
this problem, it tells you briefly what the options are, and links
somewhere for more information. This sort of thing is related to an
earlier -core thread about how to keep developers up to date..threads
like this are great but they are not a system.

In short, I don't think a package naming convention is sufficient to
achieve your goal of making this problem happen less often.

I also have no idea how often this problem happens today (and if we
can lint it, we can count those too.)
Is the problem even a big enough deal to do anything?

-A

>
> --
> Best regards,
> Michał Górny
>
>
>



Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-18 Thread Alec Warner
On Mon, Jan 18, 2021 at 8:09 PM Zac Medico  wrote:
>
> On 1/18/21 6:07 PM, Alec Warner wrote:
> > On Fri, Jan 15, 2021 at 6:47 PM Matt Turner  wrote:
> >>
> >> This set is the upgradable packages for which the highest visible
> >> version has a different subslot than the currently installed version.
> >>
> >> The primary purpose of this feature is for use in catalyst builds. We
> >> update the "seed" stage3 before using it to build a new stage1.
> >>
> >> Updating the entire stage is expensive and unnecessary (since we're
> >> going to build the latest packages in stage1 and then rebuild everything
> >> in stage3).
> >>
> >> What we definitely do need to update in the original stage3 however, is
> >> any package that would trigger a subslot rebuild.
> >>
> >> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> >> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> >> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> >> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> >> will be built and libmpfr.so.6 included into the stage1. Since the old
> >> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> >> subsequent stage builds.
> >>
> >> Our current options to update the seed are too large a hammer (e.g.,
> >> "--update --deep --newuse @world" or "--update --deep --newuse
> >> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> >> updating seed stages for no gain beyond updating only packages for whom
> >> the subslot has changed.
> >>
> >> With this set, catalyst will likely use
> >>
> >> emerge @changed-subslot --ignore-built-slot-operator-deps y
> >>
> >> to update the seed stage.
> >>
> >> Thank you to Zac Medico for showing me how to do this.
> >>
> >> Bug: https://bugs.gentoo.org/739004
> >> Signed-off-by: Matt Turner 
> >> ---
> >>  cnf/sets/portage.conf  |  5 +
> >>  lib/portage/_sets/dbapi.py | 39 +-
> >>  2 files changed, 43 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
> >> index 22f0fa3a5..5651a9c53 100644
> >> --- a/cnf/sets/portage.conf
> >> +++ b/cnf/sets/portage.conf
> >> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
> >>  [rebuilt-binaries]
> >>  class = portage.sets.dbapi.RebuiltBinaries
> >>
> >> +# Installed packages for which the subslot of the highest visible ebuild
> >> +# version is different than the currently installed version.
> >> +[changed-subslot]
> >> +class = portage.sets.dbapi.SubslotChangedSet
> >> +
> >>  # Installed packages for which the highest visible ebuild
> >>  # version is lower than the currently installed version.
> >>  [downgrade]
> >> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
> >> index 52367c4a6..46ba5c17d 100644
> >> --- a/lib/portage/_sets/dbapi.py
> >> +++ b/lib/portage/_sets/dbapi.py
> >> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
> >>  import portage
> >>
> >>  __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
> >> -   "EverythingSet", "OwnerSet", "VariableSet"]
> >> +   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
> >>
> >>  class EverythingSet(PackageSet):
> >> _operations = ["merge"]
> >> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
> >>
> >> singleBuilder = classmethod(singleBuilder)
> >>
> >> +class SubslotChangedSet(PackageSet):
> >> +
> >> +   _operations = ["merge", "unmerge"]
> >> +
> >> +   description = "Package set which contains all packages " + \
> >> +   "for which the subslot of the highest visible ebuild is " 
> >> + \
> >> +   "different than the currently installed version."
> >
> > description = ("string1",
> >"string2",
> >"string3")
> >
> > vs concat + \ for line continuation?
> &g

Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-18 Thread Alec Warner
On Fri, Jan 15, 2021 at 6:47 PM Matt Turner  wrote:
>
> This set is the upgradable packages for which the highest visible
> version has a different subslot than the currently installed version.
>
> The primary purpose of this feature is for use in catalyst builds. We
> update the "seed" stage3 before using it to build a new stage1.
>
> Updating the entire stage is expensive and unnecessary (since we're
> going to build the latest packages in stage1 and then rebuild everything
> in stage3).
>
> What we definitely do need to update in the original stage3 however, is
> any package that would trigger a subslot rebuild.
>
> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> will be built and libmpfr.so.6 included into the stage1. Since the old
> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> subsequent stage builds.
>
> Our current options to update the seed are too large a hammer (e.g.,
> "--update --deep --newuse @world" or "--update --deep --newuse
> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> updating seed stages for no gain beyond updating only packages for whom
> the subslot has changed.
>
> With this set, catalyst will likely use
>
> emerge @changed-subslot --ignore-built-slot-operator-deps y
>
> to update the seed stage.
>
> Thank you to Zac Medico for showing me how to do this.
>
> Bug: https://bugs.gentoo.org/739004
> Signed-off-by: Matt Turner 
> ---
>  cnf/sets/portage.conf  |  5 +
>  lib/portage/_sets/dbapi.py | 39 +-
>  2 files changed, 43 insertions(+), 1 deletion(-)
>
> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
> index 22f0fa3a5..5651a9c53 100644
> --- a/cnf/sets/portage.conf
> +++ b/cnf/sets/portage.conf
> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
>  [rebuilt-binaries]
>  class = portage.sets.dbapi.RebuiltBinaries
>
> +# Installed packages for which the subslot of the highest visible ebuild
> +# version is different than the currently installed version.
> +[changed-subslot]
> +class = portage.sets.dbapi.SubslotChangedSet
> +
>  # Installed packages for which the highest visible ebuild
>  # version is lower than the currently installed version.
>  [downgrade]
> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
> index 52367c4a6..46ba5c17d 100644
> --- a/lib/portage/_sets/dbapi.py
> +++ b/lib/portage/_sets/dbapi.py
> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
>  import portage
>
>  __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
> -   "EverythingSet", "OwnerSet", "VariableSet"]
> +   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
>
>  class EverythingSet(PackageSet):
> _operations = ["merge"]
> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
>
> singleBuilder = classmethod(singleBuilder)
>
> +class SubslotChangedSet(PackageSet):
> +
> +   _operations = ["merge", "unmerge"]
> +
> +   description = "Package set which contains all packages " + \
> +   "for which the subslot of the highest visible ebuild is " + \
> +   "different than the currently installed version."

description = ("string1",
   "string2",
   "string3")

vs concat + \ for line continuation?

-A

> +
> +   def __init__(self, portdb=None, vardb=None):
> +   super(SubslotChangedSet, self).__init__()
> +   self._portdb = portdb
> +   self._vardb = vardb
> +
> +   def load(self):
> +   atoms = []
> +   xmatch = self._portdb.xmatch
> +   xmatch_level = "bestmatch-visible"
> +   cp_list = self._vardb.cp_list
> +   pkg_str = self._vardb._pkg_str
> +   for cp in self._vardb.cp_all():
> +   for cpv in cp_list(cp):
> +   pkg = pkg_str(cpv, None)
> +   slot_atom = "%s:%s" % (pkg.cp, pkg.slot)
> +   ebuild = xmatch(xmatch_level, slot_atom)
> +   if not ebuild:
> +   continue
> +   if pkg.sub_slot != ebuild.sub_slot:
> +   atoms.append(slot_atom)
> +
> +   self._setAtoms(atoms)
> +
> +   def singleBuilder(cls, options, settings, trees):
> +   return cls(portdb=trees["porttree"].dbapi,
> +   vardb=trees["vartree"].dbapi)
> +
> +   singleBuilder = classmethod(singleBuilder)
> +
>  class DowngradeSet(PackageSet):
>
> _operations = ["merge", "unmerge"]
> --
> 2.26.2
>
>



Re: [gentoo-dev] [PATCH 1/2] acct-user.eclass: Support ACCT_USER_ID override

2021-01-06 Thread Alec Warner
On Wed, Jan 6, 2021 at 11:05 AM Patrick McLean  wrote:
>
> On Wed, 6 Jan 2021 15:02:12 +0100
> Thomas Deutschmann  wrote:
>
> > Hi,
> >
> > is there a specific reason why we want to support dynamic variables
> > (ACCT_USER_$foo) at all?
> >
> > Isn't package.env support enough, i.e. use ACCT_USER_ID from environment
> > if set (which we should detect and log, maybe this will require a
> > different namespace for the variables at all to be able to differentiate
> > between values set by acct-* ebuild and user override)?
> >
> > Of course this won't allow something like `ACCT_USER_ID=42 emerge
> > ` but I am not sure if
> > this is an implementation goal.
>
> This is so ACCT_USER_$foo can be set in make.conf, and not have to
> be specified as an environment variable whenever portage is run. This
> helps when automated systems are building Gentoo images or systems.
>

Not sure I follow. Whether your automation sets a variable in
/etc/portage/make.conf or /etc/portage/package.env; it's basically the
same problem space; no?

-A



Re: [gentoo-dev] [PATCH] acct-user.eclass: Support var overrides for user properties

2021-01-04 Thread Alec Warner
On Mon, Jan 4, 2021 at 9:15 AM Thomas Deutschmann  wrote:
>
> On 2021-01-04 18:08, Michał Górny wrote:
> > Introduce a few variables to allow easy overrides of common user account
> > proprerties, that is:
> >
> > - ACCT_USER__SHELL
> > - ACCT_USER__HOME
> > - ACCT_USER__HOME_OWNER
> > - ACCT_USER__HOME_PERMS
> > - ACCT_USER__GROUPS
> > - ACCT_USER__GROUPS_ADD
> >
> > The first five variables override the respective ACCT_USER_* variables,
> > with ACCT_USER_*_GROUPS being a space-separated list.  The *_GROUPS_ADD
> > variable appends to groups present in the ebuild, as this seems a common
> > necessity.
> >
> > We do realize that the original requirement of overriding ebuilds
> > in a local repository was inconvenient.  This new logic should permit
> > easy updates via make.conf.  Additionally, it has the advantage
> > of clearly reporting the changes made in the build logs.
> >
> > This does not preclude other solutions to the problem.  However, this
> > is probably the best one and it should become the current
> > recommendation.
>
> This will improve the overlay situation and can be seen as overall
> improvement but it doesn't address any shared concerns nor is it a
> replacement for the proposed 'acct-user.eclass: don't modify existing
> user by default' patch.
>

Same response from me, merge it but please also merge the other patch.

-A

>
> --
> Regards,
> Thomas Deutschmann / Gentoo Linux Developer
> fpr: C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
>



Re: [gentoo-dev] [PATCH] acct-user.eclass: don't modify existing user by default

2021-01-03 Thread Alec Warner
On Sun, Jan 3, 2021 at 6:42 PM Mike Gilbert  wrote:
>
> On Sun, Jan 3, 2021 at 8:35 PM Thomas Deutschmann  wrote:
> >
> > Modifying an existing user is a bad default and makes Gentoo
> > special because it is common for system administrators to make
> > modifications to user (i.e. putting an user into another service's
> > group to allow that user to access service in question) and it
> > would be unexpected to see these changes reverted during normal
> > world upgrade (which could break services).
> >
> > This commit will make Gentoo behave like any other Linux distribution
> > by respecting any user modifications by default. However, we will retain
> > the functionality to reset system user and groups and users interested
> > in this feature can opt-in by setting
> > ACCT_USER_ALLOW_EXISTING_USER_TO_BE_MODIFIED to a non-zero value in
> > their make.conf.
>
> So the main problem I see with doing this is that it becomes
> impossible to reliably make changes to a user in later ebuild
> revisions. Developers may want/need to deploy changes to user
> attributes. Changing group memberships seems like the best example,
> but I could foresee a want/need to change DESCRIPTION, HOME, or SHELL
> as well.

I think the TL;DR is I don't believe changes in HOME, or SHELL are
safe to do in later revisions. This means developers:
 - Need to be careful when picking em!
 - Can use later revisions to set sane defaults for fresh installs.
 - Can't touch existing installs because changing HOME and SHELL can
cause user-facing breakage.

Often if HOME points somewhere that isn't /dev/null, you have to do
some kind of migration. I'd rather not enable that sort of breakage by
default because it's a per-app / per-user migration pain.

To pick an example:

The git user has a HOME in either /var/lib/gitea or /var/lib/gitolite.
If you change it, you will break whatever service is running.
This is the same for any service account whose $HOME stores
state...which is most of them with a HOME set.

-A

>
> Because of this, I think the new behavior should be opt-in, and people
> who use it should be aware that they will need to pay attention if any
> account changes are rolled out in new ebuild versions.
>
> > diff --git a/eclass/acct-user.eclass b/eclass/acct-user.eclass
> > index 22b0038fbff7..d60b1e53b4bb 100644
> > --- a/eclass/acct-user.eclass
> > +++ b/eclass/acct-user.eclass
> > @@ -309,6 +321,20 @@ acct-user_pkg_pretend() {
> > fi
> >  }
> >
> > +# @FUNCTION: acct-user_pkg_setup
> > +# @DESCRIPTION:
> > +# Initialize internal environment variable(s).
> > +acct-user_pkg_setup() {
> > +   debug-print-function ${FUNCNAME} "${@}"
> > +
> > +   # check if user already exists
> > +   ACCT_USER_ALREADY_EXISTS=
> > +   if [[ -n $(egetent passwd "${ACCT_USER_NAME}") ]]; then
> > +   ACCT_USER_ALREADY_EXISTS=yes
> > +   fi
> > +   readonly ACCT_USER_ALREADY_EXISTS
> > +}
>
> I don't think this pkg_setup function is necessary; you could do this
> in pkg_preinst instead, before enewuser gets called.
>



[gentoo-dev] [Turndown] rsync access to VCS content is no longer available.

2020-12-31 Thread Alec Warner
TL;DR if you never used rsync:// access to download copies of data
from our VCS's you can stop paying attention. This has nothing to do
with "gentoo rsync mirrors" for ::gentoo, which is a separate service
that we are not terminating.

We previously offered rsync:// protocol access to gentoo VCS
repositories. We have stopped offering this access. This is another
service that had little to no usage according to logs, and has a bunch
of legacy dependencies. For VCS content please use anongit.gentoo.org
to access content in git, and the vcs history project to access
read-only copies of historical cvs data at
https://projects.gentoo.org. We have kept completed copies of this
data, and if the public data is missing something or is incorrect,
please follow up with us.

-A



Re: [gentoo-dev] Re: [gentoo-dev-announce] We are finally shutting down CVS

2020-12-29 Thread Alec Warner
On Sun, Dec 27, 2020 at 10:31 AM Alec Warner  wrote:
>
> On Sun, Dec 27, 2020 at 6:39 AM Ulrich Mueller  wrote:
> >
> > >>>>> On Sun, 27 Dec 2020, Max Magorsch wrote:
> >
> > > To access the old repositories you can use gitweb.gentoo.org instead.
> > > We have migrated all old cvs repositories to git. All of them are
> > > available read-only now at [0].
> >
> > I've just looked at
> > https://sources.gentoo.org/archive/cvs/gentoo.git/
> > and its commit history ends in 2004.
> >
> > Can you please reinstate CVS until a more accurate conversion is
> > available?
>
> I'm happy to make tarballs available (as discussed in a bunch of
> places on irc.) Is that sufficient or is there some particular
> requirement for the CVS protocol specifically?

So some updates:

 - We will make tarballs of the raw CVS repos available soon; I'm
working with ulm@ to remove various PII in some places; we previously
used CVS ACLs to prevent people from seeing this PII and we will
similarly remove it in these tarballs.
 - Sources.gentoo.org (used to browse cvs repositories) is now gone.
 - https://sources.gentoo.org now points at gitweb.gentoo.org, but we
may point it at https://anongit.gentoo.org in the future.
 - The viewvc instance that powered cvs browsing was deleted and isn't
coming back and the basic idea is that we will serve archival tarballs
(for really old stuff) and git (for modern stuff) and nothing else.

Thanks to ulm (for reviewing the cvs content) and arzano (who is doing
all the actual work underneath ;p)

Also some clarification here. Infra has carried some tech debt for 5
years[0] and this mostly worked OK for us. However cfengine2 is
*really* old and we need to get off of it. Most new services run on
top of 'puppet' a 'newer' management stack that helps us configure
machines and services. So this push to retire a bunch of stuff is
aligned with a push to get off of cfengine completely by EOY because
it's starting to really cause us operational challenges. Antarus is
not just arbitrarily shutting stuff off; but we are also in a bit of a
rush at the end of 2020 so the shutdowns are not as smooth as we would
like.

-A

[0] I checked, we started to migrate to puppet in *2010* and are still
migrating, 10 years later ;P

>
> -A
>
> >
> > Same applies to gentoo-x86 where the git repo misses whole categories.
> >
> > Ulrich



Re: [gentoo-dev] Re: [gentoo-dev-announce] We are finally shutting down CVS

2020-12-27 Thread Alec Warner
On Sun, Dec 27, 2020 at 6:39 AM Ulrich Mueller  wrote:
>
> > On Sun, 27 Dec 2020, Max Magorsch wrote:
>
> > To access the old repositories you can use gitweb.gentoo.org instead.
> > We have migrated all old cvs repositories to git. All of them are
> > available read-only now at [0].
>
> I've just looked at
> https://sources.gentoo.org/archive/cvs/gentoo.git/
> and its commit history ends in 2004.
>
> Can you please reinstate CVS until a more accurate conversion is
> available?

I'm happy to make tarballs available (as discussed in a bunch of
places on irc.) Is that sufficient or is there some particular
requirement for the CVS protocol specifically?

-A

>
> Same applies to gentoo-x86 where the git repo misses whole categories.
>
> Ulrich



Re: [gentoo-dev] Re: [gentoo-dev-announce] We are finally shutting down CVS

2020-12-27 Thread Alec Warner
On Sun, Dec 27, 2020 at 12:42 AM Michał Górny  wrote:
>
> On Sun, 2020-12-27 at 00:55 +, Max Magorsch wrote:
> > Hi all,
> >
> > as a quick note: We are finally shutting down all old cvs services.
> > Accordingly the old viewvc repository browser will be shut down as
> > well.
> >
> > To access the old repositories you can use gitweb.gentoo.org instead.
> > We have migrated all old cvs repositories to git. All of them are
> > available read-only now at [0].
>
> Are we keeping the original CVS archives too, for potential better
> conversions in the future?

We don't plan on deleting any data, and we have not deleted any data
to date. Most historical data is kept in Amazon S3 and on a
Gentoo-owned machine somewhere.

-A

>
> --
> Best regards,
> Michał Górny
>
>
>



[gentoo-dev] I'm looking for a Mentor

2020-12-18 Thread Alec Warner
TL;DR, I think infra needs more people who can commit to ::gentoo and
thus, I am looking for an ebuild dev mentor. I myself had commit
access in the before times (probably 2010 - 2012) but have been mostly
ignoring ebuild development since then to focus on infra and the
Foundation. However infra is using more and more software that is
maintainer-needed and so we probably need to change our focus to
giving back more to the main tree.

-A



Re: [gentoo-dev] GPG key refresh

2020-12-14 Thread Alec Warner
antarus@woodpecker ~ $ gpg --keyserver hkps://keys.gentoo.org --search-keys
1C49724D229E93A2
gpg: data source: https://[2001:470:ea4a:1:230:48ff:fef8:9fdc]:443
(1) Michael Orlitzky 
   Michael Orlitzky 
 4096 bit RSA key 1C49724D229E93A2, created: 2010-03-17, expires:
2020-12-26
Keys 1-1 of 1 for "1C49724D229E93A2".  Enter number(s), N)ext, or Q)uit > q
gpg: error searching keyserver: Operation cancelled
gpg: keyserver search failed: Operation cancelled

I think you need to push to hkps://keys.gentoo.org

-A

On Mon, Dec 14, 2020 at 11:00 AM Michael Orlitzky  wrote:

> I'm still getting this,
>
>$ git push --signed
>...
>remote: 1C49724D229E93A2 [Michael Orlitzky ] [E]
>expire:short Expiration date is too close, please renew (is 2020-12-26
>23:30:47, less than 14 days)
>remote: 1C49724D229E93A2:6F48D3DA05C2DADB [Michael Orlitzky
>] [E] expire:short Expiration date is too close,
>please renew (is 2020-12-26 23:31:43, less than 14 days)
>
> over a week after I've renewed my keys. The answer I get back from the
> keyserver(s) looks OK:
>
>$ gpg --search-keys 0x6F48D3DA05C2DADB
>gpg: data source: http://85.25.207.23:11371
>(1)   Michael Orlitzky 
>  Michael Orlitzky 
>4096 bit RSA key 0x1C49724D229E93A2, created: 2010-03-17,
>expires: 2022-12-05
>
>
> Did I forget a step? How long should it take for infra to refresh?
>
>


Re: [gentoo-dev] [PATCH 1/2] eapi8-dosym.eclass: New eclass.

2020-11-19 Thread Alec Warner
On Thu, Nov 19, 2020 at 7:28 AM Alec Warner  wrote:

>
>
> On Thu, Nov 19, 2020 at 2:32 AM Ulrich Müller  wrote:
>
>> This implements the dosym command proposed for EAPI 8 (called dosym8
>> because we cannot use the same name as the package-manager builtin).
>>
>> "dosym -r  " will expand the (apparent) path of 
>> relative to the (apparent) path of the directory containing .
>> The main aim of this is to allow for an absolute path to be specified
>> as the link target, and the function will count path components and
>> convert it into a relative path.
>>
>> Since we're inside ED at this point but the image will finally be
>> installed in EROOT, we don't try to resolve any pre-existing symlinks
>> in  or . In other words, path expansion only looks at
>> the specified apparent paths, without touching any actual files in ED
>> or EROOT.
>>
>> Signed-off-by: Ulrich Müller 
>> ---
>>  eclass/eapi8-dosym.eclass | 108 ++
>>  1 file changed, 108 insertions(+)
>>  create mode 100644 eclass/eapi8-dosym.eclass
>>
>> diff --git a/eclass/eapi8-dosym.eclass b/eclass/eapi8-dosym.eclass
>> new file mode 100644
>> index ..52f0ffe3e62b
>> --- /dev/null
>> +++ b/eclass/eapi8-dosym.eclass
>> @@ -0,0 +1,108 @@
>> +# Copyright 2020 Gentoo Authors
>> +# Distributed under the terms of the GNU General Public License v2
>> +
>> +# @ECLASS: eapi8-dosym.eclass
>> +# @MAINTAINER:
>> +# PMS team 
>> +# @AUTHOR:
>> +# Ulrich Müller 
>> +# @SUPPORTED_EAPIS: 5 6 7
>> +# @BLURB: Testing implementation of EAPI 8 dosym -r option
>> +# @DESCRIPTION:
>> +# A stand-alone implementation of the dosym command aimed for EAPI 8.
>> +# Intended to be used for wider testing of the proposed option and to
>> +# allow ebuilds to switch to the new model early, with minimal change
>> +# needed for actual EAPI 8.
>> +#
>> +# https://bugs.gentoo.org/708360
>> +
>> +case ${EAPI} in
>> +   5|6|7) ;;
>> +   *) die "${ECLASS}: EAPI=${EAPI:-0} not supported" ;;
>> +esac
>> +
>> +# @FUNCTION: _dosym8_canonicalize
>> +# @USAGE: 
>> +# @INTERNAL
>> +# @DESCRIPTION:
>> +# Transparent bash-only replacement for GNU "realpath -m -s".
>> +# Resolve references to "/./", "/../" and remove extra "/" characters
>> +# from , without touching any actual file.
>>
>
> Take this as a nit, but do we have any way to test this function
> (particularly around edge cases.)
> I see eclass/tests exists, could we add a couple for this?
>

hah, and you added them in the second patch, sorry I didn't read ahead! :)

-A


>
>
>> +_dosym8_canonicalize() {
>>
>
> in dosym8() you save and restore IFS, but you don't here, is there a
> reason for that?
>
>
>> +   local path slash i prev out IFS=/
>> +
>> +   path=( $1 )
>> +   [[ $1 == /* ]] && slash=/
>> +
>> +   while true; do
>> +   # Find first instance of non-".." path component followed
>> by "..",
>> +   # or as a special case, "/.." at the beginning of the
>> path.
>> +   # Also drop empty and "." path components as we go along.
>> +   prev=
>> +   for i in ${!path[@]}; do
>> +   if [[ -z ${path[i]} || ${path[i]} == . ]]; then
>> +   unset "path[i]"
>> +   elif [[ ${path[i]} != .. ]]; then
>> +   prev=${i}
>> +   elif [[ ${prev} || ${slash} ]]; then
>> +   # Found, remove path components and
>> reiterate
>> +   [[ ${prev} ]] && unset "path[prev]"
>> +   unset "path[i]"
>> +   continue 2
>> +   fi
>> +   done
>> +   # No (further) instance found, so we're done
>> +   break
>> +   done
>> +
>> +   out="${slash}${path[*]}"
>> +   echo "${out:-.}"
>> +}
>> +
>> +# @FUNCTION: dosym8
>> +# @USAGE: [-r]  
>> +# @DESCRIPTION:
>> +# Create a symbolic link , pointing to .  If the
>> +# directory containing the new link does not exist, create it.
>> +#
>> +# If called with option -r, expand  relative to the 

Re: [gentoo-dev] [PATCH 1/2] eapi8-dosym.eclass: New eclass.

2020-11-19 Thread Alec Warner
On Thu, Nov 19, 2020 at 2:32 AM Ulrich Müller  wrote:

> This implements the dosym command proposed for EAPI 8 (called dosym8
> because we cannot use the same name as the package-manager builtin).
>
> "dosym -r  " will expand the (apparent) path of 
> relative to the (apparent) path of the directory containing .
> The main aim of this is to allow for an absolute path to be specified
> as the link target, and the function will count path components and
> convert it into a relative path.
>
> Since we're inside ED at this point but the image will finally be
> installed in EROOT, we don't try to resolve any pre-existing symlinks
> in  or . In other words, path expansion only looks at
> the specified apparent paths, without touching any actual files in ED
> or EROOT.
>
> Signed-off-by: Ulrich Müller 
> ---
>  eclass/eapi8-dosym.eclass | 108 ++
>  1 file changed, 108 insertions(+)
>  create mode 100644 eclass/eapi8-dosym.eclass
>
> diff --git a/eclass/eapi8-dosym.eclass b/eclass/eapi8-dosym.eclass
> new file mode 100644
> index ..52f0ffe3e62b
> --- /dev/null
> +++ b/eclass/eapi8-dosym.eclass
> @@ -0,0 +1,108 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: eapi8-dosym.eclass
> +# @MAINTAINER:
> +# PMS team 
> +# @AUTHOR:
> +# Ulrich Müller 
> +# @SUPPORTED_EAPIS: 5 6 7
> +# @BLURB: Testing implementation of EAPI 8 dosym -r option
> +# @DESCRIPTION:
> +# A stand-alone implementation of the dosym command aimed for EAPI 8.
> +# Intended to be used for wider testing of the proposed option and to
> +# allow ebuilds to switch to the new model early, with minimal change
> +# needed for actual EAPI 8.
> +#
> +# https://bugs.gentoo.org/708360
> +
> +case ${EAPI} in
> +   5|6|7) ;;
> +   *) die "${ECLASS}: EAPI=${EAPI:-0} not supported" ;;
> +esac
> +
> +# @FUNCTION: _dosym8_canonicalize
> +# @USAGE: 
> +# @INTERNAL
> +# @DESCRIPTION:
> +# Transparent bash-only replacement for GNU "realpath -m -s".
> +# Resolve references to "/./", "/../" and remove extra "/" characters
> +# from , without touching any actual file.
>

Take this as a nit, but do we have any way to test this function
(particularly around edge cases.)
I see eclass/tests exists, could we add a couple for this?


> +_dosym8_canonicalize() {
>

in dosym8() you save and restore IFS, but you don't here, is there a reason
for that?


> +   local path slash i prev out IFS=/
> +
> +   path=( $1 )
> +   [[ $1 == /* ]] && slash=/
> +
> +   while true; do
> +   # Find first instance of non-".." path component followed
> by "..",
> +   # or as a special case, "/.." at the beginning of the path.
> +   # Also drop empty and "." path components as we go along.
> +   prev=
> +   for i in ${!path[@]}; do
> +   if [[ -z ${path[i]} || ${path[i]} == . ]]; then
> +   unset "path[i]"
> +   elif [[ ${path[i]} != .. ]]; then
> +   prev=${i}
> +   elif [[ ${prev} || ${slash} ]]; then
> +   # Found, remove path components and
> reiterate
> +   [[ ${prev} ]] && unset "path[prev]"
> +   unset "path[i]"
> +   continue 2
> +   fi
> +   done
> +   # No (further) instance found, so we're done
> +   break
> +   done
> +
> +   out="${slash}${path[*]}"
> +   echo "${out:-.}"
> +}
> +
> +# @FUNCTION: dosym8
> +# @USAGE: [-r]  
> +# @DESCRIPTION:
> +# Create a symbolic link , pointing to .  If the
> +# directory containing the new link does not exist, create it.
> +#
> +# If called with option -r, expand  relative to the apparent
> +# path of the directory containing .  For example, "dosym8 -r
> +# /bin/foo /usr/bin/foo" will create a link named "../../bin/foo".
> +dosym8() {
> +   local option_r
> +
> +   case $1 in
> +   -r) option_r=t; shift ;;
> +   esac
> +
> +   [[ $# -eq 2 ]] || die "${FUNCNAME}: bad number of arguments"
> +
> +   local target=$1 link=$2
> +
> +   if [[ ${option_r} ]]; then
> +   local linkdir comp
> +
> +   # Expansion makes sense only for an absolute target path
> +   [[ ${target} == /* ]] \
> +   || die "${FUNCNAME}: -r specified but no absolute
> target path"
> +
> +   target=$(_dosym8_canonicalize "${target}")
> +   linkdir=$(_dosym8_canonicalize "/${link#/}")
> +   linkdir=${linkdir%/*}   # poor man's dirname(1)
> +   linkdir=${linkdir:-/}   # always keep the initial "/"
> +
> +   local ifs_save=${IFS-$' \t\n'} IFS=/
> +   for comp in ${linkdir}; do
> +   if [[ ${target%%/*} == "${comp}" 

Re: [gentoo-dev] Automated Package Removal and Addition Tracker, for the week ending 2020-11-08 23:59 UTC

2020-11-09 Thread Alec Warner
On Mon, Nov 9, 2020 at 2:24 AM Ulrich Mueller  wrote:

> > On Mon, 09 Nov 2020, Robin H Johnson wrote:
>
> > The attached list notes all of the packages that were added or removed
> > from the tree, for the week ending 2020-11-08 23:59 UTC.
>
> > [...]
>
> > app-editors/emacs
> 20150808-20:49 robbat2  56bd759df1d
>
> Not really ...
>
> This is broken for the second week in a row now. Could you suspend these
> messages please, until the problem is fixed?
>

This is an infra job, I've disabled it for now.

-A


>
> Ulrich
>


Re: [gentoo-dev] New QA policy suggestion: Disallow "live-only" packages

2020-11-04 Thread Alec Warner
On Wed, Nov 4, 2020 at 7:38 PM Joonas Niilola  wrote:

>
>
> On 11/4/20 11:19 PM, Rich Freeman wrote:
> > Did you consider that somebody could read your email and not actually
> > agree with you?
> Impossible! My suggestion is about keeping the tree clean and to provide
> the best user experience. Who'd disagree with that?!
>
> Sure the methods for achieving that can be discussed, but if I find ~50
> % of current live-only packages being unbuildable, don't you agree
> there's a problem with them? And regardless the outcome of this policy
> suggestion, I've filed 14 bugs to maintainers notifying their --only
> packages aren't working.
>
> > Does it work?  If so, then there is no harm.  If not, then just remove
> > it - you don't need a new policy to treeclean stuff that doesn't work.
> One of the selling points to this policy suggestion is that they'd get
> CI coverage, which they currently don't. Why keep dead weight around for
> years that apparently no one uses, not even the maintainer themself.
>

> > You're saying that live-only packages should be removed because they
> > could have snapshots.  I'm saying that if you want to maintain a
> > snapshot just add it and co-maintain - you don't have to remove
> > packages that don't have them.
> I'm saying currently maintainers aren't doing very good job at
> maintaining them, and no one else uses/finds these packages.
>

Again though, this isn't about having a policy against X or Y and seems
instead to be a policy against unmaintained stuff...and I think we mostly
have a policy against that already; there is a whole team who removes stuff
(treecleaners).

The result is that we should remove badly maintained stuff; not create more
policies.

-A



>
> -- juippis
>
>
>
>


Re: [gentoo-dev] New QA policy suggestion: Disallow "live-only" packages

2020-11-04 Thread Alec Warner
On Mon, Nov 2, 2020 at 9:13 PM Joonas Niilola  wrote:

> Hey,
>
> I'm suggesting a new QA policy to disallow any "live-ebuild-only
> packages" being hosted in ::gentoo. Rationale being the same as why
> - packages can't have KEYWORDS: They are unpredictable and
> potentially insecure. Unpredictability could mean upstream repo being
> broken at any given time placing users in an awkward situation, where
> they are able to build some packages while not the others. Upstream
> repo can also be force-pushed over. I feel like packages offered in
> ::gentoo shouldn't have these issues, and the need to have at least one
> safe release available to users that's guaranteed to build.
>

I disagree. These packages are not installable by default, and must be
unmasked by users, so this tradeoff is one we expect them to make. Are
there practical problems that these packages pose to developers? You listed
a bunch of user problems, but again users are opting into these problems,
presumably.


>
> With Git-like VCS's becoming popular, it is super easy to create an
> unchanged snapshot based on a commit. Even devmanual encourages this
> with proper guide how-to:
>
> https://devmanual.gentoo.org/ebuild-writing/file-format/index.html#snapshots-and-live-ebuilds
>(https://devmanual.gentoo.org/keywording/index.html)
>
> We currently have 43 "live-ebuild-only" packages in tree. Out of those
> 19 are totally unbuildable, that's 44 % of all packages present in the
> repo. Overall the maintenance of these live-ebuild-only packages looks
> low, there's only a handful of ebuilds that have good quality and no
> issues at all. 12 of them, 28 %, are still on EAPI-5. Of all 43, only 2
> would require the maintainer to generate a tarball by themself, while
> others can utilize upstream's tagged releases, or ability to make a
> snapshot from a specific commit. Obviously if this policy passes, I'll
> be helping getting these packages keyworded.
>

But again, why are we making this a firm policy; as opposed to letting the
maintainer make their decision?


>
> And finally here's an example how to introduce new packages to tree
> without upstream releases:
>
> https://gitweb.gentoo.org/repo/gentoo.git/commit/dev-libs/rlottie?id=42873c46b7ed07d5b4f8af5fcf08d8549cb6385b
>
> https://gitweb.gentoo.org/repo/gentoo.git/commit/media-libs/rlottie?id=2de52234783be909f6e4aed333533e6a804e8e6b
>
> https://gitweb.gentoo.org/repo/gentoo.git/commit/media-libs/rlottie?id=8305f0c6cd0ce8cb5ac0f2d92569acce36a5cc0a
>etc...
>
> https://gitweb.gentoo.org/repo/gentoo.git/commit/media-libs/rlottie?id=24c48b325dd5a22284d077d6581a1a45e397e511
>
> If the only available version for a package doesn't build - or can't be
> guaranteed to build - then it should be last-rited, or not exist in
> ::gentoo repo at all.
>

We can't guarantee any package to build..so I'm not sure how this is a
practical policy goal.

-A


>
> Some history and initiative: bug #713802
>
> -- juippis
>
>


Re: [gentoo-dev] [PATCH 1/5] verify-sig.eclass: New eclass to verify OpenPGP sigs

2020-10-12 Thread Alec Warner
On Sun, Oct 11, 2020 at 7:35 AM Joonas Niilola  wrote:

>
>
> On 10/11/20 4:40 PM, Thomas Deutschmann wrote:
> >
> >
> > First of all, calm down. You are reading too much into this. Just
> > revert your own logic: You obviously like your idea, worked on this
> > and pushed it to repository. Don't you see that anyone could ask the
> > same? Who are you? Why do believe that you can force something like
> > that down to everyone's system? That feature got enabled in developers
> > profiles by default, it will blow up distfiles with useless data (a
> > view you can have when you share my concerns and don't see any
> > problems with current Manifest system; For example we banned stuff in
> > $FILESDIR before and asked developers to move them to d.g.o when >25kb
> > in total arguing that large distfile is affecting *any* user... I am
> > not comparing this 1:1 but to repeat your question: Who are you that
> > you can decide to force something similar down to everyone).
> >
> >
> > Sure, if I am the only having that opinion and most people see a value
> > in this and disagree with me...
>

I split this into three parts:

(1) Is there a problem? I like to think there is general agreement that
developers do not cryptographically verify distfiles for most upstreams
when bumping, and thus we could all agree there is room for improvement
here. In an ideal world we are relying on existing systems (https, CAs,
etc) to prevent MITM attacks on source downloads, but not much is used
beyond that.
(2) Does the proposed solution help? This is not a yes or no question; its
nuanced. Having a keyring makes verifying easier. As Whissi notes, we don't
discuss much the managing of said keyrings (or revocation) and so I think
the proposed solution does have similar problems to the existing solution.
Instead of managing and verifying upstream tarballs, we have to verify
keys. There is no automation for this either...and so we end up with a
similar attack surface. There is *improvement* (if someone MITMs your
download, the verification will notice.) Is that the most likely attack, or
is it stolen upstream signing keys? Who can really say?
(3) The implementation. This is honestly the part that I dislike the most,
particularly in the original draft, some of the problems have been fixed
already. I'm not excited about thousands of new packages, nor am I excited
about the key management in the proposal. The biggest problem (that it was
on by default) were already fixed which is good; I don't even see this as a
feature for end users at all; instead its a feature for developers and
maybe a QA bot (that verifies the distfiles.)

Leading out of 3, maybe that is a decent solution also. Can we build a QA
bot that detects bad bumps but also has not terrible key management? Is
there an automated protocol for fetching *and* verifying upstream files
like this? Could we annotate SRC_URI somehow with verification metadata?

-A



> >
> >
> I vote for disabling that USE flag in developer profile. There are more
> than 1 person using it not yet buying or understanding this "draft". I
> also believe such a profile flag change should be discussed first.
>
> -- juippis
>
>


Re: [gentoo-dev] Python 2 cleanup update

2020-09-27 Thread Alec Warner
On Sun, Sep 27, 2020 at 10:45 AM Michał Górny  wrote:

> Hello, everyone.
>
> TL;DR: we're nearing the total annihilation of Python 2 software
> in Gentoo.  Most users could safely disable py2 USE flags today.
> Python 2 vulns have been patched recently, the interpreter and a few
> packages using Python at build time (with no deps) will stay.  Should we
> change PYTHON_TARGETS now, or wait some more and just annihilate
> the py2 flag from all packages?
>
>
> Long version:
>
> We're reached the point where the majority of packages relying on py2
> have either been ported to py3, removed or masked for removal.
> As a result, I've been able to eliminate python2_7 target from the vast
> majority of dev-python/* packages.  On their next system upgrade, our
> users are going to notice most of Python 2.7 modules gone from their
> systems.
>
> However, because of their reverse dependencies a few packages can't lose
> their py2.7-iness, and therefore are going to block depcleaning Python
> 2.7 for now.  These include old versions of setuptools, numpy, pillow,
> as well as all versions of cython, nose, pykerberos, pyyaml and their
> dependencies.  The major blockers for them are:
>
> - dev-lang/gdl (py entirely optional but the package itself is seriously
> broken)
>
> - dev-db/mongodb (py3 version was just stabilized, need to decide how to
> clean old versions up)
>
> - games-engines/renpy (no py3 version yet)
>
> - media-tv/kodi (py3 version in alpha)
>
> We plan to have these packages fixed or removed by the deadline.
>
>
> However, we already know that there are some packages that use Python 2
> at build time and that will keep requiring it past the deadline.
> The initial list includes:
>
> - dev-python/pypy* (TODO: need to figure bootstrap out)
>
> - dev-lang/spidermonkey, www-client/seamonkey, www-client/firefox...
> (thank you, Mozilla)
>
> - www-client/chromium, dev-qt/qtwebengine... (thank you, Google)
>
> Sadly, the big corps are too busy improving their spying functionality
> and creating NIH programming languages to take care of such minor
> matters as cleaning up.
>


https://bugs.chromium.org/p/chromium/issues/list?q=Proj%3DPython3Migration=2
is the tracker for python3 migration for chromium. I resent the implication
that Google is 'too busy' to work on it.

E.g. on
https://bugs.chromium.org/p/chromium/issues/detail?id=941669=Proj%3DPython3Migration=2
the last commit was Sept 26, or yesterday ;p

-A


> The general rule is that py2.7 may remain in packages that use it
> at build time only (i.e. don't install anything depending on Python)
> and have no dependencies on Python packages (i.e. don't require any
> other packages to install py2.7 modules).  Or to put it otherwise,
> python-r1 and python-single-r1 will lose py2.7 support entirely, while
> python-any-r1 will retain minimal support without dependencies.
>
>
> This also implies that we're going to keep Python 2.7 itself for as long
> as necessary, and patch it if possible.  I should take this opportunity
> to remind you that it's quite possible that the interpreter itself has
> unknown vulnerabilities.  Only recently I've backported two sec fixes
> from Python 3 which no other distribution (including the one promising
> paid support for Python 2 for next years) or upstream (including all
> these boasting that they're going to maintain Python 2 themselves) has
> even noticed (to the best of my knowledge).
>
>
> An open question is whether we should remove python2_7 from
> PYTHON_TARGETS now.  If we do that, it will permit the vast majority of
> Gentoo users to depclean Python 2.7 today, independently of how long
> the maintainer of renpy is going to block it, with only a few users
> having to enable the flag manually.  However, doing this makes sense
> only if we're really going to delay the impeding doom long.
>
> I will probably prepare an updated news item for Python 2.7 removal,
> to replace the one from February with the updated plan, current
> information and helpful tips.
>
>
> Finally, I would like to thank all the helpful package maintainers, arch
> teams and other developers who have made this possible.
>
> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] [RFC] Services and software which is critical for Gentoo should be developed/run in Gentoo namespace

2020-09-16 Thread Alec Warner
On Wed, Sep 16, 2020 at 1:17 AM Kent Fredric  wrote:

> On Mon, 14 Sep 2020 10:15:31 -0400
> Rich Freeman  wrote:
>
> > It might be easier to take smaller steps, such as having a policy that
> > "any call for devs to use/test a new tool/service, or any service that
> > automatically performs transactions on bugzilla, must be FOSS, and the
> > link to the source must be included in the initial communication, and
> > it must be clear what version of the code is operating at any time."
> > That is a pretty low barrier to those creating tools, though it
> > doesn't address the infra concern.  However, it does mean that infra
> > is now free to fork the service at any time, and reduces the bus
> > factor greatly.
>
> For the situation of things that take life before being part of infra,
> I think the least we can do is recognize their utility and importance,
> and at least, have infra *offer* some sort of shared location to run a
> deployment.
>
> That I think helps everyone, gives people a place to remove their own
> bus factor, but without mandatory strongarming.
>

The main challenge is always getting stuff onto infra. The past few things
we have launched have been because the developers in question joined infra
to launch their work.

 - repomirror-ci and all the CI stuff is on infra because mgorny is also on
infra! It's not like we set his stuff up for him; instead we gave him
access to all the infra repos and he had to write his own puppet configs
and whatnot. The benefit of course is that anyone on infra can bump the
stuff and login to the machines and debug...but its not exactly a low bar.
 - packages.gentoo.org was also originally arzano pushing patches to me
(which I would merge and then release by hand.) Just like with ebuilds this
eventually became too tedious and he was onboarded as a dev and infra
member to eliminate the middleman.

I think traditionally it's been a slog for non-infra people to get infra to
host much of anything and due to difficulties with the all-or-nothing
approach we take with infra credenteials; can really set a high bar to host
much of anything these days.

-A


Re: [gentoo-dev] [RFC] Services and software which is critical for Gentoo should be developed/run in Gentoo namespace

2020-09-13 Thread Alec Warner
On Sun, Sep 13, 2020 at 11:31 AM Thomas Deutschmann 
wrote:

> Hi,
>
> TL;DR: jstein asked council [Bug 729062] for a motion that any service
> and software which is critical for Gentoo should be developed/run in
> Gentoo namespace. Because any request to council must be discussed I
> volunteered to bring this topic to the mailing list (sorry for the huge
> delay!).
>
>
> Problem
> ===
> You maybe all remember what happened to stable-bot: Years ago,
> kensington created stable-bot on his own as PoC which revolutionized the
> way how we do package stabilization in bugzilla. The service run on his
> own infrastructure. Because of the benefit of the service the bot
> provided, arch team’s workflow became dependent on stable-bot. We were
> lucky that stable-bot just worked most of the time until the service was
> down for a while. Nobody was able to help here: Kensigton himself was
> unavailable, nobody had the sources… the end of the story: mgorny
> created nattka which replaced stable-bot.
>
> However, we are still facing the same problem: Only one person is
> involved in development and knows how to run it. In case something will
> break again and Michał will be unavailable, we can’t just push a fix and
> watch a CI pipeline picking up and deploying new nattka. Instead someone
> will have to fork repository from Michał’s private repository at GitHub,
> make the changes and hope that anyone within infrastructure team can
> help to deploy fixed nattka.
>
> This is what the motion is about: This is not about that Gentoo depends
> on single persons or things like that. It’s about the idea to
> *formalize* the requirement that any service and software which is
> critical for Gentoo (think about pkgcore) should live within Gentoo
> namespace (https://gitweb.gentoo.org/), i.e. be accessible for *any*
> Gentoo developer and deployments should be based on these repositories.
> Or in other words: Make sure that we adhere to social contract even for
> critical software and services Gentoo depends on. So that we will never
> ever face the situation that something we depend on doesn’t work
> anymore. Taking care of working pipelines before something is broken
> should also help us in case something stops working so we don’t have to
> figure out how to fix and re-deploy when house is already burning (like
> portage: In case Zac can't do a release for some reason, in theory,
> every Gentoo developer would be able to roll a new release).
>

I think your examples are a bit weird.

Is openrc critical to Gentoo? it doesn't live on our infra.
Is pkgcore critical to Gentoo? it doesn't live on our infra.

Note that these are just packages, not services and the social contract
just says
"""However, Gentoo will never depend upon a piece of software or metadata
unless it conforms to the GNU General Public License, the GNU Lesser
General Public License, the Creative Commons - Attribution/Share Alike or
some other license approved by the Open Source Initiative."""

It says nothing about where things are hosted or how services are provided.

I'd consider splitting the two here. For packages I don't think it matters
as much where they are hosted. Most things can be mirrored into gentoo (if
we want a copy of the src tree) and we also have tarballs of the source
code much of the time on the mirror network.

For services, I tend to agree more with your comments; we need need
visibility and operational capability for services. When we rely on service
components where the source is not available; its bad. But we rely on
numerous services now. E.g. p.g.o relies on repology. Does that mean we
need the source code to repology? I assume not. Does that mean we need to
run our own repology? Also I assume not.

-A


>
> See also:
> =
> Bug 729062: https://bugs.gentoo.org/729062
>
>
> --
> Regards,
> Thomas Deutschmann / Gentoo Linux Developer
> C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
>
>


Re: [gentoo-dev] [PATCH] profiles/targets/desktop: Do not enable ldap USE flag by default

2020-09-10 Thread Alec Warner
On Thu, Sep 10, 2020 at 8:13 AM Michał Górny  wrote:

> On Thu, 2020-09-10 at 07:35 +0200, Hans de Graaff wrote:
> > On Wed, 2020-09-09 at 13:35 +0300, Mikle Kolyada wrote:
> > > Closes: https://bugs.gentoo.org/741380
> >
> > Could you provide a rationale for removing this? The bug only has a
> > single anecdotal report of a user who can run a desktop without it. I'm
> > not sure if that is reason enough to remove this. I guess we won't be
> > able to figure out easily how many of our desktop profile users are
> > actually using LDAP, but changing this may cause surprises and I'm not
> > sure if that's warranted.
> >
>
> 2020.  Gentoo discovers that people usually don't run LDAP on their
> desktops.  Some developers can't believe their eyes!  Gentoo forms
> a Working Group to establish whether using LDAP on desktops is really
> that uncommon.
>
> This fits nicely with the recently announced fact that Gentoo Foundation
> has too much money and is looking for funding requests.  After all, what
> could be a better use of Gentoo money than funding the work of a Working
> Group trying to establish matters of such great importance as default
> USE flags on desktop profiles.
>

I don't appreciate these comments. You can think the Foundation wastes
money on taxes and paperwork and operational expenses but i'm not going to
sit here and watch you crap all over my attempts to actually *support* the
Gentoo community by funding actual work. So please keep your negative
comments to yourself.

-A


>
> After a lot of debate the Working Group concludes that there is no
> conclusive answer whether LDAP should be enabled by default or not.
>

> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] [PATCH] profiles/targets/desktop: Do not enable ldap USE flag by default

2020-09-10 Thread Alec Warner
On Thu, Sep 10, 2020 at 1:59 AM Mikle Kolyada  wrote:

>
> On 10.09.2020 08:35, Hans de Graaff wrote:
> > On Wed, 2020-09-09 at 13:35 +0300, Mikle Kolyada wrote:
> >> Closes: https://bugs.gentoo.org/741380
> > Could you provide a rationale for removing this? The bug only has a
> > single anecdotal report of a user who can run a desktop without it. I'm
> > not sure if that is reason enough to remove this. I guess we won't be
> > able to figure out easily how many of our desktop profile users are
> > actually using LDAP, but changing this may cause surprises and I'm not
> > sure if that's warranted.
> >
> > Hans
>
>
> Hi.
>
> It is dictated by common sense.
>

> I barely can imagine a case where you need ldap support in each and
> every package you install.
>

I can, but not in a desktop environment. In an enterprise environment you
will need it (but I'd expect folks to add it.)
The challenge is just in changing the default. E.g. we might not do it
until a new profile bump (e.g. do it in a 2020 or 2021 profile?)

-A


>
> This should rather be per-package enabled as something non-trivial.
>
>
>


Re: [gentoo-dev] rfc: switching default udev provider for new systems to udev

2020-08-09 Thread Alec Warner
On Sun, Aug 9, 2020 at 11:22 AM William Hubbs  wrote:

> On Sun, Aug 09, 2020 at 06:40:07PM +0200, Thomas Deutschmann wrote:
> > On 2020-08-08 20:51, William Hubbs wrote:
> > > What do people think?
> >
> > Like others already asked: What's the reason for this?
>
>  Like others have said on the thread, the reason for the switch away
>  from udev in the past was mostly fear driven instead of fact driven. As
>  already said, if the udev developers were going to make udev unusable
>  without systemd they would have by now.
>
> > What do you expect from this change?
>
>  I expect Gentoo to use, by default, what most of the Linux community
>  uses for device management.
>

"I expect Gentoo to use, by default, what most of the Linux community uses
for init management." So we should make the systemd profile the default? :)


>
> > Is there a problem when new Gentoo installations will use EUDEV by
> > default? Or is there a benefit if new installations would use
> sys-fs/udev?
>
> Please look back at the history of why we switched away from udev. It
> was not technical. Udev did not cause any wide scale distro breakages.
> It was because some folks were very loud about a possible systemd
> consppiracy around making udev not work without systemd.
>

You asked me on IRC "how do I convince people" and part of that is to make
it easy to agree with your argument! Asking me to read a bunch of crap
isn't going to make me want to agree; its going to make me say "your
argument is poorly formed, please go away."

 - Link to the things you want me to read.
 - Summarize them so I don't have to read a 100 message long thread from 5
years ago.
 - Make an argument!

---
"I think we picked eudev as the default because of a concern that udev
would eventually require systemd for operation, you can see this from these
mailing list posts: X, Y, Z."
"The above concern has not manifested itself and I believe udev will
continue to not strictly require systemd init for various reasons (mention
list of cases here."
"Therefore I think we should change the default udev provider from eudev to
udev in the default profiles."
---

This would be what I believe is a understandable argument (provided we had
the links to the previous material.) I'm not saying I agree[0] with it; but
I'd at least understand why you want the change to happen.


> Notice again that I'm not saying we need to lastrites eudev. There are
> cases that have developed for it (mainly non-glibc systems), but I am
> saying I see no justification at this point for it being the default
> distro wide.

William
>
>
[0] I expect that most users who want udev actually also want systemd and
so will simply select the systemd profile itself, and that this choice is
immaterial to most users; so I am for keeping the status quo here.


[gentoo-portage-dev] pylint progress

2020-07-29 Thread Alec Warner
Hi,

Recently I've begun to run pylint on the portage codebase. You can see some
recent PRs on this[0][1][2]. Most of the linter errors I've fixed are what
I consider 'fairly trivial'. In general I'm happy to disable errors (or
instances of errors) in addition to resolving them. You can see some of
this in https://github.com/gentoo/portage/pull/593/files where we just
blanket disable a bunch of very numerous messages (either because I don't
expect to ever fix them, or because they are more work that I think we
should do at the moment.)

So I see essentially a few choices:
 - (a) Do we fix errors in certain classes and run the linter as part of
CI. This means that once we resolve errors of a certain class, we can
enable that class of check and prevent new occurrences.
 - (b) Do we just avoid running the linter as part of CI (perhaps because
we are not far enough along on this journey) and focus our efforts to get
to a point where we can do (a) without spamming CI?
 - (c) What messages should we bother not fixing at all so we can just not
work on those classes of error?

As an example of (c); I believe 'protected-access' is likely intrusive to
fix and pointless for portage (cat is out of the bag) where as a message
like W0612(unused-variable) is plausibly something we should fix
throughout the codebase. Similarly I think "E0602(undefined-variable)" is
often a bug in how pylint treats our lazy initialized variables, so most of
these are false positives.

I think getting these messages recorded will also enable other people
besides me to fix them (I know b-man said he was interested.) Curious to
hear thought on this.

-A

[0]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=49794e185350f0f9c8099ff84507873685b2b0f1
[1]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=550727af87d5f646617e0c19a3f3300c8117e7f5
[2]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=be20b37180f709ab0e451e4e07b6e82ac3a87b56


Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Alec Warner
On Thu, Jul 9, 2020 at 2:06 PM Chun-Yu Shei  wrote:

> Hmm, that's strange... it seems to have made it to the list archives:
> https://archives.gentoo.org/gentoo-portage-dev/message/a4db905a64e3c1f6d88c4876e8291a65
>
> (but it is entirely possible that I used "git send-email" incorrectly)
>

Ahhh it's visible there; I'll blame gMail ;)

-A


>
> On Thu, Jul 9, 2020 at 2:04 PM Alec Warner  wrote:
>
>>
>>
>> On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei  wrote:
>>
>>> Awesome!  Here's a patch that adds @lru_cache to use_reduce, vercmp, and
>>> catpkgsplit.  use_reduce was split into 2 functions, with the outer one
>>> converting lists/sets to tuples so they can be hashed and creating a
>>> copy of the returned list (since the caller seems to modify it
>>> sometimes).  I tried to select cache sizes that minimized memory use
>>> increase,
>>> while still providing about the same speedup compared to a cache with
>>> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
>>> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
>>> RES column in htop increases from 280 MB -> 290 MB.
>>>
>>> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
>>> max observed RES value actually decreases from 228 MB -> 214 MB (similar
>>> values observed across a few before/after runs).
>>>
>>> Here are the cache hit stats, max observed RES memory, and runtime in
>>> seconds  for various sizes in the update case.  Caching for each
>>> function was tested independently (only 1 function with caching enabled
>>> at a time):
>>>
>>> catpkgsplit:
>>> CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419)
>>> 270 MB
>>> 39.217
>>>
>>> CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1)
>>> 271 MB
>>> 39.112
>>>
>>> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000)
>>> 271 MB
>>> 39.217
>>>
>>> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500)
>>> 269 MB
>>> 39.438
>>>
>>> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000)
>>> 271 MB
>>> 39.348
>>>
>>> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100)
>>> 271 MB
>>> 39.487
>>>
>>>
>>> use_reduce:
>>> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561)
>>> 407 MB
>>> 35.77
>>>
>>> CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1)
>>> 353 MB
>>> 35.52
>>>
>>> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000)
>>> 335 MB
>>> 35.31
>>>
>>> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500)
>>> 318 MB
>>> 35.85
>>>
>>> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000)
>>> 301 MB
>>> 36.39
>>>
>>> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100)
>>> 299 MB
>>> 37.175
>>>
>>>
>>> I didn't bother collecting detailed stats for vercmp, since the
>>> inputs/outputs are quite small and don't cause much memory increase.
>>> Please let me know if there are any other suggestions/improvements (and
>>> thanks Sid for the lru_cache suggestion!).
>>>
>>
>> I don't see a patch attached; can you link to it?
>>
>> -A
>>
>>
>>>
>>> Thanks,
>>> Chun-Yu
>>>
>>>
>>>
>>>


Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Alec Warner
On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei  wrote:

> Awesome!  Here's a patch that adds @lru_cache to use_reduce, vercmp, and
> catpkgsplit.  use_reduce was split into 2 functions, with the outer one
> converting lists/sets to tuples so they can be hashed and creating a
> copy of the returned list (since the caller seems to modify it
> sometimes).  I tried to select cache sizes that minimized memory use
> increase,
> while still providing about the same speedup compared to a cache with
> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
> RES column in htop increases from 280 MB -> 290 MB.
>
> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
> max observed RES value actually decreases from 228 MB -> 214 MB (similar
> values observed across a few before/after runs).
>
> Here are the cache hit stats, max observed RES memory, and runtime in
> seconds  for various sizes in the update case.  Caching for each
> function was tested independently (only 1 function with caching enabled
> at a time):
>
> catpkgsplit:
> CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419)
> 270 MB
> 39.217
>
> CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1)
> 271 MB
> 39.112
>
> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000)
> 271 MB
> 39.217
>
> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500)
> 269 MB
> 39.438
>
> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000)
> 271 MB
> 39.348
>
> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100)
> 271 MB
> 39.487
>
>
> use_reduce:
> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561)
> 407 MB
> 35.77
>
> CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1)
> 353 MB
> 35.52
>
> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000)
> 335 MB
> 35.31
>
> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500)
> 318 MB
> 35.85
>
> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000)
> 301 MB
> 36.39
>
> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100)
> 299 MB
> 37.175
>
>
> I didn't bother collecting detailed stats for vercmp, since the
> inputs/outputs are quite small and don't cause much memory increase.
> Please let me know if there are any other suggestions/improvements (and
> thanks Sid for the lru_cache suggestion!).
>

I don't see a patch attached; can you link to it?

-A


>
> Thanks,
> Chun-Yu
>
>
>
>


Re: [gentoo-dev] Last rites: */* More Py2 only items

2020-06-28 Thread Alec Warner
On Sun, Jun 28, 2020 at 5:35 PM Aaron Bauman  wrote:

> # Aaron Bauman  (2020-06-28)
> # More Py2 only stuff. Plz see -dev ML for discussions
> # Remove bindings, port to Py3, etc
> # Removal in 30 days
> app-arch/deltarpm
> app-crypt/virtualsmartcard
> app-text/duali
> app-text/mftrace
> app-text/queequeg
> app-text/referencer
> dev-libs/libmacaroons
> dev-libs/tut
> dev-python/elib-intl
> dev-python/eunuchs
> dev-python/medusa
> dev-python/misaka
> dev-python/python-iwscan
> dev-util/confix
> dev-util/qmtest
> dev-util/unrpyc
> games-engines/gemrb
> media-sound/lilycomp
> media-video/tovid
> net-dns/maradns
> net-irc/irker
> net-mail/archivemail
> net-mail/getmai
> net-nds/nsscache
>

Infra uses this, we will have to look into bumping it.

-A


> net-wireless/airpwn
> sci-chemistry/bkchem
> sci-chemistry/propka
> sci-chemistry/pymol-plugins-bni-tools
> sci-chemistry/pymol-plugins-emovie
> sci-chemistry/viewmol
> sci-libs/chemkit
> sys-apps/x86info
> www-misc/surl
> x11-wm/plwm
>
> --
> Cheers,
> Aaron
>


Re: [gentoo-dev] */*: Mask Py2 only packages

2020-06-24 Thread Alec Warner
On Wed, Jun 24, 2020 at 11:29 AM Rich Freeman  wrote:

> On Wed, Jun 24, 2020 at 2:18 PM Andreas Sturmlechner 
> wrote:
> >
> > The lack of curiosity for one's own packages' python compatibility is
> not just
> > a py27 isolated issue, it was a big problem with py36 -> py37 with so
> many
> > devs simply not filing that necessary stabilisation.
>
> That suggests that if you keep doing what you're doing, you're going
> to keep hitting your head against the wall.
>
> Right now in Gentoo there isn't really even a straightforward way for
> a maintainer to cleanly obtain a list of all the packages they
> maintain, let alone whether they use python v2.
>

> Sure, you can use the portage API to find this info.  However, that is
> as easy to do for a list of all impacted packages in the tree with
> their maintainers as for any individual maintainer to obtain this info
> for their own packages.
>

You say there is not a straightforward way, but then you say there is an
api? :p


>
> I think that if you give the maintainers a bit more info, you'll find
> them being more proactive about helping you out.  Basically you would
> be helping them help you.
>
> Otherwise you're going to mask a bunch of packages and run into a
> bunch of upset devs, and as a byproduct we create a bunch of upset
> users.
>

Extend the existing QA report?

https://qa-reports.gentoo.org/output/gpyutils/py2.txt

There is a list of py2 only packages. We just need to add the maintainer
metadata?


>
> There is no reason to mask a package only to unmask it a few days
> later.  Masks are a mechanism for deprecating packages so that users
> take action.  They're not a substitute for devs talking to each other.
>

> --
> Rich
>
>


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-06-02 Thread Alec Warner
On Tue, Jun 2, 2020 at 1:44 AM Mathy Vanvoorden  wrote:

> Hi,
>
> Sorry I'm a bit late to the party here but was behind on my emails.
>
> Op wo 27 mei 2020 om 05:25 schreef Alec Warner :
>
>> The TL;DR is that a crack team of infra-folks[0] have been putting
>> together demos of CI services and things like gitlab / gitea / gerrit and
>> so on.
>>
>>
> I didn't see in the email chain any mention of the Atlassian tools. Can
> these be considered? They offer free licenses for open source projects and
> I think that most of the requirements are covered by BitBucket
> (repo-hosting, repo-serving, code review and pull requests) and Bamboo (CI).
>

I inquired about this generally on the nfp list in May:

https://archives.gentoo.org/gentoo-nfp/message/0142bac838ed7cb356e58447c6ee20b0

The conclusion was that the software needed to be released under an open
license (preferably FSF / OSI approved.) So for example, Gitlab CE is OK
(its an OSI approved licensed) but Gitlab EE is not. The source for Gitlab
EE is open (I can go read it) but it's not freely available.


>
> If desired I can setup a demo and/or help maintain the tools, this is my
> day job and for those who care about these kind of things I am Atlassian
> certified.
>
> Here is more info on their licensing for Open Source projects:
>
> https://www.atlassian.com/software/views/open-source-license-request
>

My understanding is that this is a binary-own download that, once
installed, we can request a license to use. We won't have the source code
at all and the license is not free. I don't think it's possible for us to
use this software in Gentoo for this reason.

-A


>
> br,
> Mathy
>


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-31 Thread Alec Warner
On Fri, May 29, 2020 at 11:17 PM Michał Górny  wrote:

> On Fri, 2020-05-29 at 16:34 -0700, Alec Warner wrote:
> > The pull-based mirroring is a bit sad, as it would be nice to auto-update
> > some forks, but it's not a killer feature.
>
> Exactly.  Especially that our push-based mirroring is better,
> and I think that's how we want to populate it.
>

So you want to keep gitolite then?


>
> >  I think our new SSO solution
> > could potentially be a fix for the auth subsystems, but more work there
> > will be needed.
>
> I think SSO should be the primary login to our GitLab, especially for
> our users.  GitHub login is a must.
>

I'm fairly sure both gitlab and gitea support this requirement.


>
> > Another major issue is operating the software. I haven't found anyone to
> > *run* gitlab; I'm not eager to do it. Today Gentoo is mostly distributed,
> > bugs are in bugzilla, wiki is on mediawiki, code is on gitolite with N
> > mirrors, email and lists are separate, etc. In a world where bugs, wiki,
> > code, ci, containers, PRs, are all on gitlab and it breaks and we can't
> fix
> > it; it will be bad news for all of those things. If the bugzilla machine
> > breaks we lose bugzilla; if gitlab breaks we lose the ability to edit the
> > wiki, file bugs, commit, run CI, etc.
> >
>
> But who says we want to migrate them all into GitLab?  I thought our
> primary goal was to replace today's GitHub use, i.e. provide
> an alternative pipeline for pull/merge requests.
>

Oh I don't, but we then need to disable all of them and restrict their
usage.

-A


>
> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-29 Thread Alec Warner
On Wed, May 27, 2020 at 2:31 PM Matt Turner  wrote:

> On Wed, May 27, 2020 at 1:14 AM Alec Warner  wrote:
> > On Tue, May 26, 2020, 23:08 Michał Górny  wrote:
> >>
> >> On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
> >> > The TL;DR is that a crack team of infra-folks[0] have been putting
> together
> >> > demos of CI services and things like gitlab / gitea / gerrit and so
> on.
> >> >
> >> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> >> > review / pull reqs, CI services, and deploy services.) Some of these
> are
> >> > piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
> >> > repo-hosting but CI is separate (e.g. drone.)
> >> >
> >> > On the infra-side, I think we are pretty happy with repo-hosting
> (gitolite)
> >> > and repo-serving (gitweb). We are missing a CI piece and a
> pull-request
> >> > piece. Most of the users using PRs use either a gitlab or github
> mirror.
> >> >
> >> > I think the value of CI is pretty obvious to me (and I see tons of use
> >> > cases in Infra.) We could easily build CI into our current repository
> >> > solution (e.g. gitolite.) However gitolite doesn't really support PRs
> in a
> >> > uniform way and so CI is mostly for submitted code; similar to the
> existing
> >> > ::gentoo repo CI offered by mgorny.
> >> >
> >> > If we build a code review solution (like gitea / gerrit) would people
> use
> >> > it? Would you use it if you couldn't merge (because the code review
> >> > solution can't gpg sign your commits or merges) so a tool like the
> existing
> >> > pram tool would be needed to merge?
> >> >
> >>
> >> Does GitLab count?  Gerrit is just PITA.  I think we had some concerns
> >> about Gitea, so I'd like to test it before deciding.  GitLab OTOH works
> >> just fine for a lot of projects, and seems the next best thing after
> >> GitHub
> >
> >
> > Gitlab does count (we deployed and tested an onprem version.) I think
> there are some major issues with it though.
> >  - Licensing. Gitlab-CE is available, gitlab-EE is not OSS nor OSI
> approved and many of the features we need are EE only and are not available
> in CE.
>
> It's very surprising to me that CE wouldn't work for our purposes.
> Debian, GNOME, KDE, XFCE, and FreeDesktop have all switched to GitLab
> and are using CE. It's hard to believe that Gentoo's usage or
> requirements would be so different as to make GitLab a non-viable
> option.
>
> What features of EE do you think we need?
>

I know debian spent considerable effort customizing their gitlab. I've not
heavily investigated the other deployments. We set up a demo on
gitlab.gentoo.org already as a test and there are some issues. Many of them
are related to operations and not to the app itself.

[mirroring]
 - We can't do pull-based git mirroring (
https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html#pulling-from-a-remote-repository-starter
)
[Auth] Note that we have keycloak now, so we can perhaps workaround these
LDAP issues.
 - GItlab doesn't support multiple redundant LDAP servers
 - Gitlab doesn't support LDAP group syncing
 - Gitlab doesn't support syncing admin users from LDAP
[mgmnt]
 - The gitlab terraform provider is pretty bad; and I struggled to get it
to control our admin users; did not leave me excited about managing other
resources (projects, users, hooks, etc.)

The pull-based mirroring is a bit sad, as it would be nice to auto-update
some forks, but it's not a killer feature. I think our new SSO solution
could potentially be a fix for the auth subsystems, but more work there
will be needed.

Another major issue is operating the software. I haven't found anyone to
*run* gitlab; I'm not eager to do it. Today Gentoo is mostly distributed,
bugs are in bugzilla, wiki is on mediawiki, code is on gitolite with N
mirrors, email and lists are separate, etc. In a world where bugs, wiki,
code, ci, containers, PRs, are all on gitlab and it breaks and we can't fix
it; it will be bad news for all of those things. If the bugzilla machine
breaks we lose bugzilla; if gitlab breaks we lose the ability to edit the
wiki, file bugs, commit, run CI, etc.

-A


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-27 Thread Alec Warner
On Wed, May 27, 2020 at 5:16 AM Thomas Deutschmann 
wrote:

> Hi,
>
> is this really CI _vs_ Code Review? I.e. we can only have one?
>

I'll try to come back to this. We can make computers do anything, but it's
a question of limited time for me and for Gentoo as a whole.


>
> CI is nice and helpful. For example I usually don't start review until CI
> sets green light but CI alone wouldn't be enough. Thought that Gitlab will
> support same kind of hooks similar to how current CI is integrated into
> Github, not?
>

So I'm basically back to worrying about social contract here. If its a
common Gentoo workflow to:
 - Have a user fork a repo (github)
 - Submit a PR (github)
 - Gentoo CI sees the PR against ::gentoo and runs CI (github)
 - CI turns green and comments on the PR (github)
 - Human Gentoo dev reviews PR (github) [some iterations of review and
changes with PR author]
 - Human Gentoo dev downloads PR patch and applies to Gentoo (Gentoo
Hosted.)

Does this workflow meet the social contract? I would assert it does not.
Now of course Gentoo doesn't "solely rely" on this workflow and other
workflows are available; but if this is a de-facto workflow people are
using...does not that concern the community?  It concerns me! Hey we
originally were like 'nah github is OK' and now I'm coming around and
saying 'hey maybe this isn't OK in its current form.' So one idea might be
"what can we do about this particular problem."

What others have done is typically left mirrors of their projects on
github, written bots to auto-close PRs to those repos, and funneled users
somewhere else. We could do that if we wanted.


>
> Also, when I could wish something: The problem when doing review on Github
> for me is, that we usually create new revisions. Therefore we don't see
> what's changed in new revision versus previous revision. So the change to
> review looks like an entire new file was added (which is what basically
> happened). It would be cool if our solution would be aware of this and
> could handle this somehow. But I guess we would have to create our own
> solution for this...
>

So to address your first point above, there are a bunch of (sometimes
conflicting) requirements.

[Github]
A past argument for Github was "we need to be on github because this is
where the users are." I can tell you the users have not left Github and so
without us funneling them somewhere else...I mean if we are going to
continue to accept PRs on github we should probably service them (vs
auto-closing.)

[Code Review]
Basically how do we build a system where a user can go find our code, fork
it, generate a patch, send us the patch, and then be able to run CI on it
and do a review online (as opposed to on bugzilla / in-email.) Currently
this happens on github and I think there is some support for moving those
users to some other solution.

We could:
 - Build something on gitlab CE.
 - Build something on gitea + a CI solution (drone, zuul, etc.)
 - I don't think gerrit credibly meets these requirements because users
can't fork in gerrit and it requires annoying client side tooling.
 - The main problem I think with gitlab-CE is no one wants to operate it;
its too complex of a software package and there is a major fear that once
we mgirate everyone on it, something will go wrong and then we will be in
trouble. I don't have this fear with gitea because there are fewer moving
parts.
 - The main problem with gitea is that it's relatively unproven and we
already know it needs patches for ::gentoo.

[CI-for-ebuilds]
My understanding is that there are two existing CI workflows (I'm ignoring
the new thing nattka; it's what I'd consider a different workflow.)
(1) Github PRs: A bot watches for PRs against ::gentoo on Github and runs
CI on them.
(2) Post-Submit CI: Every so often we run Ci against ::gentoo and if
something looks super bad we bisect and email that the build is broken.

So right off the bat there are some gaps.
 - There is no pre-submit CI on git.gentoo.org at all, so devs can't vet
their changes there. Would anyone like this?
 - There is no flexibility; the CI you get is whatever is configured (e.g.
what mgorny set up). This isn't meant as a dig; maybe no one wants to set
up CI at all and just wants to use this stuff; I have no idea.

[Other Projects]
Portage has had CI for ages; but it runs on github and uses travis-CI
soko.git uses hosted gitlab-ci to build, run tests, and upload containers.
https://gitweb.gentoo.org/proj/catalyst.git/ has no CI, but might like some?
https://gitweb.gentoo.org/proj/gentoo-mirrorstats.git/
 has no CI but
might like some?
https://gitweb.gentoo.org/proj/sandbox.git/ has no CI but might want some?
I could go on.

So to kind of summarize. If I wanted to just do [CI for other projects] we
could likely just build that now and not need to talk to most of the
community because the CI for other projects is fairly segmented. It's the
part that 

Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-27 Thread Alec Warner
On Wed, May 27, 2020 at 1:09 AM Brian Dolbec  wrote:

> On Tue, 26 May 2020 20:24:56 -0700
> Alec Warner  wrote:
>
> > The TL;DR is that a crack team of infra-folks[0] have been putting
> > together demos of CI services and things like gitlab / gitea / gerrit
> > and so on.
> >
> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> > review / pull reqs, CI services, and deploy services.) Some of these
> > are piecemeal (e.g. gerrit has code review, zuul has CI) and gitea
> > offers repo-hosting but CI is separate (e.g. drone.)
> >
> > On the infra-side, I think we are pretty happy with repo-hosting
> > (gitolite) and repo-serving (gitweb). We are missing a CI piece and a
> > pull-request piece. Most of the users using PRs use either a gitlab
> > or github mirror.
> >
> > I think the value of CI is pretty obvious to me (and I see tons of use
> > cases in Infra.) We could easily build CI into our current repository
> > solution (e.g. gitolite.) However gitolite doesn't really support PRs
> > in a uniform way and so CI is mostly for submitted code; similar to
> > the existing ::gentoo repo CI offered by mgorny.
> >
> > If we build a code review solution (like gitea / gerrit) would people
> > use it? Would you use it if you couldn't merge (because the code
> > review solution can't gpg sign your commits or merges) so a tool like
> > the existing pram tool would be needed to merge?
> >
> > -A
> >
> > [0] Mostly arzano, if I'm honest. I am just the point-y haired
> > manager in this effort.
>
> There are several CI systems that could be used.  As for CI, my
> experience has been with buildbot.  It would be fairly straightforward
> to integrate with the current gitolite/gitweb hosting.
> It is also extremely flexible in its capabilities, although can be
> difficult to master.  It has a good web interface, but it can even be
> run via it's API's using curses, python, bash,   There is a
> buildbot-travis plugin which is capable of running existing .travis file
> configurations.  It also extends its capabilities to include custom
> buildbot step definitions.  I have not packaged it yet for gentoo, but
> it is on my todo list. One of buildbot's capabilities is the ability to
> run tests/builds on multiple workers (different arches or whatever)
> both in parallel or series.  It could be made to integrate with our
> bugzilla using the python client (pybugs was it?).
>

My understanding is that CI systems are all quite similar. We have
configured gitlab-ci, drone, and zuul as tests and I am familiar with
travis-ci and buildbot from other projects. I'm less concerned about this
aspect tbh. I'm also not super concerned about packaging. E.g. for
gitlab-runner (the worker portion of gitlab-ci) we just deploy upstream
containers and don't package them in ::gentoo at all. Our onprem gitlab is
a container solution; gitea is a container solution; our SSO IDP is a
container solution; gerrit is currently a container solution. These don't
bother me (too much, ugh zuul uses zookeeper? ugh.)

The major differentiators for CI appear to be:
 - SCM support (we currently only really care about git compatible systems,
but some contenders only support some providers.)
 - builders / workers (how do they deploy). For example gitlab has a
container based work executor while zuul uses ansible; I suspect
 - Authentication (ideally should support SAML or openID so we can
integrate with our alpha sso solution for Gentoo.)
 - The webUI; e.g is the solution horrible to use / interact with. Hard to
say without a trial, IMHO.

-A


>
> But that still leaves a PR/code review option.
>
>


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-27 Thread Alec Warner
On Tue, May 26, 2020, 23:08 Michał Górny  wrote:

> On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
> > The TL;DR is that a crack team of infra-folks[0] have been putting
> together
> > demos of CI services and things like gitlab / gitea / gerrit and so on.
> >
> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> > review / pull reqs, CI services, and deploy services.) Some of these are
> > piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
> > repo-hosting but CI is separate (e.g. drone.)
> >
> > On the infra-side, I think we are pretty happy with repo-hosting
> (gitolite)
> > and repo-serving (gitweb). We are missing a CI piece and a pull-request
> > piece. Most of the users using PRs use either a gitlab or github mirror.
> >
> > I think the value of CI is pretty obvious to me (and I see tons of use
> > cases in Infra.) We could easily build CI into our current repository
> > solution (e.g. gitolite.) However gitolite doesn't really support PRs in
> a
> > uniform way and so CI is mostly for submitted code; similar to the
> existing
> > ::gentoo repo CI offered by mgorny.
> >
> > If we build a code review solution (like gitea / gerrit) would people use
> > it? Would you use it if you couldn't merge (because the code review
> > solution can't gpg sign your commits or merges) so a tool like the
> existing
> > pram tool would be needed to merge?
> >
>
> Does GitLab count?  Gerrit is just PITA.  I think we had some concerns
> about Gitea, so I'd like to test it before deciding.  GitLab OTOH works
> just fine for a lot of projects, and seems the next best thing after
> GitHub


Gitlab does count (we deployed and tested an onprem version.) I think there
are some major issues with it though.
 - Licensing. Gitlab-CE is available, gitlab-EE is not OSS nor OSI approved
and many of the features we need are EE only and are not available in CE.
 - Complex: Gitlab is a giant piece of software with maybe 8-12 components
(unicorn, postgres, redis, memcache, sidekiq, puma, workhouse, gitaly,
grafana, sshd,nginx, prometheus ..the list goes on)
 - I think gitlab ships with more features than we will use (CD, docker
registry, issues / bugs, wiki, analytics, snippets, milestones, repo
hosting, repo browsing, ... Again the list goes on.) I don't play to
migrate away from bugs.gentoo.org nor wiki.gentoo.org, nor gitolite. I
think if we did; then gitlab would be a more compelling option because it
is a one-stop-shop solution for those use cases.

My understanding of gitea is that it works great for not-::gentoo, but
::gentoo and gitea don't work well and it would require work upstream to
fix; other large repos seemed to work OK in gitea (based on our test
deployment and conversations with gitea upstream.)

Gerrit is widely used for large projects and I'm not worried for ::gentoo
and we have deployed gerrit and it seems to work fine. Gerrit doesn't have
CI (we would need to deploy something) and it uses gitweb for repository
browsing (which we use today.)

-A


>
> --
> Best regards,
> Michał Górny
>
>


[gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-26 Thread Alec Warner
The TL;DR is that a crack team of infra-folks[0] have been putting together
demos of CI services and things like gitlab / gitea / gerrit and so on.

Some of these come in combined (e.g. gitlab offers repo hosting, code
review / pull reqs, CI services, and deploy services.) Some of these are
piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
repo-hosting but CI is separate (e.g. drone.)

On the infra-side, I think we are pretty happy with repo-hosting (gitolite)
and repo-serving (gitweb). We are missing a CI piece and a pull-request
piece. Most of the users using PRs use either a gitlab or github mirror.

I think the value of CI is pretty obvious to me (and I see tons of use
cases in Infra.) We could easily build CI into our current repository
solution (e.g. gitolite.) However gitolite doesn't really support PRs in a
uniform way and so CI is mostly for submitted code; similar to the existing
::gentoo repo CI offered by mgorny.

If we build a code review solution (like gitea / gerrit) would people use
it? Would you use it if you couldn't merge (because the code review
solution can't gpg sign your commits or merges) so a tool like the existing
pram tool would be needed to merge?

-A

[0] Mostly arzano, if I'm honest. I am just the point-y haired manager in
this effort.


Re: [gentoo-portage-dev] [PATCH] config.environ: delay export of A and AA (bug 720180)

2020-05-26 Thread Alec Warner
On Tue, May 26, 2020 at 9:46 AM Zac Medico  wrote:

> On 5/26/20 1:43 AM, Alec Warner wrote:
> > On Mon, May 25, 2020 at 9:34 PM Zac Medico  > <mailto:zmed...@gentoo.org>> wrote:
> >
> > Since variables like A and AA can contain extremely large values
> which
> > may trigger E2BIG errors during attempts to execute subprocesses,
> delay
> > export until the last moment, and unexport when appropriate.
> >
> >
> > So I think if you want to do this because PMS says:
> >  AA should not be visible in EAPI > 3.
> >  A should only be visible in src_*, pkg_nofetch.
> >
> > That part of the patch makes sense to me. The part that is confusing to
> > me is the 'delay' part; can you explain that further? When you say
> > "delay until the last moment" what do you mean by that and what value is
> > it delivering?
>
> If we export an environment variable which contains an extremely large
> value, then there's a vulnerability in execve which causes it to fail
> with an E2BIG error. Since A and AA values can easily grow large enough
> to trigger this vulnerability, portage can protect itself from execve
> failures by delaying the export until the moment that it hands control
> to the ebuild phase.
>

> > Is it simply that we don't export these variables on the python side,
> > and we only use them in the shell portion?
>
> That's correct. Here's a test case which demonstrates the E2BIG error,
> and shows that 'export -n A' can suppress it:
>

I've run similar tests, but I'm less excited by this work around. I think
its reasonable to work toward eventually not exporting A and AA (the latter
already done in EAPIs > 3). Then when ebuilds encounter problems, we can
tell authors "Upgrade to EAPI X" (where X is >3 or >=8; in the potential
case of A.) Having a hard limit on A for EAPIs 0-7 and a hard limit on AA
for EAPIs 0-3 seems perfectly fine and we should expect authors to refactor
(as texlive did) in order to meet the existing API limitations.

Basically unless there are more practical use cases today, the delaying of
export seems like a feature no one will use and added complexity. I dunno
how large the go-mod use cases are yet; but I myself have not seen any with
this many deps.

-A


>
> $ A=$(dd if=/dev/zero bs=1M count=1 | tr '\0' ' ')
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB, 10 MiB) copied, 0.086557 s, 121 MB/s
> $ echo ${#A}
> 10485760
> $ export A
> $ ls
> bash: /bin/ls: Argument list too long
> $ export -n A
> $ /bin/echo hello world
> hello world


> --
> Thanks,
> Zac
>
>


Re: [gentoo-portage-dev] [PATCH] config.environ: delay export of A and AA (bug 720180)

2020-05-26 Thread Alec Warner
On Mon, May 25, 2020 at 9:34 PM Zac Medico  wrote:

> Since variables like A and AA can contain extremely large values which
> may trigger E2BIG errors during attempts to execute subprocesses, delay
> export until the last moment, and unexport when appropriate.
>

So I think if you want to do this because PMS says:
 AA should not be visible in EAPI > 3.
 A should only be visible in src_*, pkg_nofetch.

That part of the patch makes sense to me. The part that is confusing to me
is the 'delay' part; can you explain that further? When you say "delay
until the last moment" what do you mean by that and what value is it
delivering?

Is it simply that we don't export these variables on the python side, and
we only use them in the shell portion?

-A


> Bug: https://bugs.gentoo.org/720180
> Signed-off-by: Zac Medico 
> ---
>  bin/eapi.sh   |  9 +
>  bin/isolated-functions.sh | 34 
>  bin/phase-functions.sh| 13 +++
>  .../ebuild/_config/special_env_vars.py|  7 +++-
>  lib/portage/package/ebuild/config.py  | 39 ++-
>  5 files changed, 91 insertions(+), 11 deletions(-)
>
> diff --git a/bin/eapi.sh b/bin/eapi.sh
> index 29dfb008c..f56468e4a 100644
> --- a/bin/eapi.sh
> +++ b/bin/eapi.sh
> @@ -26,6 +26,15 @@ ___eapi_has_S_WORKDIR_fallback() {
>
>  # VARIABLES
>
> +___eapi_exports_A() {
> +   # https://bugs.gentoo.org/721088
> +   true
> +}
> +
> +___eapi_exports_AA() {
> +   [[ ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
> +}
> +
>  ___eapi_has_prefix_variables() {
> [[ ! ${1-${EAPI-0}} =~ ^(0|1|2)$ || " ${FEATURES} " == *"
> force-prefix "* ]]
>  }
> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> index fde684013..973450d86 100644
> --- a/bin/isolated-functions.sh
> +++ b/bin/isolated-functions.sh
> @@ -107,6 +107,39 @@ __bashpid() {
> sh -c 'echo ${PPID}'
>  }
>
> +# @FUNCTION: ___eapi_vars_export
> +# @DESCRIPTION:
> +# Export variables for the current EAPI. Calls to this function should
> +# be delayed until the last moment, since exporting these variables
> +# may trigger E2BIG errors suring attempts to execute subprocesses.
> +___eapi_vars_export() {
> +   source "${T}/environment.unexported" || die
> +
> +   if ___eapi_exports_A; then
> +   export A
> +   fi
> +
> +   if ___eapi_exports_AA; then
> +   export AA
> +   fi
> +}
> +
> +# @FUNCTION: ___eapi_vars_unexport
> +# @DESCRIPTION:
> +# Unexport variables that were exported for the current EAPI. This
> +# function should be called after an ebuild phase, in order to unexport
> +# variables that may trigger E2BIG errors during attempts to execute
> +# subprocesses.
> +___eapi_vars_unexport() {
> +   if ___eapi_exports_A; then
> +   export -n A
> +   fi
> +
> +   if ___eapi_exports_AA; then
> +   export -n AA
> +   fi
> +}
> +
>  __helpers_die() {
> if ___eapi_helpers_can_die && [[ ${PORTAGE_NONFATAL} != 1 ]]; then
> die "$@"
> @@ -122,6 +155,7 @@ die() {
>
> set +x # tracing only produces useless noise here
> local IFS=$' \t\n'
> +   ___eapi_vars_unexport
>
> if ___eapi_die_can_respect_nonfatal && [[ $1 == -n ]]; then
> shift
> diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
> index 90e622e75..df2c0d8de 100644
> --- a/bin/phase-functions.sh
> +++ b/bin/phase-functions.sh
> @@ -146,6 +146,7 @@ __filter_readonly_variables() {
> fi
> fi
>
> +   ___eapi_vars_unexport
> "${PORTAGE_PYTHON:-/usr/bin/python}"
> "${PORTAGE_BIN_PATH}"/filter-bash-environment.py "${filtered_vars}" || die
> "filter-bash-environment.py failed"
>  }
>
> @@ -212,6 +213,7 @@ __ebuild_phase() {
>
>  __ebuild_phase_with_hooks() {
> local x phase_name=${1}
> +   ___eapi_vars_export
> for x in {pre_,,post_}${phase_name} ; do
> __ebuild_phase ${x}
> done
> @@ -223,6 +225,7 @@ __dyn_pretend() {
> __vecho ">>> Remove '$PORTAGE_BUILDDIR/.pretended' to
> force pretend."
> return 0
> fi
> +   ___eapi_vars_export
> __ebuild_phase pre_pkg_pretend
> __ebuild_phase pkg_pretend
> >> "$PORTAGE_BUILDDIR/.pretended" || \
> @@ -236,6 +239,7 @@ __dyn_setup() {
> __vecho ">>> Remove '$PORTAGE_BUILDDIR/.setuped' to force
> setup."
> return 0
> fi
> +   ___eapi_vars_export
> __ebuild_phase pre_pkg_setup
> __ebuild_phase pkg_setup
> >> "$PORTAGE_BUILDDIR/.setuped" || \
> @@ -252,6 +256,7 @@ __dyn_unpack() {
> install -m${PORTAGE_WORKDIR_MODE:-0700} -d "${WORKDIR}" ||
> die "Failed to create dir '${WORKDIR}'"
> fi
> cd "${WORKDIR}" || die "Directory change failed: \`cd
> '${WORKDIR}'\`"
> +   ___eapi_vars_export
> __ebuild_phase 

Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-21 Thread Alec Warner
On Tue, May 19, 2020 at 5:46 AM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> On 5/19/20 7:47 AM, Michał Górny wrote:
> > Do you have any specific solution in mind?
> >
> > [1] https://gitweb.gentoo.org/archive/proj/identity.gentoo.org.git/
>
> I would suggest for SSO an implementation like the following with LDAP
> provider:
>
> https://github.com/Luzifer/nginx-sso/wiki/Auth-Provider-Configuration


Thanks for pointing this out, we might use it for legacy apps that don't
have solid saml / openid integration.
I want to link it against keycloak though because LDAP doesn't support
newer auth standards like u2f.

-A


Re: [gentoo-dev] [PSA] If you ssh interactively to git.gentoo.org (somehow) let me know.

2020-05-21 Thread Alec Warner
A bit late, but this change is now live. Please contact me if anything has
broken.

-A

On Mon, Apr 27, 2020 at 10:34 AM Alec Warner  wrote:

> On Mon, Apr 27, 2020 at 7:04 AM Kent Fredric  wrote:
>
>> On Mon, 27 Apr 2020 09:43:44 -0400
>> Mike Gilbert  wrote:
>>
>> > He was replying to me. Your master connection will continue to work
>> > just fine, as I said in my previous message.
>>
>> I must have lost something in grammar, because no matter how many times I
>> read:
>>
>> > If you are authenticating that master connection as the "git" user, I
>> > suspect it will not affect you. If you are using it to push to
>> > gentoo.git, that is almost certainly the case.
>>
>> I interpret that as:
>>
>> - Anonymous fetch is fine
>> - Authorised Push will fail
>>
>
> "If you are authenticating the master connection as the 'git' user then
> this change will not affect you.
> "If you are using controlmaster to push to git.gentoo.org, then you are
> definitely authenticating as user=git because there is no other way to
> commit to ::gentoo."
>
>
>>
>> But I guess my mistake is in that we don't push with "user@git ...", we
>> push with "git@ ... ", and the SSH key is the gate keeper of "push will
>> work", not the UID.
>>
>> Right?
>>
>
> A working ssh key for user=git is a necessary (but not sufficient)
> component of a successful git push.
>
>
>>
>> So assuming you're using git@ for fetch *and* push, *then* it will
>> continue to work.
>>
>> Right?
>>
>
> Correct.
>
>
>>
>> Forgive me for any potential idiocy, language and remembering the
>> details of everything all the time is hard.
>>
>
> I don't actually expect people to know these details.
>


Re: [gentoo-dev] [RFC] Anti-spam for goose

2020-05-21 Thread Alec Warner
On Thu, May 21, 2020 at 1:13 PM Viktar Patotski 
wrote:

> Hi all,
>
> I believe that we are all have forgotten about Donald Knuth: Premature
> optimisation is the root of all evill.
>
> We don't have "spam" yet, but we are already trying to protect. There
> might be cases when some systems will be posting stats more often than we
> want, but probably that will not harm us. Or this will be done by our main
> users who runs 1kk of gentoo installations and this "spam"  will be
> actually valuable. Moreover, nobody forces us to treat info from 'goose' as
> first priority, so we are still able to select on which packages to work.
> In short: this topic is not so important yet, I think.
>

I raised a similar question on irc and the conclusion was that 'it is good
to have ideas' and I don't necessarily disagree there[0]. We cannot build a
foolproof system but some are feasible in some scenarios[1].

[0] Gentoo offers numerous no-login-required services; most of these are
read-only but they typically don't suffer from attacks; or at least, not
attacks that we need to respond to. The most obvious one of these is our
gentoo.org mail service which accepts unauthenticated email to gentoo.org.
Our anti-email-spam countermeasures are what I would call complex, but we
still employ broad measures when needed and the tradeoffs are similar to
the options for goose; e.g. if we are too broad we can block email from
large swaths of the internet.
[1] Bugzilla *has* recently been the target of spam attacks, it *has*
logins required (e.g. to create / modify bugs) and it has not stopped the
spammers from creating accounts. We have discussed different protections
for bugzilla, as it has different parameters. A basic bugzilla account
can't do all that much (you can't modify the bugs of others easily) and
spam posts are easily identified. This is to differentiate from goose where
the powers of each token are the same (submit report) and it may be
difficult to tell an abusive report from a real report.


> Viktar
>
>
> On Thu, May 21, 2020, 16:28 Jaco Kroon  wrote:
>
>> Hi Michał,
>>
>> On 2020/05/21 13:02, Michał Górny wrote:
>> > On Thu, 2020-05-21 at 12:45 +0200, Jaco Kroon wrote:
>> >> Even for v4, as an attacker ... well, as I'm sitting here right now
>> I've
>> >> got direct access to almost a /20 (4096 addresses).  I know a number of
>> >> people with larger scopes than that.  Use bot-nets and the scope goes
>> up
>> >> even more.
>> > See how unfair the world is!  You are filling your bathtub with IP
>> > addresses, and my ISP has taken mine only recently.
>> I must admit, I work for an ISP :$
>> >>> Option 3: explicit CAPTCHA
>> >>> ==
>> >>> A traditional way of dealing with spam -- require every new system
>> >>> identifier to be confirmed by solving a CAPTCHA (or a few
>> identifiers
>> >>> for one CAPTCHA).
>> >>>
>> >>> The advantage of this method is that it requires a real human work
>> >>> to be
>> >>> performed, effectively limiting the ability to submit spam.
>> >>>
>> >> Yea.  One would think.  CAPTCHAs are massively intrusive and in my
>> >> opinion more effort than they're worth.
>> >>
>> >> This may be beneficial to *generate* a token.  In other words - when
>> >> generating a token, that token needs to be registered by way of
>> capthca.
>> >>
>> >>> Other ideas
>> >>> ===
>> >>> Do you have any other ideas on how we could resolve this?
>> >>>
>> >> Generated token + hardware based hash.
>> > How are you going to verify that the hardware-based hash is real,
>> > and not just a random value created to circumvent the protection?
>>
>> So the generation of the hash is more to validate that it's still on the
>> same installation (ie, not a cloned token).  Sorry if that wasn't clear,
>> so trying to solve two possible problems in one go.
>>
>> >
>> >>   Rate limit the combination to 1/day.
>> >>
>> >> Don't use included results until it's been kept up to date for a
>> minimum
>> >> period.  Say updated at least 20 times 30 days.
>> > For privacy reasons, we don't correlate the results.  So this is
>> > impossible to implement.
>>
>> Ok, but a token cannot (unless we issue it based on an email based
>> account) be linked back to a specific user, so does it matter if we
>> associate uploads with a token?
>>
>> >> The downside here is that many machines are not powered up at least
>> once
>> >> a day to be able to perform that initial submission sequence.  So
>> >> perhaps it's a bit stringent.
>> > Exactly.  Even once a week is a bit risky but once a day is too narrow
>> > a period.
>> >
>> > To some degree, we could decide we don't care about exact numbers
>> > as much as some degree of weighed proportions.  This would mean that,
>> > say, people who submit daily get the count of 7, at the loss of people
>> > who don't run their machines that much.  It would effectively put more
>> > emphasis on more active users.  It's debatable whether 

Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-20 Thread Alec Warner
On Wed, May 20, 2020 at 12:26 AM Michał Górny  wrote:

> On Wed, 2020-05-20 at 00:21 -0700, Alec Warner wrote:
> > On Tue, May 19, 2020 at 1:23 AM Lars Wendler 
> > wrote:
> >
> > > Hi Alec,
> > >
> > > On Mon, 18 May 2020 18:42:24 -0700 Alec Warner wrote:
> > >
> > > > TL;DR: What if we launched id.gentoo.org, an identity provider that
> > > > provides authentication for Gentoo properties? Basically, 1 username
> /
> > > > password for wiki, bugs, email, forums, and any other http
> > > > service[0][1].
> > > >
> > > > Today Gentoo has numerous systems that mostly work in a segmented
> way.
> > > >
> > > > - To connect to hosts, we use ssh keys.
> > > > - Git is authenticated via ssh keys.
> > > > - Email uses LDAP passwords.
> > > > - Bugzilla has its own identities, with their own passwords.
> > > > - Wiki is separate, with its own passwords.
> > > > - Forums are separate.
> > > > - Infra has an additional 4 systems that use separate credentials.
> > > >
> > > > Some applications support 2FA (such as wiki.)
> > > > Some applications do not support 2FA.
> > > > Applications that require 2FA have a configuration for each app, so
> you
> > > > have N configurations.
> > > >
> > > > If we configured id.gentoo.org you would have 1 identity across all
> > > > gentoo properties.
> > > >
> > > > Is this a thing people are interested in?
> > > >
> > > > [0] It's unlikely operations for git via ssh would change in this
> > > > rollout. [1] Its unclear if the scope is "gentoo developers" or "any
> > > > community member." The former have LDAP accounts and @gentoo.org
> email
> > > > addresses and so we can manage them easily; managing 1000s of other
> > > > accounts in the IDP remains to be seem.
> > >
> > > In case 2FA won't be mandatory I find this a good idea.
> > >
> >
> > 2FA is definitely a reason to deploy software like keycloak, but in the
> > first rollout I don't expect to enforce 2FA. Ideally we would deploy the
> > U2F support in keycloak and then, similar to our earlier program, offer
> > discounted or free u2f devices for Gentoo developers; this would likely
> be
> > on a 1-2 year timeframe.
> >
> > Is there some reason you don't want to use 2FA?
> >
>
> I myself would find 2FA bothersome for low importance services.  Whether
> it's U2F or OTP, I would generally find it silly to have to carry
> the hardware/software on me all the time or even use it when it's laying
> right next to me, say, just to approve a comment on a blog.
>
> But I guess if we go for SSO, it becomes a necessity to better protect
> our passwords.
>

I think each application, when it ends up integrating with keycloak, gets
to decide what security level the application wants; I think this leads to
flexibility for low-importance stuff. E.g. we may not need OTP for blogs,
or wiki. Obvious cases are apps like our AWS credentials (where theft means
financial harm for Gentoo) or the sso.gentoo.org itself (because you
probably want to require OTP to change your password, for example.)

The other common thing I've seen is some kind of longer-lived renewable
token that requires an OTP to get, but does not require an OTP to renew.
These are commonly things like "API keys" or other such credentials that
are scopeable (unlike a password) and revocable (e.g. you can go to
sso.gentoo.org and revoke your token.) This seems more common on mobile
where there is a 'setup' flow and maybe you do it once (at setup), or once
a month, or whatnot. This would mean you don't have to OTP all the time.

-A


> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-20 Thread Alec Warner
On Tue, May 19, 2020 at 1:23 AM Lars Wendler 
wrote:

> Hi Alec,
>
> On Mon, 18 May 2020 18:42:24 -0700 Alec Warner wrote:
>
> >TL;DR: What if we launched id.gentoo.org, an identity provider that
> >provides authentication for Gentoo properties? Basically, 1 username /
> >password for wiki, bugs, email, forums, and any other http
> >service[0][1].
> >
> >Today Gentoo has numerous systems that mostly work in a segmented way.
> >
> > - To connect to hosts, we use ssh keys.
> > - Git is authenticated via ssh keys.
> > - Email uses LDAP passwords.
> > - Bugzilla has its own identities, with their own passwords.
> > - Wiki is separate, with its own passwords.
> > - Forums are separate.
> > - Infra has an additional 4 systems that use separate credentials.
> >
> >Some applications support 2FA (such as wiki.)
> >Some applications do not support 2FA.
> >Applications that require 2FA have a configuration for each app, so you
> >have N configurations.
> >
> >If we configured id.gentoo.org you would have 1 identity across all
> >gentoo properties.
> >
> >Is this a thing people are interested in?
> >
> >[0] It's unlikely operations for git via ssh would change in this
> >rollout. [1] Its unclear if the scope is "gentoo developers" or "any
> >community member." The former have LDAP accounts and @gentoo.org email
> >addresses and so we can manage them easily; managing 1000s of other
> >accounts in the IDP remains to be seem.
>
> In case 2FA won't be mandatory I find this a good idea.
>

2FA is definitely a reason to deploy software like keycloak, but in the
first rollout I don't expect to enforce 2FA. Ideally we would deploy the
U2F support in keycloak and then, similar to our earlier program, offer
discounted or free u2f devices for Gentoo developers; this would likely be
on a 1-2 year timeframe.

Is there some reason you don't want to use 2FA?

-A


>
> Kind regards
> --
> Lars Wendler
> Gentoo package maintainer
> GPG: 21CC CF02 4586 0A07 ED93  9F68 498F E765 960E 9B39
>


Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-20 Thread Alec Warner
On Mon, May 18, 2020 at 11:47 PM Michał Górny  wrote:

> On Mon, 2020-05-18 at 18:42 -0700, Alec Warner wrote:
> > TL;DR: What if we launched id.gentoo.org, an identity provider that
> > provides authentication for Gentoo properties? Basically, 1 username /
> > password for wiki, bugs, email, forums, and any other http service[0][1].
> >
> > Today Gentoo has numerous systems that mostly work in a segmented way.
> >
> >  - To connect to hosts, we use ssh keys.
> >  - Git is authenticated via ssh keys.
> >  - Email uses LDAP passwords.
> >  - Bugzilla has its own identities, with their own passwords.
> >  - Wiki is separate, with its own passwords.
> >  - Forums are separate.
> >  - Infra has an additional 4 systems that use separate credentials.
> >
> > Some applications support 2FA (such as wiki.)
> > Some applications do not support 2FA.
> > Applications that require 2FA have a configuration for each app, so you
> > have N configurations.
> >
> > If we configured id.gentoo.org you would have 1 identity across all
> gentoo
> > properties.
> >
> > Is this a thing people are interested in?
> >
>
> What a coincidence I've just archived our old identity.gentoo.org [1]
> project.  And yes, we almost had this back in 2013 but Infra failed to
> deploy, and it was claimed obsolete by the time I joined Infra.
>
> Do you have any specific solution in mind?
>

Currently we have a standalone keycloak install with LDAP user federation.
We are looking to do a domain installation for redundancy purposes.
Our existing LDAP infrastructure for example (which few services use for
Auth) has at least 3 replicas.

-A


>
> [1] https://gitweb.gentoo.org/archive/proj/identity.gentoo.org.git/
>
>
> --
> Best regards,
> Michał Górny
>
>


[gentoo-dev] RFC: Gentoo Identity Provider

2020-05-18 Thread Alec Warner
TL;DR: What if we launched id.gentoo.org, an identity provider that
provides authentication for Gentoo properties? Basically, 1 username /
password for wiki, bugs, email, forums, and any other http service[0][1].

Today Gentoo has numerous systems that mostly work in a segmented way.

 - To connect to hosts, we use ssh keys.
 - Git is authenticated via ssh keys.
 - Email uses LDAP passwords.
 - Bugzilla has its own identities, with their own passwords.
 - Wiki is separate, with its own passwords.
 - Forums are separate.
 - Infra has an additional 4 systems that use separate credentials.

Some applications support 2FA (such as wiki.)
Some applications do not support 2FA.
Applications that require 2FA have a configuration for each app, so you
have N configurations.

If we configured id.gentoo.org you would have 1 identity across all gentoo
properties.

Is this a thing people are interested in?

[0] It's unlikely operations for git via ssh would change in this rollout.
[1] Its unclear if the scope is "gentoo developers" or "any community
member." The former have LDAP accounts and @gentoo.org email addresses and
so we can manage them easily; managing 1000s of other accounts in the IDP
remains to be seem.


Re: [gentoo-dev] [RFC] Ideas for gentoostats implementation

2020-05-05 Thread Alec Warner
On Mon, May 4, 2020 at 10:14 PM Matt Turner  wrote:

> On Mon, May 4, 2020 at 5:48 PM Thomas Deutschmann 
> wrote:
> >
> > On 2020-04-26 15:46, Kent Fredric wrote:
> > > On Sun, 26 Apr 2020 14:38:54 +0200
> > > Thomas Deutschmann  wrote:
> > >
> > >> Let's assume we will get reports that app-misc/foo is only installed
> 20
> > >> times. If you are going to judge based on this data, "Obviously,
> nobody
> > >> is using that package, it's stuck on ... safe to remove"
> your
> > >> view is biased:
> > >
> > > I see this as more like what bloom filters get you, but in reverse:
> > >
> > > [...]
> > >
> > > - But now, instead of having "we don't know if anybody uses this", you
> > >   *can* have a "we know for sure somebody uses this".
> >
> > But how does that information really help us to decide anything in the
> end?
> >
> > Case A, stats are showing 0 users:
> >
> > Like said, we can't know if this is true or if this package is only used
> > in setups where people don't report stats.
> >
> >
> > Case B, stats are showing x users:
> >
> > Now what? Package from case A could have similar users -- we just don't
> > know. Assume firefox has 1.000 users, chromium has 500 users and vivaldi
> > doesn't show up in stats. How does that help us? Would this allow us to
> > skip publishing GLSAs for vivalid because we assume nobody in Gentoo is
> > using vivaldi? Does it allow Python project to go forward pushing a mask
> > for removal in case vivaldi would depend on Python version, Python
> > project want to get rid of? Would this allow Gentoo PR to make a public
> > statement like "Firefox is the most popular browser in Gentoo, twice as
> > users as chromium"?
>
> I hate the saying "the perfect is the enemy of the good" but I think
> it applies here.
>
> You're of course correct that we would not have perfect information.
> But the thing about statistics is that you can still know some things
> based on a sampling of that perfect information.
>
> I would personally like to have data on whether users of my packages
> have certain USE flags enabled. Knowing that would allow me to decide
> whether its worth the maintenance burden of supporting features that I
> *think* are very rarely used. If instead the data showed me that 50%
> of users had IUSE=xyz enabled, I probably wouldn't consider removing
> it.
>
> I think your example of potential misuse of data is a bit over dramatic.
>

Let me present the same point another way.

Today we have no data, so we make an arbitrary decision. It might be right
or wrong; and we may not know until after we decide.
This is traditionally things like "break them and they will come" type of
process. "Mask it, if they complain, I'll unmask it."

In the future, we could have this package data. It may influence decision
making. However I'm not sure from a decision-making standpoint that it is
strictly worse than no data.
The danger (which is what I think Whissi's concern is) is that it could
artificially increase decision certainty.

For example, if I have to decide whether to keep a package, or a flag, or
whatever. I might make an arbitrary decision. I'm aware it's arbitrary, it
might be wrong, and so I'm not super attached to such a decision. I'm not
*certain* about it; but I have to decide one way or the other[0]. Then I
move to a world with package data. Now I'm no longer making an arbitrary
decision; I'm making a decision based on *data*. The *data* tells me my
decision is correct, resulting in a more *certain* decision outcome. I
think this is the fallacy we want to avoid. The data can be informative but
there are significant biases in it that should result in very *little*
certainty added to decision making.

Making decisions based on incomplete data is just life though, so I'm
fairly skeptical of a "we shouldn't collect any data" type of mindset. I'd
be curious to see if we can instill a *culture* component around the use of
data in our development workflows.

-A

[0] There are a bunch of other cultural components here, like different
decision types (1 vs 2) and the ability to make a mistake in public and not
feel bad about it; so I'm aware reality does not reflect this trivial
example. But those are hallmarks of cultural markets I'd like to aim for in
Gentoo, so I would prefer to discuss a world where they exist ;)


Re: [gentoo-dev] [PSA] If you ssh interactively to git.gentoo.org (somehow) let me know.

2020-04-27 Thread Alec Warner
On Mon, Apr 27, 2020 at 7:04 AM Kent Fredric  wrote:

> On Mon, 27 Apr 2020 09:43:44 -0400
> Mike Gilbert  wrote:
>
> > He was replying to me. Your master connection will continue to work
> > just fine, as I said in my previous message.
>
> I must have lost something in grammar, because no matter how many times I
> read:
>
> > If you are authenticating that master connection as the "git" user, I
> > suspect it will not affect you. If you are using it to push to
> > gentoo.git, that is almost certainly the case.
>
> I interpret that as:
>
> - Anonymous fetch is fine
> - Authorised Push will fail
>

"If you are authenticating the master connection as the 'git' user then
this change will not affect you.
"If you are using controlmaster to push to git.gentoo.org, then you are
definitely authenticating as user=git because there is no other way to
commit to ::gentoo."


>
> But I guess my mistake is in that we don't push with "user@git ...", we
> push with "git@ ... ", and the SSH key is the gate keeper of "push will
> work", not the UID.
>
> Right?
>

A working ssh key for user=git is a necessary (but not sufficient)
component of a successful git push.


>
> So assuming you're using git@ for fetch *and* push, *then* it will
> continue to work.
>
> Right?
>

Correct.


>
> Forgive me for any potential idiocy, language and remembering the
> details of everything all the time is hard.
>

I don't actually expect people to know these details.


Re: [gentoo-dev] [PSA] If you ssh interactively to git.gentoo.org (somehow) let me know.

2020-04-26 Thread Alec Warner
On Sun, Apr 26, 2020 at 11:22 AM Mike Gilbert  wrote:

> On Sun, Apr 26, 2020 at 8:38 AM Kent Fredric  wrote:
> >
> > On Sat, 25 Apr 2020 14:12:02 -0700
> > Alec Warner  wrote:
> >
> > > Thus I now plan to remove this access[0]. If you need access to ssh as
> > > something not-git to git.gentoo.org, let me know in the next week.
> >
> > Will this affect people who set up no-op SSH connections for a
> > persistent connection master, so that following git actions don't have
> > to pay the SSH connection startup penalty?
>
> If you are authenticating that master connection as the "git" user, I
> suspect it will not affect you. If you are using it to push to
> gentoo.git, that is almost certainly the case.
>
>
This is correct.

-A


Re: [gentoo-portage-dev] Need backup mentor for FUSE-based sandbox project

2020-04-25 Thread Alec Warner
On Thu, Apr 23, 2020 at 12:09 PM Michał Górny  wrote:

> Hi, everyone.
>
> It seems that we *urgently* (read: in 6 days) need to find backup
> mentors for this year's GSoC projects.  I'm mentoring the project to
> develop a FUSE-based sandbox alternative that's going to work reliably
> with more packages than the LD_PRELOAD hack [1].
>
> Would anyone volunteer to be the backup maintainer for this?
>
> [1]
> https://summerofcode.withgoogle.com/dashboard/organization/5749981018849280/proposal/5572241732927488/


I get a 403 for that.

https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2020/Ideas#FUSE-powered_sandbox
is
your writeup; I assume the above is supposed to be a link to the student's
proposal?


>
>
> --
> Best regards,
> Michał Górny
>
>


[gentoo-dev] [PSA] If you ssh interactively to git.gentoo.org (somehow) let me know.

2020-04-25 Thread Alec Warner
TL;DR: if all you do is use git to commit to git.gentoo.org, you are not
affected and can stop reading; I know folks use git+ssh://g...@git.gentoo.org
... to push commits, that will not change.

In the olden times Gentoo used cvs as its source control and people would
push their commits to the cvs server over ssh. The setup at the time was
that everyone who pushed had ssh access to cvs.gentoo.org.

However, Gentoo doesn't use cvs (and has not for many years[1]). The git
system uses 'gitolite' and people who push do so as 'g...@git.gentoo.org'
(not as themselves.) Gitolite handles the per-user multiplexing and
everything is happy.

However, we never took the ssh access to 'cvs.gentoo.org' away, most devs
can still ssh to "git.gentoo.org" as themselves. Now the access doesn't get
you much (ForceCommand in the authorized_keys file just runs a commit
wrapper, so you could try to commit to cvs or svn I guess ;p)

Thus I now plan to remove this access[0]. If you need access to ssh as
something not-git to git.gentoo.org, let me know in the next week.

[0] Infra users are not affected; they always had normal ssh access to this
host.
[1] Anonymous access to source trees (e.g. via anon* services) is
unaffected by this change.


[gentoo-dev] Re: [RFC] Adding potentially questionable license AcePerl-Indemnity

2020-04-22 Thread Alec Warner
On Wed, Apr 22, 2020 at 3:33 PM Kent Fredric  wrote:

> I've just discovered dev-perl/Ace has some fun questionable licensing
> which includes a lovely indemnity clause, which had previously gone
> unnoticed, and it stipulates additional requests for research
> publications, which is not something mentioned in any license currently
> in tree other than Tinker
>
> Following is the entire body of the license I plan to put in
> licenses/AcePerl-Indemnity ( name chosen to specifically alert people
> tempted to accept this license that Indemnification is an important
> part they should actually read )
>
> Current advice also says that due to the terms of this license, we have
> to RESTRICT="mirror" this as well, unless the Trustees want to sign off
> on potentially indemnifying CSHL
>

I think it's less about the Foundation (I'm happy to indemnify them) but
the indemnification reads as viral to me and so anyone that operated a
distfiles mirror would also implicitly indemnify[0] (through mirroring from
us) and it seems unlikely this is a thing we would encourage because our
distfiles operators are relying on some diligence on our own end to avoid
excessive liability for files we provide to them.

-A

[0] Whether this would hold up is another matter, but I'm fine with
restricting the mirroring of aceperl to avoid this complication.



>
> Also following up with CPAN because as its *currently* mirrored on
> CPAN, and has been mirrored there for at *least* 12 years, its
> potentially in a legal situation as well.
>
> ( But that's the fault of the uploader if true, because you can't
> upload anything to CPAN without mirroring being something you didn't
> expect )
>
> Once this license is added, the plan is to rework Ace-*.ebuild to be under
>
> LICENSE="|| ( Artistic GPL-1+ ) AcePerl-Indemnity"
>
> Upstream: https://metacpan.org/source/LDS/AcePerl-1.92/DISCLAIMER.txt
>
> Text Body:
>
> ==
>
> The Ace.pm package and all associated files are Copyright (c) 1998
> Cold Spring Harbor Laboratory.
>
> This library is free software; you can redistribute it and/or modify
> it under the same terms as Perl itself.  See the Artistic License file
> in the main Perl distribution for specific terms and conditions of
> use.  In addition, the following disclaimers apply:
>
> CSHL makes no representations whatsoever as to the SOFTWARE contained
> herein.  It is experimental in nature and is provided WITHOUT WARRANTY
> OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OR ANY OTHER
> WARRANTY, EXPRESS OR IMPLIED.  CSHL MAKES NO REPRESENTATION OR
> WARRANTY THAT THE USE OF THIS SOFTWARE WILL NOT INFRINGE ANY PATENT OR
> OTHER PROPRIETARY RIGHT.
>
> By downloading this SOFTWARE, your Institution hereby indemnifies CSHL
> against any loss, claim, damage or liability, of whatsoever kind or
> nature, which may arise from your Institution's respective use,
> handling or storage of the SOFTWARE.
>
> If publications result from research using this SOFTWARE, we ask that
> CSHL be acknowledged and/or credit be given to CSHL scientists, as
> scientifically appropriate.
>
> ==
>
>


Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

2020-04-19 Thread Alec Warner
On Sun, Apr 19, 2020 at 8:37 AM Michael Orlitzky  wrote:

> On 4/19/20 10:55 AM, Samuel Bernardo wrote:
> >
> > Taking into account the network sandbox requirement, sbt.eclass needs to
> > download all dependencies with some approach like EGO_SUM implementation
> > in go-module.eclass[1].
> >
>
> EGO_SUM is not a legitimate approach to packaging. You have to create
> packages for each package. I know, it sounds crazy when you say it.
>
>
EGO_SUM is great!

-A


Re: [gentoo-dev] zoom concerns

2020-04-01 Thread Alec Warner
On Wed, Apr 1, 2020 at 5:18 PM Alessandro Barbieri 
wrote:

> I have concerns about the inclusion of zoom in ::gentoo. For me it's more
> like a malware.
> From the hacker news feed you'll find out that:
>

> [1] zero day vulnerability found
>
[2] passwords are truncated to 32 bit
>
[3] previously sent data to facebook
>
[4] end to end traffic isn't encrypted
> [5] signed binary run unsigned script
>
>
[1], [2], [5] all seem like bugs and I'd expect upstream to fix at least
[1] and [5].  Note that in Gentoo [3] isn't directly relevant (this isn't
iOS) and neither is [5] in most cases as people don't run signed binaries
or use any kind of binary whitelisting in Gentoo.

[2] I think the article mentions the truncation is to 32 bytes (or '32
chars', but I assume each char is 1 byte for entropy sake.); not 32 bits.
Most password fields have a length limit (you cannot accept arbitrary long
passwords. If 32 characters isn't enough length to protect users then the
passwords are going to be useless anyway; most user passwords are
significantly less than 32 characters. This is significantly different than
limited to '32 bits' (which is 4 characters!) and would make brute forcing
passwords an obvious breeze; there is not sufficient entropy in 32 bits to
protect users.

[4] I agree the poor marketing is a problem. I think as Rich states later
in the thread it's possible we could provide more information here. As he
notes though, I'm not convinced this is reason not to package the software
in Gentoo from a policy perspective.

In general I expect that as long as Zoom has a gentoo maintainer and
upstream actually resolves outstanding security issues; I'm not really
aware of any policy hurdles they need to overcome to stay packaged in
Gentoo. Currently it has three maintainers[6]. If it sucks, convince them
to stop maintaining it ;)

-A



> 1 https://techcrunch.com/2020/04/01/zoom-doom/?guccounter=1
> 2 https://news.ycombinator.com/item?id=22749706
> 3
> https://www.vice.com/en_us/article/z3b745/zoom-removes-code-that-sends-data-to-facebook
> 4 https://theintercept.com/2020/03/31/zoom-meeting-encryption/
> 5 https://news.ycombinator.com/item?id=22746764
>

[6] https://packages.gentoo.org/packages/net-im/zoom


Re: [gentoo-dev] network sandbox challenge

2020-04-01 Thread Alec Warner
On Wed, Apr 1, 2020 at 5:14 AM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi Robin,
> On 4/1/20 6:36 AM, Robin H. Johnson wrote:
>
> Normally we don't bundle dependencies, avoiding that problem entirely.
> The Go eclasses however are badly designed, committed against protest by
> paid corporate interests, and serve only to facilitate large-scale
> copyright infringement and security vulnerabilities. If you're looking
> for a consistent explanation of how they're supposed to work with the
> rest of Gentoo, you won't find one.
>
> mjo: Can you please substantiate your claims?
>
> It would have been nice to have heard your concerns during February, any
> of one the three times that William and I posted the go-module.eclass
> EGO_SUM development work for review on this mailing list. I don't see a
> single email from you during that entire period.
>
> The EGO_SUM support explicitly ensured that upstream distfiles (for each
> dependency) remained absolutely as upstream provided them, without
> merging the distfiles together or altering their content in way (I admit
> that the exact naming of the distfiles changed, because it was terrible,
> v0.0.0-20190311183353-d8887717615a.zip for example).
>
> Forgive my noobishness in this matter that let Alec to comment over my own
> statement.
>
> Alec pointed out some very important issues in go development that break
> copyright infringement and security vulnerabilities, but I'm sure that is
> not related to the good work done in go-module.eclass to surpass all go
> mess. npm is worst and I take from go-module as a good pattern to apply
> also into there.
>
I am antarus, not mjo (but more on that below!) I don't believe bundling
presents many challenges with regards to copyright infringement. As a
package maintainer you should know the licenses used in your packages. You
are required to reflect any licenses used in the LICENSE ebuild variable.
Obviously this becomes more work if you are using a bundle due to the fact
that bundling will include more code. In the golang ecosystem there is a
tool to help maintainers do this (
https://packages.gentoo.org/packages/dev-go/golicense). I get that with
bundling we cannot share the work from previous packages because packages
are not shared in a bundled environment but I expect the golicense tool to
have good coverage in practice. If the tool does the work, sharing the work
becomes moot.

I think licensing can be more challenging in other bundling scenarios where
tooling is not provided; but note that this is not significantly different
from the unbundled scenario in terms of license discovery. If I am
packaging a new program (A) and it depends on (B,C,D) I have two options. I
can either package [A,B,C,D] (normal gentoo way) or I can package [A] (with
B,C,D bundled). The intersection of the LICENSE variables is the same
effort for both here. The benefit of the multiple packages is that future
users of B,C,D can re-use the license discovery work and that isn't nothing.

> Going back to my overlay use case, will go-modules download all modules to
> distfiles directory? The naming convention will assure that there will be
> no modules repetition?
>
What about eclean-dist, will it work as expected for those modules
> dependencies?
>
> I think some of this answers would worth mention in documentation.
>
> Sorry for anything I wrongly stated and thank you very much for your help,
>
> Samuel
>
I've chosen this part to write my treatise on packaging, but rest assured
it's mostly intended as a response to mgorny and mjo; not specifically in
response to you.

The very long answer is that Gentoo was designed around a paradigm of
programs written primarily in C. In C programs you have the ability to link
to libraries which offer APIs and in the ideal case, each API is offered
via a unique SONAME[0]. Upstream packages were written and built in this
way (with dynamic linking). So in the case of package A, that uses
libraries B, C, and D; the result in many distributions is 4 packages
(A,B,C,D) and users who want A will get B, C, and D installed. This in fact
was a major selling point of package managers at the time because finding
these dependencies by hand and building and merging them all was painful.

Many applications break this trend; I don't think golang or nodejs are
particularly new (python and ruby have had (pip, venv) and rubygems[1] for
years, for example, which are similar bundling paradigms.) The struggle as
packagers and distribution managers is when upstream decides "my software
should be installed via a bundling solution (golang, node, pip, rubygems,
and so on)" we are left to decide both whether to map this to the ebuild
paradigm (no bundling of dependencies) or omit ebuilds entirely. In the
former case we are often left working at odds with upstream (who are
confused by our decomposition of their application) and in the latter case,
users often use the bundle anyway (e.g. they install the packages by hand
or 

Re: [gentoo-dev] network sandbox challenge

2020-03-31 Thread Alec Warner
On Tue, Mar 31, 2020 at 3:21 PM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi,
> On 3/31/20 9:25 PM, Alec Warner wrote:
>
> From thirdpartymirrors file I can see more examples... The mirror type
>> can be any label that I decide to use?
>>
>
> man portage(5) says:
> Whenever portage encounters a mirror:// style URI it will look up the
> actual hosts here.  If the mirror set is not found here, it  will  check
>  the
> global  mirrors file at /var/db/repos/gentoo/profiles/thirdpartymirrors.
> You may also set a special mirror type called "local".  This list of
> mirrors will be checked before GENTOO_MIRRORS and will be used even if the
> package has RESTRICT="mirror" or RESTRICT="fetch".
>
> We have a bunch of existing mirror types (as you noticed). This lets us do
> things like "mirror://github" and then globally change the uri for github
> easily without editing the entire set of ebuilds because we late-bind the
> URI matching.
>
> Ok, so it works as an alias... The format definition led me to
> misunderstand:
>
> - mirror type followed by a list of hosts
>
> I interpreted mirror type as some internal definition not an alias.
>
>
>> Is URI mirror:// style deprecated in ebuilds and should use
>> restric="mirror" instead?
>>
>
> I don't understand how you could arrive at this conclusion. What would
> this do?
>
> RESTRICT=mirror is for things we are not legally allowed to mirror. That's
> basically all its for; it has nothing to do with custom mirrors or anything.
>
> Sorry for my dumb question, but my previous understanding of the context
> was that, since mirror:// is always accessed before the SRC_URI, there is
> no need to use mirror:// in ebuild SRC_URI, with the sources being
> downloaded from any available mirror (also remembering the dev manuals
> documentation related to  mirror://gentoo policy for auto mirroring the
> sources).
>
> But after your explanation, I understand now that mirror types provides
> alias to use in ebuild SRC_URI, specially useful for the update task
> (awesome).
>
> Looking into overlay context, I think that I could add thirpartymirrors
> file inside overlay profiles directory to use mirror:// in SRC_URI
> to simplify ebuild URI management in the end. For example, to all JetBrains
> software this will be awesome.
>
In general I'd avoid using the mirror system as URI simplification too
much; a lot of the idea is to avoid hardcoding specific hosts. E.g. for the
gentoo mirrors we want to avoid hardcoding a *specific* mirror in the
SRC_URI, so mirror://gentoo lets us say "any gentoo mirror will do." Often
other thirdpartymirror systems act similarly. You see this commonly where
mirrors are divided by region.

example.com/mirror # "global / us"
example.au/mirror # "australia"
example.co.uk/mirror # "UK"

>From a layout perspective these are the same, but you would not want to
hardcode "https://example.com/mirror; in the ebuild, because then
australian users are routed to the US..which is wasteful. So we use a
generic:

"mirror://example"

And we potentially pick the best mirror at fetch time.

This is mostly spelled out in:
https://devmanual.gentoo.org/ebuild-writing/variables/#third-party-mirrors

If you have a bunch of packages that share a common domain and its not
sharded (there is only 1 host serving it) then you should consider using an
eclass to share data between your ebuilds and not use thirdpartymirrors
just to share a hostname.

-A


>
Thanks,
>
> Samuel
>


Re: [gentoo-dev] network sandbox challenge

2020-03-31 Thread Alec Warner
On Tue, Mar 31, 2020 at 12:58 PM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi Alec,
>
> On 3/27/20 11:20 PM, Alec Warner wrote:
> >
> > I should point you at man portage(5) (search for mirrors), which has
> > more detail on how to set up a non-gentoo mirror network.
>
> Reading portage manpage about mirrors I didn't find the mirror type
> possible values. As I could understand, there is type local as a special
> mirror and then, from the example, sourceforge and gnu. Didn't
> understand right how this works.
>
> From thirdpartymirrors file I can see more examples... The mirror type
> can be any label that I decide to use?
>

man portage(5) says:
Whenever portage encounters a mirror:// style URI it will look up the
actual hosts here.  If the mirror set is not found here, it  will  check
 the
global  mirrors file at /var/db/repos/gentoo/profiles/thirdpartymirrors.
You may also set a special mirror type called "local".  This list of
mirrors will be checked before GENTOO_MIRRORS and will be used even if the
package has RESTRICT="mirror" or RESTRICT="fetch".

We have a bunch of existing mirror types (as you noticed). This lets us do
things like "mirror://github" and then globally change the uri for github
easily without editing the entire set of ebuilds because we late-bind the
URI matching.


>
> Is URI mirror:// style deprecated in ebuilds and should use
> restric="mirror" instead?
>

I don't understand how you could arrive at this conclusion. What would this
do?

RESTRICT=mirror is for things we are not legally allowed to mirror. That's
basically all its for; it has nothing to do with custom mirrors or anything.


>
> I think that, in mirrors description for /etc/portage/, first paragraph
> should be divided in two: the new one in  the phrase that starts with
> "You may also set a special mirror type called local".
>

Feel free to submit a patch; the source is on github:
https://github.com/gentoo/portage/blob/master/man/portage.5

-A


>
> Thanks,
>
> Samuel
>
>
>


Re: [gentoo-portage-dev] precisions on installed packages' dependencies

2020-03-28 Thread Alec Warner
On Fri, Mar 27, 2020 at 7:00 AM  wrote:

>
> - Alec Warner  a écrit :
> > On Tue, Mar 24, 2020 at 11:31 AM  wrote:
> > > However, I still doubt that only storing the soname dependencies is
> enough.
> > > Consider package A (that cannot be recompiled) that depends on package
> B
> > > which provides lib L.so.
> > > B is recompiled with different use flags, which put different
> > > functionalities in L.so.
> > > The dependencies of A are still satisfied (B is installed, L.so is
> > > available), but since the content of L.so changed, A cannot execute
> anymore.
> > > Hypothetically, can this scenario occur?
> > > Can this scenario occur in practice?
> > > Is there a way in emerge/portage to avoid it?
>
>
> > > You have far more experience than me on this, and it would be nice for
> me
> > > to know what I'm up against.
> >
> > A lot of this has to do with the specifics of how package managers manage
> > system state, as well as various quirks of subsets of the tree. For
> > example, a perl upgrade (X->Y) will often break perl modules who expect
> > perl-X, but get perl-Y. So one fix is to try to keep perl-X installed (so
> > we SLOT perl and have N perls installed.) Then you need to decide which
> > version of perl to build things against (X or Y, or both?) We took this
> > tactic in the python ecosystem; but perl is not slotted in Gentoo, and so
> > upgrading perl breaks all perl modules. There is a tool
> > (gentoo-perl-cleaner) that will walk the deptree and fix all of these
> > broken packages that you run after an upgrade.
> >
> > I'm not sure it's strictly avoidable. You could build perl-Y, then
> rebuild
> > all perl-modules against perl-Y, then merge the entire result to the
> > livefs. This will reduce the breakage time but likely not eliminate it;
> > plus it seems hard to implement in practice without modern filesystem
> tools
> > (overlayfs, btrfs, zfs or similar tech to make it atomic.) It also
> doesn't
> > account for executing code. What happens to perl-X code that is executing
> > when you unmerge perl-X? The short answer is that code might break and
> > 'proper' management means you should restart services after an upgrade
> > (something Gentoo doesn't typically do; but is common in Debian for
> > example.)
>
> Many thanks for this answer.
> To sum up what I understood, the problem is not really the dependencies,
> but which recompilation (and service restart) are triggered with an update.
>

Yes and no. Assume a spherical package manager that could detect
everything, we basically need to do the following for a perl X -> Y upgrade.

1 User triggers X - Y upgrade.
2 build perl-Y, but don't merge it to the livefs.
3 numerate all deps of perl-Y, build those (but don't merge them to the
livefs, but they need to build against perl-Y, not perl-X)
4 with atomic_transaction:
  4a Merge perl-Y to the livefs
  4b Merge perl-Y's dependencies to the live fs
5 restart anything that is running perl-X, so that it runs with perl-Y
6 unmerge perl-X

In practice we cannot always do 3 or 4 or 5. We will miss dependencies (due
to missing depgraph information) both in the package depgraph as well as
the service depgraph.

So in practice we do:
1 user triggers X -Y upgrade.
2 build perl-Y, merge it to the livefs, unmerge perl-X
3 run gentoo-perl-cleaner to upgrade the dependencies broken by the X - Y
upgrade (via slot deps, or whatever mechanism, it can vary.)
4 restart anything that is running perl-X

And just accept that from 2-4..well stuff will be broken. We can minimize
the time by building binpkgs ahead of time for example, and merging the
binpkgs in a parallel. Note that 5 and 4 are the same problem in both
lists. Note that per the guide linked below, sometimes 2-3 can be done 'at
once', although again practically speaking "During Perl upgrade, packages
that depend on Perl may become unavailable." even if they are all handled
by emerge, because there are many race conditions in the order that
packages get merged to the livefs.


> In gentoo, there is the ":=" slot operator (and others similar) in
> dependencies that trigger the recompilation when the dependency's slot
> change, but this is the only existing mechanism.
> And this is why every time perl changes, the compilation of its modules is
> not triggered and they are most probably broken.
> Correct?
>

They are supposed to be triggered:
https://wiki.gentoo.org/wiki/Perl#Upgrading_.28major_version.29 for example
says this. But this is up to how callers run the tool. For example gentoo
infra executes emerge via puppet, and we never execute emerge -uDNav
--with-bdeps=y --backtrack=100 --autounmask-keep-masks=y

Re: [gentoo-dev] network sandbox challenge

2020-03-27 Thread Alec Warner
On Fri, Mar 27, 2020 at 3:59 PM Alec Warner  wrote:

>
>
> On Fri, Mar 27, 2020 at 3:10 PM Samuel Bernardo <
> samuelbernardo.m...@gmail.com> wrote:
>
>> Hi Alec,
>>
>> On 3/27/20 7:27 PM, Alec Warner wrote:
>> > The Gentoo Mirror system is basically a set of scripts that syncs the
>> > ::gentoo repository, enumerates all URIs in SRC_URI for all ebuilds,
>> > and fetches them.
>> > It doesn't enumerate anything in any overlays. Overlay authors are
>> > required to point SRC_URI somewhere useful (typically to an upstream
>> URI.)
>> >
>> > Gentoo Developers use "https://dev.gentoo.org/~user/distfiles; as an
>> > origin URI for items where either Gentoo *is* the upstream, or there
>> > is no upstream (e.g. a custom Gentoo patchset.) Any origin URI can be
>> > used; this just happens to be some free hosting that Gentoo Developers
>> > have access to use.
>> >
>> > -A
>>
>> Thank you for your clarification.
>>
>> So what happens when RESTRIC=mirror is used inside ebuild for an overlay
>> in git.gentoo.org?
>>
>
> I want to again re-iterate; gentoo doesn't mirror anything outside of
> gentoo.git, even if its hosted on git.gentoo.org. gentoo.git is literally
> the only repo the Gentoo Mirror system is processing.
>
> RESTRICT itself then has two potential usage sites.
>  - MIrroring. We already determined that for overlays, no mirroring
> occurs, so we can ignore that for your case.
>  - Fetching. Basically if something is RESTRICT=mirror, then we don't need
> to bother consulting the mirror network, because we know from the RESTRICT
> that mirror network will not have a copy. Fetching then proceeds from the
> origin URI (e.g. the URI in SRC_URI.)
>

I should point you at man portage(5) (search for mirrors), which has more
detail on how to set up a non-gentoo mirror network.

-A


>
>
>>
>> So, thinking on site reliability, should it be a good choice to upload
>> the necessary tar.gz, for example, to gitlab or github community
>> services using git lfs as an alternative to "https://dev.gentoo.org/;?
>>
>
> For most content served from dev.gentoo.org its not RESTRICT=mirror, so
> the reliability of the origin URI is not an issue in most cases (because it
> should be on the gentoo mirror network.)
> Cases where it is not include:
>  - RESTRICT=mirror content.
>  - Content that is pushed to an ebuild but hasn't been mirrored yet.
>- This can take up to 4h (or longer if the origin is unavailable.)
>
> In practice this hasn't been a big enough issue to do something more
> complicated. Many proposals have already been floated but none have ever
> been put into place.
> It also turns out that most of the time dev.gentoo.org is available (and
> so again, its reliability is not a major issue.) I understand that it
> recently had an incident; I'm not really convinced it was high enough
> impact to make operational changes.
>
> -A
>
>
>>
>> Best,
>>
>> Samuel
>>
>>
>>


Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

2020-03-27 Thread Alec Warner
On Fri, Mar 27, 2020 at 3:54 PM Patrick McLean  wrote:

> On Fri, 27 Mar 2020 15:51:35 -0700
> Alec Warner  wrote:
>
> > On Fri, Mar 27, 2020 at 3:11 PM Patrick McLean 
> wrote:
> >
> > > On Fri, 27 Mar 2020 14:48:53 -0700
> > > Matt Turner  wrote:
> > >
> > > > On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean 
>
> > > wrote:
> > > > >
> > > > > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > > > > _python_impl_supported to a separate eclass, this allows overlays
> > > > > to easily support a different set of python implementations than
> > > > > ::gentoo without having to fork the entire suite of eclasses.
> > > >
> > > > (I think the issue is that you have some packages that still need
> > > > Python 3.4. Correct me if I'm wrong)
> > > >
> > > > How many packages do you need to work with Python 3.4? Presumably
> just
> > > > a couple and then a pile of dependencies in ::gentoo?
> > > >
> > >
> > > For our particular purpose, it's more complicated than that. We do not
> > > expect or want ::gentoo to try support Python 3.4 in any way. We have
> an
> > > overlay that is shared on a lot of machines, some of those machines are
> > > older than others. As such we still have ebuilds that only support
> > > python3_4 that still exist in the overlay. We would like it if the
> > > existence of these packages in the overlay, do not cause metadata
> > > generation or dependency calculation to explode on the more up-to-date
> > > machines.
> > >
> >
> > > We do not need Python 3.4 packages to be installable on the newer
> > > machines, we just need them not to explode.
> > >
> >
> > Couldn't you simply remove the ebuilds from the overlay entirely in this
> > case? It's my understanding that on the machines with the packages
> > installed, the merged package metadata is being used (which is why those
> > machines work) and since the newer machines don't have this merged
> package
> > metadata, they don't work properly.
> >
>
> Sometimes we have to fix the older machines, so we have to keep
> everything around until none of our machines are using it any more.
>

+Zac Medico 

I'm not following something then. One of the proposals earlier was "do not
generate metadata for these ebuilds" which to me reads as "keep these
ebuilds in the repo, but the ebuilds themselves will not be usable[0]."
Then you go on to say that old machines need to be fixed occasionally, so
either you need to reinstall a package or update it. The reinstall might be
strictly possible from the vdb metadata, but the upgrade would require
working repository metadata, which we said earlier we didn't want to
generate.

(There is a different question of whether you could use a binpkg binhost to
basically build versions of these packages and use those for reinstalls,
because at least the binpkg metadata *is* nominally fixed in time and can
be re-used easily and doesn't require eclass magic in theory, as the
eclasses are bound in the environment.tar?) I suspect in essence it might
be easier to just do what Robin suggested and use a frozen ::gentoo on the
older machines.

-A

[0] Perhaps the earlier proposals were ..slightly off. With more
information it seems like what you want is to be able to easily maintain
some python-3.4 ebuilds in your overlay, even though Gentoo is on to 3.6
(and soon 3.7). I personally think the conversation would have gone much
better had you just come out and said "hey we have a bunch of internal
overlays with 3.4 and we need to keep the packages for another N months,
how can we do this easily?" Instead of whatever happened. I tend to agree
with mgorny that it's not simply carrying this single patch, but in reality
it's a stronger commitment to build some kind of method to let you continue
to support older python versions. Future changes might impact your ability
to support your ebuilds and we have no real way of knowing if we broke
you..so I can see why (irrespective of the tone of the conversation) he
would be reticent to pick that up.


>
> > >
> > > I am trying to come up with a solution that *any* overlay can use, I
> > > am happy to do work towards that end. Basically, it would be very
> > > nice if there was a minimal eclass that handles all the python
> > > version compatibility. Almost everything in the eclasses does not care
> > > about versions.
> > >
> > > This is not meant to be something just for our usage, we want this to
> > > be usable for *any* overlay, and are more than happy to make things as
> > > generic as possible. If some overlay wants to support Python 3.10
> alpha,
> > > resurrect jython support, or add Ironpython support, without forking
> > > all the eclasses, I would like to think that could be doable as well.
> > >
> > >
>
>


Re: [gentoo-dev] network sandbox challenge

2020-03-27 Thread Alec Warner
On Fri, Mar 27, 2020 at 3:10 PM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi Alec,
>
> On 3/27/20 7:27 PM, Alec Warner wrote:
> > The Gentoo Mirror system is basically a set of scripts that syncs the
> > ::gentoo repository, enumerates all URIs in SRC_URI for all ebuilds,
> > and fetches them.
> > It doesn't enumerate anything in any overlays. Overlay authors are
> > required to point SRC_URI somewhere useful (typically to an upstream
> URI.)
> >
> > Gentoo Developers use "https://dev.gentoo.org/~user/distfiles; as an
> > origin URI for items where either Gentoo *is* the upstream, or there
> > is no upstream (e.g. a custom Gentoo patchset.) Any origin URI can be
> > used; this just happens to be some free hosting that Gentoo Developers
> > have access to use.
> >
> > -A
>
> Thank you for your clarification.
>
> So what happens when RESTRIC=mirror is used inside ebuild for an overlay
> in git.gentoo.org?
>

I want to again re-iterate; gentoo doesn't mirror anything outside of
gentoo.git, even if its hosted on git.gentoo.org. gentoo.git is literally
the only repo the Gentoo Mirror system is processing.

RESTRICT itself then has two potential usage sites.
 - MIrroring. We already determined that for overlays, no mirroring occurs,
so we can ignore that for your case.
 - Fetching. Basically if something is RESTRICT=mirror, then we don't need
to bother consulting the mirror network, because we know from the RESTRICT
that mirror network will not have a copy. Fetching then proceeds from the
origin URI (e.g. the URI in SRC_URI.)


>
> So, thinking on site reliability, should it be a good choice to upload
> the necessary tar.gz, for example, to gitlab or github community
> services using git lfs as an alternative to "https://dev.gentoo.org/;?
>

For most content served from dev.gentoo.org its not RESTRICT=mirror, so the
reliability of the origin URI is not an issue in most cases (because it
should be on the gentoo mirror network.)
Cases where it is not include:
 - RESTRICT=mirror content.
 - Content that is pushed to an ebuild but hasn't been mirrored yet.
   - This can take up to 4h (or longer if the origin is unavailable.)

In practice this hasn't been a big enough issue to do something more
complicated. Many proposals have already been floated but none have ever
been put into place.
It also turns out that most of the time dev.gentoo.org is available (and so
again, its reliability is not a major issue.) I understand that it recently
had an incident; I'm not really convinced it was high enough impact to make
operational changes.

-A


>
> Best,
>
> Samuel
>
>
>


Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

2020-03-27 Thread Alec Warner
On Fri, Mar 27, 2020 at 3:11 PM Patrick McLean  wrote:

> On Fri, 27 Mar 2020 14:48:53 -0700
> Matt Turner  wrote:
>
> > On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean 
> wrote:
> > >
> > > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > > _python_impl_supported to a separate eclass, this allows overlays
> > > to easily support a different set of python implementations than
> > > ::gentoo without having to fork the entire suite of eclasses.
> >
> > (I think the issue is that you have some packages that still need
> > Python 3.4. Correct me if I'm wrong)
> >
> > How many packages do you need to work with Python 3.4? Presumably just
> > a couple and then a pile of dependencies in ::gentoo?
> >
>
> For our particular purpose, it's more complicated than that. We do not
> expect or want ::gentoo to try support Python 3.4 in any way. We have an
> overlay that is shared on a lot of machines, some of those machines are
> older than others. As such we still have ebuilds that only support
> python3_4 that still exist in the overlay. We would like it if the
> existence of these packages in the overlay, do not cause metadata
> generation or dependency calculation to explode on the more up-to-date
> machines.
>

> We do not need Python 3.4 packages to be installable on the newer
> machines, we just need them not to explode.
>

Couldn't you simply remove the ebuilds from the overlay entirely in this
case? It's my understanding that on the machines with the packages
installed, the merged package metadata is being used (which is why those
machines work) and since the newer machines don't have this merged package
metadata, they don't work properly.

-A


>
> I am trying to come up with a solution that *any* overlay can use, I
> am happy to do work towards that end. Basically, it would be very
> nice if there was a minimal eclass that handles all the python
> version compatibility. Almost everything in the eclasses does not care
> about versions.
>
> This is not meant to be something just for our usage, we want this to
> be usable for *any* overlay, and are more than happy to make things as
> generic as possible. If some overlay wants to support Python 3.10 alpha,
> resurrect jython support, or add Ironpython support, without forking
> all the eclasses, I would like to think that could be doable as well.
>
>


Re: [gentoo-dev] network sandbox challenge

2020-03-27 Thread Alec Warner
On Fri, Mar 27, 2020 at 5:17 AM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> Hi again Michał,
> On 3/27/20 11:48 AM, Michał Górny wrote:
> > Nope, just ::gentoo.  Minus ebuilds with RESTRICT=mirror.
>
> I have some doubts after reading the mirror documentation[1] in the
> context of personal overlays (not official).
>
> There is two procedures defined as I could understand:
> - manually upload a file to mirror://gentoo, scp it to
> dev.gentoo.org:/space/distfiles-local
>
> - having SRC_URI defined as
> SRC_URI="https://dev.gentoo.org/~myname/distfiles/${P}.tar.gz; to avoid
> namespace collisions with RESTRIC=mirror in ebuild would be uploaded
> automatically
>
> The use of mirror://gentoo directly is a deprecated policy. So it must
> be used https instead.
>
> 1) Did I understand it right?
>
> 2) What is dev.gentoo.org:~/public_html? Do this means that is only
> available to Gentoo official developers the access to the mirror[2]?
>
>
The Gentoo Mirror system is basically a set of scripts that syncs the
::gentoo repository, enumerates all URIs in SRC_URI for all ebuilds, and
fetches them.
It doesn't enumerate anything in any overlays. Overlay authors are required
to point SRC_URI somewhere useful (typically to an upstream URI.)

Gentoo Developers use "https://dev.gentoo.org/~user/distfiles; as an origin
URI for items where either Gentoo *is* the upstream, or there is no
upstream (e.g. a custom Gentoo patchset.) Any origin URI can be used; this
just happens to be some free hosting that Gentoo Developers have access to
use.

-A



> Best,
>
> Samuel
>
> [1] https://devmanual.gentoo.org/general-concepts/mirrors/index.html
>
> [2] https://wiki.gentoo.org/wiki/Project:Infrastructure/Developer_Webspace
>
>
>


Re: [gentoo-portage-dev] precisions on installed packages' dependencies

2020-03-26 Thread Alec Warner
On Tue, Mar 24, 2020 at 11:31 AM  wrote:

>
> - Zac Medico  a écrit :
>
> > > The goal of my tool is to have correct manipulation of package
> dependencies, and in particular here, I focus on the packages that are
> installed but not in the portage tree/a local overlay anymore (the problem
> does not occur for other packages).
> > > It seems that installed packages do not store which are the actual cpv
> they depend on. Correct?
> >
> > Right, because unfortunately that's something that changes over time.
> >
> > Also, we may not be able to pin it down at any given moment if we have
> > inconsistent || preferences as described here:
> >
> >
> https://archives.gentoo.org/gentoo-dev/message/550d3859dea6d0fb0b39064628992634
>
> Hmm, I think I see what you mean.
> Storing the cpvs that was used during solving the package's dependencies
> would be too restrictive, since two different packages could provide the
> exact same functionalities/libraries.
> And so, during a system update, only looking at the cpv dependencies would
> trigger useless recompilation of the packages that depend on the updated
> packages.
> Correct?
>
> Btw, my tool's solver does not have the problem discussed in the thread
> you're mentioning: atom order in lists has no influence in my solver.
> Would fixing the inconsistent || preferences make storing cpvs for
> installed packages more realistic?
>
>
> > > Also, I wanted to use the ebuild tool to install/uninstall package,
> which is not possible with this solution apparently.
> >
> > Why not? Would the preserve-libs feature solve your problem?
>
> ... I'm sorry, I wasn't aware of this feature.
> It would definitively solve the issue (except, as described in the bug
> 459038, when external tools remove libs).
>
> This discussion is very interesting!
> If I take this double layer of dependencies, I have to check how this
> influences the theory underlying my tool.
>
> However, I still doubt that only storing the soname dependencies is enough.
> Consider package A (that cannot be recompiled) that depends on package B
> which provides lib L.so.
> B is recompiled with different use flags, which put different
> functionalities in L.so.
> The dependencies of A are still satisfied (B is installed, L.so is
> available), but since the content of L.so changed, A cannot execute anymore.
> Hypothetically, can this scenario occur?
> Can this scenario occur in practice?
> Is there a way in emerge/portage to avoid it?
>

>
> > Well, there are a lot of upgrades that can't be performed without
> > temporarily breaking something, so the "forbid broken dependencies" idea
> > doesn't sound feasible to me.
>
> Could you tell me about several instances of such needed dependency
> breakage?
> You have far more experience than me on this, and it would be nice for me
> to know what I'm up against.
>

A lot of this has to do with the specifics of how package managers manage
system state, as well as various quirks of subsets of the tree. For
example, a perl upgrade (X->Y) will often break perl modules who expect
perl-X, but get perl-Y. So one fix is to try to keep perl-X installed (so
we SLOT perl and have N perls installed.) Then you need to decide which
version of perl to build things against (X or Y, or both?) We took this
tactic in the python ecosystem; but perl is not slotted in Gentoo, and so
upgrading perl breaks all perl modules. There is a tool
(gentoo-perl-cleaner) that will walk the deptree and fix all of these
broken packages that you run after an upgrade.

I'm not sure it's strictly avoidable. You could build perl-Y, then rebuild
all perl-modules against perl-Y, then merge the entire result to the
livefs. This will reduce the breakage time but likely not eliminate it;
plus it seems hard to implement in practice without modern filesystem tools
(overlayfs, btrfs, zfs or similar tech to make it atomic.) It also doesn't
account for executing code. What happens to perl-X code that is executing
when you unmerge perl-X? The short answer is that code might break and
'proper' management means you should restart services after an upgrade
(something Gentoo doesn't typically do; but is common in Debian for
example.)

-A


> Many thanks!
> Michael
>
>


Re: [gentoo-dev] dev.gentoo.org unreachable

2020-03-23 Thread Alec Warner
On Mon, Mar 23, 2020 at 7:29 AM malc  wrote:

> On Mon, Mar 23, 2020 at 9:13 AM Jaco Kroon  wrote:
> >
> > Hi All,
> >
> > Is there a known issue with dev.gentoo.org?
>
> https://infra-status.gentoo.org/ agrees with your analysis... (at
> 10:50 UTC) I would keep checking there as infra wakes up and updates
> the status.
>
>
This is resolved now.

-A


Re: [gentoo-dev] rfc: noarch keyword

2020-03-21 Thread Alec Warner
On Sat, Mar 21, 2020 at 1:03 AM Alexander Tsoy  wrote:

> В Сб, 21/03/2020 в 00:53 -0700, Matt Turner пишет:
> > On Fri, Mar 20, 2020 at 9:55 PM Kent Fredric 
> > wrote:
> > > If X is "noarch" and its dependency Y is "amd64", then a user on
> > > "sparc"
> > > will be able to install "X", but not its dependency "Y".
> >
> > Thank you. This is a good explanation of the problem.
> >
> > How do other distributions handle this? Arch, Fedora, and Debian have
> > "noarch" packages. Surely they've found a reasonable way to make this
> > work.
>
> Binary distros usually have separate repositories for each
> architecture.
>

Pretty much this. There is not 1 repository, there are N. This means that
if leaf package A "noarch" depends on package B (only stable on x86) then
in the x86 tree, A and B will be available. In the sparc tree, B is not
available and so A is uninstallable and also not available.

We had discussed doing this in the past but in practice we use a bunch of
files to compute these boundaries on the fly and this is not particularly
cheap in the current implementation.


Re: [gentoo-dev] rfc: noarch keyword

2020-03-19 Thread Alec Warner
On Thu, Mar 19, 2020 at 6:52 AM Gerion Entrup 
wrote:

> Am Donnerstag, 19. März 2020, 02:59:56 CET schrieb Kent Fredric:
> > On Wed, 18 Mar 2020 17:52:25 +
> > James Le Cuirot  wrote:
> >
> > > Not quite. Tools like repoman will need to be updated to understand
> > > that an ebuild with KEYWORDS="amd64" can depend on another ebuild with
> > > only KEYWORDS="noarch". I do think the idea has merit though.
> >
> > But the inverse is _not_ true, an ebuild with KEYWORDS="noarch"
> > *cannot* depend on another ebuild with only KEYWORDS="amd64".
> Maybe I misunderstand something but shouldn't that be the normal case?
> Every single Python package (candidates for noarch) for example depends
> on the Python interpreter, which must have non noarch keywords.
>
>
> > Otherwise it breaks the entire stabilization graph.
> The condition would be: If the interpreter is stable for an arch, all
> it's interpreted code is also stable for this arch.
>

Much of the concern is that this condition is not true for all interpreted
code.

-A


>
>
> Best,
> Gerion
>
>


Re: [gentoo-dev] Fwd: [gentoo-commits] repo/gentoo:master commit in: app-office/calcurse/

2020-03-12 Thread Alec Warner
On Thu, Mar 12, 2020 at 9:54 AM Andreas K. Huettel 
wrote:

> Someone needs to grow up here.
>

Meh, to me (someone who can't commit to ::gentoo) I have a few concerns
here. First, I don't see a lot of QA reverts on the gentoo-dev list. Is it
common practice to post reverts publicly? Second, I'm not aware that we
would revert for things like this. Most of the items you mention look
fairly minor (maybe the python comment looks impactful?) Why can't we fix
these items in a future commit, rather than revert? What did Patrice's
commit break?

I'm also not sure intimating that people are acting like children is
helpful.

-A


>
> --  Weitergeleitete Nachricht  --
>
> Betreff: [gentoo-commits] repo/gentoo:master commit in:
> app-office/calcurse/
> Datum: Mittwoch, 11. März 2020, 09:24:16 CET
> Von: Patrice Clement 
> An: gentoo-comm...@lists.gentoo.org
>
> commit: 81a65d7da5b5e8b9e48323d13da05525d08b9551
> Author: Patrice Clement  gentoo  org>
> AuthorDate: Wed Mar 11 08:21:58 2020 +
> Commit: Patrice Clement  gentoo  org>
> CommitDate: Wed Mar 11 08:24:03 2020 +
> URL:https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=81a65d7d
>
> app-office/calcurse: drop package to m-n.
>
> Package-Manager: Portage-2.3.89, Repoman-2.3.20
> Signed-off-by: Patrice Clement  gentoo.org>
>
>  app-office/calcurse/metadata.xml | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/app-office/calcurse/metadata.xml
> b/app-office/calcurse/metadata.xml
> index d5b9396fdc3..aa7b56c65ee 100644
> --- a/app-office/calcurse/metadata.xml
> +++ b/app-office/calcurse/metadata.xml
> @@ -1,9 +1,7 @@
>  
>  http://www.gentoo.org/dtd/metadata.dtd;>
>  
> -
> -   monsie...@gentoo.org
> -
> +
>  Calcurse is a text-based personal organizer which helps
> keeping
>  track of events and everyday tasks. It contains a calendar, a 'todo'
> list,
> and
>  puts your appointments in order. The user interface is configurable, and
> one
> can
>
>
> -
> --
> Andreas K. Hüttel
> dilfri...@gentoo.org
> Gentoo Linux developer
> (council, qa, toolchain, base-system, perl, libreoffice)
>


Re: [gentoo-portage-dev] [PATCH] repoman.modules.vcs.git.changes: reindex (bug 712106)

2020-03-11 Thread Alec Warner
On Wed, Mar 11, 2020 at 12:16 AM Zac Medico  wrote:

> For files returned by git diff-index, call git update-index in order
> to ensure that the index reflects the state on disk. This will prevent
> incorrect assumptions in cases where the index is missing or stale for
> some reason. Since repoman uses this information to decide when to
> update copyright header dates, this can prevent spurious copyright
> header updates.
>
> Signed-off-by: Zac Medico 
> Bug: https://bugs.gentoo.org/712106
> ---
>  repoman/lib/repoman/modules/vcs/git/changes.py | 15 ---
>  1 file changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/repoman/lib/repoman/modules/vcs/git/changes.py
> b/repoman/lib/repoman/modules/vcs/git/changes.py
> index 7e9ac1eb5..ebf770d53 100644
> --- a/repoman/lib/repoman/modules/vcs/git/changes.py
> +++ b/repoman/lib/repoman/modules/vcs/git/changes.py
> @@ -29,8 +29,14 @@ class Changes(ChangesBase):
> '''
> super(Changes, self).__init__(options, repo_settings)
>
> -   def _scan(self):
> -   '''VCS type scan function, looks for all detectable
> changes'''
> +   def _scan(self, _reindex=True):
>

Why the underscore prefix?

-A


> +   '''
> +   VCS type scan function, looks for all detectable changes
> +
> +   @param _reindex: ensure that the git index reflects the
> state on
> +   disk for files returned by git diff-index
> +   @type _reindex: bool
> +   '''
> with repoman_popen(
> "git diff-index --name-only "
> "--relative --diff-filter=M HEAD") as f:
> @@ -51,6 +57,9 @@ class Changes(ChangesBase):
> removed = f.readlines()
> self.removed = ["./" + elem[:-1] for elem in removed]
> del removed
> +   if _reindex and (self.changed or self.new or self.removed):
> +   self.update_index([], self.changed + self.new +
> self.removed)
> +   self._scan(_reindex=False)
>
> @property
> def unadded(self):
> @@ -91,7 +100,7 @@ class Changes(ChangesBase):
> # of the working tree.
> myfiles = mymanifests + myupdates
> myfiles.sort()
> -   update_index_cmd = ["git", "update-index"]
> +   update_index_cmd = ["git", "update-index", "--add",
> "--remove"]
> update_index_cmd.extend(f.lstrip("./") for f in myfiles)
> if self.options.pretend:
> print("(%s)" % (" ".join(update_index_cmd),))
> --
> 2.24.1
>
>
>


Re: [gentoo-dev] Last rites: dev-python/*, python-maintained, py3.6-only, no-revdep

2020-03-07 Thread Alec Warner
On Sat, Mar 7, 2020 at 9:57 AM Andreas Sturmlechner 
wrote:

> On Samstag, 7. März 2020 18:49:25 CET Ulrich Mueller wrote:
> > Just the ebuild being outdated doesn't sound like a sufficient reason
> > for removal of a package, at least not for those packages that install
> > applications for the end user.
>
> They are python packages and as such they block cleanup of old python
> versions. Someone has to actually put effort into each of them to keep
> them
> alive.
>

I think the idea is that this is all implicit in the notification, rather
than being explicit, which muddles the messaging. I *suspect* that py3.6
will get dropped eventually as its no longer developed (but is security
supported by upstream through 2021.) If we just came out and said "Hey we
plan on dropping python-3.6 in X[0] months, here are a bunch of packages on
py-3.6, we need to either drop them or update them" the conversation would
be slightly different.

I also suspect the conversation did not go this way because then instead of
discussing who would maintain these packages, we would be discussing "why
python-3.6 should not be dropped until the last possible day in 2021" which
sounds reminiscent of python2 ;)

I think however, that we can do both. Python-3.6 will get removed
eventually; there are dozens of packages that need help and I don't think
it is incumbent on the python team to do all of the work, hence this thread
notifying folks that "hey if you use these packages they will need help to
stay around."

[0] We could all argue over some value of X; but on another thread please ;p


>
> > > But that's by no means all ebuilds like that, just a subset Python
> > > team doesn't see much of a point maintaining.
> >
> > Why had they been added then, in the first place?
>
> Packages get added by someone, then that someone does not care about them
> anymore and they fall behind. Is that really news?
>
> People who see a need for some of those can pick them up as maintainers
> after
> all.
>
> Regards,
> Andreas


Re: [gentoo-dev] Changes made by acct-* ebuilds

2020-02-12 Thread Alec Warner
On Wed, Feb 12, 2020 at 9:59 PM Michał Górny  wrote:

> On Thu, 2020-02-13 at 02:32 +0100, Thomas Deutschmann wrote:
> > Hi,
> >
> > thank you for bringing this to the list.
> >
> > I have the same experience which is the reason why I haven't migrated
> > most of my packages yet (which is not a good move because I also didn't
> > post the problem to the list like I wanted *yet*, I only talked to
> > people via private mail, chat or at FOSDEM about that and was working on
> > a proposal I wanted to show next week when I am hopefully healthy again).
> >
>
> In other words, instead of bringing the problem up to the person who
> created the GLEP and the eclasses and/or community at large, you've been
> conspiring behind their back.  Yes, that's really the procommunity
> attitude we expect from Council members.  Thanks for showing it again.
>

This is a bit of a double standard. Either people are 'conspiring behind
your back' or they 'are gathering data for a counterproposal.' There is no
need to paint a negative narrative here.

-A


>
> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] Changes made by acct-* ebuilds

2020-02-12 Thread Alec Warner
On Wed, Feb 12, 2020 at 10:02 AM Christopher Head  wrote:

> Hi all,
> Yesterday something surprised me. I updated my system and got the
> acct-{user,group}/lighttpd for the first time. Because lighttpd was
> running, package installation failed to change the home directory—fine, it
> printed an error message, I stopped the server, changed the home directory
> by hand, and started the server back up.
>
> What I didn’t realize was that it also, successfully, removed the lighttpd
> user from a couple of auxiliary groups I had put it in. It did this without
> telling me, without printing any messages. I only noticed because I
> happened to look at syslog and discovered that usermod or gpasswd or
> whatever it called had logged the changes. Presumably this has broken a
> service or two (nothing too critical) since now Lighttpd won’t be able to
> connect to SCGI sockets any more.
>

I'm not convinced this behavior is correct, so we may be able to just fix
it.

-A


>
> Does it make sense for these ebuilds to print out all the changes they
> make to existing users and groups, so that the sysadmin can see what
> happened and immediately look into alternative solutions if it breaks
> something, rather than silently changing things? Maybe this could even be
> limited to cases where the package is being newly installed (not upgraded)
> and the user or group already exists, to ease migration from the old world
> where sysadmins are easily able to do anything we want with our users and
> groups to the new world where we’re expected to leave them alone as the
> ebuilds make them, or worst case make out changes in an overlay.
>
> Thoughts?
> --
> Christopher Head


Re: [gentoo-dev] [PATCH 0/2] allow acct-user home directories in /home

2020-01-20 Thread Alec Warner
On Mon, Jan 20, 2020 at 6:20 AM Michael Orlitzky  wrote:

> On 1/20/20 2:02 AM, Ulrich Mueller wrote:
> >> On Mon, 20 Jan 2020, Michael Orlitzky wrote:
> >
> >>   install-qa-check.d: allow acct-user home directories under /home.
> >
> > Nope. As you've been told, /home is site specific and can be setup in
> > multiple ways that are incompatible with the package manager installing
> > things there (the only exception being baselayout creating the directory
> > itself).
>
> I haven't been given a single technical reason why using /home would
> cause a problem. What specific incompatibilities are you talking about?
>

So I can describe in detail one example, but its not running Gentoo; so I'm
not sure if you care in practice.

At work we had sec=krb5 NFS v3 mounted home directories. They were mounted
in /home (via the automounter.) So if these machines ran Gentoo and you
went to do something like "create /home/amavisd" it would fail because the
root user doesn't have the ability to make home directories in /home (uid=0
is mapped to nobody, who doesn't have +w on /home.) All home directories
were created by a business application and there were specific hosts where
root was not squashed (and we used sec=sys instead of krb5) and so root on
the admin host would have +w on /home and not be squashed to nobody.)

In practice in that enterprise environment, if we needed something like
/home/web/ (which I think did exist at one point) we would create a role
account in LDAP (www-data is a common user for example), assign it a uid,
create the homedirectory (/home/web) and it would be owned by
www-data:www-data. Then we would configure the web front ends to use
www-data instead of the normal user (apache or nginx or whatever.)

In practice:
(1) These environments are what I'd consider legacy; if I was crafting an
enterprise environment today I would not design one quite like this[0].
(2) I don't think most people running Gentoo are running these
environments, which is why you don't see many practical objections on the
list. I think it's reasonable to avoid service account homedirs in /home
not because of fancy examples like above (that maybe 10 companies in the
world run) and instead just focus on this idea that "system stuff doesn't
go in /home." Its somewhat arbitrary as mgorny points out earlier in the
thread.

-A

[0] Linux has really poor machine trust by default and while you can build
a ragtag set of primitives to trust machines and identities; I think the
effort is better spent shelling out money for some kind of real identity
management provider that isn't just 'hey here is a uid + ip' which is how
we did things in the 90s man. It was an innocent time ;)



>
> > Quoting FHS-3.0 again:
> >
> > | On large systems (especially when the /home directories are shared
> > | amongst many hosts using NFS) it is useful to subdivide user home
> > | directories. Subdivision may be accomplished by using subdirectories
> > | such as /home/staff, /home/guests, /home/students, etc.
> >
> > So, how are you going to detect if such a scheme is used on the system,
> > and in which subdirectory the amavis user should be placed?
>
> The same way we detect that scheme before setting a home directory to
> /var/lib/whatever, which you may notice, is not under /home/guests or
> anything like that. Does this cause a real technical problem, or is it
> just more FUD?
>
>
> > I also wonder why you would send this patch, when there wasn't a single
> > voice supporting your proposition in the other thread and several
> > opposing ones.
>
> I don't want to just complain without offering a solution.
>
> No one has pointed out any problems with it.
>
> This stuff is already in /home, and I'd like to get off user.eclass
> without introducing a new QA warning for a keepdir file.
>
>


Re: [gentoo-dev] GLEP81 and /home

2020-01-19 Thread Alec Warner
On Sun, Jan 19, 2020 at 11:28 AM Michael Orlitzky  wrote:

> On 1/19/20 2:19 PM, Alec Warner wrote:
> >
> > All I want to do is *set* a user's home directory to /home/foo.
> >
> > Why wouldn't you set the homedirectory to /dev/null then?
> >
>
> Because /dev/null is not /home/foo? Is this a trick question? =)
>
>
 Ahh quoting, I'll try with more context ;)

Earlier you wrote:

 * The daemon DOES NOT need a home directory for its user.
  * I DO NOT want to install anything to anyone's home directory.
  * With respect to user.eclass, I'm proposing that /home be treated
EXACTLY THE SAME as it always has been.
---

So my question is "why not leave the homedir unset, or set it to /dev/null?"
The claim is that the daemon doesn't need a home directory, so why are we
trying to make one?

-A


Re: [gentoo-dev] GLEP81 and /home

2020-01-19 Thread Alec Warner
On Sat, Jan 18, 2020 at 6:50 PM Michael Orlitzky  wrote:

> On 1/18/20 7:21 PM, Rich Freeman wrote:
> > On Sat, Jan 18, 2020 at 6:38 PM Michael Orlitzky  wrote:
> >>
> >> But now users have to follow one more step (create /home/amavis) when
> >> setting up amavisd-new. Is the QA check really assuring a quality user
> >> experience here?
> >>
> >
> > Lots of daemons need a home directory for their users, and usually
> > they manage to get by in /var/lib.  It really seems like a bad
> > practice to start having packages creating stuff in /home.  Certainly
> > I don't want random daemons sticking stuff in /home, which I manage
> >
>
> Let's restart:
>
>   * The daemon DOES NOT need a home directory for its user.
>   * I DO NOT want to install anything to anyone's home directory.
>   * With respect to user.eclass, I'm proposing that /home be treated
> EXACTLY THE SAME as it always has been.
>
> All I want to do is *set* a user's home directory to /home/foo.
>

Why wouldn't you set the homedirectory to /dev/null then?

-A


>
> Some people want to configure amavis to launch a program like pyzor,
> which uses per-user configuration files. If you want to do that, you
> first log in as amavis, and run some command like
>
>   $ pyzor discover
>
> which then finds some servers and creates configuration files for you in
> $HOME/.pyzor.
>
> This is user configuration, not daemon configuration. It doesn't belong
> in the daemon's working directory. In particular, you should not be
> dicking around in the daemon's working directory when you run "su
> amavis..." because what you're doing is irrelevant to the daemon.
>
> Spamassassin itself is a better example than amavis. We already set the
> spamd user's home directory to /home/spamd with user.eclass. We don't
> install anything there, and this works fine. If a human logs into that
> account and generates some configuration in ~/.spamassassin, then it's
> within the spirit of the FHS to have that data stored in /home.
>
> Now, with GLEP 81, we get a QA warning for that, because the eclass
> installs a keepdir file there. I would like things to remain exactly as
> they are, with no QA warning. Otherwise, we have to tell users to move
> /home/spamd to /var/lib/spamd-home or something stupid like that. I can
> think of no good reason to do so.
>
>
> > It seems like the straightforward solution is to stick everything in
> > /var/lib/amavis, and fix things so that everything has the right
> > permissions regardless of install order.
>
> Please stop telling people to do this. Calling chown on the live
> filesystem is rarely safe, and I already fixed one root exploit (bug
> 630836) in the amavisd-new ebuild from the last guy who tried to adjust
> wrong permissions after the fact.
>
>
> > Another option is to have /var/lib/amavis/home and /var/lib/amavis/work.
>
> This is better-looking than /var/lib/amavis-home, but it's still
> violating the spirit of the FHS to avoid triggering a warning on a dummy
> file.
>
>


Re: [gentoo-dev] GLEP81 and /home

2020-01-18 Thread Alec Warner
On Sat, Jan 18, 2020 at 9:52 AM Michael Orlitzky  wrote:

> We forbid packages from installing to /home for good reason: for most of
> history, users (and their home directories) were outside the purview of
> the package manager. But with GLEP81, that's changed: the package
> manager is now in charge of creating each system user's home directory
> and of giving it the correct permissions and ownership.
>
> Is the policy against installing to /home still consistent?
>
> For example: the mail-filter/amavisd-new daemon needs a user, typically
> called "amavis". The daemon also needs a working directory that it can
> write to. The obvious choice for a working directory is /var/lib/amavis,
> but there's a catch: spamassassin, razor, pyzor, et cetera (which are
> all used by amavis) store their configuration in the current user's home
> directory, and not in some daemon-specific location. So "amavis" needs a
> home directory, because that's where much of the configuration for
> amavisd goes.
>
> Where do we put amavis's home directory?
>
>   1 /var/lib/amavis is a bad idea, because it conflicts with the working
> directory (we don't want the two packages to get out of sync, nor do
> we want to keep them in-sync manually).
>
>   2 /var/lib/amavis/home was my next choice, because logically it puts
> the amavisd configuration in a subdirectory of the place where all
> of the other amavis stuff goes, and because it doesn't have the
> same issue that (1) does.
>
> But there's a problem: if we create /var/lib/amavis/home before
> amavisd-new is installed (as happens when you emerge amavisd-new),
> then /var/lib/amavis winds up root:root and the installation of
> amavisd-new doesn't change that. So now amavisd-new doesn't work,
> because it can't write to its working directory.
>
> This is a combination of an implementation detail and the fact that
> the PMS doesn't cover directories; but ultimately, it just doesn't
> work reliably.
>
>   3 /home/amavis also seems fine to me, except for the fact that it's a
> QA violation to install there.
>
> Note that we could always set system users' home directories to
> /home/whatever. It has only become a QA violation with GLEP81 because
> the eclass calls keepdir on the user's home directory.
>
> Should option (3) be viable, or do I go back to the drawing board?
>

I tend to agree that in theory keeping the working directory and home
directory the same is not ideal. However  I'm not really aware of any
practical problems. Haven't we basically run in this configuration for
years? What kind of issues does it pose (outside of "well it sounds like
not the best idea?")

Agreeing with ulm here. I think the potential struggle for (3) is that
conceptually /home is not always system specific. If /home is shared, you
could end up with a bad time (e.g. I *don't* want /home/amavis shared
across all my hosts, how would I manage multiple versions? Multiple
configs?) Often all of /home is mounted and it becomes tricky to build
workarounds. You could do things like "symlink /home/amavisd to some local
path". At work we use /usr/local/google/home/$USER for this purpose; but it
is basically a vestigial artifact from an earlier time than something we
explicitly designed.

Even the symlink solution is not ideal. NFS can present some exciting
reliability issues and so having to touch NFS just to read the symlink
(pointing locally) is probably not something I'd advise at scale; it would
be simpler to change the users homedir to be elsewhere. Note that I'd
expect large scale installs to do this anyway (I wouldn't rely on GLEP81 to
manage users in an enterprise environment) but i'm not sure if the entire
community shares that belief ;)

-A


[gentoo-dev] Perl upgrade on git.gentoo.org

2020-01-16 Thread Alec Warner
It looks like it took, please report bugs on bugs.gentoo.org if you
encounter any weirdness with git.

-A


[gentoo-dev] Archives Test Mail [Please Ignore]

2020-01-14 Thread Alec Warner



Re: [gentoo-dev] Keywordreqs and slacking arch teams

2019-12-28 Thread Alec Warner
On Sat, Dec 28, 2019 at 3:43 AM Kent Fredric  wrote:

> On Sat, 28 Dec 2019 11:35:49 +
> Michael 'veremitz' Everitt  wrote:
>
> > Note:  we're nnot acttually talking about replacing portage here, just
> > creating a tool thiink php script web tthingy)) that will do some of
> > the pre-screeninng wok that AT hate (eg what kensiington did
> with
> > stable-bbot)
> >
> > * with apologies for keyboard/remote-access la creating typo hell.
>
> But, doing that requires viewing realised copies of ebuilds, which
> requires interpreting eclasses and variable interpolation, which
> requires bash sourcing, which requires a mountain of portage hell.
>

> Yes, sharing the stable-bot logic would probably be fine.
>
> But it doesn't use a database AFAIK, it would likely just be making use
> of the MD5Cache (either directly, or indirectly via portage APIs)
>
> But I won't be volunteering, because I won't touch python.
>

You could of course work together with someone else to write the tool?

-A


Re: [gentoo-dev] [RFC] Revisiting GLEP 81 (acct-*) policies (reviews, cross-distro syncing)

2019-12-09 Thread Alec Warner
On Mon, Dec 9, 2019 at 12:17 AM Michał Górny  wrote:

> Hello,
>
> I think the policies proposed in GLEP 81 [1] were overenthusiastic
> and they don't stand collision with sad Gentoo developer reality.
> Instead of improving the quality of resulting packages, they rather
> hamper their adoption and cause growing frustration.
>
> The problems I see today are:
>
>
> 1. Mailing list reviews hamper the adoption of new user packages.
>
> Firstly, there are a few developers who obstructively refuse to
> communicate with others and especially to use the public mailing lists.
> While this is a separate problem, and a problem that needs to be
> resolved, this GLEP can't resolve it.  Of course, there is no reason to
> believe that removing review requirement will actually make them migrate
> their packages but it's at least one obstacle out of the way.
>
> Secondly, even developers capable of communication find the two stage
> request-wait-commit workflow inconvenient.  At any time, there are
> at least a few requests waiting for being committed, possibly with
> the developers forgetting about them.
>
>
> 2. Mailing list reviews don't serve their original purpose.
>
> The original purpose of mailing list reviews was to verify that
> the developers use new packages correctly.  For example, Michael
> Orlitzky has found a lot of unnecessary home directories specified.
> Of course, that works only if people submit *ebuilds* for review.
>
> However, at some points developers arbitrarily decided to send only
> numbers for review.  This defeats the purpose of the review in the first
> place.
>
>
> 3. Cross-distro syncing has no purpose.
>
> One of the original ideas was to reuse UID/GID numbers from other
> distros when available to improve sync.  However, given the collisions
> between old Gentoo UIDs and other distros, other distros themselves,
> non-overlapping user/group names, etc. there seems to be little reason
> to actually do it.  If we even managed some overlap, it would be minimal
> and quasi-random.
>
> While other distros provide a cheap way of choosing new UID/GID, it
> doesn't seem that many people actually use it.  Then we hit pretty
> absurd situations when someone chooses one UID/GID, somebody else tells
> him to use the one from other distro.
>
>
> 4. Assignment mechanism is not collision-prone.
>
> The secondary goal of mailing list reviews is to prevent UID/GID
> collisions.  Sadly, it doesn't work there either.  Sometimes two people
> request the same UID/GID, and only sometimes somebody else notices.
> In the end, people have hard time figuring out which number is the 'next
> free', sometimes they discover the number's been taken when somebody
> else commits it first.
>
>
> All that considered, I'd like to open discussion how we could improve
> things.
>
> My proposal would be to:
>
> a. split the UID/GID range into 'high' (app) and 'low' (system)
> assignments, 'high' being >=100 and 'low' <100 (matching Apache suEXEC
> defaults),
>
> b. UIDs/GIDs in the 'high' range can be taken arbitrarily (recommending
> taking highest free), while in the 'low' range must be approved by QA,
>
> c. no review requirement for the 'high' range, just choose your UID/GID
> straight of uid-gid.txt and commit it.
>

What is the mechanism to keep the uid-gid.txt aligned with tree content? is
there a CI check that says I am using the new acct-* eclasses AND I have a
UID / GID assigned that is not matching uid-gid.txt? I see the CI has
"ConflictingAccountIdentifiers", is this already doing this work (checking
that the ebuild matchines uid-gid.txt), or just scanning the whole tree and
ensuring that 2 packages don't re-use the same ID?

-A


>
> d. strong recommendation to use matching UID/GID for the same user/group
> name.
>
> WDYT?
>
>
> [1] https://www.gentoo.org/glep/glep-0081.html
>
> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] [PATCH] package.deprecated: Create initial template

2019-12-05 Thread Alec Warner
On Thu, Dec 5, 2019 at 8:10 AM Michał Górny  wrote:

> Signed-off-by: Michał Górny 
> ---
>  profiles/package.deprecated | 17 +
>  1 file changed, 17 insertions(+)
>  create mode 100644 profiles/package.deprecated
>
>
This looks great Michał, thanks for putting it together.

-A


> diff --git a/profiles/package.deprecated b/profiles/package.deprecated
> new file mode 100644
> index ..b4803a4ce68f
> --- /dev/null
> +++ b/profiles/package.deprecated
> @@ -0,0 +1,17 @@
> +
> +#
> +# This file specifies packages that are considered deprecated (but not
> +# masked yet).  It will trigger pkgcheck warnings whenever other
> +# packages depend on them.
> +#
> +# When you add an entry to the top of this file, add your name, the date
> +# in the UTC timezone, and an explanation of why something is getting
> +# deprecated.
> +#
> +## Example:
> +##
> +## # Dev E. Loper  (2019-07-01)
> +## # This is no longer supported upstream, please switch to dev-foo/bar.
> +## dev-foo/foo
> +
> +#--- END OF EXAMPLES ---
> --
> 2.24.0
>
>
>


Re: [gentoo-dev] Usefulness of HOMEPAGE=https://www.gentoo.org

2019-12-04 Thread Alec Warner
On Wed, Dec 4, 2019 at 9:26 AM Michał Górny  wrote:

> On Wed, 2019-12-04 at 17:24 +0200, Joonas Niilola wrote:
> > On 12/4/19 5:21 PM, Kent Fredric wrote:
> > > On Wed, 04 Dec 2019 13:36:07 +0100
> > > Michał Górny  wrote:
> > >
> > > > My point is: gentoo.org as a HOMEPAGE sucks.  Please use something
> more
> > > > specific instead.  Even link to gitweb would be more helpful because
> it
> > > > would at least be relevant to the package in question.
> > > I agree so much I would support the addition of a QA check for this.
> > >
> > I take it you haven't checked the CI results lately? Reaction to that
> > probably spawned this ML thread.
> >
> > https://qa-reports.gentoo.org/output/gentoo-ci/output.html
>
> Actually, I've requested that check.  However, I didn't expect that many
> packages to be affected.
>
> Given that it's open season on me lately, and apparently people feel
> offended when bugs are reported for their packages, I've decided to
> start by trying to make people realize the problem globally first.
>

When QA was run by Diego, he suffered some of the same problems. A lot of
this comes down to three factors (IMHO.)

 - Lack of buy-in from developers. When you add a QA thing, you are asking
people to do more work. If they don't agree with the work, they have no
real incentive to do it. I don't see a lot of incentive building here and
so for some efforts adoption of fixes is slow / low. In addition,
expectations are often not set (at all[1]) or not shared with the group
(e.g. QA and the community disagree on the expectation; often in relation
to timelines or end goals.)
 - The above leads to the stick instead of the carrot. Instead of helping
people adhere to the policy and recruiting the community to do the work, QA
takes an adversarial approach where the policy is wielded as a cudgel to
'force' people to do the work. This then leads to the comments like the
above (e.g. "its open season on mgorny") because often forcing people to do
work on a tight timetable does not generate trust or goodwill and
encourages the adversarial relationship between the community and QA.
 - This perception that perfection is required and imperfect packages are
ripe for removal. This again creates this air of anxiety between a package
maintainer and QA where QA can basically invent new reasons to mask
arbitrary[0] packages.

-A

[0] I'm not suggesting this is the intent of the QA team, but it's one
narrative that a non-QA member might have and the QA team is fairly
adversarial and often takes little action to dissuade this narrative from
taking hold.
[1] Some good examples are things like EAPI deprecation
https://archives.gentoo.org/gentoo-dev/message/aef37db23c862865fffdd24071fce1ec.
You notice that Andreas has articulated some goal (no more EAPI2), has
clearly specified the packages that need work, and has encouraged people to
help achieve the goal. Even the tone is positive. I want to help! This is
different from messaging like "Hey you have 7 days to fix your
EAPI2 packages or I will mask them!". This may encourage me to save my
packages (from the evil QA team) but it doesn't make me love the QA team at
all; it makes me feel negative feelings.


> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] Last rites: ruby24-only packages

2019-10-21 Thread Alec Warner
On Sun, Oct 20, 2019 at 10:31 PM Hans de Graaff  wrote:

> On Sun, 2019-10-20 at 12:15 -0700, Alec Warner wrote:
>
> Infra uses thin a lot, is there a replacement?
>
>
> www-servers/puma would be a good replacement.
>
> Feel free to unmask it for now if that helps infra to transition. Upstream
> EOL for ruby 2.4 is March 2020, so we could wait until then if needed.
>

Nope, keep it masked in ::gentoo please.


>
> Hans
>
>


Re: [gentoo-dev] Last rites: ruby24-only packages

2019-10-20 Thread Alec Warner
Infra uses thin a lot, is there a replacement?

On Sun, Oct 20, 2019, 05:41 Hans de Graaff  wrote:

> # Hans de Graaff  (2019-10-19)
> # ruby24-only packages with no reverse dependencies and no recent
> # releases.
> dev-ruby/cocaine
> dev-ruby/debugger-linecache
> dev-ruby/escape_utils
> dev-ruby/http-form_data:1.0
> dev-ruby/ruby-beautify
> dev-ruby/termcolor
> dev-ruby/terrapin
> dev-ruby/vcard
> www-servers/thin
>


Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Alec Warner
On Sat, Oct 19, 2019 at 4:24 PM Joshua Kinard  wrote:

> On 10/18/2019 09:41, Michał Górny wrote:
> > Hi, everybody.
> >
> > It is my pleasure to announce that yesterday (EU) evening we've switched
> > to a new distfile mirror layout.  Users will be switching to the new
> > layout either as they upgrade Portage to 2.3.77 or -- if they upgraded
> > already -- as their caches expire (24hrs).
> >
> > The new layout is mostly a bow towards mirror admins, for some of whom
> > having a 6+ files in a single directory have been a problem.
> > However, I suppose some of you also found e.g. the directory index
> > hardly usable due to its size.
> >
> > Throughout a transitional period (whose exact length hasn't been decided
> > yet), both layouts will be available.  Afterwards, the old layout will
> > be removed from mirrors.  This has a few implications:
> >
> > 1. Users who don't upgrade their package managers in time will lose
> > the ability of fetching from Gentoo mirrors.  This shouldn't be that
> > much of a problem given that the core software needed to upgrade Portage
> > should all have reliable upstream SRC_URIs.
> >
> > 2. mirror://gentoo/file URIs will stop working.  While technically you
> > could use mirror://gentoo/XX/file, I'd rather recommend finally
> > discarding its usage and moving distfiles to devspace.
> >
> > 3. Directly fetching files from distfiles.gentoo.org will become
> > a little harder.  To fetch a distfile named 'foo-1.tar.gz', you'd have
> > to use something like:
> >
> > $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2
> > 1b
> > $ wget http://distfiles.gentoo.org/distfiles/1b/foo-1.tar.gz
> > ...
> >
> >
> > Alternatively, you can:
> >
> > $ wget http://distfiles.gentoo.org/distfiles/INDEX
> >
> > and grep for the right path there.  This INDEX is also a more
> > lightweight alternative to HTML indexes generated by the servers.
> >
> >
> > If you're interested in more background details and some plots, see [1].
> >
> > [1]
> https://dev.gentoo.org/~mgorny/articles/improving-distfile-mirror-structure.html
> >
>
> So the answer I didn't really see directly stated here is, where do new
> distfiles need to go //now//?  E.g., if on woodpecker, I currently cp a
> distfile to /space/distfiles-local.  What is the new directory I need to
> use?  And if mirror://gentoo/${FOO} is going away, for the new distfiles
> target, what would be the applicable prefix to use?
>




>
> Directly using devspace seems like a bad idea, IMHO.  Once long ago, we all
> got chastised for doing exactly that.  Too much possibility of
> fragmentation
> as devs retire or package maintainership changes hands.
>
> I looked at the whitepaper'ish-like writeup, and I kinda don't like using a
> hash-based naming scheme on the new distfiles layout.  I really kind prefer
> breaking the directories up based on the first letter of the distfiles in
> question, factoring case-sensitivity in (so you'd have 52 top-level
> directories for A-Z and a-z, plus 10 more for 0-9).  Under each of those
> directories, additional subdirectories for the next few letters (say,
> letters 2-3).  Yes, this leads to some orphan cases where a distfile might
> live on its own, but from a direct navigation standpoint, it's easy to find
> for someone browsing the distfiles server and easy to predict where a
> distfile is at.
>
> No math, statistical analysis, or deep-rooted knowledge of filesystems
> behind that paragraph.  Just a plain old unfiltered opinion.  Sometimes, I
> need to go get a distfile off the Gentoo mirrors, and being able to quickly
> find it in the mirror root is great.  Having to do hash calculations to
> work
> out the file path will be *really* annoying.
>

So if you want a tool that "downloads a distfile off of the mirrors" we
should be able to build such a utility.

I'm not really sure why that tool needs to be:
*copy DISTFILENAME*
wget distilfes.gentoo.org/$PASTE

It could just `ebuild portageq download $DISTFILENAME or similar.`

-A






>
> --
> Joshua Kinard
> Gentoo/MIPS
> ku...@gentoo.org
> rsa6144/5C63F4E3F5C6C943 2015-04-27
> 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943
>
> "The past tempts us, the present confuses us, the future frightens us.  And
> our lives slip away, moment by moment, lost in that vast, terrible
> in-between."
>
> --Emperor Turhan, Centauri Republic
>
>


Re: [gentoo-dev] The demotivating process of contributing to devmanual

2019-10-15 Thread Alec Warner
On Tue, Oct 15, 2019 at 1:59 PM Michał Górny  wrote:

> On Tue, 2019-10-15 at 16:47 -0400, Mike Gilbert wrote:
> > On Tue, Oct 15, 2019 at 4:35 PM Michał Górny  wrote:
> > > Hello, everyone.
> > >
> > > I'd like to highlight a major problem with devmanual.  For a basic
> > > policy & developer documentation thingie, it's quality is so-so at
> best.
> > > A lot of stuff is missing, lots of things are outdated or even
> > > incorrect.  Not many people are contributing, and those who try quickly
> > > resign.
> >
> > Maybe you should join the project? Especially if you are making major
> > contributions.
>
> Are you suggesting that I join the project and start committing without
> review, or disregarding review?  I have serious doubts on joining
> the project if I am repeatedly proven to be doing things wrong --
> whether the issues were serious or not.
>
> >
> > > Most of my pull requests were apparently approved, so they might be
> > > finally merged some day.
> >
> > I believe all devs have push access to that repo, so you could just
> > push the changes yourself if there are no reasonable objections.
>
> I never realized that.
>
> >
> > Minor mistakes happen, and can be corrected after the fact.
> >
>

One tactic here is to just timebound the reviews. 2 weeks between posting a
PR and getting a review is too long IMHO. Post a PR and say you will merge
it in 72 hours or something. If it's wrong, it can be fixed after the fact
as floppym notes.

If I'm at work and someone has sent me a patch and the patch is good but
there are some minor spelling / grammar fixes they can make I will
basically reply pointing out the problems (so they can fix them) but I also
tell them to merge once the fixes are applied. This means they don't need
to wait for me to "review" the spelling fixes. Obviously there is both
trust (in that I assume they did what I asked) and tooling (we have a tool
where I write comments like "you spelled foobare wrong here, should be
'foobar'" and they have to click "RESOLVE" on each item; you can't submit a
PR with 'unresolved' items open) so there is some pressure to "do the right
thing."

-A


>
> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] [PATCH 1/1] kernel-2.eclass: Support move of genpatches tarballs from /space/distfiles-local to devspace

2019-10-11 Thread Alec Warner
On Thu, Oct 10, 2019 at 3:04 AM Mike Pagano  wrote:

> On Wed, Oct 09, 2019 at 06:23:06PM -0700, Alec Warner wrote:
> >On Wed, Oct 9, 2019 at 10:01 AM Mike Pagano <[1]mpag...@gentoo.org>
> >wrote:
> >
> >  This change will support moving the genpatches tarballs from
> >  /space/distfiles-local to
> >  the devspace ~developer/public_html/dist/genpatches
> >
> >I think it would help if you discussed why we were making this change.
> >(I mean I can guess why, but it's not obvious.)
> >-A
> >Ā
>
> I was informed that use of /space/distfiles-local is deprecated in favor
> of devspace.
>
> https://devmanual.gentoo.org/general-concepts/mirrors/index.html


That policy was made in 2011; so clearly it's not super urgent nor
stringently applied. Which is to say, I'm not sure it needs to be a policy.

-A


>
>
>
>
>
>
> >
> >  Signed-off-by: Mike Pagano <[2]mpag...@gentoo.org>
> >  ---
> >  Ā eclass/kernel-2.eclass | 4 +++-
> >  Ā 1 file changed, 3 insertions(+), 1 deletion(-)
> >  diff --git a/eclass/kernel-2.eclass b/eclass/kernel-2.eclass
> >  index c5f35cd3e..0bc4f35de 100644
> >  --- a/eclass/kernel-2.eclass
> >  +++ b/eclass/kernel-2.eclass
> >  @@ -295,7 +295,9 @@ handle_genpatches() {
> >  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  UNIPATCH_LIST_GENPATCHES+="
> >  ${DISTDIR}/${tarball}"
> >  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  debug-print "genpatches tarball:
> >  $tarball"
> >  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  fi
> >  -Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā GENPATCHES_URI+="
> >  ${use_cond_start}mirror://gentoo/${tarball}${use_cond_end}"
> >  +Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā GENPATCHES_URI+="
> >  ${use_cond_start}[3]
> https://dev.gentoo.org/~mpagano/dist/genpatches/
> >  ${tarball}${use_cond_end}
> >  +Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā
> >  Ā ${use_cond_start}[4]
> https://dev.gentoo.org/~whissi/dist/genpatches
> >  /${tarball}${use_cond_end}
> >  +Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā  Ā
> >  Ā ${use_cond_start}[5]
> https://dev.gentoo.org/~alicef/dist/genpatches
> >  /${tarball}${use_cond_end}"
> >  Ā  Ā  Ā  Ā  done
> >  Ā }
> >  --
> >  2.21.0
> >  --
> >  Mike Pagano
> >  Gentoo Developer - Kernel Project
> >  Gentoo Sources - Member
> >  E-MailĀ  Ā  Ā : [6]mpag...@gentoo.org
> >  GnuPG FPĀ  Ā : EEE2 601D 0763 B60F 848CĀ  9E14 3C33 C650 B576 E4E3
> >  Public Key :
> >  [7]http://pgp.mit.edu:11371/pks/lookup?search=0xB576E4E3=index
> >
> > References
> >
> >1. mailto:mpag...@gentoo.org
> >2. mailto:mpag...@gentoo.org
> >3.
> https://dev.gentoo.org/~mpagano/dist/genpatches/${tarball}${use_cond_end}
> >4.
> https://dev.gentoo.org/~whissi/dist/genpatches/${tarball}${use_cond_end}
> >5.
> https://dev.gentoo.org/~alicef/dist/genpatches/${tarball}${use_cond_end}
> >6. mailto:mpag...@gentoo.org
> >7. http://pgp.mit.edu:11371/pks/lookup?search=0xB576E4E3=index
>
> --
> Mike Pagano
> Gentoo Developer - Kernel Project
> Gentoo Sources - Member
> E-Mail : mpag...@gentoo.org
> GnuPG FP   : EEE2 601D 0763 B60F 848C  9E14 3C33 C650 B576 E4E3
> Public Key :
> http://pgp.mit.edu:11371/pks/lookup?search=0xB576E4E3=index
>
>


Re: [gentoo-dev] [PATCH 1/1] kernel-2.eclass: Support move of genpatches tarballs from /space/distfiles-local to devspace

2019-10-09 Thread Alec Warner
On Wed, Oct 9, 2019 at 10:01 AM Mike Pagano  wrote:

> This change will support moving the genpatches tarballs from
> /space/distfiles-local to
> the devspace ~developer/public_html/dist/genpatches
>

I think it would help if you discussed why we were making this change. (I
mean I can guess why, but it's not obvious.)

-A


>
> Signed-off-by: Mike Pagano 
> ---
>  eclass/kernel-2.eclass | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/eclass/kernel-2.eclass b/eclass/kernel-2.eclass
> index c5f35cd3e..0bc4f35de 100644
> --- a/eclass/kernel-2.eclass
> +++ b/eclass/kernel-2.eclass
> @@ -295,7 +295,9 @@ handle_genpatches() {
> UNIPATCH_LIST_GENPATCHES+=" ${DISTDIR}/${tarball}"
> debug-print "genpatches tarball: $tarball"
> fi
> -   GENPATCHES_URI+="
> ${use_cond_start}mirror://gentoo/${tarball}${use_cond_end}"
> +   GENPATCHES_URI+=" ${use_cond_start}
> https://dev.gentoo.org/~mpagano/dist/genpatches/${tarball}${use_cond_end}
> +   ${use_cond_start}
> https://dev.gentoo.org/~whissi/dist/genpatches/${tarball}${use_cond_end}
> +   ${use_cond_start}
> https://dev.gentoo.org/~alicef/dist/genpatches/${tarball}${use_cond_end};
> done
>  }
>
> --
> 2.21.0
>
> --
> Mike Pagano
> Gentoo Developer - Kernel Project
> Gentoo Sources - Member
> E-Mail : mpag...@gentoo.org
> GnuPG FP   : EEE2 601D 0763 B60F 848C  9E14 3C33 C650 B576 E4E3
> Public Key :
> http://pgp.mit.edu:11371/pks/lookup?search=0xB576E4E3=index
>
>


Re: [gentoo-dev] Re: stable-bot is down. Temporary? Forever? Can we have a contacts page for it?

2019-10-08 Thread Alec Warner
On Tue, Oct 8, 2019 at 4:57 AM Michael Palimaka 
wrote:

> On 10/8/19 7:21 AM, Andreas K. Huettel wrote:
> > In any case, since many people *do* rely on it, maybe we should declare
> it
> > official? [+]
> >
> > And, if that's OK with both of you, move it onto infra hardware?
> >
> > Happy to sponsor both for the next council meeting agenda.
> >
> >
> > [+] At some point the one remaining whiner doesnt count anymore.
> >
>
> In the past, infra has been understandably hesitant to take on new
> services due to staffing issues.
>
> Additionally, I understand that the current infra design does not easily
> allow granular access control, preventing non-infra members from easily
> performing maintenance on individual services.
>
> Has this situation changed? I doubt infra want to take responsibility
> for the bot, and I don't fancy the hassle of trying to find people to
> poke things on my behalf.
>

Things have not changed. We don't need to run the bot, we just need some
clearer contact info for it IMHO.

I don't think the reliability of the bot is really that different from
official infra services, but it was unclear who owned it and so there was
confusion; and I think the confusion is the key thing we are looking to
resolve here.

-A


Re: [gentoo-portage-dev] [PATCH v2] fetch: Support GLEP 75 mirror structure

2019-10-03 Thread Alec Warner
On Thu, Oct 3, 2019 at 9:37 AM Michał Górny  wrote:

> Add a support for the subset of GLEP 75 needed by Gentoo Infra.  This
> includes fetching and parsing layout.conf, and support for flat layout
> and filename-hash layout with cutoffs being multiplies of 4.
>
> Bug: https://bugs.gentoo.org/646898
> Signed-off-by: Michał Górny 
> ---
>  lib/portage/package/ebuild/fetch.py | 139 +++-
>  1 file changed, 135 insertions(+), 4 deletions(-)
>
> Changes in v2: switched to a more classy layout to make the code
> reusable in emirrordist.
>
> diff --git a/lib/portage/package/ebuild/fetch.py
> b/lib/portage/package/ebuild/fetch.py
> index 227bf45ae..18e3d390a 100644
> --- a/lib/portage/package/ebuild/fetch.py
> +++ b/lib/portage/package/ebuild/fetch.py
> @@ -7,12 +7,15 @@ __all__ = ['fetch']
>
>  import errno
>  import io
> +import itertools
> +import json
>  import logging
>  import random
>  import re
>  import stat
>  import sys
>  import tempfile
> +import time
>
>  from collections import OrderedDict
>
> @@ -27,14 +30,17 @@ portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.package.ebuild.doebuild:doebuild_environment,' + \
> '_doebuild_spawn',
> 'portage.package.ebuild.prepare_build_dirs:prepare_build_dirs',
> +
>  'portage.util.configparser:SafeConfigParser,read_configs,NoOptionError',
> +   'portage.util._urlopen:urlopen',
>  )
>
>  from portage import os, selinux, shutil, _encodings, \
> _movefile, _shell_quote, _unicode_encode
>  from portage.checksum import (get_valid_checksum_keys, perform_md5,
> verify_all,
> -   _filter_unaccelarated_hashes, _hash_filter, _apply_hash_filter)
> +   _filter_unaccelarated_hashes, _hash_filter, _apply_hash_filter,
> +   checksum_str)
>  from portage.const import BASH_BINARY, CUSTOM_MIRRORS_FILE, \
> -   GLOBAL_CONFIG_PATH
> +   GLOBAL_CONFIG_PATH, CACHE_PATH
>  from portage.data import portage_gid, portage_uid, secpass,
> userpriv_groups
>  from portage.exception import FileNotFound, OperationNotPermitted, \
> PortageException, TryAgain
> @@ -253,6 +259,130 @@ _size_suffix_map = {
> 'Y' : 80,
>  }
>
> +
> +class FlatLayout(object):
> +   def get_path(self, filename):
> +   return filename
> +
> +
> +class FilenameHashLayout(object):
> +   def __init__(self, algo, cutoffs):
> +   self.algo = algo
> +   self.cutoffs = [int(x) for x in cutoffs.split(':')]
> +
> +   def get_path(self, filename):
> +   fnhash = checksum_str(filename.encode('utf8'), self.algo)
> +   ret = ''
> +   for c in self.cutoffs:
> +   assert c % 4 == 0
>

I'm not quite sure what this assert is doing. I'm not super in favor of
asserts (I'd rather see an exception like raise FooError("..."), but if you
are going to use it please use something like:

assert c %4 == 0, "Some description of why we put this assert here so if it
fires we can do something useful."

+   c = c // 4
> +   ret += fnhash[:c] + '/'
> +   fnhash = fnhash[c:]
> +   return ret + filename
> +
> +
> +class MirrorLayoutConfig(object):
> +   """
> +   Class to read layout.conf from a mirror.
> +   """
> +
> +   def __init__(self):
> +   self.structure = ()
> +
> +   def read_from_file(self, f):
> +   cp = SafeConfigParser()
> +   read_configs(cp, [f])
> +   vals = []
> +   for i in itertools.count():
> +   try:
> +   vals.append(tuple(cp.get('structure', '%d'
> % i).split()))
> +   except NoOptionError:
> +   break
> +   self.structure = tuple(vals)
> +
> +   def serialize(self):
> +   return self.structure
> +
> +   def deserialize(self, data):
> +   self.structure = data
> +
> +   @staticmethod
> +   def validate_structure(val):
> +   if val == ('flat',):
> +   return True
> +   if val[0] == 'filename-hash' and len(val) == 3:
> +   if val[1] not in get_valid_checksum_keys():
> +   return False
> +   # validate cutoffs
> +   for c in val[2].split(':'):
> +   try:
> +   c = int(c)
> +   except ValueError:
> +   break
> +   else:
> +   if c % 4 != 0:
> +   break
> +   else:
> +   return True
> +   return False
> +   return False
> +
> +   def get_best_supported_layout(self):
> +   

Re: [gentoo-portage-dev] [PATCH] fetch: Support GLEP 75 mirror structure

2019-10-03 Thread Alec Warner
On Thu, Oct 3, 2019 at 7:52 AM Michał Górny  wrote:

> Add a support for the subset of GLEP 75 needed by Gentoo Infra.  This
> includes fetching and parsing layout.conf, and support for flat layout
> and filename-hash layout with cutoffs being multiplies of 4.
>
> Bug: https://bugs.gentoo.org/646898
> Signed-off-by: Michał Górny 
> ---
>  lib/portage/package/ebuild/fetch.py | 113 +++-
>  1 file changed, 109 insertions(+), 4 deletions(-)
>
> diff --git a/lib/portage/package/ebuild/fetch.py
> b/lib/portage/package/ebuild/fetch.py
> index 227bf45ae..692efcc01 100644
> --- a/lib/portage/package/ebuild/fetch.py
> +++ b/lib/portage/package/ebuild/fetch.py
> @@ -7,12 +7,15 @@ __all__ = ['fetch']
>
>  import errno
>  import io
> +import itertools
> +import json
>  import logging
>  import random
>  import re
>  import stat
>  import sys
>  import tempfile
> +import time
>
>  from collections import OrderedDict
>
> @@ -27,14 +30,17 @@ portage.proxy.lazyimport.lazyimport(globals(),
> 'portage.package.ebuild.doebuild:doebuild_environment,' + \
> '_doebuild_spawn',
> 'portage.package.ebuild.prepare_build_dirs:prepare_build_dirs',
> +
>  'portage.util.configparser:SafeConfigParser,read_configs,NoOptionError',
> +   'portage.util._urlopen:urlopen',
>  )
>
>  from portage import os, selinux, shutil, _encodings, \
> _movefile, _shell_quote, _unicode_encode
>  from portage.checksum import (get_valid_checksum_keys, perform_md5,
> verify_all,
> -   _filter_unaccelarated_hashes, _hash_filter, _apply_hash_filter)
> +   _filter_unaccelarated_hashes, _hash_filter, _apply_hash_filter,
> +   checksum_str)
>  from portage.const import BASH_BINARY, CUSTOM_MIRRORS_FILE, \
> -   GLOBAL_CONFIG_PATH
> +   GLOBAL_CONFIG_PATH, CACHE_PATH
>  from portage.data import portage_gid, portage_uid, secpass,
> userpriv_groups
>  from portage.exception import FileNotFound, OperationNotPermitted, \
> PortageException, TryAgain
> @@ -253,6 +259,104 @@ _size_suffix_map = {
> 'Y' : 80,
>  }
>
> +
> +def filename_hash_path(filename, algo, cutoffs):
> +   """
> +   Get directory path for filename in filename-hash mirror structure.
> +
> +   @param filename: Filename to fetch
> +   @param algo: Hash algorithm
> +   @param cutoffs: Cutoff values (n:n...)
> +   @return: Directory path
> +   """
> +
> +   fnhash = checksum_str(filename.encode('utf8'), algo)
> +   ret = ''
> +   for c in cutoffs.split(':'):
> +   c = int(c) // 4
> +   ret += fnhash[:c] + '/'
>

When making a path, please use os.path.join()


> +   fnhash = fnhash[c:]
> +   return ret
> +
> +
> +def get_mirror_url(mirror_url, filename, eroot):
> +   """
> +   Get correct fetch URL for a given file, accounting for mirror
> +   layout configuration.
> +
> +   @param mirror_url: Base URL to the mirror (without '/distfiles')
> +   @param filename: Filename to fetch
> +   @param eroot: EROOT to use for the cache file
> +   @return: Full URL to fetch
> +   """
>
+
> +   cache_file = os.path.join(eroot, CACHE_PATH,
> 'mirror-metadata.json')
> +   try:
> +   with open(cache_file, 'r') as f:
> +   cache = json.load(f)
> +   except (IOError, ValueError):
> +   cache = {}
>

I'm a bit worried that we are opening this cache file off of disk every
time we call get_mirror_url(). Can we just cache the contents in memory
between calls; or even better pass the cache in as argument rather than it
be contained in get_mirror_url?


> +
> +   ts, layout = cache.get(mirror_url, (0, None))
> +   # refresh at least daily
> +   if ts < time.time() - 86400:
> +   # the default
> +   layout = ('flat',)
> +
> +   try:
> +   f = urlopen(mirror_url + '/distfiles/layout.conf')
> +   try:
> +   data = io.StringIO(f.read().decode('utf8'))
> +   finally:
> +   f.close()
> +   cp = SafeConfigParser()
> +   read_configs(cp, [data])
> +
> +   for i in itertools.count():
> +   try:
> +   val = tuple(cp.get('structure',
> '%d' % i).split())
> +   if val == ('flat',):
> +   pass
> +   elif val[0] == 'filename-hash' and
> len(val) == 3:
> +   if val[1] not in
> get_valid_checksum_keys():
> +   continue
> +   # validate cutoffs
> +   cutoffs_good = False
> +  

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

2019-09-18 Thread Alec Warner
On Wed, Sep 18, 2019 at 12:15 PM Michael Orlitzky  wrote:

> On 9/18/19 2:04 PM, Alec Warner wrote:
> >
> > I'm actually pretty fine with this wording, upstream has said not to
> > dynamically link in these use cases.
> >
>
> Respectfully, the fact that you're OK with it doesn't make it not BS. It
> reads like "there's no way we can fix this!" when really it means "we
> don't feel like doing this properly!"
>
> Upstreams suggest dumb stuff all the time. We fix it. That's, like, what
> we do here.
>

> >
> > So if the package *maintainer* bumps each package every time it, or a
> > dep has a security issue; then updating will work fine.
> >
>
> Simply not true. If there's a security problem in a dependency and if
> you bump the packages that depend on it... nothing happens. Everyone
> reinstalls the vulnerable dependency, because the vulnerable dependency

is bundled in every single one of those packages.
>


I think the problem I have with this conversation is that I am discussing
things that are technically possible (e.g. we can in fact propagate
security fixes to all go packages, same as dynamically linked packages)
with things we do not think we will do.

If A deps on B and B has a sec vuln we can modify A's go.mod files to
depend on B-next (with security fixes), vendor that in, and bump A.

We don't do this, not because it's not possible, but because it's expensive
and people don't want to do it. The benefit of such a discussion is that
when we don't do this work, we can describe it to end users and say "hey
this is what it takes to run these packages securely, Gentoo has chosen not
to do it, but if you want to use these packages here is the work necessary."

I think that presents a better message than "Upstream is crap" or "these
packages are crap but we are forced to carry them for $reason so use them
at your own risk!".

-A


  1   2   3   4   5   6   7   8   9   10   >