Re: [gentoo-dev] Repository mirrors & CI are looking for a new "maintainer"

2022-03-24 Thread Alec Warner
On Thu, Mar 24, 2022 at 3:41 PM Michał Górny  wrote:
>
> Hi, everyone.
>
> TL;DR: I need someone to take over the job of minimal "maintenance" work
> around repo mirrors & CI.  I can still do the needed code changes, I
> just need replacement for stuff like filing bugs and taking care of
> immediate issues.
>
>
> I've started the repo-mirror-ci project back in 2015 to provide QA
> checks for the Gentoo repositories (as listed in repositories.xml).
> Originally, it was supposed to run repoman (later: pkgcheck) on all
> repos.  You can guess how that went.
>
> Fast-forward, the project roughly broke into three somewhat
> complementary parts:
>
> 1) Repository mirrors that sync, do basic "is it working" checks
> and generate md5-cache (which also catches some ebuild problems) for all
> repos.
>
> 2) Gentoo CI that runs pkgcheck against ::gentoo.

Happy to take this.

>
> 3) Pull request CI that runs pkgcheck against pull requests on GitHub.

Happy to take this too.

Both are part of work we would need to do to migrate CI to gitlab
(should we eventually move stuff to gitlab.)

>
> Nowadays, everything is running on Gentoo Infra, lots of things are
> automated but it still requires some manual maintenance.  Most notably,
> this includes:
>
> a. periodically running a script to file or update bugs about problems
> with overlays,
>
> b. pinging overlay maintainers and removing obsolete overlays if
> necessary (this part seems to have been partially taken over),
>
> c. occasionally dealing with weird failures blocking repos pipeline --
> usually through adding repos to bla... I mean, blocklist until I manage
> to fix the scripts.
>
>
> While admittedly this isn't that much work, I no longer wish to do it.
> I still feel responsible for setting this up, so I'll do my best to keep
> the code working but I need somebody else to do the maintenance work.
>
> If you're interested, please ping me on IRC and I'll give you a quick
> run-around and add you where necessary.  I don't think you need strictly
> to be a dev but you'll need editbugs.
>
> --
> Best regards,
> Michał Górny
>
>



Re: [gentoo-dev] Deprecating repoman

2022-03-11 Thread Alec Warner
On Fri, Mar 11, 2022 at 9:14 AM Peter Stuge  wrote:
>
> Matt Turner wrote:
> > repoman is inferior to other tooling mentioned. The other tooling is
> > actually run in CI.
>
> The problem seems to be that CI is running something other than
> developers run, not the other way around.
>
>
> > Developers should get the same warnings locally as in CI.
>
> I agree. And developers and their tools should not have to bend to
> whatever CI does, I think the other way around makes more sense.
>
>
> CI doesn't use repoman because of performance issues.

I think the problem is that making committers use the CI tool is
technically a fairly straightforward change; while everything you
discuss further in the thread is not; because it implies people will
work on improving tools that have shown little or no progress on
improving in the 15 years I've been in Gentoo.

>
> What if pkgcore evolves to provide a portage-compatible API?

No API is planned and no one is working on it.

>
> Then CI could use repoman without performance problems, and
> developers would also enjoy improved performance, without spending
> time on replacing tooling which already works for them.

The benefit of getting people to change their behavior is that the
benefits (less breakage, better tooling) are achieved now; as opposed
to in some hypothetical future where repoman is improved.
Note that we have been waiting for 'improved repoman' for about 8
years; so..I'm not willing to bet on it when we have better tooling
that works today and the primary complaint is that...what exactly?

The new workflow with pkgcheck was announced at the end of 2019:
https://blogs.gentoo.org/mgorny/2019/12/12/a-better-ebuild-workflow-with-pure-git-and-pkgcheck

It's been 2 years, I think we can bring everyone into the fold here.

>
> Looking into the future then maybe portage could even come to use
> pkgcore for the low-level things that pkgcore does, then even users
> could enjoy improved performance.

Many things could happen but again, if you talk to people who work on
pkgcore; there is no concrete plan for this. So I don't see why we are
betting on it happening.

If you read radhermit's blog:
https://pkgcraft.github.io/posts/timesink-chronicles/ you can get one
take on the project and why it struggled.

-A

>
>
> Right?
>
> //Peter
>



Re: [gentoo-dev] Deprecating repoman

2022-03-10 Thread Alec Warner
On Thu, Mar 10, 2022 at 10:28 AM Joshua Kinard  wrote:
>
> On 3/9/2022 16:47, Matt Turner wrote:
> > On Wed, Mar 9, 2022 at 1:37 PM Matthias Maier  wrote:
> >>
> >>
> >> Just a quick though:
> >>
> >> Looking at the man page of repoman it doesn't look to difficult to
> >> replicate the behavior with pkgcheck. Meaning, we could think of
> >> creating a drop-in replacement for "repoman [full]" (which would just
> >> call pkgcheck) and "repoman commit" which actually does much more than
> >> just prepending the git commit summary line: repoman commit does
> >>
> >>- update the manifest
> >>- bail out if files are not correctly "git add"ed
> >>- add the output of [pkgcheck] as a comment to the git commit
> >>  description
> >
> > I wouldn't block anyone from doing this, but it's not something I'm
> > personally interested in pursuing. I see very little value here.
>
> First, you're trying to justify replacing repoman on an entirely subjective
> opinion of "I think  is superior", then when you get feedback on the
> idea, you dismiss that feedback with more opinion.
>
> Why do you not see value here?  The actions described are actually quite a
> few useful steps in the process of checking a change into the tree.  If you
> expect developers to do those steps on their own, that increases, not
> decreases, the chances of making a mistake.  Or are these steps already
> handled by one of the other scripts in the replacement packages you propose?
>  If so, please say so and identify which one(s).
>
> My opinion is that any tool that replaces repoman should, at a minimum,
> replace like functionality with like functionality, plus benefits or
> enhancements.  This looks more like a step backwards, not a step forwards.

I'd be interested in hearing your workflow, so we can capture it in
the table (mentioned earlier) so its clear how your existing workflow
will work with the new tools (or perhaps there is a gap, or we need to
craft / add additional tools?) I agree on the face it may not be
obvious what workflows look like.

-A

>
> --
> Joshua Kinard
> Gentoo/MIPS
> ku...@gentoo.org
> rsa6144/5C63F4E3F5C6C943 2015-04-27
> 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943
>
> "The past tempts us, the present confuses us, the future frightens us.  And
> our lives slip away, moment by moment, lost in that vast, terrible 
> in-between."
>
> --Emperor Turhan, Centauri Republic
>



Re: [gentoo-dev] Deprecating repoman

2022-03-10 Thread Alec Warner
On Thu, Mar 10, 2022 at 10:27 AM Joshua Kinard  wrote:
>
> On 3/9/2022 16:00, Matt Turner wrote:
> > I'd like to deprecate and ultimately remove repoman. I believe that
> > dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
> > are far superior replacements, and it makes sense to have people using
> > the same tool and seeing the same warnings as in the CI.
> >
> > Are there any useful checks or behaviors of repoman that are missing
> > from pkgcheck and pkgcommit?
> >
> > Thanks,
> > Matt
>
> repoman has been //the// goto tool for checking in a change since before I
> was a developer in 2003.  It does everything we need, in one simple tool.
> Your proposal looks to replace repoman's functionality (and snark) with two
> or more packages, including tools from a package named after a fellow 
> developer.
>
> Additionally, "I believe that  are far superior replacements" is an
> entirely subjective opinion and, frankly, completely invalid as
> justification to replace a tool that has worked fine for the last 20+ years.
>  What is so fundamentally broken about repoman that cannot be fixed such
> that it needs total replacement by multiple independent tools?  Please
> document. the pros and cons here so that we can all make an informed decision.

So here is the more basic argument, you can agree or disagree.

*The goal I want is for people to commit to the tree and not break things.*

If we could accomplish this with no tooling at all, that would be
great. Sadly humans are fallible and so we have tools to check their
work. Currently this tooling has two parts:
 - pre-push tooling that you run prior to pushing.
 - post-push CI tooling that informs you when you break the tree.

So holding to our goal of not breaking the tree, it's better for
developers to run the same QA tool in pre-push that CI is using,
because our metric for 'breaking the tree' is the CI tool and if the
CI tool passes locally, there is a strong likelihood it will not break
in CI either. Note this argument is generic, I'm not even saying which
tools are in use, or which tools are better, or anything like that.

Here we see Matt discussing a workflow he finds frustrating.
 - A developer does a push.
 - Their push breaks CI.
 - He inquires about their workflow.
 - He learns they did not run the CI tool in their pre-push workflow.
 - He tells them they should run the CI tool in their pre-push workflow.
 - This happens many times, causing this thread.

The point of the thread then is to convince people to run the CI tool
in pre-push, as a matter of policy, to reduce tree breakage and reduce
the occurrence of the above conversation.

So the generic argument above, we now get into the specifics.

>
> I'm not opposed to making our tools better, but I think before anything can
> replace the RepoMan, several more boxes need to be ticked:
>
> - functionality from multiple tools should be packaged into a single tool.
>   * caveat: at least provide a wrapper that, using args, can invoke the
> individual tools if needed.

I do not believe it's reasonable to provide a 'drop-in' backwards
compatibility tool guarantee.
I believe we should provide a table of common repoman actions, with a
mapping to their replacements; so that users can understand how to do
similar actions.

>
> - app-portage/mgorny-dev-scripts needs a new name.  It's fine if it's
> intended to only be a collection of useful scripts for individual developers
> on an as-needed basis, but if it's to be the Official Tool(TM), then it
> needs to have a proper name.  If not all of the scripts contained within it
> are applicable to the sole function of checking a change into the tree, then
> only the scripts that deal with change validation and committing should be
> broken out into a separate package, and ostensibly combined with any other
> tools/scripts into a single package, and preferably a single tool, to get
> the job done.

I don't consider this a blocker, but I think it's mostly a discussion
between mattst88 and mgorny.

>
> - all of our developer documentation needs to be updated to reference the
> new tool and the new way of doing things, as well as a warning not to use
> repoman any further after a set date.  Additionally, a news item is probably
> advisable so that developers and proxy maintainers alike can get the message.

We should hold Matt accountable for updating any relevant development
documentation, including the table I mentioned above.
I'm not sure a news item is strictly necessary, we might just p.mask
repoman with a link to the guide that Matt will need to write about
how repoman is being replaced.

>
> - we need the snark.  Gentoo is all about personality, and RepoMan has been
> scouring our neighborhoods for two decades and change, and while some may
> think this is utterly frivolous, I will actually miss seeing those messages
> on the console every time I commit a change.  They give the RepoMan
> personality and a soul, and thus, contribute to the 

Re: [gentoo-portage-dev] [PATCH 2/4] portage.eapi: use tuple instead of str for namedtuple definition

2022-03-08 Thread Alec Warner
Looks merge-able.

On Wed, Feb 23, 2022 at 8:15 PM Matt Turner  wrote:
>
> From: "Wolfgang E. Sanyer" 
>
> Reviewed-by: Matt Turner 
> Signed-off-by: Wolfgang E. Sanyer 
> ---
>  lib/portage/eapi.py | 52 -
>  1 file changed, 33 insertions(+), 19 deletions(-)
>
> diff --git a/lib/portage/eapi.py b/lib/portage/eapi.py
> index adee87d00..18069b04b 100644
> --- a/lib/portage/eapi.py
> +++ b/lib/portage/eapi.py
> @@ -288,25 +288,39 @@ def eapi_has_sysroot(eapi):
>
>  _eapi_attrs = collections.namedtuple(
>  "_eapi_attrs",
> -"allows_package_provided "
> -"bdepend "
> -"broot "
> -"dots_in_PN dots_in_use_flags "
> -"exports_AA "
> -"exports_EBUILD_PHASE_FUNC "
> -"exports_ECLASSDIR "
> -"exports_KV "
> -"exports_merge_type "
> -"exports_PORTDIR "
> -"exports_replace_vars "
> -"feature_flag_test "
> -"idepend iuse_defaults iuse_effective posixish_locale "
> -"path_variables_end_with_trailing_slash "
> -"prefix "
> -"repo_deps required_use required_use_at_most_one_of "
> -"selective_src_uri_restriction slot_operator slot_deps "
> -"src_uri_arrows strong_blocks use_deps use_dep_defaults "
> -"empty_groups_always_true sysroot",
> +(
> +"allows_package_provided",
> +"bdepend",
> +"broot",
> +"dots_in_PN",
> +"dots_in_use_flags",
> +"exports_AA",
> +"exports_EBUILD_PHASE_FUNC",
> +"exports_ECLASSDIR",
> +"exports_KV",
> +"exports_merge_type",
> +"exports_PORTDIR",
> +"exports_replace_vars",
> +"feature_flag_test",
> +"idepend",
> +"iuse_defaults",
> +"iuse_effective",
> +"posixish_locale",
> +"path_variables_end_with_trailing_slash",
> +"prefix",
> +"repo_deps",
> +"required_use",
> +"required_use_at_most_one_of",
> +"selective_src_uri_restriction",
> +"slot_operator",
> +"slot_deps",
> +"src_uri_arrows",
> +"strong_blocks",
> +"use_deps",
> +"use_dep_defaults",
> +"empty_groups_always_true",
> +"sysroot",
> +),
>  )
>
>
> --
> 2.34.1
>
>



Re: [gentoo-portage-dev] [PATCH 1/4] portage.dep.Atom: Clean up __new__ parameters

2022-03-08 Thread Alec Warner
maybe *unused_args, **unused_kwargs, unsure on the style guide for
that (normally its _)

But feel free to merge as-is.

-A

On Wed, Feb 23, 2022 at 8:15 PM Matt Turner  wrote:
>
> From: "Wolfgang E. Sanyer" 
>
> Reviewed-by: Matt Turner 
> Signed-off-by: Wolfgang E. Sanyer 
> ---
>  lib/portage/dep/__init__.py | 12 +---
>  1 file changed, 1 insertion(+), 11 deletions(-)
>
> diff --git a/lib/portage/dep/__init__.py b/lib/portage/dep/__init__.py
> index 3b3577025..13c0f4ef7 100644
> --- a/lib/portage/dep/__init__.py
> +++ b/lib/portage/dep/__init__.py
> @@ -1489,17 +1489,7 @@ class Atom(str):
>  def __init__(self, forbid_overlap=False):
>  self.overlap = self._overlap(forbid=forbid_overlap)
>
> -def __new__(
> -cls,
> -s,
> -unevaluated_atom=None,
> -allow_wildcard=False,
> -allow_repo=None,
> -_use=None,
> -eapi=None,
> -is_valid_flag=None,
> -allow_build_id=None,
> -):
> +def __new__(cls, s, *args, **kwargs):
>  return str.__new__(cls, s)
>
>  def __init__(
> --
> 2.34.1
>
>



Re: [gentoo-dev] Feature request: auto-CC for bugs modified via commit tags

2022-01-25 Thread Alec Warner
On Tue, Jan 25, 2022 at 9:00 AM Michael Orlitzky  wrote:
>
> Can I request that Bug: and Closes: tags in our commits automatically
> CC the committer on the bug that is modified?
>
> Use case: I often fix (sci-*) bugs that I'm not CCed on, and a user
> will leave a comment like "it still crashes on x86" that I never see.
> Of course, I could manually CC myself on every bug. But that will send
> everyone an extra email, and is forgettable. Plus, avoiding the manual
> step is kind of the point of the automation, right?

Just to clarify here:
For your own commits (e.g. fixing a package you own) you are already
typically on the bug..right?
I assume the major use case here is proxying commits for others (where
they are on the bug, but you are not, either directly, or via an
alias?)

>
> One potential downside is that the commit author could wind up CCed
> twice via an alias, but that could be solved with a sufficiently clever
> implementation. Or disregarded if it's not too much of a problem in
> practice; the bugs will usually be closed, after all.

The hooks are all public:

https://gitweb.gentoo.org/infra/githooks.git/tree/local/postrecv-bugs

Submit a patch and we can update the hook.

-A

>
>
>



Re: [gentoo-dev] [RFC] making rust-bin ordered first in virtual/rust

2022-01-21 Thread Alec Warner
On Mon, Jan 17, 2022 at 3:24 PM Georgy Yakovlev  wrote:
>
> Hi,
>
> I've been approached multiple times with that request, and a lot of
> time I see new users completely destroyed by rust build time and disk
> space requirements.

What does this actually mean? Can we more clearly document the negatives?

Is it that Rust takes a long time to build? This seems like just a
consequence of a sourced based distribution. Many things take a long
time to build. I suspect rust is not the only package and again we are
back to staffing binhost like projects for much of the tree so that
this particular complaint can be mitigated, not just for rust, but for
any package.

Is it that Rust build fails because users lack the resources to build
it? We should be using check-reqs here to detect this pre-build and
give the user choices (like telling them rust can't be compiled from
source on their machine and to use rust-bin.)

I tend to empathize more with Peter Böhm later in the thread, in that
Gentoo is sourced based and I'd expect portage to be installing things
from source by default. Building from source is resource intensive and
expensive, yes. I think if you want a binary-based distro you are
using the wrong distro at the moment.

-A

>
> WDYT about switching order of rusts in a virtual?
>
> RDEPEND="|| (
> ~dev-lang/rust-${PV}
> ~dev-lang/rust-bin-${PV}
> )"
>
>
> becomes
>
> RDEPEND="|| (
> ~dev-lang/rust-bin-${PV}
> ~dev-lang/rust-${PV}
> )"
>
>
> Existing installs should be unaffected ofc.
> But portage may prefer to depclean rust and not rust-bin if both are
> present.
> Users who wish to use source version at all times can just add it to
> world file.
>
> I see both positives and negatives of doing that, but would like to
> reach out to community first.
>
> Thanks!
>
> --
> Georgy
>



Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

2022-01-05 Thread Alec Warner
On Tue, Jan 4, 2022 at 3:03 PM Sam James  wrote:
>
>
>
> On 4 Jan 2022, at 22:58, Sam James  wrote:
>
> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> amount of RAM available (uses amount declared as needed
> in the ebuild). Typically should be ~2GB per job.
>
> Bug: https://bugs.gentoo.org/570534
> Signed-off-by: Sam James 
> ---
> eclass/check-reqs.eclass | 42 +---
> 1 file changed, 39 insertions(+), 3 deletions(-)
>
>
> Note that we discussed this on GitHub a bit when I just posted it there
> for some rough feedback: https://github.com/gentoo/gentoo/pull/23311.
>
> I think this is valuable for reducing invalid bug reports from OOM and
> easing user experience.
>
> Still kind of a WIP/rough draft, but may be ready in this state. Need
> more testing, so not planning on pushing yet or anything.

I'm still not sure I grasp why we cannot make OOMs easier to discover
from portage.

Most packages don't even use check-reqs, so your solution is very
narrow (and I get why, because you get a lot of bug reports from the
big packages that do use it.)

Can we write a build log analyzer?

-A

PS: If this was a global change I'd downvote it. It's only for
check-reqs though and most packages don't use check-reqs; I don't
really care. I'd be concerned about adopting this kind of approach
wider; its very much a bandaid.



Re: [gentoo-dev] Rationalizing USE flags by narrowing the scope of them.

2022-01-03 Thread Alec Warner
On Mon, Jan 3, 2022 at 7:39 PM Sam James  wrote:
>
>
>
> On 3 Jan 2022, at 17:16, Alec Warner  wrote:
>
> [snip]
>
>
> I'm trying to understand your principles here. Like on what basis do
> you remove or add flags (in general).
>
> I want to remove:
> - bash-completion
>
>
> FWIW, I've managed to remove basically all instances of {bash,zsh}-completion
> and made upstream PRs (all of one of which have been merged!) for fixing
> `./configure` behaviour accordingly.
>
> This is in align with our small files policy: 
> https://projects.gentoo.org/qa/policy-guide/installed-files.html#pg0301.
>
> - acl
> - ldap
>
>
> ACL is kind of similar to what Ionen said for PAM, i.e. sometimes
> people may want to turn it off and it makes sense to expose
> this option for those who do, but we don't need to try support it.

I feel like 'acl' or 'pam' or 'policykit' are not really USE flags in
the general sense. They are more like profile settings. You don't want
to toggle these flags, you want to switch profiles. I think
conceptually profile migrations are larger changes that require a
rebuild of a bunch of stuff, and typically have downtime (e.g. where
your system isn't expected to work the entire time.)

There are practical problems with profile proliferation, but I think
it is closer to what these flags represent, if that makes sense.

-A


>
> I think some of this kind of comes back to "how do we better
> make clear what is/isn't okay (supported)to customise?"
>
> LDAP is a fun one because IIRC we had it enabled by default
> for too long and it didn't really break anything when we turned
> It off.
>
> Overall, I think we kind of come back to this idea of
> trying to just set better IUSE defaults rather than
> in profiles, so that it's per-package where possible.
>
> - policykit
>
>
> This is a reasonable flag to keep given the heavy polkit
> dependency of spidermonkey (for now) and it's somewhat
> heavy-handed as a concept anyway.
>
> - readline
>
>
> readline is a tricky one because of its relation with libedit too.
>
> - sound
>
> (Part of this is just to have a meta discussion so we settle on some
> driving principles on why we keep one flag over the other.)
>
> I can easily craft a narrative for getting rid of ipv6, for example,
> but I cannot really craft a good narrative for getting rid of pam, or
> policykit, or ldap as flags. So why do we keep some and remove others?
>
>
>
> It's very hard to quantify :(
>
>
> Best,
> sam
>



Re: [gentoo-dev] Rationalizing USE flags by narrowing the scope of them.

2022-01-03 Thread Alec Warner
On Mon, Jan 3, 2022 at 4:29 PM Michael Orlitzky  wrote:
>
> On Mon, 2022-01-03 at 21:29 +0100, Piotr Karbowski wrote:
> >
> > Perhaps the 'pam' example was extreme, but ipv6, or threads as Sam
> > shared, does not make sense to be togglable.
> >
>
> Many packages need their ipv6 code disabled if the kernel has no ipv6
> support, and enabling ipv6 in the kernel is a pointless security risk
> for pretty much anyone in the United States.

https://www.google.com/intl/en/ipv6/statistics.html#tab=per-country-ipv6-adoption

Go troll somewhere else?

-A

>
>
>



Re: [gentoo-dev] Rationalizing USE flags by narrowing the scope of them.

2022-01-03 Thread Alec Warner
On Sat, Jan 1, 2022 at 2:22 PM Piotr Karbowski  wrote:
>
> Hi,
>
> I'd like to get some insight how others see the concept of narrowing the
> scope of USE flags in Gentoo.
>
> Taking a quote from devmanual:
>
>> USE flags are to control optional dependencies and settings which
> the user may reasonably want to select.
>
> I'd like to focus on the 'reasonably want' here. While it is commonly
> agreed on that we interface as USE flags only things that make sense to
> be togglable, it is not always the case. It is not uncommon to see
> packages that puts every possible option as USE flag which hardly
> benefit anyone in some cases.
>
> It creates artificial choice of USE flag that makes as much sense as
> building and trying to use solar-powered night vision googles. Possible
> to be engineered, but makes absolute no sense to exist, yet, there will
> be someone who will go with it and then things will not work in desired
> way, bugs will be reported, effort will be wasted on investigation and
> patching things up.
>
> As example I'd like to use 'ipv6' USE flag, at the moment of writing
> this email there's 351 ebuilds in tree that expose ipv6 as USE flag,
> allow it to be disabled.
>
> The thing is, it's 2022, and it does not make any sense to *not* support
> IPv6, even if one does not connect to any network with IPv6, there's no
> harm to just have it there.
>
> While I am all for choice, I am for choice on things that do make sense.
> For instance, Linux kernel can be built with CONFIG_MULTIUSER=n, someone
> could argue that since Linux kernel, that is user-configured in Gentoo,
> can be built without support for other than UID 0, then Gentoo should
> support it. One of the extreme examples of not supporting something that
> does not make sense to be supported.
>
> Beside 'ipv6', there are other USE flags that I have on mind. 'pam'
> being another of them.
>
> Whats your view on it?

I'm trying to understand your principles here. Like on what basis do
you remove or add flags (in general).

I want to remove:
 - bash-completion
 - acl
 - ldap
 - policykit
 - readline
 - sound

(Part of this is just to have a meta discussion so we settle on some
driving principles on why we keep one flag over the other.)

I can easily craft a narrative for getting rid of ipv6, for example,
but I cannot really craft a good narrative for getting rid of pam, or
policykit, or ldap as flags. So why do we keep some and remove others?

-A

>
> -- Piotr.
>



Re: [gentoo-dev] [PATCH] profiles/default/linux: set gl_cv_type_time_t_bits_macro=no

2021-12-17 Thread Alec Warner
Can you put the bug # in the comment in the file?

On Fri, Dec 17, 2021, 09:42 Mike Gilbert  wrote:

> This is intended to prevent packages from automatically switching to
> 64-bit time_t on 32-bit ABIs. Making this switch in an uncontrolled
> manner will lead to inconsistent library ABIs that fail at runtime.
>
> At a later time, we will introduce new profiles to enable 64-bit time_t
> distro-wide.
>
> https://wiki.gentoo.org/wiki/Project:Toolchain/time64_migration
>
> Bug: https://bugs.gentoo.org/828001
> Signed-off-by: Mike Gilbert 
> ---
>  profiles/default/linux/make.defaults | 4 
>  1 file changed, 4 insertions(+)
>
> diff --git a/profiles/default/linux/make.defaults
> b/profiles/default/linux/make.defaults
> index 6ae7cf297cf..53ace7e229c 100644
> --- a/profiles/default/linux/make.defaults
> +++ b/profiles/default/linux/make.defaults
> @@ -53,3 +53,7 @@ VIDEO_CARDS="dummy fbdev v4l"
>  # Note that adding LDFLAGS="-Wl,-O1 ${LDFLAGS}" breaks
> dev-util/boost-build
>  # because of whitespace.
>  LDFLAGS="-Wl,-O1 -Wl,--as-needed"
> +
> +# Mike Gilbert  (2021-12-17)
> +# Prevent automagic use of 64-bit time_t.
> +gl_cv_type_time_t_bits_macro="no"
> --
> 2.34.1
>
>
>


Re: [gentoo-dev] Common options missed in OpenRC declarative scripts and how to improve them

2021-12-02 Thread Alec Warner
On Thu, Dec 2, 2021 at 4:34 AM Michael Orlitzky  wrote:
>
> On 2021-12-01 21:02:20, Brian Evans wrote:
> > After a cursory scan of the Gentoo repository, I've noticed an
> > overabundance of start_stop_daemon_args being declared in scripts committed.
> >
> > I would like to draw attention and see if we can clean these up together.
>
> A lot of this is covered in the service script guide:
>
>   https://github.com/OpenRC/openrc/blob/master/service-script-guide.md
>
> There's a 2.5-year old bug to mention it in the devmanual:
>
>   https://bugs.gentoo.org/684354
>

Can we automate any of it? Emit QA warnings? etc.

-A



Re: [gentoo-dev] rfc: allow -1 for ACCT_USER_ID and ACCT_GROUP_ID in ::gentoo

2021-11-30 Thread Alec Warner
On Tue, Nov 30, 2021 at 10:16 PM Jaco Kroon  wrote:
>
> Hi,
>
> On 2021/12/01 03:32, William Hubbs wrote:
> > This is the part of this that I don't understand. If we aren't enforcing
> > an ID, why do we care which ID to try first? It seems to be an
> > unnecessary step since users can pick the IDs they want by putting
> > settings in make.conf.
>
> Because when running clusters of hosts it's useful to have the UIDs for
> "system" users match.  Yes, I know this won't match in a multi-distro
> setup, but at least for those of us with clusters consisting only of
> Gentoo hosts it will *usually* match.  Changing these are possible, but
> a nuisance, so having it "just work" for the usual case is great IMHO.

So questions from my side are:
Does your cluster not have human users?
Do the userids for the human users also not have to match between
hosts in the cluster?

-A

>
> If I'm not mistaken there is a setting to REQUIRE the ID, and that could
> even be set from make.conf, or env/ so for those packages that we care
> about that (eg, mailman running on top of glusterfs) we use that.
>
> Kind Regards,
> Jaco
>
>



Re: [gentoo-dev] rfc: allow -1 for ACCT_USER_ID and ACCT_GROUP_ID in ::gentoo

2021-11-29 Thread Alec Warner
On Mon, Nov 29, 2021 at 2:25 AM Ulrich Mueller  wrote:
>
> >>>>> On Mon, 29 Nov 2021, Alec Warner wrote:
>
> > - If Gentoo adds an acct-user/foo user, and that user already exists
> > on my system with the wrong UID: the eclass dies[0].
>
> I don't think that's correct. The eclass will just use the already
> existing UID then (except for the very few acct-user packages that
> define ACCT_USER_ENFORCE_ID).
>
> > - If Gentoo adds an acct-user/foo user, with uid=12345, and that uid
> > is assigned to a user on my system already, the eclass dies.
>
> Similar to above, the eclass will dynamically allocate another UID that
> is free.

Oh good I misread it, you are right; my apologies.

>
> > - Some environments are very old, and so real users have unexpected
> > uids; this includes Gentoo itself.
> >- Gentoo (the community) used to allocate UIDs to devs in the
> > 500-1000 range and we have 17 active developers with UIDs in that
> > range. So for example if we allocate one of these UIDs to an acct-*
> > package; that package will not be installable on woodpecker without
> > modification because those UIDs are already taken.
>
> See above.
>
> Also, why would one allocate UIDs in the 500..999 range (1000 is fine,
> actually)? Gentoo always had UID_MIN=1000 and SYS_UID_MAX=999.

A bunch of reasons.
 - In the case of gentoo.org specifically I am guessing bugs and / or
ignorance (as we discussed on IRC.) enewuser / useradd / the normal
utilities lack the permissions to add users (because they cannot write
to LDAP without credentials) and so currently we have a tool
(perl_ldap); from 2006 onward it looks for the highest uidNumber in
LDAP and adds 1 to it. I don't have the source code for earlier
versions, but code comments implied the uids were entered by people;
not machines. People are really bad at consistently allocating UIDs
and are bad at following standards :)
 - In my previous work, the uid automation would routinely have bugs
(we did not have good unit or functional testing) and often the uid
range requirements were either not implemented (oops) or were buggy
(also oops.) We often fixed weird bugs by hand (if we noticed that
e.g. an account had some weird problem and it was someone's first day;
redoing their account is cheap.) But if the bug was in the past; it
was often too expensive to fix anything; so our user accounts had many
exciting quirks of names, odd assignments, etc.

This is why I say that conceptually the 'identity provider' is
external to Gentoo (because we all have our weird site-specific
quirks.) As you note above though, most acct-* packages will not break
and will just assign some other uid / gid; so only the FORCE_ID
packages matter and there are only 3 of those...so I mostly concede on
that basis provided we avoid adding more FORCE'd packages.

-A

>
> Ulrich



Re: [gentoo-dev] rfc: allow -1 for ACCT_USER_ID and ACCT_GROUP_ID in ::gentoo

2021-11-28 Thread Alec Warner
On Sun, Nov 28, 2021 at 8:07 PM Michał Górny  wrote:
>
> On Sun, 2021-11-28 at 16:31 -0600, William Hubbs wrote:
> > All,
> >
> > I want to discuss why we ban -1 as the ACCT_USER_ID and ACCT_GROUP_ID 
> > setting
> > for all acct-user and acct-group packages in ::gentoo.
> >
> > Here are my thoughts about it.
> >
> > - As Gordon pointed out, it isn't necessary for us to care about UIDS/GIDS
> >   most of the time.
> >
> > - I realize that our settings are suggestions, but the values we can
> >   suggest are not infinite. We have run out once, and it is only a matter of
> >   time until we do again.
> >
> > - If an end user needs to care about the UID/GID, they can easily override
> >   the settings in make.conf.
> >
> > In short, I don't think we should be forcing maintainers to pick a
> > specific UID/GID for every package that needs a user/group. Most of the
> > time they can set ACCT_USER_ID and ACCT_GROUP_ID to -1.
> >
> > Thoughts? In particular, I want to hear from folks who disagree with me
> > about using -1 in the main tree for most packages.
> >
>
> Let me put this bluntly.  Yes, most users don't care.  However:
>
> 1) if we don't assign static UIDs/GIDs, the few users who care are gonna
> be in hell having to assign them all manually.  Every single one of
> them, on every one of their systems.

I think the challenge here for me is twofold:

 - I need some source of truth in gid / uids. Most places already have
one (some kind of identity provider.)
 - I need some way to distribute the truth to my machines. In industry
we use nsscache, or sssd, or ldap, or nis+, or yp, or samba, or you
hardcode uid / gids in our configuration management pipeline (ansible,
chef, puppet, our container build scripts!) For instance Gentoo Infra
does a mix of LDAP (our source of truth for uid / gid) and nsscache
(to distribute the uids and gids to machines) as well as hardcoded uid
/ gids (often for services.)

Many people have both of these already. They have some identity
provider (LDAP, AD, FreeIPA, a yaml file!) and they have some way of
getting those identities to their boxes. So when you say things like
"well these people have to do this manually" I struggle to really
understand who these people are; and how things like a yaml file +
ansible script is not sufficient to address this problem for them.
Syncing uid / gids across machines is a solved problem. Some of the
solutions (freeIPA, LDAP) are gross for home use; but ansible is
pretty lightweight, for example.

>
> 2) if we do assign static UIDs/GIDs... what's the problem, again?
> Little extra work for a few devs?

I think we are back to like "why do we care about the uid / gid ranges
at all?" and the answer is because many people have an existing
identity scheme and the gentoo scheme interacts poorly with it.

- If Gentoo adds an acct-user/foo user, and that user already exists
on my system with the wrong UID: the eclass dies[0].
- If Gentoo adds an acct-user/foo user, with uid=12345, and that uid
is assigned to a user on my system already, the eclass dies.
- Some environments are very old, and so real users have unexpected
uids; this includes Gentoo itself.
   - Gentoo (the community) used to allocate UIDs to devs in the
500-1000 range and we have 17 active developers with UIDs in that
range. So for example if we allocate one of these UIDs to an acct-*
package; that package will not be installable on woodpecker without
modification because those UIDs are already taken.
   - Other organizations are similarly encumbered with state; and
these systems do not interact well with the current defaults.

I'm looking for a documented way to say "hey I want dynamic allocation
on this machine; even in ::gentoo, because I have an existing scheme I
want to use." I want to do it in one place, and not have to set dozens
of package.env files, or copy ebuilds, or hack the eclasses. One idea
I had was some bashrc hacks to always set ACCT_USER_ID=-1 (and
ACCT_GROUP_ID=-1); as this might achieve that goal...but it would be
nicer to probably just set some accepted make.conf var.

-A

[0] Also there are just hilarious stories of weird names in industry.
We had one person who joined who got assigned the username 'lib' and
so their homedirectory was '/home/lib/'. All kinds of random crap
broke with that name and we had to bughunting in some apps...and we
reserved a bunch of similar names so they would not be taken. Or
another case: our uid allocator had a bug and we assigned a real
person the nobody uid; turns out NFS does not work well when you do
that.



>
> --
> Best regards,
> Michał Górny
>
>



Re: [gentoo-dev] [PATCH] go-module.eclass: Add GO_OPTIONAL flag

2021-11-28 Thread Alec Warner
On Sun, Nov 28, 2021 at 11:52 AM William Hubbs  wrote:
>
> On Sun, Nov 28, 2021 at 11:23:16AM -0800, Zac Medico wrote:
> > On 11/21/21 02:57, Florian Schmaus wrote:
> > > Following the pattern found in other eclasses, add GO_OPTIONAL to the
> > > go-module eclass. This allows to inherit the eclass without pulling
> > > its dependencies. See, e.g., bug #775779 for the motivation.
> > >
> > > Signed-off-by: Florian Schmaus 
> > > ---
> > >   eclass/go-module.eclass | 31 ++-
> > >   1 file changed, 22 insertions(+), 9 deletions(-)
> > >
> > > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > > index 3ad8542a28ae..c9eb90ac62ea 100644
> > > --- a/eclass/go-module.eclass
> > > +++ b/eclass/go-module.eclass
> > > @@ -1,4 +1,4 @@
> > > -# Copyright 2019-2020 Gentoo Authors
> > > +# Copyright 2019-2021 Gentoo Authors
> > >   # Distributed under the terms of the GNU General Public License v2
> > >
> > >   # @ECLASS: go-module.eclass
> > > @@ -55,13 +55,17 @@ if [[ -z ${_GO_MODULE} ]]; then
> > >
> > >   _GO_MODULE=1
> > >
> > > -BDEPEND=">=dev-lang/go-1.12"
> > > +if [[ ! ${GO_OPTIONAL} ]]; then
> > > +   BDEPEND=">=dev-lang/go-1.12"
> > >
> > > -# Workaround for pkgcheck false positive: 
> > > https://github.com/pkgcore/pkgcheck/issues/214
> > > -# MissingUnpackerDep: version ...: missing BDEPEND="app-arch/unzip"
> > > -# Added here rather than to each affected package, so it can be cleaned 
> > > up just
> > > -# once when pkgcheck is improved.
> > > -BDEPEND+=" app-arch/unzip"
> > > +   # Workaround for pkgcheck false positive: 
> > > https://github.com/pkgcore/pkgcheck/issues/214
> > > +   # MissingUnpackerDep: version ...: missing BDEPEND="app-arch/unzip"
> > > +   # Added here rather than to each affected package, so it can be 
> > > cleaned up just
> > > +   # once when pkgcheck is improved.
> > > +   BDEPEND+=" app-arch/unzip"
> > > +
> > > +   EXPORT_FUNCTIONS src_unpack
> > > +fi
> > >
> > >   # Force go to build in module mode.
> > >   # In this mode the GOPATH environment variable is ignored.
> > > @@ -83,8 +87,6 @@ QA_FLAGS_IGNORED='.*'
> > >   # Go packages should not be stripped with strip(1).
> > >   RESTRICT+=" strip"
> > >
> > > -EXPORT_FUNCTIONS src_unpack
> > > -
> > >   # @ECLASS-VARIABLE: EGO_SUM
> > >   # @DESCRIPTION:
> > >   # This is an array based on the go.sum content from inside the target 
> > > package.
> > > @@ -147,6 +149,17 @@ EXPORT_FUNCTIONS src_unpack
> > >   # directory structure.
> > >   declare -A -g _GOMODULE_GOSUM_REVERSE_MAP
> > >
> > > +# @ECLASS-VARIABLE: GO_OPTIONAL
> > > +# @DEFAULT_UNSET
> > > +# @PRE_INHERIT
> > > +# @DESCRIPTION:
> > > +# If set to a non-null value before inherit, then the Go part of the
> > > +# ebuild will be considered optional. No dependencies will be added and
> > > +# no phase functions will be exported.
> > > +#
> > > +# If you enable GO_OPTIONAL, you have to set BDEPEND on 
> > > >=dev-lang/go-1.12
> > > +# for your package and call go-module_src_unpack manually.
> > > +
> > >   # @FUNCTION: go-module_set_globals
> > >   # @DESCRIPTION:
> > >   # Convert the information in EGO_SUM for other usage in the ebuild.
> > >
> >
> > How about if we also add a GO_DEPEND variable, so that eclasshi
> > consumers can do something like BDEPEND="go? ( ${GO_DEPEND} )" ?
> > --
> > Thanks,
> > Zac
>
> this is on my radar. I haven't read the bug yet, but I'll look at it, if
> not today, sometime this week.
>
> Without looking at the bug, I'm not sure why you would want to use this
> eclass without depending on dev-lang/go.

I was going to say "just read the bug" but bugs have been misbehaving
recently, so I will summarize.

A package has an optional component that is golang based; users can
enable the component via a USE flag.
That component needs the go eclasses to build. While we can have USE
flag'd components, it's a QA violation to conditionally inherit an
eclass. This results in packages of this type needed to always inherit
the golang eclasses; even if the user has not enabled the golang
functionality.
The eclass always adds a dep on dev-lang/go
(https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/go-module.eclass)
This change let's ebuild callers control that golang dep; because it
should only be added when it's required, not on the eclass inherit.

You could also do other stuff (like not modify DEPEND in global scope
in the go eclasses), which is what Zac was suggesting. Many ways to
skin a cat and all that.

-A

>
> Also, if you have to write src_unpack you can call go-module_setup_proxy
> in src_unpack to set things up.
>
> William
>



[gentoo-dev] [FYI] Archives.gentoo.org has moved physical hosts

2021-11-23 Thread Alec Warner
Hello community,

I believe I have copied all email from the old physical host to the
new host, and that the indexing of the mail (so it appears in the web
archive) is completed. Please let me know if you see any weird
behavior or missing emails on archives.gentoo.org; and I can
investigate and see if they are a result of this migration.

Thanks,

-A



Re: [gentoo-dev] You currently cannot smoothly upgrade a 4 months old Gentoo system

2021-11-03 Thread Alec Warner
On Wed, Nov 3, 2021 at 2:50 PM Andreas K. Huettel  wrote:
>
> Am Mittwoch, 3. November 2021, 22:39:41 CET schrieb Ulrich Mueller:
> > > On Wed, 03 Nov 2021, Andreas K Huettel wrote:
> >
> > > The mistake was to allow the use of EAPI=8 too early. In the future,
> > > we should have a new EAPI supported by portage for at least some
> > > months before the EAPI is even used in the main tree. Not even
> > > speaking about stable here.
> >
> > I tend to disagree. Adding an ebuild with a new EAPI cannot break
> > anything, because it will simply be invisible to old package managers.
>
> Except that you need to keep track of version dependencies across the whole
> tree.
>
> So, yes, this is in principle correct, and in practice with our current
> tooling more or less impossible to do reliably.

I think the obvious easy solution here is to run a CI that is using
older stuff, and report problems when commits break that.

It's less clear what to do about it though; the problems Whissi
raised.. it's not like we didn't know about them (they were known),
but we chose not to do anything about it?
Or we learned about them too late (and figured the majority of users
had seen it; so fixing them was not necessary?)

-A

>
> [Part of the output Whissi pasted was (more or less) that a Perl upgrade
> required a rebuild of Perl modules. Unfortunately, even a single one that
> is not available for rebuild makes the emerge bail out.]
>
>
> --
> Andreas K. Hüttel
> dilfri...@gentoo.org
> Gentoo Linux developer
> (council, toolchain, base-system, perl, libreoffice)



Re: [gentoo-portage-dev] [PATCH] estrip: rework hard link logic in save_elf_debug

2021-10-27 Thread Alec Warner
On Mon, Oct 25, 2021 at 10:10 AM Mike Gilbert  wrote:
>
> GDB loads debug files based on the file name given in the .gnu_debuglink
> section, prepended with /usr/lib/debug/${dirname}, where dirname is the
> absolute path to the parent directory of the binary being executed.
>
> For each unique inode as input, we need a link to the debug file with
> the GNU debuglink as its basename. A link to the debug file should exist
> for each directory in which the input inode exists.
>
> The debug link names should be based on the .gnu_debuglink value instead
> of the name of the file we are processing as input.
>
> The .gnu_debuglink value is based on the name of the first link
> processed for each inode. We save this value as a symlink, and then read
> it back as we process subsequent links.
>
> For example, given the following input:
>
> INODE PATH
> 1 /usr/bin/git
> 1 /usr/libexec/git-core/git-add
> 2 /usr/bin/git-shell
> 2 /usr/libexec/git-core/git-shell
>
> We generate the following inodes for the debug files:
>
> INODE DEBUGLINK
> 3 git.debug
> 4 git-shell.debug
>
> We should generate the following links:
>
> INODE PATH
> 3 /usr/lib/debug/usr/bin/git.debug
> 3 /usr/lib/debug/usr/libexec/git-core/git.debug
> 4 /usr/bin/debug/usr/bin/git-shell.debug
> 4 /usr/bin/debug/usr/libexec/git-core/git-shell.debug
>
> The previous code would have generated this broken output:
>
> INODE PATH
> 3 /usr/lib/debug/usr/bin/git.debug
> 3 /usr/lib/debug/usr/libexec/git-core/git-add.debug (*)
> 4 /usr/bin/debug/usr/bin/git-shell.debug
> 4 /usr/bin/debug/usr/libexec/git-core/git-shell.debug
>
> (*) This link has the wrong name.
>
> Bug: https://bugs.gentoo.org/820107
> Signed-off-by: Mike Gilbert 
> ---
>  bin/estrip | 31 ++-
>  1 file changed, 22 insertions(+), 9 deletions(-)
>
> diff --git a/bin/estrip b/bin/estrip
> index abe523fff..8134020fd 100755
> --- a/bin/estrip
> +++ b/bin/estrip
> @@ -201,17 +201,29 @@ save_elf_debug() {
> local x=$1
> local inode_debug=$2
> local splitdebug=$3
> -   local d_noslash=${D%/}
> -   local y=${ED%/}/usr/lib/debug/${x:${#d_noslash}}.debug
> +
> +   local x_basename=${x##*/}
> +   local x_dirname=${x%/*}
> +   local y_dirname=${ED%/}/usr/lib/debug/${x_dirname#${D%/}/}
> +   local y_basename y

Can we do better than X and Y here?

>
> # dont save debug info twice
> [[ ${x} == *".debug" ]] && return 0
>
> -   mkdir -p "${y%/*}"
> +   mkdir -p "${y_dirname}" || die
> +
> +   if [[ -L ${inode_debug} ]] ; then
> +   y_basename=$(readlink "${inode_debug}") || die
> +   y_basename=${y_basename##*/}
> +   y=${y_dirname}/${y_basename}
>
> -   if [ -f "${inode_debug}" ] ; then
> -   ln "${inode_debug}" "${y}" || die "ln failed unexpectedly"
> +   if [[ ! -e ${y} ]]; then
> +   ln -L "${inode_debug}" "${y}" || die
> +   fi
> else
> +   y_basename=${x_basename}.debug
> +   y=${y_dirname}/${y_basename}
> +
> if [[ -n ${splitdebug} ]] ; then
> mv "${splitdebug}" "${y}"
> else
> @@ -222,11 +234,12 @@ save_elf_debug() {
> fi
> # Only do the following if the debug file was
> # successfully created (see bug #446774).
> -   if [ $? -eq 0 ] ; then
> +   if [[ $? -eq 0 ]] ; then
> local args="a-x,o-w"
> [[ -g ${x} || -u ${x} ]] && args+=",go-r"
> chmod ${args} "${y}"
> -   ln "${y}" "${inode_debug}" || die "ln failed 
> unexpectedly"
> +   # symlink so we can read the name back.
> +   ln -s "${y}" "${inode_debug}" || die
> fi
> fi
>
> @@ -239,8 +252,8 @@ save_elf_debug() {
> local 
> buildid_dir="${ED%/}/usr/lib/debug/.build-id/${buildid:0:2}"
> local buildid_file="${buildid_dir}/${buildid:2}"
> mkdir -p "${buildid_dir}"
> -   [ -L "${buildid_file}".debug ] || ln -s 
> "../../${x:$((${#d_noslash} + 1))}.debug" "${buildid_file}.debug"
> -   [ -L "${buildid_file}" ] || ln -s "/${x:$((${#d_noslash} + 
> 1))}" "${buildid_file}"
> +   [[ -L "${buildid_file}".debug ]] || ln -s 
> "../../${x#${D%/}/}.debug" "${buildid_file}.debug"
> +   [[ -L "${buildid_file}" ]] || ln -s 
> "../../../../../${x#${D%/}/}" "${buildid_file}"
> fi
>  }
>
> --
> 2.33.1
>
>



Re: [gentoo-portage-dev] [RFC] LTS branch of Portage

2021-10-19 Thread Alec Warner
On Mon, Oct 18, 2021 at 4:25 PM Francesco Riosa  wrote:
>
>
> Il giorno mar 5 ott 2021 alle ore 10:31 Michał Górny  ha 
> scritto:
>>
>> Hi, everyone.
>>
>> I've been thinking about this for some time already, and the recent
>> FILESDIR mess seems to confirm it: I'd like to start a more stable LTS
>> branch of Portage.
>>
>> Roughly, the idea is that:
>>
>> - master becomes 3.1.x, and primary development happens there
>>
>> - 3.0.x becomes the LTS branch and only major bugfixes are backported
>> there
>>
>> As things settle down in the future, master would become 3.2.x, 3.1.x
>> would become LTS, 3.0.x will be discontinued and so on.
>>
>> WDYT?
>
>
> Sorry but portage is too strictly related to the ebuilds in tree, recent 
> removal of EAPI=5 from most eclasses underlined that.
> Or to put id differently if you want a LTS portage you also need a certain 
> number of "protected" eclasses and ebuilds
> It seems a lot of (very appreciated but don't count on me) work

I think this is backwards a bit. The idea is to backport things from
the main (development) branch to the LTS branch such that the tree
continues to work for both; no?

This seems mostly related to "what is a bugfix and will be backported
into LTS" and "what is a feature and is not backportable for LTS" in
terms of what the tree will rely on.

-A

>
>
>>
>>
>> --
>> Best regards,
>> Michał Górny
>>
>>
>>



Re: [gentoo-portage-dev] [PATCH] lib/_emerge/actions.py: warn on missing /run

2021-10-07 Thread Alec Warner
On Wed, Oct 6, 2021 at 8:20 PM Sam James  wrote:
>
> Newer versions of build-docbook-catalog use
> /run/lock. This exposed that we weren't
> asking users to mount /run in the handbook.
>
> Check if it exists and warn if it doesn't.
>
> This should primarily (exclusively?) be a
> problem in chroots given an init system
> should be creating this.
>
> Bug: https://bugs.gentoo.org/816303
> Signed-off-by: Sam James 
> ---
>  lib/_emerge/actions.py | 34 ++
>  1 file changed, 22 insertions(+), 12 deletions(-)
>
> diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
> index 05a115250..1b40bebd3 100644
> --- a/lib/_emerge/actions.py
> +++ b/lib/_emerge/actions.py
> @@ -2986,17 +2986,26 @@ def validate_ebuild_environment(trees):
>  check_locale()
>
>
> -def check_procfs():
> -procfs_path = "/proc"
> -if platform.system() not in ("Linux",) or os.path.ismount(procfs_path):
> -return os.EX_OK
> -msg = "It seems that %s is not mounted. You have been warned." % 
> procfs_path
> -writemsg_level(
> -"".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
> -level=logging.ERROR,
> -noiselevel=-1,
> -)
> -return 1

Please add a docstring (""" comment) to the function describing why
it's doing what its doing.

> +def check_mounted_fs():
> +paths = {"/proc": False, "/run": False}

So on Linux, proc is necessary. Isn't /run necessary regardless of OS?
Will docbook build on prefix, or on BSD without /run (I'm guessing
not?)

We might go with something like:

def check_mounted_fs():
  """Cheeky comment explaining why we probably need these mounts."""
required_paths = ['/run']
if platform.system() == 'Linux':
  required_paths.append('/proc')

missing_mounts = False
for path in required_paths:
  if not os.path.ismount(path):
msg = 
writemsg = ...
missing_mounts = True
  # else we do nothing, the mount is where we want it, etc.

return missing_mounts # true if mounts are missing, false otherwise.

> +
> +for path in paths.keys():
> +if platform.system() not in ("Linux",) or os.path.ismount(path):
> +paths[path] = True
> +continue
> +
> +msg = "It seems that %s is not mounted. You have been warned." % path
> +writemsg_level(
> +"".join("!!! %s\n" % l for l in textwrap.wrap(msg, 70)),
> +level=logging.ERROR,
> +noiselevel=-1,
> +)

Duncan covered this already, but I will +1 his concern about better
messaging being an option ;)

> +
> +# Were any of the mounts we were looking for missing?
> +if False in paths.values():
> +return 1

So a couple of nitpicks here. This is a common sort of "collect"
pattern; and I'd use python's any() / all() helpers for a more
pythonic experience.

E.g.

items = [some, list, of, items]
results = []
for j in items:
  results.append(some_computation(j))

# Did any of the computations succeed?
return any(results) # at least 1 True
# Did all of the computations succeed?
return all(results) # all True
# Did none of the computations succeed?
return not any(results) # all False

So in this example you have elected to store the results in a dict,
and we want to return true if all the results are true:

return all(paths.values())

But see also earlier notes above.

> +
> +return os.EX_OK

This isn't shell, so return True or False (not 0 or 1.) (but see above comment.)

>
>
>  def config_protect_check(trees):
> @@ -3474,7 +3483,8 @@ def run_action(emerge_config):
>  repo_name_check(emerge_config.trees)
>  repo_name_duplicate_check(emerge_config.trees)
>  config_protect_check(emerge_config.trees)
> -check_procfs()
> +
> +check_mounted_fs()
>
>  for mytrees in emerge_config.trees.values():
>  mydb = mytrees["porttree"].dbapi
> --
> 2.33.0
>
>



Re: [gentoo-dev] [PATCH 0/1] Add 'notes' element to metadata.xml (GLEP 68)

2021-10-06 Thread Alec Warner
On Tue, Oct 5, 2021 at 10:37 PM Ulrich Mueller  wrote:
>
> >>>>> On Tue, 05 Oct 2021, Alec Warner wrote:
>
> > I'd argue we can add NOTES.md to packages (e.g. allow those files.)
> > Then we modify packages.gentoo.org to render the markdown; or users
> > can render locally or read unrendered.
>
> How would you deal with translations? One NOTES file for every language?
>
> Ulrich

The notes files are for devs, not for users. Do we have non-english
speaking developers?

-A



Re: [gentoo-portage-dev] [RFC] LTS branch of Portage

2021-10-05 Thread Alec Warner
On Tue, Oct 5, 2021 at 1:31 AM Michał Górny  wrote:
>
> Hi, everyone.
>
> I've been thinking about this for some time already, and the recent
> FILESDIR mess seems to confirm it: I'd like to start a more stable LTS
> branch of Portage.
>
> Roughly, the idea is that:
>
> - master becomes 3.1.x, and primary development happens there
>
> - 3.0.x becomes the LTS branch and only major bugfixes are backported
> there
>
> As things settle down in the future, master would become 3.2.x, 3.1.x
> would become LTS, 3.0.x will be discontinued and so on.

I'd appreciate a brief overview of how we would expect users to
install each branch.

E.g. one way might be:

sys-apps/portage- # dev branch
sys-apps/portage-3.1.x # LTS branch
sys-apps/portage-3.0.x # old branch; unmaintained

-A

>
> WDYT?
>
> --
> Best regards,
> Michał Górny
>
>
>



Re: [gentoo-dev] [PATCH 0/1] Add 'notes' element to metadata.xml (GLEP 68)

2021-10-05 Thread Alec Warner
On Tue, Oct 5, 2021 at 1:36 PM Sam James  wrote:
>
>
>
> > On 5 Oct 2021, at 21:29, Alec Warner  wrote:
> >
> > I thought we were going to go with the github-pages type route
> > (markdown, rendered online or locally.)
> >
>
> So, the thinking was, we could allow somewhat shorthand
> notes or for the people who want to invest more time, allow
> the GitHub-pages type route.
>
> But I'm open to the idea of just recommending people
> use that if there's no appetite for mixed types.

I'd argue we can add NOTES.md to packages (e.g. allow those files.)
Then we modify packages.gentoo.org to render the markdown; or users
can render locally or read unrendered.

WDYT?

-A

>
> > -A
> >
> > On Tue, Oct 5, 2021 at 11:28 AM Sam James  wrote:
> >>
> >> This is a preliminary version/draft of a proposed change to
> >> GLEP 68.
> >>
> >> I'd like to introduce a method for developers to signal anything
> >> special about a package and how to correctly maintain it.
> >>
> >> Sam James (1):
> >>  glep-0068: Add notes element for package maintenance instructions
> >>
> >> glep-0068.rst | 26 +++---
> >> 1 file changed, 23 insertions(+), 3 deletions(-)
> >>
> >> --
> >> 2.33.0
> >>
> >>
> >
>



Re: [gentoo-dev] [PATCH 0/1] Add 'notes' element to metadata.xml (GLEP 68)

2021-10-05 Thread Alec Warner
I thought we were going to go with the github-pages type route
(markdown, rendered online or locally.)

-A

On Tue, Oct 5, 2021 at 11:28 AM Sam James  wrote:
>
> This is a preliminary version/draft of a proposed change to
> GLEP 68.
>
> I'd like to introduce a method for developers to signal anything
> special about a package and how to correctly maintain it.
>
> Sam James (1):
>   glep-0068: Add notes element for package maintenance instructions
>
>  glep-0068.rst | 26 +++---
>  1 file changed, 23 insertions(+), 3 deletions(-)
>
> --
> 2.33.0
>
>



Re: [gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg

2021-10-04 Thread Alec Warner
On Sat, Oct 2, 2021 at 1:11 PM Sam James  wrote:
>
> This is part of a series of fixes for the linked bug (failure
> to preserve libraries in some situations).
>
> When unpacking a binpkg to be installed, we should check
> for the existence of PROVIDES if we're installing any
> dynamic libraries. If PROVIDES does not exist in that case,
> this suggests that e.g. scanelf malfunctioned or some corruption occurred.
>
> Bug: https://bugs.gentoo.org/811462
> Signed-off-by: Sam James 
> ---
>  lib/_emerge/Binpkg.py | 28 +++-
>  1 file changed, 27 insertions(+), 1 deletion(-)
>
> diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py
> index c7dde69bd..9b876f354 100644
> --- a/lib/_emerge/Binpkg.py
> +++ b/lib/_emerge/Binpkg.py
> @@ -2,7 +2,7 @@
>  # Distributed under the terms of the GNU General Public License v2
>
>  import functools
> -
> +import glob
>  import _emerge.emergelog
>  from _emerge.EbuildPhase import EbuildPhase
>  from _emerge.BinpkgFetcher import BinpkgFetcher
> @@ -13,6 +13,7 @@ from _emerge.EbuildMerge import EbuildMerge
>  from _emerge.EbuildBuildDir import EbuildBuildDir
>  from _emerge.SpawnProcess import SpawnProcess
>  from portage.eapi import eapi_exports_replace_vars
> +from portage.output import colorize
>  from portage.util import ensure_dirs
>  from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
>  import portage
> @@ -425,6 +426,31 @@ class Binpkg(CompositeTask):
>  self._async_unlock_builddir(returncode=self.returncode)
>  return
>
> +# Before anything else, let's do an integrity check.
> +(provides,) = self._bintree.dbapi.aux_get(self.pkg.cpv, ["PROVIDES"])
> +if not provides:
> +# Let's check if we've got inconsistent results.
> +# If we're installing dynamic libraries (.so files), we should
> +# really have a PROVIDES.
> +# (This is a complementary check at the point of ingestion for 
> the
> +# creation check in doebuild.py)
> +# Note: we could check a non-empty PROVIDES against the list of 
> .sos,
> +# but this doesn't gain us anything. We're interested in failure
> +# to properly parse the installed files at all, which should 
> really
> +# be a global problem (e.g. bug #811462)
> +installed_dynlibs = glob.glob(
> +self.settings["D"] + "/**/*.so", recursive=True
> +)
> +if installed_dynlibs:
> +self._writemsg_level(
> +colorize(
> +"BAD",
> +"!!! Error! Installing dynamic libraries (.so) with 
> blank PROVIDES!",
> +),
> +noiselevel=-1,
> +level=logging.ERROR,
> +)
> +

Here you have written the same code (and long comment block) as the
last patch...I'd take that as an argument to extract it into a
function (check_provides or whatever) and not copy and paste the code
around.

-A

>  try:
>  with io.open(
>  _unicode_encode(
> --
> 2.33.0
>
>



Re: [gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install

2021-10-04 Thread Alec Warner
On Sat, Oct 2, 2021 at 1:11 PM Sam James  wrote:
>
> This is part of a series of fixes for the linked bug (failure
> to preserve libraries in some situations).
>
> At the point of installation (even if not merging), we need
> to detect inconsistent metadata: PROVIDES should be populated
> if we're installing any dynamic libraries. This suggests that
> e.g. scanelf malfunctioned or some corruption occurred.
>
> Bug: https://bugs.gentoo.org/811462
> Signed-off-by: Sam James 
> ---
>  lib/portage/package/ebuild/doebuild.py | 24 +++-
>  1 file changed, 23 insertions(+), 1 deletion(-)
>
> diff --git a/lib/portage/package/ebuild/doebuild.py 
> b/lib/portage/package/ebuild/doebuild.py
> index 9650a8444..dc3fe3d97 100644
> --- a/lib/portage/package/ebuild/doebuild.py
> +++ b/lib/portage/package/ebuild/doebuild.py
> @@ -3,6 +3,7 @@
>
>  __all__ = ["doebuild", "doebuild_environment", "spawn", "spawnebuild"]
>
> +import glob
>  import grp
>  import gzip
>  import errno
> @@ -3079,7 +3080,7 @@ def _post_src_install_soname_symlinks(mysettings, out):
>  ) as f:
>  f.write(soname_deps.requires)
>
> -if soname_deps.provides is not None:
> +if soname_deps.provides:

The previous code checked if soname_deps.provides was None (or not.)
You have changed it to check if soname_deps.provides is true-ish (or not.)

Why did you change it?

>  with io.open(
>  _unicode_encode(
>  os.path.join(build_info_dir, "PROVIDES"),
> @@ -3091,6 +3092,27 @@ def _post_src_install_soname_symlinks(mysettings, out):
>  errors="strict",
>  ) as f:
>  f.write(soname_deps.provides)
> +else:
> +# Let's check if we've got inconsistent results.
> +# If we're installing dynamic libraries (.so files), we should
> +# really have a PROVIDES.
> +# (This is a complementary check at the point of creation for the
> +# ingestion check in Binpkg.py)
> +# Note: we could check a non-empty PROVIDES against the list of .sos,
> +# but this doesn't gain us anything. We're interested in failure
> +# to properly parse the installed files at all, which should really
> +# be a global problem (e.g. bug #811462)
> +installed_dynlibs = glob.glob(image_dir + "/**/*.so", recursive=True)
> +
> +if installed_dynlibs:
> +self._writemsg_level(
> +colorize(
> +"BAD",
> +"!!! Error! Installing dynamic libraries (.so) with 
> blank PROVIDES!",
> +),
> +noiselevel=-1,
> +level=logging.ERROR,
> +)
>
>  if unrecognized_elf_files:
>  qa_msg = ["QA Notice: Unrecognized ELF file(s):"]
> --
> 2.33.0
>
>



Re: [gentoo-dev] [RFC] Portage einfo, elog... output format change

2021-10-01 Thread Alec Warner
On Fri, Oct 1, 2021 at 11:30 AM A Schenck  wrote:
>
> On 9/29/21 11:44 PM, Michał Górny wrote:
> > On Thu, 2021-09-30 at 08:40 +0200, Fabian Groffen wrote:
> >> Hi,
> >>
> >> Would it be possible to have some switch (e.g. --style=legacy) that
> >> controls this new vs. the old behaviour?  Would perhaps allow
> >> applications that parse the output to work via setting this in the
> >> global opts.
> > Patches welcome.  It shouldn't be hard, my commit shows which files need
> > to be edited to alter the prefixes and how to pass them into ebd.
>
> Would it be possible to get this patch in an format that it can be
> interacted with with open tools? Per the other branch of this thread,
> we'd be happy to make this an opt-in so as to not break existing people.

I'm not sure what you mean; you can download the PR...

https://github.com/gentoo/portage/pull/759

-A

>
> -A
>
>



[gentoo-dev] Re: Guidance on distributed patented software

2021-09-26 Thread Alec Warner
On Mon, Sep 20, 2021 at 11:30 AM Ulrich Mueller  wrote:
>
> >>>>> On Mon, 20 Sep 2021, Alec Warner wrote:
>
> > The devmanual discusses licensing as a core concept
> > (https://devmanual.gentoo.org/general-concepts/licenses/index.html)
> > but does not cover patents. My understanding is that we:
>
> >  - set RESTRICT=bindist when we are unable to redistribute binaries
> > (e.g. due to a license or patent restriction.)
> >  - set RESTRICT=mirror when we are unable to redistribute source code
> > (e.g. due to a license of patent restriction.)
>
> IANAL, but IIUC patents only apply to programs that can run on a
> computer. This is the case for binaries but not for source code.
>
> In other words, we don't need mirror restriction for source tarballs
> because of patents.
>
> >  - Sometimes, we remove patent encumbered source code from packages
> > (e.g. with USE=bindist) so that we can build redistributable binaries
> > with the patented features removed.
>
> We do, but normally this doesn't prevent us from distributing the source
> code.

Great, I'll send a patch to devmanual to add details about how we
treat patented software in Gentoo (the original goal of the thread; I
don't care what happens to openssl as much ;p)

-A

>
> > Could we add some text to the license concepts covering patents? It
> > seems to have been omitted?
> > Is my understanding of how we manage patented software correct?



[gentoo-dev] Guidance on distributed patented software

2021-09-20 Thread Alec Warner
The devmanual discusses licensing as a core concept
(https://devmanual.gentoo.org/general-concepts/licenses/index.html)
but does not cover patents. My understanding is that we:

 - set RESTRICT=bindist when we are unable to redistribute binaries
(e.g. due to a license or patent restriction.)
 - set RESTRICT=mirror when we are unable to redistribute source code
(e.g. due to a license of patent restriction.)
 - Sometimes, we remove patent encumbered source code from packages
(e.g. with USE=bindist) so that we can build redistributable binaries
with the patented features removed.

Could we add some text to the license concepts covering patents? It
seems to have been omitted?
Is my understanding of how we manage patented software correct?

-A



Re: [gentoo-portage-dev] Performance tuning and parallelisation

2021-08-27 Thread Alec Warner
On Thu, Aug 26, 2021 at 4:03 AM Ed W  wrote:
>
> Hi All
>
> Consider this a tentative first email to test the water, but I have started 
> to look at performance
> of particularly the install phase of the emerge utility and I could use some 
> guidance on where to go
> next

To clarify; the 'install' phase installs the package into ${D}. The
'qmerge' phase is the phase that merges to the livefs.

>
> Firstly, to define the "problem": I have found gentoo to be a great base for 
> building custom
> distributions and I use it to build a small embedded distro which runs on a 
> couple of different
> architectures. (Essentially just a "ROOT=/something emerge $some_packages"). 
> However, I use some
> packaging around binpackages to avoid uncessary rebuilds, and this highlights 
> that "building" a
> complete install using only binary packages rarely gets over a load of 1. Can 
> we do better than
> this? Seems to be highly serialised on the install phase of copying the files 
> to the disk?

In terms of parallelism it's not safe to run multiple phase functions
simultaneously. This is a problem in theory and occasionally in
practice (recently discussed in #gentoo-dev.)
The phase functions run arbitrary code that modifies the livefs (as
pre / post install and rm can touch $ROOT.) As an example we observed
recently; font ebuilds will generate font related metadata. If 2
ebuilds try to generate the metadata at the same time; they can race
and cause unexpected results. Sometimes this is caught in the ebuild
(e.g. they wrote code like rebuild_indexes || die and the indexer
returned non-zero) but can simply result in silent data corruption
instead; particularly if the races go undetected.

>
> (Note I use parallel build and parallel-install flags, plus --jobs=N. If 
> there is code to compile
> then load will shoot up, but simply installing binpackages struggles to get 
> the load over about
> 0.7-1.1, so presumably single threaded in all parts?)
>
>
> Now, this is particularly noticeable where I cheated to build my arm install 
> and just used qemu
> user-mode on an amd64 host (rather than using cross-compile). Here it's very 
> noticeable that the
> install/merge phase of the build is consuming much/most of the install time.
>
> eg, random example (under qemu user mode)

I think perhaps a simpler test is to use qmerge (from portage-utils)?
If you can use emerge (e.g. in --pretend mode) to generate a package
list to merge; you can simply merge them with qmerge. I suspect qmerge
will both (a) be faster and (b) be less safe than emerge; as emerge is
doing a bunch of extra work you may or may not care about. You can
also consider running N qmerge's (again less sure how safe this is; as
the writes by qmerge may be racy.) Note again that this speed may not
come for free and you may end up with a corrupt image afterwards.

I'm not sure if folks are running qmerge in production like this
(maybe others on the list have experience.)

>
> # time ROOT=/tmp/timetest emerge -1k --nodeps openssl
>
> >>> Emerging binary (1 of 1) dev-libs/openssl-1.1.1k-r1::gentoo for 
> >>> /tmp/timetest/
> ...
> real0m30.145s
> user0m29.066s
> sys0m1.685s
>
>
> Running the same on the native host is about 5-6sec, (and I find this ratio 
> fairly consistent for
> qemu usermode, about 5-6x slower than native)
>
> If I pick another package with fewer files, then I will see this 5-6 secs 
> drop, suggesting (without
> offering proof) that the bulk of the time here is some "per file" processing.
>
> Note this machine is a 12 core AMD ryzen 3900x with SSDs that bench around 
> the 4GB/s+. So really 5-6
> seconds to install a few files is relatively "slow". Random benchmark on this 
> machine might be that
> I can backup 4.5GB of chroot with tar+zstd in about 4 seconds.
>
>
> So the question is: I assume that further parallelisation of the install 
> phase will be difficult,
> therefore the low hanging fruit here seems to be the install/merge phase and 
> why there seems to be
> quite a bit of CPU "per file installed"? Can anyone give me a leg up on how I 
> could benchmark this
> further and look for the hotspot? Perhaps someone understand the architecture 
> of this point more
> intimately and could point at whether there are opportunities to do some of 
> the processing on mass,
> rather than per file?
>
> I'm not really a python guru, but interested to poke further to see where the 
> time is going.
>
>
> Many thanks
>
> Ed W
>
>
>
>
>



[gentoo-dev] Access to dipper.gentoo.org

2021-08-19 Thread Alec Warner
Previously due to an exciting misconfiguration; access to rsync
services to dipper.gentoo.org were open to the internet[0]. This was
caught in a recent audit and we have disallowed access from the
internet to rsync services on this host.

Please continue to use our public rsync services (rsync.gentoo.org or
similar rotation.) to sync data from us. We are sorry for the change
in behavior.

-A

[0] The repos available are the same repos; so it's not a security
problem; just a problem of load management; we want users to sync from
the edges of our network, not from the core.



Re: [gentoo-dev] RFC: new category for container related packages, instead of app-emulation

2021-08-06 Thread Alec Warner
On Fri, Aug 6, 2021 at 2:12 PM William Hubbs  wrote:
>
> On Thu, Aug 05, 2021 at 05:57:06PM -0700, Alec Warner wrote:
> > On Thu, Aug 5, 2021 at 2:44 PM Georgy Yakovlev  wrote:
> > >
> > > Hi,
> > >
> > > We've been collecting more and more container related packages in
> > >  app-emulation/*
> > >
> > > What do you think about finally moving those packages to separate 
> > > category?
> >
> > As always my opinion is that:
> >
> > (a) Categories were a design mistake.
> > (b) The mistake is hard to fix.
> > (c) It's basically low-value to try to 'correctly' categorize packages
> > because of A.
>
> I disagree that categories are a mistake. For one thing, they help with
> situations where different upstream packages have the same name (we have
> two docker packages for example, both named docker by their upstreams),
> and in my opinion they help keep the tree organized. It is nice to be
> able to do something like `ls -d ` and check out somewhat
> related groups of packages.

Sure, but you can build a much better mechanism to disambiguate
packages than a category[0].
Grouping packages together is again better served by a different
system. If you want to group packages using categories:

 - A package can only be in 1 group (its category).
 - It's hard to have custom groupings (basically requires I make an
overlay, move the package into a different group, and then do a
pkgmove in my overlay? Does this even work?)
 - Moving packages between groups is not a trivial operation.

>
> I will agree that C above is subjective.
>
> > (d) Recategorizing means a bunch of stuff has to be updated.
>
> All of the updates are much easier to do since we use git, so this isn't
> a concern. You can set this up so it hits the tree all at once.

I think you misunderstand the cost you are imposing on users. Updating
git is the easy part!

You add the pkgmove.
You rewrite some DEPEND atoms.
You bump.

But the impact on *downstream*.

Everyone else who uses those atoms also needs to update them.

For portage-visible names portage will update them (so e.g. it will do
the pkgmove in the VDB, in your binpkg repo, in your metadata, in
/etc/portage/*, world, sets, etc.)
For non-portage visible names you are stuck updating it yourself.

 - Overlays (all need to update their atoms)
 - Any kind of scripts that drive your systems (e.g. "install
foo/bar-123 needs to say 'baz/bar-123' instead")

So my point is that it isn't free. That doesn't mean don't do it; just
understand that it isn't a free operation; or even cheap!

-A

[0] When I say it's a design mistake I am basically talking about
encoding metadata in a name. It's a common design mistake because then
the metadata changes and so does the name. Very common with naming in
computer science. I'm all for names that provide unambiguous packages;
I just want to be able to reference a package in a stable way.

>
> > Do people actually care what category things are in? I just use
> > --search or eix or whatever and the category is just this...bad
> > concept we attach to packages for silly historical reasons..
>
> Yes, I think they are useful like I said above.
>
> So add me to the list of folks supporting recatagorizing the packages.
>
> William



Re: [gentoo-portage-dev] In what phase are file "merged"?

2021-08-06 Thread Alec Warner
On Thu, Aug 5, 2021 at 9:23 PM Ulrich Mueller  wrote:
>
> > On Fri, 06 Aug 2021, Joakim Tjernlund wrote:
>
> > On Wed, 2021-06-23 at 13:33 +0200, Michał Górny wrote:
> >> On Wed, 2021-06-23 at 12:40 +0200, Ulrich Mueller wrote:
> >> > I don't think that the ebuild can rely on any particular status of
> >> > the new package in pkg_*rm (of the old package), or the status of
> >> > the old package in pkg_*inst (of the new package).
> >>
> >> I would even say that it can't rely on the particular status of the
> >> old package in any case, if it's meant to be removed.  In particular,
> >> its dependencies can be unmerged before the package itself.
>
> > Stubled ove this mail again and noticed "its dependencies can be
> > unmerged before the package itself" stmt. That does not make sense to
> > me. Deps should be unmerged after any pkg that depends on them?
>
> A popular workflow is "emerge -c -p" followed by "emerge -C" on entries
> of the list shown. IIUC emerge -C doesn't do any dependency resolution,
> therefore ebuilds cannot rely on any removal order.

Not quite sure I follow. Let's assume I have A -> B -> C.

Is it legal for A to call a binary packaged in A in A's pkg_prerm?
If yes, then B and C have to be on the livefs at least until A's
pkg_prerm has run; right?; otherwise if we unmerged B or C before then
we might break A's binaries?

-A

>
> Ulrich



Re: [gentoo-dev] RFC: new category for container related packages, instead of app-emulation

2021-08-05 Thread Alec Warner
On Thu, Aug 5, 2021 at 2:44 PM Georgy Yakovlev  wrote:
>
> Hi,
>
> We've been collecting more and more container related packages in
>  app-emulation/*
>
> What do you think about finally moving those packages to separate category?

As always my opinion is that:

(a) Categories were a design mistake.
(b) The mistake is hard to fix.
(c) It's basically low-value to try to 'correctly' categorize packages
because of A.
(d) Recategorizing means a bunch of stuff has to be updated.

Do people actually care what category things are in? I just use
--search or eix or whatever and the category is just this...bad
concept we attach to packages for silly historical reasons..

-A

> probably app-containers/
>
> Here's the tentative list, most tools have description attached for easier
> review, 36 candidates so far, there may be more around, I only looked at
> app-emulation.
>
> app-emulation/buildah - A tool that facilitates building OCI images
> app-emulation/cadvisor - container analyzer
> app-emulation/conmon - An OCI container runtime monitor
> app-emulation/containerd - A daemon to control runC
> app-emulation/containers-storage - containers/storage library
> app-emulation/cri-o - OCI-based implementation of Kubernetes CRI
> app-emulation/cri-tools - CLI and validation tools for CRI
> app-emulation/crun - OCI Container Runtime fully written in C
> app-emulation/distrobuilder - System container image builder for LXC and LXD
> app-emulation/docker - Main offender
> app-emulation/docker-bench-security - Test for best docker practices
> app-emulation/docker-cli - Main offender cli
> app-emulation/docker-compose - Multi-container orchestration for Docker
> app-emulation/docker-credential-helpers -
> app-emulation/docker-gc - Docker garbage collection of containers and images
> app-emulation/docker-proxy - Docker container networking
> app-emulation/docker-registry -
> app-emulation/docker-swarm -
> app-emulation/flannel - An etcd backed network fabric for containers
> app-emulation/go-secbench - run and evaluate the docker security benchmark
> app-emulation/img - Standalone Dockerfile and OCI container image builder
> app-emulation/k3d - creates k3s clusters in docker
> app-emulation/kompose - Tool to move from docker-compose to Kubernetes
> app-emulation/lxc - A userspace interface for the Linux kernel containment
> app-emulation/lxc-templates -
> app-emulation/lxd - Fast, dense and secure container management
> app-emulation/nerdctl - Docker-compatible CLI for containerd
> app-emulation/podman - Main offender killer
> app-emulation/reg - Docker registry v2 command line client
> app-emulation/runc - another docker thing
> app-emulation/s6-overlay - an s6-based init system for containers
> app-emulation/sen - Terminal User Interface for docker engine
> app-emulation/skopeo - Utility for operations on container images/repositories
> app-emulation/slirp4netns - User-mode networking for unpriv network namespaces
> app-emulation/snapd - Service and tools for management of snap packages
> app-emulation/umoci - Manipulation tool for OCI images
>
> Those 4 are technically emulation related, so I'm not sure about category:
> app-emulation/docker-machine
> app-emulation/docker-machine-kvm
> app-emulation/hyperd
> app-emulation/runv



Re: [gentoo-dev] [PATCH] metadata/install-qa-check.d: add 60tmpfiles-path QA check

2021-08-04 Thread Alec Warner
On Tue, Aug 3, 2021 at 7:56 PM Sam James  wrote:
>
>
>
> > On 4 Aug 2021, at 03:39, Thomas Deutschmann  wrote:
> >
> > On 2021-08-01 01:56, Sam James wrote:
> >> 1) Verify packages don't install tmpfiles to /etc/tmpfiles.d, which
> >> is a deprecated location;
> >
> > This location is _not_ deprecated.
> >
> > This is the location for the local system administrator to override 
> > tmpfiles.d specifications installed by packages which should install below 
> > /usr.
> >
> >
>
> Sure, thanks for the clarification. It's deprecated in the sense of ebuilds 
> installing to it though, right?

What if I use ebuilds to configure my local system settings? ;)

-A

>
> best,
> sam



[gentoo-dev] [FYI] Gentoo Service instability June 28 - 29

2021-06-28 Thread Alec Warner
Hi,

Many of you reported that various infra-owned services were not functioning
properly these past two days. We had some package upgrades roll out that
were bad, and a subset of services were non-functional while we worked
through the incident.

(a) The upgrades did not appear to cause any data loss, but service
availability was reduced (500s / 502s for some HTTP services.)
(b) Some hosts may have had problems running some commands. E.g. I know
that 'grep' didn't work for a while. If you observed errors related to
missing 'GLIBC_2.33' symbols then this was likely related.
(c) Many replicated services (including infra-status) rely on gitweb to
take updates from git; and gitweb was not up, so they were unable to
receive updates.
(d) More impacted notes will follow in a postmortem that will come later
this week for this incident.

Thanks to all of you who reported problems and apologies for the service
disruptions.

-A


[gentoo-dev] FYI: Unexpected outage May 11 04:40 - 20:00 UTC

2021-05-12 Thread Alec Warner
The following nodes were offline:

bittern.gentoo.org (blogs, Bouncer, devmanual)
bobolink.gentoo.org (ns1; dns)
devbox.amd64.dev.gentoo.org
elivepatch.amd64.dev.gentoo.org
motmot.gentoo.org (qa-reports, keys.g.o)
roverlay.dev.gentoo.org
woodpecker.gentoo.org (dev.g.o, smtp.g.o)

These are all VMs; the host running them failed and it took a while to
bring the failed host back up[0]. I believe all services are restored,
sorry for the inconvenience. Mainly I'd expect a delay in email; as
email is a store-and-forward system and our mailserver was down for
half a day.

-A

[0] Infra members were busy in that AM with work meetings and the
remote login documentation was a bit out of date; we are working to
update that playbook.



Re: [gentoo-dev] [PATCH v2 3/3] gnuconfig.eclass: use BDEPEND, BROOT where available (drop support for EAPI <4)

2021-04-09 Thread Alec Warner
On Fri, Apr 9, 2021 at 5:19 PM Sam James  wrote:
>
>
>
> > On 10 Apr 2021, at 01:13, Michael Orlitzky  wrote:
> >
> > On Sat, 2021-04-10 at 00:32 +0100, Sam James wrote:
> >>
> >>
> >> Yes, this is the part I find difficult too. The important
> >> distinction here was *bootstrapping* (which I missed)
> >> but I think at least we should make a list of packages generally considered
> >> critical for bootstrap.
> >>
> >
> > What is a bootstrap package?
> >
> > There is some chicken-and-egg problem to be solved, but I don't think
> > that we should be assuming that e.g. GNU grep is always present just
> > because, during the base case of some recursive process, POSIX grep
> > must be available temporarily.
> >
> > Anyway, https://bugs.gentoo.org/485356 awaits reopening if you make any
> > progress on this.
> >
>
> Oh, I agree completely. CCed myself on the bug and added to the list
> to think about/work on.
>
> I’m pleased a bug existed in the past! I don’t agree with it being closed 
> though:
> documentation issues can exist without a patch existing to fix them yet, 
> right?

I worry a lot about more complex dependencies trees (imagine all of
the exciting cycles in the currently excluded @system depgraph.)
Remember that while in theory it would be great if portage knew about
all of them; computing these nodes and edges isn't free. I question
what we are really buying with the extra complexity.

-A



Re: [gentoo-dev] timezone configuration - why copying, not symlinking /etc/localtime ?

2021-03-20 Thread Alec Warner
On Sat, Mar 20, 2021 at 10:27 PM Ulrich Mueller  wrote:
>
> >>>>> On Sun, 21 Mar 2021, Alec Warner wrote:
>
> >> Which doesn't imply that we deliberately break things.
>
> > Not sure I follow.. how is updating the handbook breaking anything?
>
> Both configurations (regular file and symlink) work just fine, and
> sys-libs/timezone-data supports them. I don't see a good reason why we
> would suddenly declare the regular file to be an invalid option.

https://bugs.gentoo.org/737914 seems to imply for some upstreams, it
being a file is not a valid option anymore?

(I'm ignoring the logic of that decision of course, but this was the
original reason this was raised.)

-A

>
> Ulrich



Re: [gentoo-dev] timezone configuration - why copying, not symlinking /etc/localtime ?

2021-03-20 Thread Alec Warner
On Sat, Mar 20, 2021 at 9:19 PM Ulrich Mueller  wrote:
>
> > On Sat, 20 Mar 2021, William Hubbs wrote:
>
> > /etc/localtime should definitely be a symlink to the proper file in
> > /usr/share/zoneinfo.
>
> > This works fine if /usr is on a separate partition *and* you are using
> > an initramfs. The only time it doesn't work is if /usr is separate
> > without using an initramfs.
>
> > Council decided years ago that we don't support separate /usr without
> > an initramfs, but we haven't completed that transition yet.
>
> Which doesn't imply that we deliberately break things.

Not sure I follow.. how is updating the handbook breaking anything?

-A

>
> Ulrich



Re: [gentoo-dev] [PATCH] gstreamer-meson.eclass: New eclass required for gstreamer-1.18.0+

2021-03-17 Thread Alec Warner
On Tue, Mar 16, 2021 at 6:49 PM Haelwenn (lanodan) Monnier
 wrote:
>
> Gstreamer switched to meson in 1.16.0 and removed autotools support in 1.18.0,
> this eclass is an update of gstreamer.eclass.
>
> One significant change between autotools and meson is that in the latter we
> don't have easily extractable semantics in the buildsystem to get a list
> of plugins with extraneous dependencies that we currently split in other
> packages.
> Hence the rather ugly but currently required GST_PLUGINS_DISABLED block.
>
> Fixes: https://bugs.gentoo.org/690468
>
> Signed-off-by: Haelwenn (lanodan) Monnier 
> ---
>  eclass/gstreamer-meson.eclass | 293 ++
>  1 file changed, 293 insertions(+)
>  create mode 100644 eclass/gstreamer-meson.eclass
>
> diff --git a/eclass/gstreamer-meson.eclass b/eclass/gstreamer-meson.eclass
> new file mode 100644
> index 000..263deeb08aa
> --- /dev/null
> +++ b/eclass/gstreamer-meson.eclass
> @@ -0,0 +1,293 @@
> +# Copyright 1999-2021 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: gstreamer-meson.eclass
> +# @MAINTAINER:
> +# gstrea...@gentoo.org
> +# @AUTHOR:
> +# Michał Górny 
> +# Gilles Dartiguelongue 
> +# Saleem Abdulrasool 
> +# foser 
> +# zaheerm 
> +# Steven Newbury
> +# Haelwenn (lanodan) Monnier 
> +# @SUPPORTED_EAPIS: 5 6
> +# @BLURB: Helps building core & split gstreamer plugins.
> +# @DESCRIPTION:
> +# Eclass to make external gst-plugins emergable on a per-plugin basis
> +# and to solve the problem with gst-plugins generating far too much
> +# unneeded dependencies.
> +#
> +# GStreamer consuming applications should depend on the specific plugins
> +# they need as defined in their source code. Usually you can find that
> +# out by grepping the source tree for 'factory_make'. If it uses playbin
> +# plugin, consider adding media-plugins/gst-plugins-meta dependency, but
> +# also list any packages that provide explicitly requested plugins.
> +
> +inherit eutils multilib meson multilib-minimal toolchain-funcs versionator 
> xdg-utils
> +
> +case "${EAPI:-0}" in
> +   5|6)
> +   ;;
> +   0|1|2|3|4)
> +   die "EAPI=\"${EAPI:-0}\" is not supported anymore"
> +   ;;
> +   *)
> +   die "EAPI=\"${EAPI}\" is not supported yet"
> +   ;;
> +esac
> +
> +# @ECLASS-VARIABLE: GST_PLUGINS_ENABLED
> +# @DESCRIPTION:
> +# Defines the plugins to be built.
> +# May be set by an ebuild and contain more than one indentifier, space
> +# seperated (only src_configure can handle mutiple plugins at this time).
> +: ${GST_PLUGINS_ENABLED:=${PN/gst-plugins-/}}
> +
> +# @ECLASS-VARIABLE: GST_PLUGINS_DISABLED
> +# @DESCRIPTION:
> +# Defines the plugins to not be built, GST_PLUGINS_ENABLED overrides it.
> +# May be set by an ebuild and contain more than one indentifier, space
> +# seperated (only src_configure can handle mutiple plugins at this time).
> +case "${GST_ORG_MODULE}" in
> +   # copied GST_PLUGINS_DISABLED from media-libs/${GST_ORG_MODULE} then 
> added GST_PLUGINS_ENABLED
> +   gst-plugins-bad)
> +   # removed from list: shm ipcpipeline gl
> +   GST_PLUGINS_DISABLED="aom avtp androidmedia applemedia 
> assrender bluez bs2b bz2 chromaprint closedcaption colormanagement curl 
> curl-ssh2 d3dvideosink d3d11 dash dc1394 decklink directfb directsound dtls 
> dts dvb faac faad fbdev fdkaac flite fluidsynth gme gsm iqa kate kms ladspa 
> libde265 libmms lv2 mediafoundation microdns modplug mpeg2enc mplex msdk 
> musepack neon nvcodec ofa openal openexr openh264 openjpeg openmpt openni2 
> opensles opus resindvd rsvg rtmp sbc sctp smoothstreaming sndfile soundtouch 
> spandsp srt srtp svthevcenc teletext tinyalsa transcode ttml uvch264 va 
> voaacenc voamrwbenc vulkan wasapi wasapi2 webp webrtc webrtcdsp wildmidi 
> winks winscreencap x265 zbar zxing wpe magicleap v4l2codecs hls opencv"
> +   GST_PLUGINS_DISABLED="${GST_PLUGINS_DISABLED} accurip 
> adpcmdec adpcmenc aiff asfmux audiobuffersplit audiofxbad audiolatency 
> audiomixmatrix audiovisualizers autoconvert bayer camerabin2 coloreffects deb 
> ugutils dvbsubenc dvbsuboverlay dvdspu faceoverlay festival fieldanalysis 
> freeverb frei0r gaudieffects gdp geometrictransform id3tag inter interlace 
> ivfpars e ivtc jp2kdecimator jpegformat librfb midi mpegdemux mpegpsmux 
> mpegtsdemux mpegtsmux mxf netsim onvif pcapparse pnm proxy rawparse 
> removesilence rist rtmp2 rtp sdp segmentclip siren smooth speed subenc 
> switchbin timecode videofilters videoframe_audiolevel videoparsers 
> videosignal vmnc y4m"
> +   ;;
> +   gst-plugins-base)
> +   GST_PLUGINS_DISABLED="cdparanoia libvisual opus tremor"
> +   GST_PLUGINS_DISABLED="${GST_PLUGINS_DISABLED} adder app 
> audioconvert audiomixer audiorate audioresample audiotestsrc compositor 
> encoding gio gio-typefinder overlaycomposition pbtypes playback 

Re: [gentoo-dev] Fwd: [Python-Dev] Move support of legacy platforms/architectures outside Python

2021-02-22 Thread Alec Warner
So my question is basically, how much does Gentoo care about these
ports? Should we be funding ports of rust to these arches (assuming
that would continue ensure gentoo works on those arches.)

-A


On Sun, Feb 21, 2021 at 5:01 AM Michał Górny  wrote:
>
> Hi,
>
> FYI, a few member of Python upstream are continuing their crusade
> against minor architectures not supported by Rust.  This time, they're
> discussing actively removing support for platforms they don't officially
> support, and requiring people to maintain external patches forever.
>
>
>  Forwarded Message 
> From: Victor Stinner 
> To: Python Dev 
> Subject: [Python-Dev] Move support of legacy platforms/architectures
> outside Python
> Date: Sun, 21 Feb 2021 13:13:45 +0100
>
> > Hi,
> >
> > I propose to actively remove support for *legacy* platforms and
> > architectures which are not supported by Python according to PEP 11
> > rules: hardware no longer sold and end-of-life operating systems. The
> > removal should be discussed on a case by case basis, but I would like
> > to get an agreement on the overall idea first. Hobbyists wanting to
> > support these platforms/archs can continue to support them with
> > patches maintained outside Python. For example, I consider that the
> > 16-bit m68k architecture is legacy, whereas the OpenBSD platform is
> > still actively maintained.
> >
> > I already know that there will be a strike back: "oh no, you must
> > continue to support my architecture" and "their existing code should
> > stay and doesn't cost anything to maintain". Python is maintained by
> > volunteers, the majority is contributing in their free time, so people
> > are free to use their free time as they want. You cannot ask core
> > developers to support your favorite *legacy* platform/architecture if
> > they don't want to.
> >
> > In short, I propose to move maintenance of the legacy platforms/archs
> > outside Python: people are free to continue support them as patches.
> >
> > --
> >
> > Concrete example: Christian Heimes proposed to drop support for 31-bit
> > s390 Linux:
> > https://bugs.python.org/issue43179
> >
> > The lack of clear definition on how a platform is supported or not
> > confuses users who consider that their favorite platform/arch is
> > supported, whereas core developers don't want to support it since it
> > would be too much work.
> >
> > In fact, the PEP 11 has clear and explicit rules:
> > https://www.python.org/dev/peps/pep-0011/#supporting-platforms
> >
> > A platform is only considered as supported if the following two
> > conditions are met:
> >
> > 1) a core developer needs to volunteer to maintain platform-specific code
> > 2) a stable buildbot must be provided
> >
> > Last October, I proposed to drop Solaris support (bpo-42173). Jakub
> > Kulik stepped in and proposed some Solaris patches, so I abandoned my
> > idea. But I still don't see any running Solaris buildbot worker, and
> > there is still no core developer volunteer to maintain Solaris
> > support. It's unclear to me if Python has "best effort" support for
> > Solaris, of if Solaris is "not supported".
> >
> > --
> >
> > Over the years, Python was ported to tons of platforms and CPU
> > architectures. It didn't matter if the platform or the architecture
> > was commonly used or not. 30 years later, Python still has the code
> > for many legacy platforms and architectures. Some hardware is no
> > longer sold but kept alive by hobbyists, especially members of retro
> > computing groups.
> >
> > Some Linux distributions like Gentoo and Debian are trying to support
> > most architectures which are supported by these hobbyist groups,
> > whereas some other distributions like Ubuntu are limited to a few
> > platforms. For example, Ubuntu 20.4.2 LTS server supports 4
> > architectures: x86-64, AArch64 (ARM), POWER and s390x. I guess that
> > the difference between Debian and Ubuntu is that Ubuntu is a Canonical
> > product, Canonical sells professional support and so cannot support
> > too many architectures. Each architecture support requires to build
> > all packages on it, tests the packages, have experts who fix issues
> > specific to this arch, etc.
> >
> > --
> >
> > Python has different kinds of platform and architecture supports. In
> > practice, I would say that we have:
> >
> > * (1) Fully supported. Platform/architecture used by core developers
> > and have at least one working buildbot worker: fully supported. Since
> > core developers want to use Python on their machine, they fix issues
> > as soon as they notice them. Examples: x86-64 on Linux, Windows and
> > macOS.
> >
> > * (2) Best effort. Platform/architecture which has a buildbot worker
> > usually not used by core developers. Regressions (buildbot failures)
> > are reported to bugs.python.org, if someone investigates and provides
> > a fix, the fix is merged. But there is usually no "proactive" work to
> > ensure that Python works "perfectly" on these platforms. 

Re: [gentoo-dev] [RFC] Timeline for Python 3.9 switch and Python 3.7 removal

2021-02-18 Thread Alec Warner
On Thu, Feb 18, 2021 at 3:45 PM Michał Górny  wrote:
>
> Hi,
>
> I'd like to discuss a rough timeline for switching the default
> PYTHON_TARGETS to python3.9.
>
> According to the upstream release schedule [1], the last bugfix release
> is planned for May.  Afterwards, upstream will release only security
> fixes.  At the same time, our rough current schedule [2] suggests that
> we'll start last-riting stuff May 1st, so the 3.7 target would be
> removed in 15-30 days.
>
> Do you think 2021-06-01 would be a reasonable planned switchover date?

So you are saying we will start masking packages may 1st and we will
drop 3.7 targets 30 days later (june 1?)

(I felt like this was a bit ambiguous?)

-A

>
> [1] https://www.python.org/dev/peps/pep-0569/#bugfix-releases
> [2] https://wiki.gentoo.org/wiki/Project:Python/Implementations
>
> --
> Best regards,
> Michał Górny
>
>
>



Re: [gentoo-dev] dev-python/cryptography to use rust, effectively killing alpha, hppa, ia64, m68k, s390

2021-02-08 Thread Alec Warner
On Mon, Feb 8, 2021 at 9:56 AM Alessandro Barbieri
 wrote:
>
> Il Lun 8 Feb 2021, 12:19 Michał Górny  ha scritto:
>>
>> Hi,
>>
>> FYI the developers of dev-python/cryptography decided that Rust is going
>> to be mandatory for 1.5+ versions.  It's unlikely that they're going to
>> provide LTS support or security fixes for the old versions.
>>
>> Since cryptography is a very important package in the Python ecosystem,
>> and it is an indirect dependency of Portage, this means that we will
>> probably have to entirely drop support for architectures that are not
>> supported by Rust.
>>
>> According to upstream platform support information [1], this probably
>> means (eventually) entirely removing the following architectures:
>> - alpha (stable)
>> - hppa (stable)
>> - ia64 (stable)
>> - m68k (exp)
>> - s390 (except for s390x, exp)
>>
>> Furthermore, the Gentoo Rust packages are missing support
>> for the following platforms, apparently supported upstream:
>> - mips (exp)
>> - ppc (32) (stable)
>> - sparc (stable)
>> - s390x (exp)
>> - riscv (stable)
>>
>> Apparently it's non-trivial to bootstrap Rust on these platforms,
>> so it's unclear when Gentoo is going to start providing Rust on them.
>>
>> I've raised a protest on the cryptography bug tracker [2] but apparently
>> upstream considers Rust's 'memory safety' more important than ability to
>> actually use the package.
>>
>> Honestly, I don't think it likely that Rust will gain support for these
>> platforms.  This involves a lot of work, starting with writing a new
>> LLVM backend and getting it accepted (getting new code into LLVM is very
>> hard unless you're doing that on behalf one of the big companies).  You
>> can imagine how much effort that involves compared to rewriting the new
>> code from Cryptography into C.
>>
>> If we can't convince upstream, I'm afraid we'll either have to drop
>> these architectures entirely or fork Cryptography.
>>
>>
>> [1] https://doc.rust-lang.org/nightly/rustc/platform-support.html
>> [2] https://github.com/pyca/cryptography/issues/5771
>>
>> --
>> Best regards,
>> Michał Górny
>
>
> Should we shed tears for those legacy architectures or move forward? Does 
> anyone really use them in production?

Many users don't use gentoo in 'production' so I'm not sure how that
matters in our decision making. I'm not sure the trade off here
(taking rust as a core dep) is reasonable and it looks like we can
avoid it.

-A



Re: [gentoo-dev] GTK:2 EOL and incoming migration to GTK:3

2021-02-08 Thread Alec Warner
On Mon, Feb 8, 2021 at 6:59 AM Peter Stuge  wrote:
>
> Hanno Böck wrote:
> > > "It does mean, however, that GTK 2 has reached the end of its life.
> > > We will do one final 2.x release in the coming days, and we encourage
> > > everybody to port their GTK 2 applications to GTK 3 or 4."
> >
> > I read that as there will be one more gtk2 release and none after that.
> >
> > This seems to imply:
> > * When there's a security flaw in gtk2 there won't be a fix from
> >   upstream.
> > * When there's an incompatibility with new infrastructure (e.g. new gcc
> >   version / new glibc / changing API of libraries gtk depends on) there
> >   will be no updates from upstream.
> >
> > This means in all those instances maintainers will have to get patches
> > from somewhere. We'll likely end up with some form of
> > gtk-2.x-r[largenumber] with a large patchset at some point.
> > Maintaining that will be an increasing burden.
> >
> > No urgency, but a sign to slowly move off gtk2.
>
> Until there's a relevant flaw that will remain unfixed or there is
> significant incompatibility with infrastructure (recurse my argument)
> no signs actually exist.

So I think as the next post in the thread hints at:

 - I expect gtk2 (the library) to be around for a while. As written it
gets at least one more release.
 - I expect Gentoo to come after gtk2-only leaf packages pretty hard;
either to get upstream to port, or to remove them.
   - This is true even if the packages are fully functional with gtk2,
or don't have other bugs.
   - This is because we will eventually remove gtk2 from the tree
(which will make these packages unbuildable, and cause their removal.)

I'm less clear why we would keep libgtk2 in the tree for years and
years (just to keep nominally unmaintained gtk2 leaf packages
buildable?)

>
> Assuming that there will be a significant maintenance burden which
> affects all uses doesn't seem rational - hence my question.

I think you need to keep gtk2 (the library) for a fair bit (just like
we kept python2.7; the interpreter; for a fair while after its EOL.)

>
> The blog post shouldn't be misunderstood. The intended audience seems
> to be application developers, encouraging them to port applications,
> not so much distributions.
>
> Distributions quite often overlook that they wield much power, and
> thus also have much responsibility.
>
> Of course, GTK maintainers in Gentoo choose what to work on, and have
> made many (only?) excellent choices.
>
> I'm merely pleading for rational choices based on actual problems.
>
>
> //Peter
>



Re: [gentoo-dev] [RFC] Moving subslot-only virtuals to a separate category to reduce confusion

2021-01-30 Thread Alec Warner
On Sat, Jan 30, 2021 at 9:36 AM Michał Górny  wrote:
>
> Hi,
>
> TL;DR: I'd like to move virtual/libjpeg, virtual/libudev and so on to
> another category (e.g. lib-sover/*) to make it clear that they are used
> for := deps and have valid use even with a single provider.
>
>
> Right now we have at least a few packages that provide more than one
> valid consumer-intended library, each with a different SOVERSION that is
> bumped independently.  To list a few examples:
>
> - media-libs/libjpeg-turbo: libturbojpeg.so.0, libjpeg.so.62
> - sys-apps/systemd: libsystemd.so.0, libudev.so.1
> - app-text/poppler: libpoppler.so.106, libpoppler-cpp.so.0, libpoppler-
> glib.so.8, libpoppler-qt5.so.1
>
> The problem is that the current subslot mechanism doesn't really provide
> for that.  Laying aside possible future EAPI extensions (that are
> debatable due to added complexity to an already complex syntax), there
> are three commonly used possibilities here:
>
> a. bumping subslot whenever any of SOVERSIONs change -- meaning possibly
> unnecessary rebuilds,
>
> b. using subslot for one of the SOVERSIONs and assuming the rest stable
> -- this is what we do for poppler, and it's mildly confusing,
>
> c. using additional packages to represent SOVERSION of individual
> libraries -- this is e.g. used for libudev or libjpeg.
>
>
> I'd like to discuss option c. in more detail.  Right now, we're creating
> additional packages in virtual/ category.  This was mostly fine,
> as the libraries in question (libjpeg, libudev) had multiple providers.
> However, now that we've removed the old media-libs/jpeg, people get
> the obvious idea that the virtual is no longer necessary -- except that
> it is!
>
> The rough idea is that the subslot of libjpeg-turbo is supposed to
> represent the ABI version of libjpeg-turbo -- that is actually rarely
> used.  On the other hand, the subslot of libjpeg is represented by
> virtual/jpeg.  So if your package is using libjpeg.so, it should :=-
> depend on the latter and not on the former.
>
> The problem is that this is confusing, and if somebody doesn't know
> the secret, he can easily consider the virtual obsolete and depend
> on the library directly.
>
> To make this SOVERSION-virtual concept more visible and easily
> distinguishable from regular virtuals, I'd like to propose that we start
> moving them into a dedicated category.  For example, 'lib-sover' comes
> to my mind.  While this ofc doesn't guarantee that people will do things
> right, it will at least make it clear that these packages are somewhat
> different from regular virtuals and hopefully avoid making wrong
> assumptions.
>
> WDYT?

I'm not sure I understand the problem with (a). But this is partly my
bias here. I expect to rebuild with Gentoo; often; its literally the
whole point. So when the developer community is like "Well we want
fewer rebuilds" I sort of scoff. "This is Gentoo, rebuilds are a
thing." I wish I was more in sync with the community on this ;(

For (c) I'd rather see something like a linter:
 - If I dep on libjpeg-turbo, raise a linter hint that says "hey if
this dep is for libjpeg.so, if you want that, swap this *DEPEND for
virtual/jpeg: read this wiki page for more information..."
 - Its not meant as a QA warning (the hint will not prevent submission
or block CI.)
 - But its a mechanization of this problem and solution. Instead of
reading some thread on -dev,  there is a tool to detect instances of
this problem, it tells you briefly what the options are, and links
somewhere for more information. This sort of thing is related to an
earlier -core thread about how to keep developers up to date..threads
like this are great but they are not a system.

In short, I don't think a package naming convention is sufficient to
achieve your goal of making this problem happen less often.

I also have no idea how often this problem happens today (and if we
can lint it, we can count those too.)
Is the problem even a big enough deal to do anything?

-A

>
> --
> Best regards,
> Michał Górny
>
>
>



Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-18 Thread Alec Warner
On Mon, Jan 18, 2021 at 8:09 PM Zac Medico  wrote:
>
> On 1/18/21 6:07 PM, Alec Warner wrote:
> > On Fri, Jan 15, 2021 at 6:47 PM Matt Turner  wrote:
> >>
> >> This set is the upgradable packages for which the highest visible
> >> version has a different subslot than the currently installed version.
> >>
> >> The primary purpose of this feature is for use in catalyst builds. We
> >> update the "seed" stage3 before using it to build a new stage1.
> >>
> >> Updating the entire stage is expensive and unnecessary (since we're
> >> going to build the latest packages in stage1 and then rebuild everything
> >> in stage3).
> >>
> >> What we definitely do need to update in the original stage3 however, is
> >> any package that would trigger a subslot rebuild.
> >>
> >> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> >> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> >> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> >> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> >> will be built and libmpfr.so.6 included into the stage1. Since the old
> >> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> >> subsequent stage builds.
> >>
> >> Our current options to update the seed are too large a hammer (e.g.,
> >> "--update --deep --newuse @world" or "--update --deep --newuse
> >> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> >> updating seed stages for no gain beyond updating only packages for whom
> >> the subslot has changed.
> >>
> >> With this set, catalyst will likely use
> >>
> >> emerge @changed-subslot --ignore-built-slot-operator-deps y
> >>
> >> to update the seed stage.
> >>
> >> Thank you to Zac Medico for showing me how to do this.
> >>
> >> Bug: https://bugs.gentoo.org/739004
> >> Signed-off-by: Matt Turner 
> >> ---
> >>  cnf/sets/portage.conf  |  5 +
> >>  lib/portage/_sets/dbapi.py | 39 +-
> >>  2 files changed, 43 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
> >> index 22f0fa3a5..5651a9c53 100644
> >> --- a/cnf/sets/portage.conf
> >> +++ b/cnf/sets/portage.conf
> >> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
> >>  [rebuilt-binaries]
> >>  class = portage.sets.dbapi.RebuiltBinaries
> >>
> >> +# Installed packages for which the subslot of the highest visible ebuild
> >> +# version is different than the currently installed version.
> >> +[changed-subslot]
> >> +class = portage.sets.dbapi.SubslotChangedSet
> >> +
> >>  # Installed packages for which the highest visible ebuild
> >>  # version is lower than the currently installed version.
> >>  [downgrade]
> >> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
> >> index 52367c4a6..46ba5c17d 100644
> >> --- a/lib/portage/_sets/dbapi.py
> >> +++ b/lib/portage/_sets/dbapi.py
> >> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
> >>  import portage
> >>
> >>  __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
> >> -   "EverythingSet", "OwnerSet", "VariableSet"]
> >> +   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
> >>
> >>  class EverythingSet(PackageSet):
> >> _operations = ["merge"]
> >> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
> >>
> >> singleBuilder = classmethod(singleBuilder)
> >>
> >> +class SubslotChangedSet(PackageSet):
> >> +
> >> +   _operations = ["merge", "unmerge"]
> >> +
> >> +   description = "Package set which contains all packages " + \
> >> +   "for which the subslot of the highest visible ebuild is " 
> >> + \
> >> +   "different than the currently installed version."
> >
> > description = ("string1",
> >"string2",
> >"string3")
> >
> > vs concat + \ for line continuation?
> &g

Re: [gentoo-portage-dev] [PATCH] Add @changed-subslot package set

2021-01-18 Thread Alec Warner
On Fri, Jan 15, 2021 at 6:47 PM Matt Turner  wrote:
>
> This set is the upgradable packages for which the highest visible
> version has a different subslot than the currently installed version.
>
> The primary purpose of this feature is for use in catalyst builds. We
> update the "seed" stage3 before using it to build a new stage1.
>
> Updating the entire stage is expensive and unnecessary (since we're
> going to build the latest packages in stage1 and then rebuild everything
> in stage3).
>
> What we definitely do need to update in the original stage3 however, is
> any package that would trigger a subslot rebuild.
>
> For example: gcc links with libmpfr.so from dev-libs/mpfr. mpfr's SONAME
> changes from libmpfr.so.4 (SLOT="0/4") to libmpfr.so.6 (SLOT="0/6"). If
> the seed stage's dev-libs/mpfr is not updated before emerging gcc, gcc
> will link with libmpfr.so.4, but the latest version of dev-libs/mpfr
> will be built and libmpfr.so.6 included into the stage1. Since the old
> libmpfr.so.4 is not included in the stage1, gcc will not work, breaking
> subsequent stage builds.
>
> Our current options to update the seed are too large a hammer (e.g.,
> "--update --deep --newuse @world" or "--update --deep --newuse
> --complete-graph --rebuild-if-new-ver gcc") and spend too much time
> updating seed stages for no gain beyond updating only packages for whom
> the subslot has changed.
>
> With this set, catalyst will likely use
>
> emerge @changed-subslot --ignore-built-slot-operator-deps y
>
> to update the seed stage.
>
> Thank you to Zac Medico for showing me how to do this.
>
> Bug: https://bugs.gentoo.org/739004
> Signed-off-by: Matt Turner 
> ---
>  cnf/sets/portage.conf  |  5 +
>  lib/portage/_sets/dbapi.py | 39 +-
>  2 files changed, 43 insertions(+), 1 deletion(-)
>
> diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
> index 22f0fa3a5..5651a9c53 100644
> --- a/cnf/sets/portage.conf
> +++ b/cnf/sets/portage.conf
> @@ -84,6 +84,11 @@ exclude-files = /usr/bin/Xorg
>  [rebuilt-binaries]
>  class = portage.sets.dbapi.RebuiltBinaries
>
> +# Installed packages for which the subslot of the highest visible ebuild
> +# version is different than the currently installed version.
> +[changed-subslot]
> +class = portage.sets.dbapi.SubslotChangedSet
> +
>  # Installed packages for which the highest visible ebuild
>  # version is lower than the currently installed version.
>  [downgrade]
> diff --git a/lib/portage/_sets/dbapi.py b/lib/portage/_sets/dbapi.py
> index 52367c4a6..46ba5c17d 100644
> --- a/lib/portage/_sets/dbapi.py
> +++ b/lib/portage/_sets/dbapi.py
> @@ -15,7 +15,7 @@ from portage._sets import SetConfigError, get_boolean
>  import portage
>
>  __all__ = ["CategorySet", "ChangedDepsSet", "DowngradeSet",
> -   "EverythingSet", "OwnerSet", "VariableSet"]
> +   "EverythingSet", "OwnerSet", "SubslotChangedSet", "VariableSet"]
>
>  class EverythingSet(PackageSet):
> _operations = ["merge"]
> @@ -167,6 +167,43 @@ class VariableSet(EverythingSet):
>
> singleBuilder = classmethod(singleBuilder)
>
> +class SubslotChangedSet(PackageSet):
> +
> +   _operations = ["merge", "unmerge"]
> +
> +   description = "Package set which contains all packages " + \
> +   "for which the subslot of the highest visible ebuild is " + \
> +   "different than the currently installed version."

description = ("string1",
   "string2",
   "string3")

vs concat + \ for line continuation?

-A

> +
> +   def __init__(self, portdb=None, vardb=None):
> +   super(SubslotChangedSet, self).__init__()
> +   self._portdb = portdb
> +   self._vardb = vardb
> +
> +   def load(self):
> +   atoms = []
> +   xmatch = self._portdb.xmatch
> +   xmatch_level = "bestmatch-visible"
> +   cp_list = self._vardb.cp_list
> +   pkg_str = self._vardb._pkg_str
> +   for cp in self._vardb.cp_all():
> +   for cpv in cp_list(cp):
> +   pkg = pkg_str(cpv, None)
> +   slot_atom = "%s:%s" % (pkg.cp, pkg.slot)
> +   ebuild = xmatch(xmatch_level, slot_atom)
> +   if not ebuild:
> +   continue
> +   if pkg.sub_slot != ebuild.sub_slot:
> +   atoms.append(slot_atom)
> +
> +   self._setAtoms(atoms)
> +
> +   def singleBuilder(cls, options, settings, trees):
> +   return cls(portdb=trees["porttree"].dbapi,
> +   vardb=trees["vartree"].dbapi)
> +
> +   singleBuilder = classmethod(singleBuilder)
> +
>  class DowngradeSet(PackageSet):
>
> _operations = ["merge", "unmerge"]
> --
> 2.26.2
>
>



Re: [gentoo-dev] [PATCH 1/2] acct-user.eclass: Support ACCT_USER_ID override

2021-01-06 Thread Alec Warner
On Wed, Jan 6, 2021 at 11:05 AM Patrick McLean  wrote:
>
> On Wed, 6 Jan 2021 15:02:12 +0100
> Thomas Deutschmann  wrote:
>
> > Hi,
> >
> > is there a specific reason why we want to support dynamic variables
> > (ACCT_USER_$foo) at all?
> >
> > Isn't package.env support enough, i.e. use ACCT_USER_ID from environment
> > if set (which we should detect and log, maybe this will require a
> > different namespace for the variables at all to be able to differentiate
> > between values set by acct-* ebuild and user override)?
> >
> > Of course this won't allow something like `ACCT_USER_ID=42 emerge
> > ` but I am not sure if
> > this is an implementation goal.
>
> This is so ACCT_USER_$foo can be set in make.conf, and not have to
> be specified as an environment variable whenever portage is run. This
> helps when automated systems are building Gentoo images or systems.
>

Not sure I follow. Whether your automation sets a variable in
/etc/portage/make.conf or /etc/portage/package.env; it's basically the
same problem space; no?

-A



Re: [gentoo-dev] [PATCH] acct-user.eclass: Support var overrides for user properties

2021-01-04 Thread Alec Warner
On Mon, Jan 4, 2021 at 9:15 AM Thomas Deutschmann  wrote:
>
> On 2021-01-04 18:08, Michał Górny wrote:
> > Introduce a few variables to allow easy overrides of common user account
> > proprerties, that is:
> >
> > - ACCT_USER__SHELL
> > - ACCT_USER__HOME
> > - ACCT_USER__HOME_OWNER
> > - ACCT_USER__HOME_PERMS
> > - ACCT_USER__GROUPS
> > - ACCT_USER__GROUPS_ADD
> >
> > The first five variables override the respective ACCT_USER_* variables,
> > with ACCT_USER_*_GROUPS being a space-separated list.  The *_GROUPS_ADD
> > variable appends to groups present in the ebuild, as this seems a common
> > necessity.
> >
> > We do realize that the original requirement of overriding ebuilds
> > in a local repository was inconvenient.  This new logic should permit
> > easy updates via make.conf.  Additionally, it has the advantage
> > of clearly reporting the changes made in the build logs.
> >
> > This does not preclude other solutions to the problem.  However, this
> > is probably the best one and it should become the current
> > recommendation.
>
> This will improve the overlay situation and can be seen as overall
> improvement but it doesn't address any shared concerns nor is it a
> replacement for the proposed 'acct-user.eclass: don't modify existing
> user by default' patch.
>

Same response from me, merge it but please also merge the other patch.

-A

>
> --
> Regards,
> Thomas Deutschmann / Gentoo Linux Developer
> fpr: C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
>



Re: [gentoo-dev] [PATCH] acct-user.eclass: don't modify existing user by default

2021-01-03 Thread Alec Warner
On Sun, Jan 3, 2021 at 6:42 PM Mike Gilbert  wrote:
>
> On Sun, Jan 3, 2021 at 8:35 PM Thomas Deutschmann  wrote:
> >
> > Modifying an existing user is a bad default and makes Gentoo
> > special because it is common for system administrators to make
> > modifications to user (i.e. putting an user into another service's
> > group to allow that user to access service in question) and it
> > would be unexpected to see these changes reverted during normal
> > world upgrade (which could break services).
> >
> > This commit will make Gentoo behave like any other Linux distribution
> > by respecting any user modifications by default. However, we will retain
> > the functionality to reset system user and groups and users interested
> > in this feature can opt-in by setting
> > ACCT_USER_ALLOW_EXISTING_USER_TO_BE_MODIFIED to a non-zero value in
> > their make.conf.
>
> So the main problem I see with doing this is that it becomes
> impossible to reliably make changes to a user in later ebuild
> revisions. Developers may want/need to deploy changes to user
> attributes. Changing group memberships seems like the best example,
> but I could foresee a want/need to change DESCRIPTION, HOME, or SHELL
> as well.

I think the TL;DR is I don't believe changes in HOME, or SHELL are
safe to do in later revisions. This means developers:
 - Need to be careful when picking em!
 - Can use later revisions to set sane defaults for fresh installs.
 - Can't touch existing installs because changing HOME and SHELL can
cause user-facing breakage.

Often if HOME points somewhere that isn't /dev/null, you have to do
some kind of migration. I'd rather not enable that sort of breakage by
default because it's a per-app / per-user migration pain.

To pick an example:

The git user has a HOME in either /var/lib/gitea or /var/lib/gitolite.
If you change it, you will break whatever service is running.
This is the same for any service account whose $HOME stores
state...which is most of them with a HOME set.

-A

>
> Because of this, I think the new behavior should be opt-in, and people
> who use it should be aware that they will need to pay attention if any
> account changes are rolled out in new ebuild versions.
>
> > diff --git a/eclass/acct-user.eclass b/eclass/acct-user.eclass
> > index 22b0038fbff7..d60b1e53b4bb 100644
> > --- a/eclass/acct-user.eclass
> > +++ b/eclass/acct-user.eclass
> > @@ -309,6 +321,20 @@ acct-user_pkg_pretend() {
> > fi
> >  }
> >
> > +# @FUNCTION: acct-user_pkg_setup
> > +# @DESCRIPTION:
> > +# Initialize internal environment variable(s).
> > +acct-user_pkg_setup() {
> > +   debug-print-function ${FUNCNAME} "${@}"
> > +
> > +   # check if user already exists
> > +   ACCT_USER_ALREADY_EXISTS=
> > +   if [[ -n $(egetent passwd "${ACCT_USER_NAME}") ]]; then
> > +   ACCT_USER_ALREADY_EXISTS=yes
> > +   fi
> > +   readonly ACCT_USER_ALREADY_EXISTS
> > +}
>
> I don't think this pkg_setup function is necessary; you could do this
> in pkg_preinst instead, before enewuser gets called.
>



[gentoo-dev] [Turndown] rsync access to VCS content is no longer available.

2020-12-31 Thread Alec Warner
TL;DR if you never used rsync:// access to download copies of data
from our VCS's you can stop paying attention. This has nothing to do
with "gentoo rsync mirrors" for ::gentoo, which is a separate service
that we are not terminating.

We previously offered rsync:// protocol access to gentoo VCS
repositories. We have stopped offering this access. This is another
service that had little to no usage according to logs, and has a bunch
of legacy dependencies. For VCS content please use anongit.gentoo.org
to access content in git, and the vcs history project to access
read-only copies of historical cvs data at
https://projects.gentoo.org. We have kept completed copies of this
data, and if the public data is missing something or is incorrect,
please follow up with us.

-A



Re: [gentoo-dev] Re: [gentoo-dev-announce] We are finally shutting down CVS

2020-12-29 Thread Alec Warner
On Sun, Dec 27, 2020 at 10:31 AM Alec Warner  wrote:
>
> On Sun, Dec 27, 2020 at 6:39 AM Ulrich Mueller  wrote:
> >
> > >>>>> On Sun, 27 Dec 2020, Max Magorsch wrote:
> >
> > > To access the old repositories you can use gitweb.gentoo.org instead.
> > > We have migrated all old cvs repositories to git. All of them are
> > > available read-only now at [0].
> >
> > I've just looked at
> > https://sources.gentoo.org/archive/cvs/gentoo.git/
> > and its commit history ends in 2004.
> >
> > Can you please reinstate CVS until a more accurate conversion is
> > available?
>
> I'm happy to make tarballs available (as discussed in a bunch of
> places on irc.) Is that sufficient or is there some particular
> requirement for the CVS protocol specifically?

So some updates:

 - We will make tarballs of the raw CVS repos available soon; I'm
working with ulm@ to remove various PII in some places; we previously
used CVS ACLs to prevent people from seeing this PII and we will
similarly remove it in these tarballs.
 - Sources.gentoo.org (used to browse cvs repositories) is now gone.
 - https://sources.gentoo.org now points at gitweb.gentoo.org, but we
may point it at https://anongit.gentoo.org in the future.
 - The viewvc instance that powered cvs browsing was deleted and isn't
coming back and the basic idea is that we will serve archival tarballs
(for really old stuff) and git (for modern stuff) and nothing else.

Thanks to ulm (for reviewing the cvs content) and arzano (who is doing
all the actual work underneath ;p)

Also some clarification here. Infra has carried some tech debt for 5
years[0] and this mostly worked OK for us. However cfengine2 is
*really* old and we need to get off of it. Most new services run on
top of 'puppet' a 'newer' management stack that helps us configure
machines and services. So this push to retire a bunch of stuff is
aligned with a push to get off of cfengine completely by EOY because
it's starting to really cause us operational challenges. Antarus is
not just arbitrarily shutting stuff off; but we are also in a bit of a
rush at the end of 2020 so the shutdowns are not as smooth as we would
like.

-A

[0] I checked, we started to migrate to puppet in *2010* and are still
migrating, 10 years later ;P

>
> -A
>
> >
> > Same applies to gentoo-x86 where the git repo misses whole categories.
> >
> > Ulrich



Re: [gentoo-dev] Re: [gentoo-dev-announce] We are finally shutting down CVS

2020-12-27 Thread Alec Warner
On Sun, Dec 27, 2020 at 6:39 AM Ulrich Mueller  wrote:
>
> > On Sun, 27 Dec 2020, Max Magorsch wrote:
>
> > To access the old repositories you can use gitweb.gentoo.org instead.
> > We have migrated all old cvs repositories to git. All of them are
> > available read-only now at [0].
>
> I've just looked at
> https://sources.gentoo.org/archive/cvs/gentoo.git/
> and its commit history ends in 2004.
>
> Can you please reinstate CVS until a more accurate conversion is
> available?

I'm happy to make tarballs available (as discussed in a bunch of
places on irc.) Is that sufficient or is there some particular
requirement for the CVS protocol specifically?

-A

>
> Same applies to gentoo-x86 where the git repo misses whole categories.
>
> Ulrich



Re: [gentoo-dev] Re: [gentoo-dev-announce] We are finally shutting down CVS

2020-12-27 Thread Alec Warner
On Sun, Dec 27, 2020 at 12:42 AM Michał Górny  wrote:
>
> On Sun, 2020-12-27 at 00:55 +, Max Magorsch wrote:
> > Hi all,
> >
> > as a quick note: We are finally shutting down all old cvs services.
> > Accordingly the old viewvc repository browser will be shut down as
> > well.
> >
> > To access the old repositories you can use gitweb.gentoo.org instead.
> > We have migrated all old cvs repositories to git. All of them are
> > available read-only now at [0].
>
> Are we keeping the original CVS archives too, for potential better
> conversions in the future?

We don't plan on deleting any data, and we have not deleted any data
to date. Most historical data is kept in Amazon S3 and on a
Gentoo-owned machine somewhere.

-A

>
> --
> Best regards,
> Michał Górny
>
>
>



[gentoo-dev] I'm looking for a Mentor

2020-12-18 Thread Alec Warner
TL;DR, I think infra needs more people who can commit to ::gentoo and
thus, I am looking for an ebuild dev mentor. I myself had commit
access in the before times (probably 2010 - 2012) but have been mostly
ignoring ebuild development since then to focus on infra and the
Foundation. However infra is using more and more software that is
maintainer-needed and so we probably need to change our focus to
giving back more to the main tree.

-A



Re: [gentoo-dev] GPG key refresh

2020-12-14 Thread Alec Warner
antarus@woodpecker ~ $ gpg --keyserver hkps://keys.gentoo.org --search-keys
1C49724D229E93A2
gpg: data source: https://[2001:470:ea4a:1:230:48ff:fef8:9fdc]:443
(1) Michael Orlitzky 
   Michael Orlitzky 
 4096 bit RSA key 1C49724D229E93A2, created: 2010-03-17, expires:
2020-12-26
Keys 1-1 of 1 for "1C49724D229E93A2".  Enter number(s), N)ext, or Q)uit > q
gpg: error searching keyserver: Operation cancelled
gpg: keyserver search failed: Operation cancelled

I think you need to push to hkps://keys.gentoo.org

-A

On Mon, Dec 14, 2020 at 11:00 AM Michael Orlitzky  wrote:

> I'm still getting this,
>
>$ git push --signed
>...
>remote: 1C49724D229E93A2 [Michael Orlitzky ] [E]
>expire:short Expiration date is too close, please renew (is 2020-12-26
>23:30:47, less than 14 days)
>remote: 1C49724D229E93A2:6F48D3DA05C2DADB [Michael Orlitzky
>] [E] expire:short Expiration date is too close,
>please renew (is 2020-12-26 23:31:43, less than 14 days)
>
> over a week after I've renewed my keys. The answer I get back from the
> keyserver(s) looks OK:
>
>$ gpg --search-keys 0x6F48D3DA05C2DADB
>gpg: data source: http://85.25.207.23:11371
>(1)   Michael Orlitzky 
>  Michael Orlitzky 
>4096 bit RSA key 0x1C49724D229E93A2, created: 2010-03-17,
>expires: 2022-12-05
>
>
> Did I forget a step? How long should it take for infra to refresh?
>
>


Re: [gentoo-dev] [PATCH 1/2] eapi8-dosym.eclass: New eclass.

2020-11-19 Thread Alec Warner
On Thu, Nov 19, 2020 at 7:28 AM Alec Warner  wrote:

>
>
> On Thu, Nov 19, 2020 at 2:32 AM Ulrich Müller  wrote:
>
>> This implements the dosym command proposed for EAPI 8 (called dosym8
>> because we cannot use the same name as the package-manager builtin).
>>
>> "dosym -r  " will expand the (apparent) path of 
>> relative to the (apparent) path of the directory containing .
>> The main aim of this is to allow for an absolute path to be specified
>> as the link target, and the function will count path components and
>> convert it into a relative path.
>>
>> Since we're inside ED at this point but the image will finally be
>> installed in EROOT, we don't try to resolve any pre-existing symlinks
>> in  or . In other words, path expansion only looks at
>> the specified apparent paths, without touching any actual files in ED
>> or EROOT.
>>
>> Signed-off-by: Ulrich Müller 
>> ---
>>  eclass/eapi8-dosym.eclass | 108 ++
>>  1 file changed, 108 insertions(+)
>>  create mode 100644 eclass/eapi8-dosym.eclass
>>
>> diff --git a/eclass/eapi8-dosym.eclass b/eclass/eapi8-dosym.eclass
>> new file mode 100644
>> index ..52f0ffe3e62b
>> --- /dev/null
>> +++ b/eclass/eapi8-dosym.eclass
>> @@ -0,0 +1,108 @@
>> +# Copyright 2020 Gentoo Authors
>> +# Distributed under the terms of the GNU General Public License v2
>> +
>> +# @ECLASS: eapi8-dosym.eclass
>> +# @MAINTAINER:
>> +# PMS team 
>> +# @AUTHOR:
>> +# Ulrich Müller 
>> +# @SUPPORTED_EAPIS: 5 6 7
>> +# @BLURB: Testing implementation of EAPI 8 dosym -r option
>> +# @DESCRIPTION:
>> +# A stand-alone implementation of the dosym command aimed for EAPI 8.
>> +# Intended to be used for wider testing of the proposed option and to
>> +# allow ebuilds to switch to the new model early, with minimal change
>> +# needed for actual EAPI 8.
>> +#
>> +# https://bugs.gentoo.org/708360
>> +
>> +case ${EAPI} in
>> +   5|6|7) ;;
>> +   *) die "${ECLASS}: EAPI=${EAPI:-0} not supported" ;;
>> +esac
>> +
>> +# @FUNCTION: _dosym8_canonicalize
>> +# @USAGE: 
>> +# @INTERNAL
>> +# @DESCRIPTION:
>> +# Transparent bash-only replacement for GNU "realpath -m -s".
>> +# Resolve references to "/./", "/../" and remove extra "/" characters
>> +# from , without touching any actual file.
>>
>
> Take this as a nit, but do we have any way to test this function
> (particularly around edge cases.)
> I see eclass/tests exists, could we add a couple for this?
>

hah, and you added them in the second patch, sorry I didn't read ahead! :)

-A


>
>
>> +_dosym8_canonicalize() {
>>
>
> in dosym8() you save and restore IFS, but you don't here, is there a
> reason for that?
>
>
>> +   local path slash i prev out IFS=/
>> +
>> +   path=( $1 )
>> +   [[ $1 == /* ]] && slash=/
>> +
>> +   while true; do
>> +   # Find first instance of non-".." path component followed
>> by "..",
>> +   # or as a special case, "/.." at the beginning of the
>> path.
>> +   # Also drop empty and "." path components as we go along.
>> +   prev=
>> +   for i in ${!path[@]}; do
>> +   if [[ -z ${path[i]} || ${path[i]} == . ]]; then
>> +   unset "path[i]"
>> +   elif [[ ${path[i]} != .. ]]; then
>> +   prev=${i}
>> +   elif [[ ${prev} || ${slash} ]]; then
>> +   # Found, remove path components and
>> reiterate
>> +   [[ ${prev} ]] && unset "path[prev]"
>> +   unset "path[i]"
>> +   continue 2
>> +   fi
>> +   done
>> +   # No (further) instance found, so we're done
>> +   break
>> +   done
>> +
>> +   out="${slash}${path[*]}"
>> +   echo "${out:-.}"
>> +}
>> +
>> +# @FUNCTION: dosym8
>> +# @USAGE: [-r]  
>> +# @DESCRIPTION:
>> +# Create a symbolic link , pointing to .  If the
>> +# directory containing the new link does not exist, create it.
>> +#
>> +# If called with option -r, expand  relative to the 

Re: [gentoo-dev] [PATCH 1/2] eapi8-dosym.eclass: New eclass.

2020-11-19 Thread Alec Warner
On Thu, Nov 19, 2020 at 2:32 AM Ulrich Müller  wrote:

> This implements the dosym command proposed for EAPI 8 (called dosym8
> because we cannot use the same name as the package-manager builtin).
>
> "dosym -r  " will expand the (apparent) path of 
> relative to the (apparent) path of the directory containing .
> The main aim of this is to allow for an absolute path to be specified
> as the link target, and the function will count path components and
> convert it into a relative path.
>
> Since we're inside ED at this point but the image will finally be
> installed in EROOT, we don't try to resolve any pre-existing symlinks
> in  or . In other words, path expansion only looks at
> the specified apparent paths, without touching any actual files in ED
> or EROOT.
>
> Signed-off-by: Ulrich Müller 
> ---
>  eclass/eapi8-dosym.eclass | 108 ++
>  1 file changed, 108 insertions(+)
>  create mode 100644 eclass/eapi8-dosym.eclass
>
> diff --git a/eclass/eapi8-dosym.eclass b/eclass/eapi8-dosym.eclass
> new file mode 100644
> index ..52f0ffe3e62b
> --- /dev/null
> +++ b/eclass/eapi8-dosym.eclass
> @@ -0,0 +1,108 @@
> +# Copyright 2020 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: eapi8-dosym.eclass
> +# @MAINTAINER:
> +# PMS team 
> +# @AUTHOR:
> +# Ulrich Müller 
> +# @SUPPORTED_EAPIS: 5 6 7
> +# @BLURB: Testing implementation of EAPI 8 dosym -r option
> +# @DESCRIPTION:
> +# A stand-alone implementation of the dosym command aimed for EAPI 8.
> +# Intended to be used for wider testing of the proposed option and to
> +# allow ebuilds to switch to the new model early, with minimal change
> +# needed for actual EAPI 8.
> +#
> +# https://bugs.gentoo.org/708360
> +
> +case ${EAPI} in
> +   5|6|7) ;;
> +   *) die "${ECLASS}: EAPI=${EAPI:-0} not supported" ;;
> +esac
> +
> +# @FUNCTION: _dosym8_canonicalize
> +# @USAGE: 
> +# @INTERNAL
> +# @DESCRIPTION:
> +# Transparent bash-only replacement for GNU "realpath -m -s".
> +# Resolve references to "/./", "/../" and remove extra "/" characters
> +# from , without touching any actual file.
>

Take this as a nit, but do we have any way to test this function
(particularly around edge cases.)
I see eclass/tests exists, could we add a couple for this?


> +_dosym8_canonicalize() {
>

in dosym8() you save and restore IFS, but you don't here, is there a reason
for that?


> +   local path slash i prev out IFS=/
> +
> +   path=( $1 )
> +   [[ $1 == /* ]] && slash=/
> +
> +   while true; do
> +   # Find first instance of non-".." path component followed
> by "..",
> +   # or as a special case, "/.." at the beginning of the path.
> +   # Also drop empty and "." path components as we go along.
> +   prev=
> +   for i in ${!path[@]}; do
> +   if [[ -z ${path[i]} || ${path[i]} == . ]]; then
> +   unset "path[i]"
> +   elif [[ ${path[i]} != .. ]]; then
> +   prev=${i}
> +   elif [[ ${prev} || ${slash} ]]; then
> +   # Found, remove path components and
> reiterate
> +   [[ ${prev} ]] && unset "path[prev]"
> +   unset "path[i]"
> +   continue 2
> +   fi
> +   done
> +   # No (further) instance found, so we're done
> +   break
> +   done
> +
> +   out="${slash}${path[*]}"
> +   echo "${out:-.}"
> +}
> +
> +# @FUNCTION: dosym8
> +# @USAGE: [-r]  
> +# @DESCRIPTION:
> +# Create a symbolic link , pointing to .  If the
> +# directory containing the new link does not exist, create it.
> +#
> +# If called with option -r, expand  relative to the apparent
> +# path of the directory containing .  For example, "dosym8 -r
> +# /bin/foo /usr/bin/foo" will create a link named "../../bin/foo".
> +dosym8() {
> +   local option_r
> +
> +   case $1 in
> +   -r) option_r=t; shift ;;
> +   esac
> +
> +   [[ $# -eq 2 ]] || die "${FUNCNAME}: bad number of arguments"
> +
> +   local target=$1 link=$2
> +
> +   if [[ ${option_r} ]]; then
> +   local linkdir comp
> +
> +   # Expansion makes sense only for an absolute target path
> +   [[ ${target} == /* ]] \
> +   || die "${FUNCNAME}: -r specified but no absolute
> target path"
> +
> +   target=$(_dosym8_canonicalize "${target}")
> +   linkdir=$(_dosym8_canonicalize "/${link#/}")
> +   linkdir=${linkdir%/*}   # poor man's dirname(1)
> +   linkdir=${linkdir:-/}   # always keep the initial "/"
> +
> +   local ifs_save=${IFS-$' \t\n'} IFS=/
> +   for comp in ${linkdir}; do
> +   if [[ ${target%%/*} == "${comp}" 

Re: [gentoo-dev] Automated Package Removal and Addition Tracker, for the week ending 2020-11-08 23:59 UTC

2020-11-09 Thread Alec Warner
On Mon, Nov 9, 2020 at 2:24 AM Ulrich Mueller  wrote:

> > On Mon, 09 Nov 2020, Robin H Johnson wrote:
>
> > The attached list notes all of the packages that were added or removed
> > from the tree, for the week ending 2020-11-08 23:59 UTC.
>
> > [...]
>
> > app-editors/emacs
> 20150808-20:49 robbat2  56bd759df1d
>
> Not really ...
>
> This is broken for the second week in a row now. Could you suspend these
> messages please, until the problem is fixed?
>

This is an infra job, I've disabled it for now.

-A


>
> Ulrich
>


Re: [gentoo-dev] New QA policy suggestion: Disallow "live-only" packages

2020-11-04 Thread Alec Warner
On Wed, Nov 4, 2020 at 7:38 PM Joonas Niilola  wrote:

>
>
> On 11/4/20 11:19 PM, Rich Freeman wrote:
> > Did you consider that somebody could read your email and not actually
> > agree with you?
> Impossible! My suggestion is about keeping the tree clean and to provide
> the best user experience. Who'd disagree with that?!
>
> Sure the methods for achieving that can be discussed, but if I find ~50
> % of current live-only packages being unbuildable, don't you agree
> there's a problem with them? And regardless the outcome of this policy
> suggestion, I've filed 14 bugs to maintainers notifying their --only
> packages aren't working.
>
> > Does it work?  If so, then there is no harm.  If not, then just remove
> > it - you don't need a new policy to treeclean stuff that doesn't work.
> One of the selling points to this policy suggestion is that they'd get
> CI coverage, which they currently don't. Why keep dead weight around for
> years that apparently no one uses, not even the maintainer themself.
>

> > You're saying that live-only packages should be removed because they
> > could have snapshots.  I'm saying that if you want to maintain a
> > snapshot just add it and co-maintain - you don't have to remove
> > packages that don't have them.
> I'm saying currently maintainers aren't doing very good job at
> maintaining them, and no one else uses/finds these packages.
>

Again though, this isn't about having a policy against X or Y and seems
instead to be a policy against unmaintained stuff...and I think we mostly
have a policy against that already; there is a whole team who removes stuff
(treecleaners).

The result is that we should remove badly maintained stuff; not create more
policies.

-A



>
> -- juippis
>
>
>
>


Re: [gentoo-dev] New QA policy suggestion: Disallow "live-only" packages

2020-11-04 Thread Alec Warner
On Mon, Nov 2, 2020 at 9:13 PM Joonas Niilola  wrote:

> Hey,
>
> I'm suggesting a new QA policy to disallow any "live-ebuild-only
> packages" being hosted in ::gentoo. Rationale being the same as why
> - packages can't have KEYWORDS: They are unpredictable and
> potentially insecure. Unpredictability could mean upstream repo being
> broken at any given time placing users in an awkward situation, where
> they are able to build some packages while not the others. Upstream
> repo can also be force-pushed over. I feel like packages offered in
> ::gentoo shouldn't have these issues, and the need to have at least one
> safe release available to users that's guaranteed to build.
>

I disagree. These packages are not installable by default, and must be
unmasked by users, so this tradeoff is one we expect them to make. Are
there practical problems that these packages pose to developers? You listed
a bunch of user problems, but again users are opting into these problems,
presumably.


>
> With Git-like VCS's becoming popular, it is super easy to create an
> unchanged snapshot based on a commit. Even devmanual encourages this
> with proper guide how-to:
>
> https://devmanual.gentoo.org/ebuild-writing/file-format/index.html#snapshots-and-live-ebuilds
>(https://devmanual.gentoo.org/keywording/index.html)
>
> We currently have 43 "live-ebuild-only" packages in tree. Out of those
> 19 are totally unbuildable, that's 44 % of all packages present in the
> repo. Overall the maintenance of these live-ebuild-only packages looks
> low, there's only a handful of ebuilds that have good quality and no
> issues at all. 12 of them, 28 %, are still on EAPI-5. Of all 43, only 2
> would require the maintainer to generate a tarball by themself, while
> others can utilize upstream's tagged releases, or ability to make a
> snapshot from a specific commit. Obviously if this policy passes, I'll
> be helping getting these packages keyworded.
>

But again, why are we making this a firm policy; as opposed to letting the
maintainer make their decision?


>
> And finally here's an example how to introduce new packages to tree
> without upstream releases:
>
> https://gitweb.gentoo.org/repo/gentoo.git/commit/dev-libs/rlottie?id=42873c46b7ed07d5b4f8af5fcf08d8549cb6385b
>
> https://gitweb.gentoo.org/repo/gentoo.git/commit/media-libs/rlottie?id=2de52234783be909f6e4aed333533e6a804e8e6b
>
> https://gitweb.gentoo.org/repo/gentoo.git/commit/media-libs/rlottie?id=8305f0c6cd0ce8cb5ac0f2d92569acce36a5cc0a
>etc...
>
> https://gitweb.gentoo.org/repo/gentoo.git/commit/media-libs/rlottie?id=24c48b325dd5a22284d077d6581a1a45e397e511
>
> If the only available version for a package doesn't build - or can't be
> guaranteed to build - then it should be last-rited, or not exist in
> ::gentoo repo at all.
>

We can't guarantee any package to build..so I'm not sure how this is a
practical policy goal.

-A


>
> Some history and initiative: bug #713802
>
> -- juippis
>
>


Re: [gentoo-dev] [PATCH 1/5] verify-sig.eclass: New eclass to verify OpenPGP sigs

2020-10-12 Thread Alec Warner
On Sun, Oct 11, 2020 at 7:35 AM Joonas Niilola  wrote:

>
>
> On 10/11/20 4:40 PM, Thomas Deutschmann wrote:
> >
> >
> > First of all, calm down. You are reading too much into this. Just
> > revert your own logic: You obviously like your idea, worked on this
> > and pushed it to repository. Don't you see that anyone could ask the
> > same? Who are you? Why do believe that you can force something like
> > that down to everyone's system? That feature got enabled in developers
> > profiles by default, it will blow up distfiles with useless data (a
> > view you can have when you share my concerns and don't see any
> > problems with current Manifest system; For example we banned stuff in
> > $FILESDIR before and asked developers to move them to d.g.o when >25kb
> > in total arguing that large distfile is affecting *any* user... I am
> > not comparing this 1:1 but to repeat your question: Who are you that
> > you can decide to force something similar down to everyone).
> >
> >
> > Sure, if I am the only having that opinion and most people see a value
> > in this and disagree with me...
>

I split this into three parts:

(1) Is there a problem? I like to think there is general agreement that
developers do not cryptographically verify distfiles for most upstreams
when bumping, and thus we could all agree there is room for improvement
here. In an ideal world we are relying on existing systems (https, CAs,
etc) to prevent MITM attacks on source downloads, but not much is used
beyond that.
(2) Does the proposed solution help? This is not a yes or no question; its
nuanced. Having a keyring makes verifying easier. As Whissi notes, we don't
discuss much the managing of said keyrings (or revocation) and so I think
the proposed solution does have similar problems to the existing solution.
Instead of managing and verifying upstream tarballs, we have to verify
keys. There is no automation for this either...and so we end up with a
similar attack surface. There is *improvement* (if someone MITMs your
download, the verification will notice.) Is that the most likely attack, or
is it stolen upstream signing keys? Who can really say?
(3) The implementation. This is honestly the part that I dislike the most,
particularly in the original draft, some of the problems have been fixed
already. I'm not excited about thousands of new packages, nor am I excited
about the key management in the proposal. The biggest problem (that it was
on by default) were already fixed which is good; I don't even see this as a
feature for end users at all; instead its a feature for developers and
maybe a QA bot (that verifies the distfiles.)

Leading out of 3, maybe that is a decent solution also. Can we build a QA
bot that detects bad bumps but also has not terrible key management? Is
there an automated protocol for fetching *and* verifying upstream files
like this? Could we annotate SRC_URI somehow with verification metadata?

-A



> >
> >
> I vote for disabling that USE flag in developer profile. There are more
> than 1 person using it not yet buying or understanding this "draft". I
> also believe such a profile flag change should be discussed first.
>
> -- juippis
>
>


Re: [gentoo-dev] Python 2 cleanup update

2020-09-27 Thread Alec Warner
On Sun, Sep 27, 2020 at 10:45 AM Michał Górny  wrote:

> Hello, everyone.
>
> TL;DR: we're nearing the total annihilation of Python 2 software
> in Gentoo.  Most users could safely disable py2 USE flags today.
> Python 2 vulns have been patched recently, the interpreter and a few
> packages using Python at build time (with no deps) will stay.  Should we
> change PYTHON_TARGETS now, or wait some more and just annihilate
> the py2 flag from all packages?
>
>
> Long version:
>
> We're reached the point where the majority of packages relying on py2
> have either been ported to py3, removed or masked for removal.
> As a result, I've been able to eliminate python2_7 target from the vast
> majority of dev-python/* packages.  On their next system upgrade, our
> users are going to notice most of Python 2.7 modules gone from their
> systems.
>
> However, because of their reverse dependencies a few packages can't lose
> their py2.7-iness, and therefore are going to block depcleaning Python
> 2.7 for now.  These include old versions of setuptools, numpy, pillow,
> as well as all versions of cython, nose, pykerberos, pyyaml and their
> dependencies.  The major blockers for them are:
>
> - dev-lang/gdl (py entirely optional but the package itself is seriously
> broken)
>
> - dev-db/mongodb (py3 version was just stabilized, need to decide how to
> clean old versions up)
>
> - games-engines/renpy (no py3 version yet)
>
> - media-tv/kodi (py3 version in alpha)
>
> We plan to have these packages fixed or removed by the deadline.
>
>
> However, we already know that there are some packages that use Python 2
> at build time and that will keep requiring it past the deadline.
> The initial list includes:
>
> - dev-python/pypy* (TODO: need to figure bootstrap out)
>
> - dev-lang/spidermonkey, www-client/seamonkey, www-client/firefox...
> (thank you, Mozilla)
>
> - www-client/chromium, dev-qt/qtwebengine... (thank you, Google)
>
> Sadly, the big corps are too busy improving their spying functionality
> and creating NIH programming languages to take care of such minor
> matters as cleaning up.
>


https://bugs.chromium.org/p/chromium/issues/list?q=Proj%3DPython3Migration=2
is the tracker for python3 migration for chromium. I resent the implication
that Google is 'too busy' to work on it.

E.g. on
https://bugs.chromium.org/p/chromium/issues/detail?id=941669=Proj%3DPython3Migration=2
the last commit was Sept 26, or yesterday ;p

-A


> The general rule is that py2.7 may remain in packages that use it
> at build time only (i.e. don't install anything depending on Python)
> and have no dependencies on Python packages (i.e. don't require any
> other packages to install py2.7 modules).  Or to put it otherwise,
> python-r1 and python-single-r1 will lose py2.7 support entirely, while
> python-any-r1 will retain minimal support without dependencies.
>
>
> This also implies that we're going to keep Python 2.7 itself for as long
> as necessary, and patch it if possible.  I should take this opportunity
> to remind you that it's quite possible that the interpreter itself has
> unknown vulnerabilities.  Only recently I've backported two sec fixes
> from Python 3 which no other distribution (including the one promising
> paid support for Python 2 for next years) or upstream (including all
> these boasting that they're going to maintain Python 2 themselves) has
> even noticed (to the best of my knowledge).
>
>
> An open question is whether we should remove python2_7 from
> PYTHON_TARGETS now.  If we do that, it will permit the vast majority of
> Gentoo users to depclean Python 2.7 today, independently of how long
> the maintainer of renpy is going to block it, with only a few users
> having to enable the flag manually.  However, doing this makes sense
> only if we're really going to delay the impeding doom long.
>
> I will probably prepare an updated news item for Python 2.7 removal,
> to replace the one from February with the updated plan, current
> information and helpful tips.
>
>
> Finally, I would like to thank all the helpful package maintainers, arch
> teams and other developers who have made this possible.
>
> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] [RFC] Services and software which is critical for Gentoo should be developed/run in Gentoo namespace

2020-09-16 Thread Alec Warner
On Wed, Sep 16, 2020 at 1:17 AM Kent Fredric  wrote:

> On Mon, 14 Sep 2020 10:15:31 -0400
> Rich Freeman  wrote:
>
> > It might be easier to take smaller steps, such as having a policy that
> > "any call for devs to use/test a new tool/service, or any service that
> > automatically performs transactions on bugzilla, must be FOSS, and the
> > link to the source must be included in the initial communication, and
> > it must be clear what version of the code is operating at any time."
> > That is a pretty low barrier to those creating tools, though it
> > doesn't address the infra concern.  However, it does mean that infra
> > is now free to fork the service at any time, and reduces the bus
> > factor greatly.
>
> For the situation of things that take life before being part of infra,
> I think the least we can do is recognize their utility and importance,
> and at least, have infra *offer* some sort of shared location to run a
> deployment.
>
> That I think helps everyone, gives people a place to remove their own
> bus factor, but without mandatory strongarming.
>

The main challenge is always getting stuff onto infra. The past few things
we have launched have been because the developers in question joined infra
to launch their work.

 - repomirror-ci and all the CI stuff is on infra because mgorny is also on
infra! It's not like we set his stuff up for him; instead we gave him
access to all the infra repos and he had to write his own puppet configs
and whatnot. The benefit of course is that anyone on infra can bump the
stuff and login to the machines and debug...but its not exactly a low bar.
 - packages.gentoo.org was also originally arzano pushing patches to me
(which I would merge and then release by hand.) Just like with ebuilds this
eventually became too tedious and he was onboarded as a dev and infra
member to eliminate the middleman.

I think traditionally it's been a slog for non-infra people to get infra to
host much of anything and due to difficulties with the all-or-nothing
approach we take with infra credenteials; can really set a high bar to host
much of anything these days.

-A


Re: [gentoo-dev] [RFC] Services and software which is critical for Gentoo should be developed/run in Gentoo namespace

2020-09-13 Thread Alec Warner
On Sun, Sep 13, 2020 at 11:31 AM Thomas Deutschmann 
wrote:

> Hi,
>
> TL;DR: jstein asked council [Bug 729062] for a motion that any service
> and software which is critical for Gentoo should be developed/run in
> Gentoo namespace. Because any request to council must be discussed I
> volunteered to bring this topic to the mailing list (sorry for the huge
> delay!).
>
>
> Problem
> ===
> You maybe all remember what happened to stable-bot: Years ago,
> kensington created stable-bot on his own as PoC which revolutionized the
> way how we do package stabilization in bugzilla. The service run on his
> own infrastructure. Because of the benefit of the service the bot
> provided, arch team’s workflow became dependent on stable-bot. We were
> lucky that stable-bot just worked most of the time until the service was
> down for a while. Nobody was able to help here: Kensigton himself was
> unavailable, nobody had the sources… the end of the story: mgorny
> created nattka which replaced stable-bot.
>
> However, we are still facing the same problem: Only one person is
> involved in development and knows how to run it. In case something will
> break again and Michał will be unavailable, we can’t just push a fix and
> watch a CI pipeline picking up and deploying new nattka. Instead someone
> will have to fork repository from Michał’s private repository at GitHub,
> make the changes and hope that anyone within infrastructure team can
> help to deploy fixed nattka.
>
> This is what the motion is about: This is not about that Gentoo depends
> on single persons or things like that. It’s about the idea to
> *formalize* the requirement that any service and software which is
> critical for Gentoo (think about pkgcore) should live within Gentoo
> namespace (https://gitweb.gentoo.org/), i.e. be accessible for *any*
> Gentoo developer and deployments should be based on these repositories.
> Or in other words: Make sure that we adhere to social contract even for
> critical software and services Gentoo depends on. So that we will never
> ever face the situation that something we depend on doesn’t work
> anymore. Taking care of working pipelines before something is broken
> should also help us in case something stops working so we don’t have to
> figure out how to fix and re-deploy when house is already burning (like
> portage: In case Zac can't do a release for some reason, in theory,
> every Gentoo developer would be able to roll a new release).
>

I think your examples are a bit weird.

Is openrc critical to Gentoo? it doesn't live on our infra.
Is pkgcore critical to Gentoo? it doesn't live on our infra.

Note that these are just packages, not services and the social contract
just says
"""However, Gentoo will never depend upon a piece of software or metadata
unless it conforms to the GNU General Public License, the GNU Lesser
General Public License, the Creative Commons - Attribution/Share Alike or
some other license approved by the Open Source Initiative."""

It says nothing about where things are hosted or how services are provided.

I'd consider splitting the two here. For packages I don't think it matters
as much where they are hosted. Most things can be mirrored into gentoo (if
we want a copy of the src tree) and we also have tarballs of the source
code much of the time on the mirror network.

For services, I tend to agree more with your comments; we need need
visibility and operational capability for services. When we rely on service
components where the source is not available; its bad. But we rely on
numerous services now. E.g. p.g.o relies on repology. Does that mean we
need the source code to repology? I assume not. Does that mean we need to
run our own repology? Also I assume not.

-A


>
> See also:
> =
> Bug 729062: https://bugs.gentoo.org/729062
>
>
> --
> Regards,
> Thomas Deutschmann / Gentoo Linux Developer
> C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
>
>


Re: [gentoo-dev] [PATCH] profiles/targets/desktop: Do not enable ldap USE flag by default

2020-09-10 Thread Alec Warner
On Thu, Sep 10, 2020 at 8:13 AM Michał Górny  wrote:

> On Thu, 2020-09-10 at 07:35 +0200, Hans de Graaff wrote:
> > On Wed, 2020-09-09 at 13:35 +0300, Mikle Kolyada wrote:
> > > Closes: https://bugs.gentoo.org/741380
> >
> > Could you provide a rationale for removing this? The bug only has a
> > single anecdotal report of a user who can run a desktop without it. I'm
> > not sure if that is reason enough to remove this. I guess we won't be
> > able to figure out easily how many of our desktop profile users are
> > actually using LDAP, but changing this may cause surprises and I'm not
> > sure if that's warranted.
> >
>
> 2020.  Gentoo discovers that people usually don't run LDAP on their
> desktops.  Some developers can't believe their eyes!  Gentoo forms
> a Working Group to establish whether using LDAP on desktops is really
> that uncommon.
>
> This fits nicely with the recently announced fact that Gentoo Foundation
> has too much money and is looking for funding requests.  After all, what
> could be a better use of Gentoo money than funding the work of a Working
> Group trying to establish matters of such great importance as default
> USE flags on desktop profiles.
>

I don't appreciate these comments. You can think the Foundation wastes
money on taxes and paperwork and operational expenses but i'm not going to
sit here and watch you crap all over my attempts to actually *support* the
Gentoo community by funding actual work. So please keep your negative
comments to yourself.

-A


>
> After a lot of debate the Working Group concludes that there is no
> conclusive answer whether LDAP should be enabled by default or not.
>

> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] [PATCH] profiles/targets/desktop: Do not enable ldap USE flag by default

2020-09-10 Thread Alec Warner
On Thu, Sep 10, 2020 at 1:59 AM Mikle Kolyada  wrote:

>
> On 10.09.2020 08:35, Hans de Graaff wrote:
> > On Wed, 2020-09-09 at 13:35 +0300, Mikle Kolyada wrote:
> >> Closes: https://bugs.gentoo.org/741380
> > Could you provide a rationale for removing this? The bug only has a
> > single anecdotal report of a user who can run a desktop without it. I'm
> > not sure if that is reason enough to remove this. I guess we won't be
> > able to figure out easily how many of our desktop profile users are
> > actually using LDAP, but changing this may cause surprises and I'm not
> > sure if that's warranted.
> >
> > Hans
>
>
> Hi.
>
> It is dictated by common sense.
>

> I barely can imagine a case where you need ldap support in each and
> every package you install.
>

I can, but not in a desktop environment. In an enterprise environment you
will need it (but I'd expect folks to add it.)
The challenge is just in changing the default. E.g. we might not do it
until a new profile bump (e.g. do it in a 2020 or 2021 profile?)

-A


>
> This should rather be per-package enabled as something non-trivial.
>
>
>


Re: [gentoo-dev] rfc: switching default udev provider for new systems to udev

2020-08-09 Thread Alec Warner
On Sun, Aug 9, 2020 at 11:22 AM William Hubbs  wrote:

> On Sun, Aug 09, 2020 at 06:40:07PM +0200, Thomas Deutschmann wrote:
> > On 2020-08-08 20:51, William Hubbs wrote:
> > > What do people think?
> >
> > Like others already asked: What's the reason for this?
>
>  Like others have said on the thread, the reason for the switch away
>  from udev in the past was mostly fear driven instead of fact driven. As
>  already said, if the udev developers were going to make udev unusable
>  without systemd they would have by now.
>
> > What do you expect from this change?
>
>  I expect Gentoo to use, by default, what most of the Linux community
>  uses for device management.
>

"I expect Gentoo to use, by default, what most of the Linux community uses
for init management." So we should make the systemd profile the default? :)


>
> > Is there a problem when new Gentoo installations will use EUDEV by
> > default? Or is there a benefit if new installations would use
> sys-fs/udev?
>
> Please look back at the history of why we switched away from udev. It
> was not technical. Udev did not cause any wide scale distro breakages.
> It was because some folks were very loud about a possible systemd
> consppiracy around making udev not work without systemd.
>

You asked me on IRC "how do I convince people" and part of that is to make
it easy to agree with your argument! Asking me to read a bunch of crap
isn't going to make me want to agree; its going to make me say "your
argument is poorly formed, please go away."

 - Link to the things you want me to read.
 - Summarize them so I don't have to read a 100 message long thread from 5
years ago.
 - Make an argument!

---
"I think we picked eudev as the default because of a concern that udev
would eventually require systemd for operation, you can see this from these
mailing list posts: X, Y, Z."
"The above concern has not manifested itself and I believe udev will
continue to not strictly require systemd init for various reasons (mention
list of cases here."
"Therefore I think we should change the default udev provider from eudev to
udev in the default profiles."
---

This would be what I believe is a understandable argument (provided we had
the links to the previous material.) I'm not saying I agree[0] with it; but
I'd at least understand why you want the change to happen.


> Notice again that I'm not saying we need to lastrites eudev. There are
> cases that have developed for it (mainly non-glibc systems), but I am
> saying I see no justification at this point for it being the default
> distro wide.

William
>
>
[0] I expect that most users who want udev actually also want systemd and
so will simply select the systemd profile itself, and that this choice is
immaterial to most users; so I am for keeping the status quo here.


[gentoo-portage-dev] pylint progress

2020-07-29 Thread Alec Warner
Hi,

Recently I've begun to run pylint on the portage codebase. You can see some
recent PRs on this[0][1][2]. Most of the linter errors I've fixed are what
I consider 'fairly trivial'. In general I'm happy to disable errors (or
instances of errors) in addition to resolving them. You can see some of
this in https://github.com/gentoo/portage/pull/593/files where we just
blanket disable a bunch of very numerous messages (either because I don't
expect to ever fix them, or because they are more work that I think we
should do at the moment.)

So I see essentially a few choices:
 - (a) Do we fix errors in certain classes and run the linter as part of
CI. This means that once we resolve errors of a certain class, we can
enable that class of check and prevent new occurrences.
 - (b) Do we just avoid running the linter as part of CI (perhaps because
we are not far enough along on this journey) and focus our efforts to get
to a point where we can do (a) without spamming CI?
 - (c) What messages should we bother not fixing at all so we can just not
work on those classes of error?

As an example of (c); I believe 'protected-access' is likely intrusive to
fix and pointless for portage (cat is out of the bag) where as a message
like W0612(unused-variable) is plausibly something we should fix
throughout the codebase. Similarly I think "E0602(undefined-variable)" is
often a bug in how pylint treats our lazy initialized variables, so most of
these are false positives.

I think getting these messages recorded will also enable other people
besides me to fix them (I know b-man said he was interested.) Curious to
hear thought on this.

-A

[0]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=49794e185350f0f9c8099ff84507873685b2b0f1
[1]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=550727af87d5f646617e0c19a3f3300c8117e7f5
[2]
https://gitweb.gentoo.org/proj/portage.git/commit/?id=be20b37180f709ab0e451e4e07b6e82ac3a87b56


Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Alec Warner
On Thu, Jul 9, 2020 at 2:06 PM Chun-Yu Shei  wrote:

> Hmm, that's strange... it seems to have made it to the list archives:
> https://archives.gentoo.org/gentoo-portage-dev/message/a4db905a64e3c1f6d88c4876e8291a65
>
> (but it is entirely possible that I used "git send-email" incorrectly)
>

Ahhh it's visible there; I'll blame gMail ;)

-A


>
> On Thu, Jul 9, 2020 at 2:04 PM Alec Warner  wrote:
>
>>
>>
>> On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei  wrote:
>>
>>> Awesome!  Here's a patch that adds @lru_cache to use_reduce, vercmp, and
>>> catpkgsplit.  use_reduce was split into 2 functions, with the outer one
>>> converting lists/sets to tuples so they can be hashed and creating a
>>> copy of the returned list (since the caller seems to modify it
>>> sometimes).  I tried to select cache sizes that minimized memory use
>>> increase,
>>> while still providing about the same speedup compared to a cache with
>>> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
>>> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
>>> RES column in htop increases from 280 MB -> 290 MB.
>>>
>>> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
>>> max observed RES value actually decreases from 228 MB -> 214 MB (similar
>>> values observed across a few before/after runs).
>>>
>>> Here are the cache hit stats, max observed RES memory, and runtime in
>>> seconds  for various sizes in the update case.  Caching for each
>>> function was tested independently (only 1 function with caching enabled
>>> at a time):
>>>
>>> catpkgsplit:
>>> CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419)
>>> 270 MB
>>> 39.217
>>>
>>> CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1)
>>> 271 MB
>>> 39.112
>>>
>>> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000)
>>> 271 MB
>>> 39.217
>>>
>>> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500)
>>> 269 MB
>>> 39.438
>>>
>>> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000)
>>> 271 MB
>>> 39.348
>>>
>>> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100)
>>> 271 MB
>>> 39.487
>>>
>>>
>>> use_reduce:
>>> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561)
>>> 407 MB
>>> 35.77
>>>
>>> CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1)
>>> 353 MB
>>> 35.52
>>>
>>> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000)
>>> 335 MB
>>> 35.31
>>>
>>> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500)
>>> 318 MB
>>> 35.85
>>>
>>> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000)
>>> 301 MB
>>> 36.39
>>>
>>> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100)
>>> 299 MB
>>> 37.175
>>>
>>>
>>> I didn't bother collecting detailed stats for vercmp, since the
>>> inputs/outputs are quite small and don't cause much memory increase.
>>> Please let me know if there are any other suggestions/improvements (and
>>> thanks Sid for the lru_cache suggestion!).
>>>
>>
>> I don't see a patch attached; can you link to it?
>>
>> -A
>>
>>
>>>
>>> Thanks,
>>> Chun-Yu
>>>
>>>
>>>
>>>


Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Alec Warner
On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei  wrote:

> Awesome!  Here's a patch that adds @lru_cache to use_reduce, vercmp, and
> catpkgsplit.  use_reduce was split into 2 functions, with the outer one
> converting lists/sets to tuples so they can be hashed and creating a
> copy of the returned list (since the caller seems to modify it
> sometimes).  I tried to select cache sizes that minimized memory use
> increase,
> while still providing about the same speedup compared to a cache with
> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
> RES column in htop increases from 280 MB -> 290 MB.
>
> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
> max observed RES value actually decreases from 228 MB -> 214 MB (similar
> values observed across a few before/after runs).
>
> Here are the cache hit stats, max observed RES memory, and runtime in
> seconds  for various sizes in the update case.  Caching for each
> function was tested independently (only 1 function with caching enabled
> at a time):
>
> catpkgsplit:
> CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419)
> 270 MB
> 39.217
>
> CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1)
> 271 MB
> 39.112
>
> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000)
> 271 MB
> 39.217
>
> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500)
> 269 MB
> 39.438
>
> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000)
> 271 MB
> 39.348
>
> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100)
> 271 MB
> 39.487
>
>
> use_reduce:
> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561)
> 407 MB
> 35.77
>
> CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1)
> 353 MB
> 35.52
>
> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000)
> 335 MB
> 35.31
>
> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500)
> 318 MB
> 35.85
>
> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000)
> 301 MB
> 36.39
>
> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100)
> 299 MB
> 37.175
>
>
> I didn't bother collecting detailed stats for vercmp, since the
> inputs/outputs are quite small and don't cause much memory increase.
> Please let me know if there are any other suggestions/improvements (and
> thanks Sid for the lru_cache suggestion!).
>

I don't see a patch attached; can you link to it?

-A


>
> Thanks,
> Chun-Yu
>
>
>
>


Re: [gentoo-dev] Last rites: */* More Py2 only items

2020-06-28 Thread Alec Warner
On Sun, Jun 28, 2020 at 5:35 PM Aaron Bauman  wrote:

> # Aaron Bauman  (2020-06-28)
> # More Py2 only stuff. Plz see -dev ML for discussions
> # Remove bindings, port to Py3, etc
> # Removal in 30 days
> app-arch/deltarpm
> app-crypt/virtualsmartcard
> app-text/duali
> app-text/mftrace
> app-text/queequeg
> app-text/referencer
> dev-libs/libmacaroons
> dev-libs/tut
> dev-python/elib-intl
> dev-python/eunuchs
> dev-python/medusa
> dev-python/misaka
> dev-python/python-iwscan
> dev-util/confix
> dev-util/qmtest
> dev-util/unrpyc
> games-engines/gemrb
> media-sound/lilycomp
> media-video/tovid
> net-dns/maradns
> net-irc/irker
> net-mail/archivemail
> net-mail/getmai
> net-nds/nsscache
>

Infra uses this, we will have to look into bumping it.

-A


> net-wireless/airpwn
> sci-chemistry/bkchem
> sci-chemistry/propka
> sci-chemistry/pymol-plugins-bni-tools
> sci-chemistry/pymol-plugins-emovie
> sci-chemistry/viewmol
> sci-libs/chemkit
> sys-apps/x86info
> www-misc/surl
> x11-wm/plwm
>
> --
> Cheers,
> Aaron
>


Re: [gentoo-dev] */*: Mask Py2 only packages

2020-06-24 Thread Alec Warner
On Wed, Jun 24, 2020 at 11:29 AM Rich Freeman  wrote:

> On Wed, Jun 24, 2020 at 2:18 PM Andreas Sturmlechner 
> wrote:
> >
> > The lack of curiosity for one's own packages' python compatibility is
> not just
> > a py27 isolated issue, it was a big problem with py36 -> py37 with so
> many
> > devs simply not filing that necessary stabilisation.
>
> That suggests that if you keep doing what you're doing, you're going
> to keep hitting your head against the wall.
>
> Right now in Gentoo there isn't really even a straightforward way for
> a maintainer to cleanly obtain a list of all the packages they
> maintain, let alone whether they use python v2.
>

> Sure, you can use the portage API to find this info.  However, that is
> as easy to do for a list of all impacted packages in the tree with
> their maintainers as for any individual maintainer to obtain this info
> for their own packages.
>

You say there is not a straightforward way, but then you say there is an
api? :p


>
> I think that if you give the maintainers a bit more info, you'll find
> them being more proactive about helping you out.  Basically you would
> be helping them help you.
>
> Otherwise you're going to mask a bunch of packages and run into a
> bunch of upset devs, and as a byproduct we create a bunch of upset
> users.
>

Extend the existing QA report?

https://qa-reports.gentoo.org/output/gpyutils/py2.txt

There is a list of py2 only packages. We just need to add the maintainer
metadata?


>
> There is no reason to mask a package only to unmask it a few days
> later.  Masks are a mechanism for deprecating packages so that users
> take action.  They're not a substitute for devs talking to each other.
>

> --
> Rich
>
>


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-06-02 Thread Alec Warner
On Tue, Jun 2, 2020 at 1:44 AM Mathy Vanvoorden  wrote:

> Hi,
>
> Sorry I'm a bit late to the party here but was behind on my emails.
>
> Op wo 27 mei 2020 om 05:25 schreef Alec Warner :
>
>> The TL;DR is that a crack team of infra-folks[0] have been putting
>> together demos of CI services and things like gitlab / gitea / gerrit and
>> so on.
>>
>>
> I didn't see in the email chain any mention of the Atlassian tools. Can
> these be considered? They offer free licenses for open source projects and
> I think that most of the requirements are covered by BitBucket
> (repo-hosting, repo-serving, code review and pull requests) and Bamboo (CI).
>

I inquired about this generally on the nfp list in May:

https://archives.gentoo.org/gentoo-nfp/message/0142bac838ed7cb356e58447c6ee20b0

The conclusion was that the software needed to be released under an open
license (preferably FSF / OSI approved.) So for example, Gitlab CE is OK
(its an OSI approved licensed) but Gitlab EE is not. The source for Gitlab
EE is open (I can go read it) but it's not freely available.


>
> If desired I can setup a demo and/or help maintain the tools, this is my
> day job and for those who care about these kind of things I am Atlassian
> certified.
>
> Here is more info on their licensing for Open Source projects:
>
> https://www.atlassian.com/software/views/open-source-license-request
>

My understanding is that this is a binary-own download that, once
installed, we can request a license to use. We won't have the source code
at all and the license is not free. I don't think it's possible for us to
use this software in Gentoo for this reason.

-A


>
> br,
> Mathy
>


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-31 Thread Alec Warner
On Fri, May 29, 2020 at 11:17 PM Michał Górny  wrote:

> On Fri, 2020-05-29 at 16:34 -0700, Alec Warner wrote:
> > The pull-based mirroring is a bit sad, as it would be nice to auto-update
> > some forks, but it's not a killer feature.
>
> Exactly.  Especially that our push-based mirroring is better,
> and I think that's how we want to populate it.
>

So you want to keep gitolite then?


>
> >  I think our new SSO solution
> > could potentially be a fix for the auth subsystems, but more work there
> > will be needed.
>
> I think SSO should be the primary login to our GitLab, especially for
> our users.  GitHub login is a must.
>

I'm fairly sure both gitlab and gitea support this requirement.


>
> > Another major issue is operating the software. I haven't found anyone to
> > *run* gitlab; I'm not eager to do it. Today Gentoo is mostly distributed,
> > bugs are in bugzilla, wiki is on mediawiki, code is on gitolite with N
> > mirrors, email and lists are separate, etc. In a world where bugs, wiki,
> > code, ci, containers, PRs, are all on gitlab and it breaks and we can't
> fix
> > it; it will be bad news for all of those things. If the bugzilla machine
> > breaks we lose bugzilla; if gitlab breaks we lose the ability to edit the
> > wiki, file bugs, commit, run CI, etc.
> >
>
> But who says we want to migrate them all into GitLab?  I thought our
> primary goal was to replace today's GitHub use, i.e. provide
> an alternative pipeline for pull/merge requests.
>

Oh I don't, but we then need to disable all of them and restrict their
usage.

-A


>
> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-29 Thread Alec Warner
On Wed, May 27, 2020 at 2:31 PM Matt Turner  wrote:

> On Wed, May 27, 2020 at 1:14 AM Alec Warner  wrote:
> > On Tue, May 26, 2020, 23:08 Michał Górny  wrote:
> >>
> >> On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
> >> > The TL;DR is that a crack team of infra-folks[0] have been putting
> together
> >> > demos of CI services and things like gitlab / gitea / gerrit and so
> on.
> >> >
> >> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> >> > review / pull reqs, CI services, and deploy services.) Some of these
> are
> >> > piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
> >> > repo-hosting but CI is separate (e.g. drone.)
> >> >
> >> > On the infra-side, I think we are pretty happy with repo-hosting
> (gitolite)
> >> > and repo-serving (gitweb). We are missing a CI piece and a
> pull-request
> >> > piece. Most of the users using PRs use either a gitlab or github
> mirror.
> >> >
> >> > I think the value of CI is pretty obvious to me (and I see tons of use
> >> > cases in Infra.) We could easily build CI into our current repository
> >> > solution (e.g. gitolite.) However gitolite doesn't really support PRs
> in a
> >> > uniform way and so CI is mostly for submitted code; similar to the
> existing
> >> > ::gentoo repo CI offered by mgorny.
> >> >
> >> > If we build a code review solution (like gitea / gerrit) would people
> use
> >> > it? Would you use it if you couldn't merge (because the code review
> >> > solution can't gpg sign your commits or merges) so a tool like the
> existing
> >> > pram tool would be needed to merge?
> >> >
> >>
> >> Does GitLab count?  Gerrit is just PITA.  I think we had some concerns
> >> about Gitea, so I'd like to test it before deciding.  GitLab OTOH works
> >> just fine for a lot of projects, and seems the next best thing after
> >> GitHub
> >
> >
> > Gitlab does count (we deployed and tested an onprem version.) I think
> there are some major issues with it though.
> >  - Licensing. Gitlab-CE is available, gitlab-EE is not OSS nor OSI
> approved and many of the features we need are EE only and are not available
> in CE.
>
> It's very surprising to me that CE wouldn't work for our purposes.
> Debian, GNOME, KDE, XFCE, and FreeDesktop have all switched to GitLab
> and are using CE. It's hard to believe that Gentoo's usage or
> requirements would be so different as to make GitLab a non-viable
> option.
>
> What features of EE do you think we need?
>

I know debian spent considerable effort customizing their gitlab. I've not
heavily investigated the other deployments. We set up a demo on
gitlab.gentoo.org already as a test and there are some issues. Many of them
are related to operations and not to the app itself.

[mirroring]
 - We can't do pull-based git mirroring (
https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html#pulling-from-a-remote-repository-starter
)
[Auth] Note that we have keycloak now, so we can perhaps workaround these
LDAP issues.
 - GItlab doesn't support multiple redundant LDAP servers
 - Gitlab doesn't support LDAP group syncing
 - Gitlab doesn't support syncing admin users from LDAP
[mgmnt]
 - The gitlab terraform provider is pretty bad; and I struggled to get it
to control our admin users; did not leave me excited about managing other
resources (projects, users, hooks, etc.)

The pull-based mirroring is a bit sad, as it would be nice to auto-update
some forks, but it's not a killer feature. I think our new SSO solution
could potentially be a fix for the auth subsystems, but more work there
will be needed.

Another major issue is operating the software. I haven't found anyone to
*run* gitlab; I'm not eager to do it. Today Gentoo is mostly distributed,
bugs are in bugzilla, wiki is on mediawiki, code is on gitolite with N
mirrors, email and lists are separate, etc. In a world where bugs, wiki,
code, ci, containers, PRs, are all on gitlab and it breaks and we can't fix
it; it will be bad news for all of those things. If the bugzilla machine
breaks we lose bugzilla; if gitlab breaks we lose the ability to edit the
wiki, file bugs, commit, run CI, etc.

-A


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-27 Thread Alec Warner
On Wed, May 27, 2020 at 5:16 AM Thomas Deutschmann 
wrote:

> Hi,
>
> is this really CI _vs_ Code Review? I.e. we can only have one?
>

I'll try to come back to this. We can make computers do anything, but it's
a question of limited time for me and for Gentoo as a whole.


>
> CI is nice and helpful. For example I usually don't start review until CI
> sets green light but CI alone wouldn't be enough. Thought that Gitlab will
> support same kind of hooks similar to how current CI is integrated into
> Github, not?
>

So I'm basically back to worrying about social contract here. If its a
common Gentoo workflow to:
 - Have a user fork a repo (github)
 - Submit a PR (github)
 - Gentoo CI sees the PR against ::gentoo and runs CI (github)
 - CI turns green and comments on the PR (github)
 - Human Gentoo dev reviews PR (github) [some iterations of review and
changes with PR author]
 - Human Gentoo dev downloads PR patch and applies to Gentoo (Gentoo
Hosted.)

Does this workflow meet the social contract? I would assert it does not.
Now of course Gentoo doesn't "solely rely" on this workflow and other
workflows are available; but if this is a de-facto workflow people are
using...does not that concern the community?  It concerns me! Hey we
originally were like 'nah github is OK' and now I'm coming around and
saying 'hey maybe this isn't OK in its current form.' So one idea might be
"what can we do about this particular problem."

What others have done is typically left mirrors of their projects on
github, written bots to auto-close PRs to those repos, and funneled users
somewhere else. We could do that if we wanted.


>
> Also, when I could wish something: The problem when doing review on Github
> for me is, that we usually create new revisions. Therefore we don't see
> what's changed in new revision versus previous revision. So the change to
> review looks like an entire new file was added (which is what basically
> happened). It would be cool if our solution would be aware of this and
> could handle this somehow. But I guess we would have to create our own
> solution for this...
>

So to address your first point above, there are a bunch of (sometimes
conflicting) requirements.

[Github]
A past argument for Github was "we need to be on github because this is
where the users are." I can tell you the users have not left Github and so
without us funneling them somewhere else...I mean if we are going to
continue to accept PRs on github we should probably service them (vs
auto-closing.)

[Code Review]
Basically how do we build a system where a user can go find our code, fork
it, generate a patch, send us the patch, and then be able to run CI on it
and do a review online (as opposed to on bugzilla / in-email.) Currently
this happens on github and I think there is some support for moving those
users to some other solution.

We could:
 - Build something on gitlab CE.
 - Build something on gitea + a CI solution (drone, zuul, etc.)
 - I don't think gerrit credibly meets these requirements because users
can't fork in gerrit and it requires annoying client side tooling.
 - The main problem I think with gitlab-CE is no one wants to operate it;
its too complex of a software package and there is a major fear that once
we mgirate everyone on it, something will go wrong and then we will be in
trouble. I don't have this fear with gitea because there are fewer moving
parts.
 - The main problem with gitea is that it's relatively unproven and we
already know it needs patches for ::gentoo.

[CI-for-ebuilds]
My understanding is that there are two existing CI workflows (I'm ignoring
the new thing nattka; it's what I'd consider a different workflow.)
(1) Github PRs: A bot watches for PRs against ::gentoo on Github and runs
CI on them.
(2) Post-Submit CI: Every so often we run Ci against ::gentoo and if
something looks super bad we bisect and email that the build is broken.

So right off the bat there are some gaps.
 - There is no pre-submit CI on git.gentoo.org at all, so devs can't vet
their changes there. Would anyone like this?
 - There is no flexibility; the CI you get is whatever is configured (e.g.
what mgorny set up). This isn't meant as a dig; maybe no one wants to set
up CI at all and just wants to use this stuff; I have no idea.

[Other Projects]
Portage has had CI for ages; but it runs on github and uses travis-CI
soko.git uses hosted gitlab-ci to build, run tests, and upload containers.
https://gitweb.gentoo.org/proj/catalyst.git/ has no CI, but might like some?
https://gitweb.gentoo.org/proj/gentoo-mirrorstats.git/
 has no CI but
might like some?
https://gitweb.gentoo.org/proj/sandbox.git/ has no CI but might want some?
I could go on.

So to kind of summarize. If I wanted to just do [CI for other projects] we
could likely just build that now and not need to talk to most of the
community because the CI for other projects is fairly segmented. It's the
part that 

Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-27 Thread Alec Warner
On Wed, May 27, 2020 at 1:09 AM Brian Dolbec  wrote:

> On Tue, 26 May 2020 20:24:56 -0700
> Alec Warner  wrote:
>
> > The TL;DR is that a crack team of infra-folks[0] have been putting
> > together demos of CI services and things like gitlab / gitea / gerrit
> > and so on.
> >
> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> > review / pull reqs, CI services, and deploy services.) Some of these
> > are piecemeal (e.g. gerrit has code review, zuul has CI) and gitea
> > offers repo-hosting but CI is separate (e.g. drone.)
> >
> > On the infra-side, I think we are pretty happy with repo-hosting
> > (gitolite) and repo-serving (gitweb). We are missing a CI piece and a
> > pull-request piece. Most of the users using PRs use either a gitlab
> > or github mirror.
> >
> > I think the value of CI is pretty obvious to me (and I see tons of use
> > cases in Infra.) We could easily build CI into our current repository
> > solution (e.g. gitolite.) However gitolite doesn't really support PRs
> > in a uniform way and so CI is mostly for submitted code; similar to
> > the existing ::gentoo repo CI offered by mgorny.
> >
> > If we build a code review solution (like gitea / gerrit) would people
> > use it? Would you use it if you couldn't merge (because the code
> > review solution can't gpg sign your commits or merges) so a tool like
> > the existing pram tool would be needed to merge?
> >
> > -A
> >
> > [0] Mostly arzano, if I'm honest. I am just the point-y haired
> > manager in this effort.
>
> There are several CI systems that could be used.  As for CI, my
> experience has been with buildbot.  It would be fairly straightforward
> to integrate with the current gitolite/gitweb hosting.
> It is also extremely flexible in its capabilities, although can be
> difficult to master.  It has a good web interface, but it can even be
> run via it's API's using curses, python, bash,   There is a
> buildbot-travis plugin which is capable of running existing .travis file
> configurations.  It also extends its capabilities to include custom
> buildbot step definitions.  I have not packaged it yet for gentoo, but
> it is on my todo list. One of buildbot's capabilities is the ability to
> run tests/builds on multiple workers (different arches or whatever)
> both in parallel or series.  It could be made to integrate with our
> bugzilla using the python client (pybugs was it?).
>

My understanding is that CI systems are all quite similar. We have
configured gitlab-ci, drone, and zuul as tests and I am familiar with
travis-ci and buildbot from other projects. I'm less concerned about this
aspect tbh. I'm also not super concerned about packaging. E.g. for
gitlab-runner (the worker portion of gitlab-ci) we just deploy upstream
containers and don't package them in ::gentoo at all. Our onprem gitlab is
a container solution; gitea is a container solution; our SSO IDP is a
container solution; gerrit is currently a container solution. These don't
bother me (too much, ugh zuul uses zookeeper? ugh.)

The major differentiators for CI appear to be:
 - SCM support (we currently only really care about git compatible systems,
but some contenders only support some providers.)
 - builders / workers (how do they deploy). For example gitlab has a
container based work executor while zuul uses ansible; I suspect
 - Authentication (ideally should support SAML or openID so we can
integrate with our alpha sso solution for Gentoo.)
 - The webUI; e.g is the solution horrible to use / interact with. Hard to
say without a trial, IMHO.

-A


>
> But that still leaves a PR/code review option.
>
>


Re: [gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-27 Thread Alec Warner
On Tue, May 26, 2020, 23:08 Michał Górny  wrote:

> On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
> > The TL;DR is that a crack team of infra-folks[0] have been putting
> together
> > demos of CI services and things like gitlab / gitea / gerrit and so on.
> >
> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> > review / pull reqs, CI services, and deploy services.) Some of these are
> > piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
> > repo-hosting but CI is separate (e.g. drone.)
> >
> > On the infra-side, I think we are pretty happy with repo-hosting
> (gitolite)
> > and repo-serving (gitweb). We are missing a CI piece and a pull-request
> > piece. Most of the users using PRs use either a gitlab or github mirror.
> >
> > I think the value of CI is pretty obvious to me (and I see tons of use
> > cases in Infra.) We could easily build CI into our current repository
> > solution (e.g. gitolite.) However gitolite doesn't really support PRs in
> a
> > uniform way and so CI is mostly for submitted code; similar to the
> existing
> > ::gentoo repo CI offered by mgorny.
> >
> > If we build a code review solution (like gitea / gerrit) would people use
> > it? Would you use it if you couldn't merge (because the code review
> > solution can't gpg sign your commits or merges) so a tool like the
> existing
> > pram tool would be needed to merge?
> >
>
> Does GitLab count?  Gerrit is just PITA.  I think we had some concerns
> about Gitea, so I'd like to test it before deciding.  GitLab OTOH works
> just fine for a lot of projects, and seems the next best thing after
> GitHub


Gitlab does count (we deployed and tested an onprem version.) I think there
are some major issues with it though.
 - Licensing. Gitlab-CE is available, gitlab-EE is not OSS nor OSI approved
and many of the features we need are EE only and are not available in CE.
 - Complex: Gitlab is a giant piece of software with maybe 8-12 components
(unicorn, postgres, redis, memcache, sidekiq, puma, workhouse, gitaly,
grafana, sshd,nginx, prometheus ..the list goes on)
 - I think gitlab ships with more features than we will use (CD, docker
registry, issues / bugs, wiki, analytics, snippets, milestones, repo
hosting, repo browsing, ... Again the list goes on.) I don't play to
migrate away from bugs.gentoo.org nor wiki.gentoo.org, nor gitolite. I
think if we did; then gitlab would be a more compelling option because it
is a one-stop-shop solution for those use cases.

My understanding of gitea is that it works great for not-::gentoo, but
::gentoo and gitea don't work well and it would require work upstream to
fix; other large repos seemed to work OK in gitea (based on our test
deployment and conversations with gitea upstream.)

Gerrit is widely used for large projects and I'm not worried for ::gentoo
and we have deployed gerrit and it seems to work fine. Gerrit doesn't have
CI (we would need to deploy something) and it uses gitweb for repository
browsing (which we use today.)

-A


>
> --
> Best regards,
> Michał Górny
>
>


[gentoo-dev] Value of Continuous integration vs Code Review / Pull Requests

2020-05-26 Thread Alec Warner
The TL;DR is that a crack team of infra-folks[0] have been putting together
demos of CI services and things like gitlab / gitea / gerrit and so on.

Some of these come in combined (e.g. gitlab offers repo hosting, code
review / pull reqs, CI services, and deploy services.) Some of these are
piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
repo-hosting but CI is separate (e.g. drone.)

On the infra-side, I think we are pretty happy with repo-hosting (gitolite)
and repo-serving (gitweb). We are missing a CI piece and a pull-request
piece. Most of the users using PRs use either a gitlab or github mirror.

I think the value of CI is pretty obvious to me (and I see tons of use
cases in Infra.) We could easily build CI into our current repository
solution (e.g. gitolite.) However gitolite doesn't really support PRs in a
uniform way and so CI is mostly for submitted code; similar to the existing
::gentoo repo CI offered by mgorny.

If we build a code review solution (like gitea / gerrit) would people use
it? Would you use it if you couldn't merge (because the code review
solution can't gpg sign your commits or merges) so a tool like the existing
pram tool would be needed to merge?

-A

[0] Mostly arzano, if I'm honest. I am just the point-y haired manager in
this effort.


Re: [gentoo-portage-dev] [PATCH] config.environ: delay export of A and AA (bug 720180)

2020-05-26 Thread Alec Warner
On Tue, May 26, 2020 at 9:46 AM Zac Medico  wrote:

> On 5/26/20 1:43 AM, Alec Warner wrote:
> > On Mon, May 25, 2020 at 9:34 PM Zac Medico  > <mailto:zmed...@gentoo.org>> wrote:
> >
> > Since variables like A and AA can contain extremely large values
> which
> > may trigger E2BIG errors during attempts to execute subprocesses,
> delay
> > export until the last moment, and unexport when appropriate.
> >
> >
> > So I think if you want to do this because PMS says:
> >  AA should not be visible in EAPI > 3.
> >  A should only be visible in src_*, pkg_nofetch.
> >
> > That part of the patch makes sense to me. The part that is confusing to
> > me is the 'delay' part; can you explain that further? When you say
> > "delay until the last moment" what do you mean by that and what value is
> > it delivering?
>
> If we export an environment variable which contains an extremely large
> value, then there's a vulnerability in execve which causes it to fail
> with an E2BIG error. Since A and AA values can easily grow large enough
> to trigger this vulnerability, portage can protect itself from execve
> failures by delaying the export until the moment that it hands control
> to the ebuild phase.
>

> > Is it simply that we don't export these variables on the python side,
> > and we only use them in the shell portion?
>
> That's correct. Here's a test case which demonstrates the E2BIG error,
> and shows that 'export -n A' can suppress it:
>

I've run similar tests, but I'm less excited by this work around. I think
its reasonable to work toward eventually not exporting A and AA (the latter
already done in EAPIs > 3). Then when ebuilds encounter problems, we can
tell authors "Upgrade to EAPI X" (where X is >3 or >=8; in the potential
case of A.) Having a hard limit on A for EAPIs 0-7 and a hard limit on AA
for EAPIs 0-3 seems perfectly fine and we should expect authors to refactor
(as texlive did) in order to meet the existing API limitations.

Basically unless there are more practical use cases today, the delaying of
export seems like a feature no one will use and added complexity. I dunno
how large the go-mod use cases are yet; but I myself have not seen any with
this many deps.

-A


>
> $ A=$(dd if=/dev/zero bs=1M count=1 | tr '\0' ' ')
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB, 10 MiB) copied, 0.086557 s, 121 MB/s
> $ echo ${#A}
> 10485760
> $ export A
> $ ls
> bash: /bin/ls: Argument list too long
> $ export -n A
> $ /bin/echo hello world
> hello world


> --
> Thanks,
> Zac
>
>


Re: [gentoo-portage-dev] [PATCH] config.environ: delay export of A and AA (bug 720180)

2020-05-26 Thread Alec Warner
On Mon, May 25, 2020 at 9:34 PM Zac Medico  wrote:

> Since variables like A and AA can contain extremely large values which
> may trigger E2BIG errors during attempts to execute subprocesses, delay
> export until the last moment, and unexport when appropriate.
>

So I think if you want to do this because PMS says:
 AA should not be visible in EAPI > 3.
 A should only be visible in src_*, pkg_nofetch.

That part of the patch makes sense to me. The part that is confusing to me
is the 'delay' part; can you explain that further? When you say "delay
until the last moment" what do you mean by that and what value is it
delivering?

Is it simply that we don't export these variables on the python side, and
we only use them in the shell portion?

-A


> Bug: https://bugs.gentoo.org/720180
> Signed-off-by: Zac Medico 
> ---
>  bin/eapi.sh   |  9 +
>  bin/isolated-functions.sh | 34 
>  bin/phase-functions.sh| 13 +++
>  .../ebuild/_config/special_env_vars.py|  7 +++-
>  lib/portage/package/ebuild/config.py  | 39 ++-
>  5 files changed, 91 insertions(+), 11 deletions(-)
>
> diff --git a/bin/eapi.sh b/bin/eapi.sh
> index 29dfb008c..f56468e4a 100644
> --- a/bin/eapi.sh
> +++ b/bin/eapi.sh
> @@ -26,6 +26,15 @@ ___eapi_has_S_WORKDIR_fallback() {
>
>  # VARIABLES
>
> +___eapi_exports_A() {
> +   # https://bugs.gentoo.org/721088
> +   true
> +}
> +
> +___eapi_exports_AA() {
> +   [[ ${1-${EAPI-0}} =~ ^(0|1|2|3)$ ]]
> +}
> +
>  ___eapi_has_prefix_variables() {
> [[ ! ${1-${EAPI-0}} =~ ^(0|1|2)$ || " ${FEATURES} " == *"
> force-prefix "* ]]
>  }
> diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
> index fde684013..973450d86 100644
> --- a/bin/isolated-functions.sh
> +++ b/bin/isolated-functions.sh
> @@ -107,6 +107,39 @@ __bashpid() {
> sh -c 'echo ${PPID}'
>  }
>
> +# @FUNCTION: ___eapi_vars_export
> +# @DESCRIPTION:
> +# Export variables for the current EAPI. Calls to this function should
> +# be delayed until the last moment, since exporting these variables
> +# may trigger E2BIG errors suring attempts to execute subprocesses.
> +___eapi_vars_export() {
> +   source "${T}/environment.unexported" || die
> +
> +   if ___eapi_exports_A; then
> +   export A
> +   fi
> +
> +   if ___eapi_exports_AA; then
> +   export AA
> +   fi
> +}
> +
> +# @FUNCTION: ___eapi_vars_unexport
> +# @DESCRIPTION:
> +# Unexport variables that were exported for the current EAPI. This
> +# function should be called after an ebuild phase, in order to unexport
> +# variables that may trigger E2BIG errors during attempts to execute
> +# subprocesses.
> +___eapi_vars_unexport() {
> +   if ___eapi_exports_A; then
> +   export -n A
> +   fi
> +
> +   if ___eapi_exports_AA; then
> +   export -n AA
> +   fi
> +}
> +
>  __helpers_die() {
> if ___eapi_helpers_can_die && [[ ${PORTAGE_NONFATAL} != 1 ]]; then
> die "$@"
> @@ -122,6 +155,7 @@ die() {
>
> set +x # tracing only produces useless noise here
> local IFS=$' \t\n'
> +   ___eapi_vars_unexport
>
> if ___eapi_die_can_respect_nonfatal && [[ $1 == -n ]]; then
> shift
> diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
> index 90e622e75..df2c0d8de 100644
> --- a/bin/phase-functions.sh
> +++ b/bin/phase-functions.sh
> @@ -146,6 +146,7 @@ __filter_readonly_variables() {
> fi
> fi
>
> +   ___eapi_vars_unexport
> "${PORTAGE_PYTHON:-/usr/bin/python}"
> "${PORTAGE_BIN_PATH}"/filter-bash-environment.py "${filtered_vars}" || die
> "filter-bash-environment.py failed"
>  }
>
> @@ -212,6 +213,7 @@ __ebuild_phase() {
>
>  __ebuild_phase_with_hooks() {
> local x phase_name=${1}
> +   ___eapi_vars_export
> for x in {pre_,,post_}${phase_name} ; do
> __ebuild_phase ${x}
> done
> @@ -223,6 +225,7 @@ __dyn_pretend() {
> __vecho ">>> Remove '$PORTAGE_BUILDDIR/.pretended' to
> force pretend."
> return 0
> fi
> +   ___eapi_vars_export
> __ebuild_phase pre_pkg_pretend
> __ebuild_phase pkg_pretend
> >> "$PORTAGE_BUILDDIR/.pretended" || \
> @@ -236,6 +239,7 @@ __dyn_setup() {
> __vecho ">>> Remove '$PORTAGE_BUILDDIR/.setuped' to force
> setup."
> return 0
> fi
> +   ___eapi_vars_export
> __ebuild_phase pre_pkg_setup
> __ebuild_phase pkg_setup
> >> "$PORTAGE_BUILDDIR/.setuped" || \
> @@ -252,6 +256,7 @@ __dyn_unpack() {
> install -m${PORTAGE_WORKDIR_MODE:-0700} -d "${WORKDIR}" ||
> die "Failed to create dir '${WORKDIR}'"
> fi
> cd "${WORKDIR}" || die "Directory change failed: \`cd
> '${WORKDIR}'\`"
> +   ___eapi_vars_export
> __ebuild_phase 

Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-21 Thread Alec Warner
On Tue, May 19, 2020 at 5:46 AM Samuel Bernardo <
samuelbernardo.m...@gmail.com> wrote:

> On 5/19/20 7:47 AM, Michał Górny wrote:
> > Do you have any specific solution in mind?
> >
> > [1] https://gitweb.gentoo.org/archive/proj/identity.gentoo.org.git/
>
> I would suggest for SSO an implementation like the following with LDAP
> provider:
>
> https://github.com/Luzifer/nginx-sso/wiki/Auth-Provider-Configuration


Thanks for pointing this out, we might use it for legacy apps that don't
have solid saml / openid integration.
I want to link it against keycloak though because LDAP doesn't support
newer auth standards like u2f.

-A


Re: [gentoo-dev] [PSA] If you ssh interactively to git.gentoo.org (somehow) let me know.

2020-05-21 Thread Alec Warner
A bit late, but this change is now live. Please contact me if anything has
broken.

-A

On Mon, Apr 27, 2020 at 10:34 AM Alec Warner  wrote:

> On Mon, Apr 27, 2020 at 7:04 AM Kent Fredric  wrote:
>
>> On Mon, 27 Apr 2020 09:43:44 -0400
>> Mike Gilbert  wrote:
>>
>> > He was replying to me. Your master connection will continue to work
>> > just fine, as I said in my previous message.
>>
>> I must have lost something in grammar, because no matter how many times I
>> read:
>>
>> > If you are authenticating that master connection as the "git" user, I
>> > suspect it will not affect you. If you are using it to push to
>> > gentoo.git, that is almost certainly the case.
>>
>> I interpret that as:
>>
>> - Anonymous fetch is fine
>> - Authorised Push will fail
>>
>
> "If you are authenticating the master connection as the 'git' user then
> this change will not affect you.
> "If you are using controlmaster to push to git.gentoo.org, then you are
> definitely authenticating as user=git because there is no other way to
> commit to ::gentoo."
>
>
>>
>> But I guess my mistake is in that we don't push with "user@git ...", we
>> push with "git@ ... ", and the SSH key is the gate keeper of "push will
>> work", not the UID.
>>
>> Right?
>>
>
> A working ssh key for user=git is a necessary (but not sufficient)
> component of a successful git push.
>
>
>>
>> So assuming you're using git@ for fetch *and* push, *then* it will
>> continue to work.
>>
>> Right?
>>
>
> Correct.
>
>
>>
>> Forgive me for any potential idiocy, language and remembering the
>> details of everything all the time is hard.
>>
>
> I don't actually expect people to know these details.
>


Re: [gentoo-dev] [RFC] Anti-spam for goose

2020-05-21 Thread Alec Warner
On Thu, May 21, 2020 at 1:13 PM Viktar Patotski 
wrote:

> Hi all,
>
> I believe that we are all have forgotten about Donald Knuth: Premature
> optimisation is the root of all evill.
>
> We don't have "spam" yet, but we are already trying to protect. There
> might be cases when some systems will be posting stats more often than we
> want, but probably that will not harm us. Or this will be done by our main
> users who runs 1kk of gentoo installations and this "spam"  will be
> actually valuable. Moreover, nobody forces us to treat info from 'goose' as
> first priority, so we are still able to select on which packages to work.
> In short: this topic is not so important yet, I think.
>

I raised a similar question on irc and the conclusion was that 'it is good
to have ideas' and I don't necessarily disagree there[0]. We cannot build a
foolproof system but some are feasible in some scenarios[1].

[0] Gentoo offers numerous no-login-required services; most of these are
read-only but they typically don't suffer from attacks; or at least, not
attacks that we need to respond to. The most obvious one of these is our
gentoo.org mail service which accepts unauthenticated email to gentoo.org.
Our anti-email-spam countermeasures are what I would call complex, but we
still employ broad measures when needed and the tradeoffs are similar to
the options for goose; e.g. if we are too broad we can block email from
large swaths of the internet.
[1] Bugzilla *has* recently been the target of spam attacks, it *has*
logins required (e.g. to create / modify bugs) and it has not stopped the
spammers from creating accounts. We have discussed different protections
for bugzilla, as it has different parameters. A basic bugzilla account
can't do all that much (you can't modify the bugs of others easily) and
spam posts are easily identified. This is to differentiate from goose where
the powers of each token are the same (submit report) and it may be
difficult to tell an abusive report from a real report.


> Viktar
>
>
> On Thu, May 21, 2020, 16:28 Jaco Kroon  wrote:
>
>> Hi Michał,
>>
>> On 2020/05/21 13:02, Michał Górny wrote:
>> > On Thu, 2020-05-21 at 12:45 +0200, Jaco Kroon wrote:
>> >> Even for v4, as an attacker ... well, as I'm sitting here right now
>> I've
>> >> got direct access to almost a /20 (4096 addresses).  I know a number of
>> >> people with larger scopes than that.  Use bot-nets and the scope goes
>> up
>> >> even more.
>> > See how unfair the world is!  You are filling your bathtub with IP
>> > addresses, and my ISP has taken mine only recently.
>> I must admit, I work for an ISP :$
>> >>> Option 3: explicit CAPTCHA
>> >>> ==
>> >>> A traditional way of dealing with spam -- require every new system
>> >>> identifier to be confirmed by solving a CAPTCHA (or a few
>> identifiers
>> >>> for one CAPTCHA).
>> >>>
>> >>> The advantage of this method is that it requires a real human work
>> >>> to be
>> >>> performed, effectively limiting the ability to submit spam.
>> >>>
>> >> Yea.  One would think.  CAPTCHAs are massively intrusive and in my
>> >> opinion more effort than they're worth.
>> >>
>> >> This may be beneficial to *generate* a token.  In other words - when
>> >> generating a token, that token needs to be registered by way of
>> capthca.
>> >>
>> >>> Other ideas
>> >>> ===
>> >>> Do you have any other ideas on how we could resolve this?
>> >>>
>> >> Generated token + hardware based hash.
>> > How are you going to verify that the hardware-based hash is real,
>> > and not just a random value created to circumvent the protection?
>>
>> So the generation of the hash is more to validate that it's still on the
>> same installation (ie, not a cloned token).  Sorry if that wasn't clear,
>> so trying to solve two possible problems in one go.
>>
>> >
>> >>   Rate limit the combination to 1/day.
>> >>
>> >> Don't use included results until it's been kept up to date for a
>> minimum
>> >> period.  Say updated at least 20 times 30 days.
>> > For privacy reasons, we don't correlate the results.  So this is
>> > impossible to implement.
>>
>> Ok, but a token cannot (unless we issue it based on an email based
>> account) be linked back to a specific user, so does it matter if we
>> associate uploads with a token?
>>
>> >> The downside here is that many machines are not powered up at least
>> once
>> >> a day to be able to perform that initial submission sequence.  So
>> >> perhaps it's a bit stringent.
>> > Exactly.  Even once a week is a bit risky but once a day is too narrow
>> > a period.
>> >
>> > To some degree, we could decide we don't care about exact numbers
>> > as much as some degree of weighed proportions.  This would mean that,
>> > say, people who submit daily get the count of 7, at the loss of people
>> > who don't run their machines that much.  It would effectively put more
>> > emphasis on more active users.  It's debatable whether 

Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-20 Thread Alec Warner
On Wed, May 20, 2020 at 12:26 AM Michał Górny  wrote:

> On Wed, 2020-05-20 at 00:21 -0700, Alec Warner wrote:
> > On Tue, May 19, 2020 at 1:23 AM Lars Wendler 
> > wrote:
> >
> > > Hi Alec,
> > >
> > > On Mon, 18 May 2020 18:42:24 -0700 Alec Warner wrote:
> > >
> > > > TL;DR: What if we launched id.gentoo.org, an identity provider that
> > > > provides authentication for Gentoo properties? Basically, 1 username
> /
> > > > password for wiki, bugs, email, forums, and any other http
> > > > service[0][1].
> > > >
> > > > Today Gentoo has numerous systems that mostly work in a segmented
> way.
> > > >
> > > > - To connect to hosts, we use ssh keys.
> > > > - Git is authenticated via ssh keys.
> > > > - Email uses LDAP passwords.
> > > > - Bugzilla has its own identities, with their own passwords.
> > > > - Wiki is separate, with its own passwords.
> > > > - Forums are separate.
> > > > - Infra has an additional 4 systems that use separate credentials.
> > > >
> > > > Some applications support 2FA (such as wiki.)
> > > > Some applications do not support 2FA.
> > > > Applications that require 2FA have a configuration for each app, so
> you
> > > > have N configurations.
> > > >
> > > > If we configured id.gentoo.org you would have 1 identity across all
> > > > gentoo properties.
> > > >
> > > > Is this a thing people are interested in?
> > > >
> > > > [0] It's unlikely operations for git via ssh would change in this
> > > > rollout. [1] Its unclear if the scope is "gentoo developers" or "any
> > > > community member." The former have LDAP accounts and @gentoo.org
> email
> > > > addresses and so we can manage them easily; managing 1000s of other
> > > > accounts in the IDP remains to be seem.
> > >
> > > In case 2FA won't be mandatory I find this a good idea.
> > >
> >
> > 2FA is definitely a reason to deploy software like keycloak, but in the
> > first rollout I don't expect to enforce 2FA. Ideally we would deploy the
> > U2F support in keycloak and then, similar to our earlier program, offer
> > discounted or free u2f devices for Gentoo developers; this would likely
> be
> > on a 1-2 year timeframe.
> >
> > Is there some reason you don't want to use 2FA?
> >
>
> I myself would find 2FA bothersome for low importance services.  Whether
> it's U2F or OTP, I would generally find it silly to have to carry
> the hardware/software on me all the time or even use it when it's laying
> right next to me, say, just to approve a comment on a blog.
>
> But I guess if we go for SSO, it becomes a necessity to better protect
> our passwords.
>

I think each application, when it ends up integrating with keycloak, gets
to decide what security level the application wants; I think this leads to
flexibility for low-importance stuff. E.g. we may not need OTP for blogs,
or wiki. Obvious cases are apps like our AWS credentials (where theft means
financial harm for Gentoo) or the sso.gentoo.org itself (because you
probably want to require OTP to change your password, for example.)

The other common thing I've seen is some kind of longer-lived renewable
token that requires an OTP to get, but does not require an OTP to renew.
These are commonly things like "API keys" or other such credentials that
are scopeable (unlike a password) and revocable (e.g. you can go to
sso.gentoo.org and revoke your token.) This seems more common on mobile
where there is a 'setup' flow and maybe you do it once (at setup), or once
a month, or whatnot. This would mean you don't have to OTP all the time.

-A


> --
> Best regards,
> Michał Górny
>
>


Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-20 Thread Alec Warner
On Tue, May 19, 2020 at 1:23 AM Lars Wendler 
wrote:

> Hi Alec,
>
> On Mon, 18 May 2020 18:42:24 -0700 Alec Warner wrote:
>
> >TL;DR: What if we launched id.gentoo.org, an identity provider that
> >provides authentication for Gentoo properties? Basically, 1 username /
> >password for wiki, bugs, email, forums, and any other http
> >service[0][1].
> >
> >Today Gentoo has numerous systems that mostly work in a segmented way.
> >
> > - To connect to hosts, we use ssh keys.
> > - Git is authenticated via ssh keys.
> > - Email uses LDAP passwords.
> > - Bugzilla has its own identities, with their own passwords.
> > - Wiki is separate, with its own passwords.
> > - Forums are separate.
> > - Infra has an additional 4 systems that use separate credentials.
> >
> >Some applications support 2FA (such as wiki.)
> >Some applications do not support 2FA.
> >Applications that require 2FA have a configuration for each app, so you
> >have N configurations.
> >
> >If we configured id.gentoo.org you would have 1 identity across all
> >gentoo properties.
> >
> >Is this a thing people are interested in?
> >
> >[0] It's unlikely operations for git via ssh would change in this
> >rollout. [1] Its unclear if the scope is "gentoo developers" or "any
> >community member." The former have LDAP accounts and @gentoo.org email
> >addresses and so we can manage them easily; managing 1000s of other
> >accounts in the IDP remains to be seem.
>
> In case 2FA won't be mandatory I find this a good idea.
>

2FA is definitely a reason to deploy software like keycloak, but in the
first rollout I don't expect to enforce 2FA. Ideally we would deploy the
U2F support in keycloak and then, similar to our earlier program, offer
discounted or free u2f devices for Gentoo developers; this would likely be
on a 1-2 year timeframe.

Is there some reason you don't want to use 2FA?

-A


>
> Kind regards
> --
> Lars Wendler
> Gentoo package maintainer
> GPG: 21CC CF02 4586 0A07 ED93  9F68 498F E765 960E 9B39
>


Re: [gentoo-dev] RFC: Gentoo Identity Provider

2020-05-20 Thread Alec Warner
On Mon, May 18, 2020 at 11:47 PM Michał Górny  wrote:

> On Mon, 2020-05-18 at 18:42 -0700, Alec Warner wrote:
> > TL;DR: What if we launched id.gentoo.org, an identity provider that
> > provides authentication for Gentoo properties? Basically, 1 username /
> > password for wiki, bugs, email, forums, and any other http service[0][1].
> >
> > Today Gentoo has numerous systems that mostly work in a segmented way.
> >
> >  - To connect to hosts, we use ssh keys.
> >  - Git is authenticated via ssh keys.
> >  - Email uses LDAP passwords.
> >  - Bugzilla has its own identities, with their own passwords.
> >  - Wiki is separate, with its own passwords.
> >  - Forums are separate.
> >  - Infra has an additional 4 systems that use separate credentials.
> >
> > Some applications support 2FA (such as wiki.)
> > Some applications do not support 2FA.
> > Applications that require 2FA have a configuration for each app, so you
> > have N configurations.
> >
> > If we configured id.gentoo.org you would have 1 identity across all
> gentoo
> > properties.
> >
> > Is this a thing people are interested in?
> >
>
> What a coincidence I've just archived our old identity.gentoo.org [1]
> project.  And yes, we almost had this back in 2013 but Infra failed to
> deploy, and it was claimed obsolete by the time I joined Infra.
>
> Do you have any specific solution in mind?
>

Currently we have a standalone keycloak install with LDAP user federation.
We are looking to do a domain installation for redundancy purposes.
Our existing LDAP infrastructure for example (which few services use for
Auth) has at least 3 replicas.

-A


>
> [1] https://gitweb.gentoo.org/archive/proj/identity.gentoo.org.git/
>
>
> --
> Best regards,
> Michał Górny
>
>


[gentoo-dev] RFC: Gentoo Identity Provider

2020-05-18 Thread Alec Warner
TL;DR: What if we launched id.gentoo.org, an identity provider that
provides authentication for Gentoo properties? Basically, 1 username /
password for wiki, bugs, email, forums, and any other http service[0][1].

Today Gentoo has numerous systems that mostly work in a segmented way.

 - To connect to hosts, we use ssh keys.
 - Git is authenticated via ssh keys.
 - Email uses LDAP passwords.
 - Bugzilla has its own identities, with their own passwords.
 - Wiki is separate, with its own passwords.
 - Forums are separate.
 - Infra has an additional 4 systems that use separate credentials.

Some applications support 2FA (such as wiki.)
Some applications do not support 2FA.
Applications that require 2FA have a configuration for each app, so you
have N configurations.

If we configured id.gentoo.org you would have 1 identity across all gentoo
properties.

Is this a thing people are interested in?

[0] It's unlikely operations for git via ssh would change in this rollout.
[1] Its unclear if the scope is "gentoo developers" or "any community
member." The former have LDAP accounts and @gentoo.org email addresses and
so we can manage them easily; managing 1000s of other accounts in the IDP
remains to be seem.


Re: [gentoo-dev] [RFC] Ideas for gentoostats implementation

2020-05-05 Thread Alec Warner
On Mon, May 4, 2020 at 10:14 PM Matt Turner  wrote:

> On Mon, May 4, 2020 at 5:48 PM Thomas Deutschmann 
> wrote:
> >
> > On 2020-04-26 15:46, Kent Fredric wrote:
> > > On Sun, 26 Apr 2020 14:38:54 +0200
> > > Thomas Deutschmann  wrote:
> > >
> > >> Let's assume we will get reports that app-misc/foo is only installed
> 20
> > >> times. If you are going to judge based on this data, "Obviously,
> nobody
> > >> is using that package, it's stuck on ... safe to remove"
> your
> > >> view is biased:
> > >
> > > I see this as more like what bloom filters get you, but in reverse:
> > >
> > > [...]
> > >
> > > - But now, instead of having "we don't know if anybody uses this", you
> > >   *can* have a "we know for sure somebody uses this".
> >
> > But how does that information really help us to decide anything in the
> end?
> >
> > Case A, stats are showing 0 users:
> >
> > Like said, we can't know if this is true or if this package is only used
> > in setups where people don't report stats.
> >
> >
> > Case B, stats are showing x users:
> >
> > Now what? Package from case A could have similar users -- we just don't
> > know. Assume firefox has 1.000 users, chromium has 500 users and vivaldi
> > doesn't show up in stats. How does that help us? Would this allow us to
> > skip publishing GLSAs for vivalid because we assume nobody in Gentoo is
> > using vivaldi? Does it allow Python project to go forward pushing a mask
> > for removal in case vivaldi would depend on Python version, Python
> > project want to get rid of? Would this allow Gentoo PR to make a public
> > statement like "Firefox is the most popular browser in Gentoo, twice as
> > users as chromium"?
>
> I hate the saying "the perfect is the enemy of the good" but I think
> it applies here.
>
> You're of course correct that we would not have perfect information.
> But the thing about statistics is that you can still know some things
> based on a sampling of that perfect information.
>
> I would personally like to have data on whether users of my packages
> have certain USE flags enabled. Knowing that would allow me to decide
> whether its worth the maintenance burden of supporting features that I
> *think* are very rarely used. If instead the data showed me that 50%
> of users had IUSE=xyz enabled, I probably wouldn't consider removing
> it.
>
> I think your example of potential misuse of data is a bit over dramatic.
>

Let me present the same point another way.

Today we have no data, so we make an arbitrary decision. It might be right
or wrong; and we may not know until after we decide.
This is traditionally things like "break them and they will come" type of
process. "Mask it, if they complain, I'll unmask it."

In the future, we could have this package data. It may influence decision
making. However I'm not sure from a decision-making standpoint that it is
strictly worse than no data.
The danger (which is what I think Whissi's concern is) is that it could
artificially increase decision certainty.

For example, if I have to decide whether to keep a package, or a flag, or
whatever. I might make an arbitrary decision. I'm aware it's arbitrary, it
might be wrong, and so I'm not super attached to such a decision. I'm not
*certain* about it; but I have to decide one way or the other[0]. Then I
move to a world with package data. Now I'm no longer making an arbitrary
decision; I'm making a decision based on *data*. The *data* tells me my
decision is correct, resulting in a more *certain* decision outcome. I
think this is the fallacy we want to avoid. The data can be informative but
there are significant biases in it that should result in very *little*
certainty added to decision making.

Making decisions based on incomplete data is just life though, so I'm
fairly skeptical of a "we shouldn't collect any data" type of mindset. I'd
be curious to see if we can instill a *culture* component around the use of
data in our development workflows.

-A

[0] There are a bunch of other cultural components here, like different
decision types (1 vs 2) and the ability to make a mistake in public and not
feel bad about it; so I'm aware reality does not reflect this trivial
example. But those are hallmarks of cultural markets I'd like to aim for in
Gentoo, so I would prefer to discuss a world where they exist ;)


Re: [gentoo-dev] [PSA] If you ssh interactively to git.gentoo.org (somehow) let me know.

2020-04-27 Thread Alec Warner
On Mon, Apr 27, 2020 at 7:04 AM Kent Fredric  wrote:

> On Mon, 27 Apr 2020 09:43:44 -0400
> Mike Gilbert  wrote:
>
> > He was replying to me. Your master connection will continue to work
> > just fine, as I said in my previous message.
>
> I must have lost something in grammar, because no matter how many times I
> read:
>
> > If you are authenticating that master connection as the "git" user, I
> > suspect it will not affect you. If you are using it to push to
> > gentoo.git, that is almost certainly the case.
>
> I interpret that as:
>
> - Anonymous fetch is fine
> - Authorised Push will fail
>

"If you are authenticating the master connection as the 'git' user then
this change will not affect you.
"If you are using controlmaster to push to git.gentoo.org, then you are
definitely authenticating as user=git because there is no other way to
commit to ::gentoo."


>
> But I guess my mistake is in that we don't push with "user@git ...", we
> push with "git@ ... ", and the SSH key is the gate keeper of "push will
> work", not the UID.
>
> Right?
>

A working ssh key for user=git is a necessary (but not sufficient)
component of a successful git push.


>
> So assuming you're using git@ for fetch *and* push, *then* it will
> continue to work.
>
> Right?
>

Correct.


>
> Forgive me for any potential idiocy, language and remembering the
> details of everything all the time is hard.
>

I don't actually expect people to know these details.


Re: [gentoo-dev] [PSA] If you ssh interactively to git.gentoo.org (somehow) let me know.

2020-04-26 Thread Alec Warner
On Sun, Apr 26, 2020 at 11:22 AM Mike Gilbert  wrote:

> On Sun, Apr 26, 2020 at 8:38 AM Kent Fredric  wrote:
> >
> > On Sat, 25 Apr 2020 14:12:02 -0700
> > Alec Warner  wrote:
> >
> > > Thus I now plan to remove this access[0]. If you need access to ssh as
> > > something not-git to git.gentoo.org, let me know in the next week.
> >
> > Will this affect people who set up no-op SSH connections for a
> > persistent connection master, so that following git actions don't have
> > to pay the SSH connection startup penalty?
>
> If you are authenticating that master connection as the "git" user, I
> suspect it will not affect you. If you are using it to push to
> gentoo.git, that is almost certainly the case.
>
>
This is correct.

-A


Re: [gentoo-portage-dev] Need backup mentor for FUSE-based sandbox project

2020-04-25 Thread Alec Warner
On Thu, Apr 23, 2020 at 12:09 PM Michał Górny  wrote:

> Hi, everyone.
>
> It seems that we *urgently* (read: in 6 days) need to find backup
> mentors for this year's GSoC projects.  I'm mentoring the project to
> develop a FUSE-based sandbox alternative that's going to work reliably
> with more packages than the LD_PRELOAD hack [1].
>
> Would anyone volunteer to be the backup maintainer for this?
>
> [1]
> https://summerofcode.withgoogle.com/dashboard/organization/5749981018849280/proposal/5572241732927488/


I get a 403 for that.

https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2020/Ideas#FUSE-powered_sandbox
is
your writeup; I assume the above is supposed to be a link to the student's
proposal?


>
>
> --
> Best regards,
> Michał Górny
>
>


[gentoo-dev] [PSA] If you ssh interactively to git.gentoo.org (somehow) let me know.

2020-04-25 Thread Alec Warner
TL;DR: if all you do is use git to commit to git.gentoo.org, you are not
affected and can stop reading; I know folks use git+ssh://g...@git.gentoo.org
... to push commits, that will not change.

In the olden times Gentoo used cvs as its source control and people would
push their commits to the cvs server over ssh. The setup at the time was
that everyone who pushed had ssh access to cvs.gentoo.org.

However, Gentoo doesn't use cvs (and has not for many years[1]). The git
system uses 'gitolite' and people who push do so as 'g...@git.gentoo.org'
(not as themselves.) Gitolite handles the per-user multiplexing and
everything is happy.

However, we never took the ssh access to 'cvs.gentoo.org' away, most devs
can still ssh to "git.gentoo.org" as themselves. Now the access doesn't get
you much (ForceCommand in the authorized_keys file just runs a commit
wrapper, so you could try to commit to cvs or svn I guess ;p)

Thus I now plan to remove this access[0]. If you need access to ssh as
something not-git to git.gentoo.org, let me know in the next week.

[0] Infra users are not affected; they always had normal ssh access to this
host.
[1] Anonymous access to source trees (e.g. via anon* services) is
unaffected by this change.


[gentoo-dev] Re: [RFC] Adding potentially questionable license AcePerl-Indemnity

2020-04-22 Thread Alec Warner
On Wed, Apr 22, 2020 at 3:33 PM Kent Fredric  wrote:

> I've just discovered dev-perl/Ace has some fun questionable licensing
> which includes a lovely indemnity clause, which had previously gone
> unnoticed, and it stipulates additional requests for research
> publications, which is not something mentioned in any license currently
> in tree other than Tinker
>
> Following is the entire body of the license I plan to put in
> licenses/AcePerl-Indemnity ( name chosen to specifically alert people
> tempted to accept this license that Indemnification is an important
> part they should actually read )
>
> Current advice also says that due to the terms of this license, we have
> to RESTRICT="mirror" this as well, unless the Trustees want to sign off
> on potentially indemnifying CSHL
>

I think it's less about the Foundation (I'm happy to indemnify them) but
the indemnification reads as viral to me and so anyone that operated a
distfiles mirror would also implicitly indemnify[0] (through mirroring from
us) and it seems unlikely this is a thing we would encourage because our
distfiles operators are relying on some diligence on our own end to avoid
excessive liability for files we provide to them.

-A

[0] Whether this would hold up is another matter, but I'm fine with
restricting the mirroring of aceperl to avoid this complication.



>
> Also following up with CPAN because as its *currently* mirrored on
> CPAN, and has been mirrored there for at *least* 12 years, its
> potentially in a legal situation as well.
>
> ( But that's the fault of the uploader if true, because you can't
> upload anything to CPAN without mirroring being something you didn't
> expect )
>
> Once this license is added, the plan is to rework Ace-*.ebuild to be under
>
> LICENSE="|| ( Artistic GPL-1+ ) AcePerl-Indemnity"
>
> Upstream: https://metacpan.org/source/LDS/AcePerl-1.92/DISCLAIMER.txt
>
> Text Body:
>
> ==
>
> The Ace.pm package and all associated files are Copyright (c) 1998
> Cold Spring Harbor Laboratory.
>
> This library is free software; you can redistribute it and/or modify
> it under the same terms as Perl itself.  See the Artistic License file
> in the main Perl distribution for specific terms and conditions of
> use.  In addition, the following disclaimers apply:
>
> CSHL makes no representations whatsoever as to the SOFTWARE contained
> herein.  It is experimental in nature and is provided WITHOUT WARRANTY
> OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OR ANY OTHER
> WARRANTY, EXPRESS OR IMPLIED.  CSHL MAKES NO REPRESENTATION OR
> WARRANTY THAT THE USE OF THIS SOFTWARE WILL NOT INFRINGE ANY PATENT OR
> OTHER PROPRIETARY RIGHT.
>
> By downloading this SOFTWARE, your Institution hereby indemnifies CSHL
> against any loss, claim, damage or liability, of whatsoever kind or
> nature, which may arise from your Institution's respective use,
> handling or storage of the SOFTWARE.
>
> If publications result from research using this SOFTWARE, we ask that
> CSHL be acknowledged and/or credit be given to CSHL scientists, as
> scientifically appropriate.
>
> ==
>
>


Re: [gentoo-dev] Re: [PR] ivy, mvn, sbt, gradle builders improvement for ebuild development

2020-04-19 Thread Alec Warner
On Sun, Apr 19, 2020 at 8:37 AM Michael Orlitzky  wrote:

> On 4/19/20 10:55 AM, Samuel Bernardo wrote:
> >
> > Taking into account the network sandbox requirement, sbt.eclass needs to
> > download all dependencies with some approach like EGO_SUM implementation
> > in go-module.eclass[1].
> >
>
> EGO_SUM is not a legitimate approach to packaging. You have to create
> packages for each package. I know, it sounds crazy when you say it.
>
>
EGO_SUM is great!

-A


  1   2   3   4   5   6   7   8   9   10   >