Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Magnus Granberg]

måndag 23 januari 2017 kl. 13:56:02 CET skrev  Rich Freeman:
> On Mon, Jan 23, 2017 at 4:23 AM, Michał Górny  wrote:
> > I've written a short proposal that aims to provide basic infrastructure
> > for defining mix-in profiles in Gentoo. I've tried to keep it simple,
> > and backwards compatible. The main goal is to be able to start defining
> > some mix-ins without having to reinvent the whole profile tree.
> 
> Would it actually make sense to reinvent more of the profile tree
> while we're at it?  So, have a few categories of mixins like kernel,
> arch, and some category that covers really invasive stuff like
> hardened/libc/etc?
> 
i think it would make sense to reinvent/split it up to a few categories/mixins 
like
base
arch
  abi
libc
kernel
init/udev
desktop
server
security

> Those might be 1-of-n selections.
> 
> Then we could have the fluff that sits on top and just set some rules
> about what they can do.
> 
> Part of me wonders if some of this could also fit in with the use of
> virtuals (think foo-meta virtuals but bigger).  A virtual can of
> course pull in USE dependencies, and a lot of other stuff.  We could
> have convenience virtuals that all the profiles pull in by default but
> which gets stuff like openssh out of @system.
> 
> I'm only suggesting the last bit to the extent where we see tie-ins to
> improve the initial mix-in implementation.  A lot of that is probably
> an expansion in scope, and to that extent I'm not suggesting that it
> needs to be addressed.  I just want to think about it broadly at first
> to make sure we're not missing something.

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[M. J. Everitt]

On 24/01/17 17:20, Jason Zaman wrote:
> This is awesome! thanks for working on it, ive wanted it for a while
> now.
>
> The main issue I see with it is ordering. For the hardened and selinux
> profiles, the order matters a lot. eg hardened defaults the jit useflag
> off and the desktop profile defaults on which causes problems with PaX.
> If these two mixins would end up in a kind of random order then we'll
> have issues. See https://bugs.gentoo.org/492312 for a specific issue.
>
> It could be as simple as when putting in the make.profile list, do it in
> the order they are defined in the profiles.mixin file, then we can just
> have hardened and selinux last and problem solved. Or could go with a
> priority field but then again there is the problem of what happens when
> two have the same priority?
>
> Also how will profiles.mixin interact between different overlays? you
> can manually set a parents file with features/desktop::gentoo and
> feature/foo::overlay already. if foo::myoverlay is in the desktop group,
> does that merge together with the desktop group ones in ::gentoo?
>
> -- Jason
>
Here's one suggestion .. how about we use a system similar to the old
init naming scheme of:

[]<2-digit>

where the 2-digit defines the first-order sorting, and then by
alphabetical thereafter?

Just a random idea...



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Jason Zaman]

On Mon, Jan 23, 2017 at 10:23:35AM +0100, Michał Górny wrote:
> Hi, everyone.
> 
> I've written a short proposal that aims to provide basic infrastructure
> for defining mix-in profiles in Gentoo. I've tried to keep it simple,
> and backwards compatible. The main goal is to be able to start defining
> some mix-ins without having to reinvent the whole profile tree.
> 
> Most important points:
> 
> 1. Mix-ins are applied on top of base profile (which works the same as
> before),
> 
> 2. Mix-ins are supported via 'eselect profile'
> replacing /etc/portage/make.profile symlink with a directory, without
> need for Portage patching (this is how Funtoo does it),
> 
> 3. Most important mix-ins are used to construct base profiles which
> provides both backwards compatibility and proper targets for repoman
> (to avoid having to check all possible mix-in combinations).

This is awesome! thanks for working on it, ive wanted it for a while
now.

The main issue I see with it is ordering. For the hardened and selinux
profiles, the order matters a lot. eg hardened defaults the jit useflag
off and the desktop profile defaults on which causes problems with PaX.
If these two mixins would end up in a kind of random order then we'll
have issues. See https://bugs.gentoo.org/492312 for a specific issue.

It could be as simple as when putting in the make.profile list, do it in
the order they are defined in the profiles.mixin file, then we can just
have hardened and selinux last and problem solved. Or could go with a
priority field but then again there is the problem of what happens when
two have the same priority?

Also how will profiles.mixin interact between different overlays? you
can manually set a parents file with features/desktop::gentoo and
feature/foo::overlay already. if foo::myoverlay is in the desktop group,
does that merge together with the desktop group ones in ::gentoo?

-- Jason

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Alexis Ballier]

On Tue, 24 Jan 2017 09:04:37 -0600
Matthias Maier  wrote:

> > well then 'ihateudev' masking udev, 'ihateeudev' masking eudev and
> > 'ihatesystemd' masking systemd; what are the blockers here?  
> 
> You make three profiles, 'udev', 'eudev', 'systemd' and put them in
> one group and let them block said group.


so... the mixins I proposed above shouldn't be allowed, right ?
why ? what is the logic ?


Note: It might seem I dislike the idea of mixins. In fact, it is the
contrary, I think those are deeply needed. I'm simply pointing out that
making them too expressive means giving up on some static checks. Maybe
that's ok, I don't know, maybe it is possible to keep completeness of
repoman checks while still having enough expressiveness: In my very
first answer, I throwed the idea that mixins shouldnt change packages
visibility. It is probably too restrictive and you've just shown that
it is possible to do it correctly without that restriction. However, I
still don't have a good idea of what would be the rule applied to
mixins to ensure that.

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Matthias Maier]

> well then 'ihateudev' masking udev, 'ihateeudev' masking eudev and
> 'ihatesystemd' masking systemd; what are the blockers here?

You make three profiles, 'udev', 'eudev', 'systemd' and put them in one
group and let them block said group.

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Alexis Ballier]

On Mon, 23 Jan 2017 20:40:00 +0100
Michał Górny  wrote:

> On Mon, 23 Jan 2017 20:30:38 +0100
> Alexis Ballier  wrote:
> 
> > On Mon, 23 Jan 2017 18:37:15 +0100
> > Michał Górny  wrote:  
> > > > For example, if you allow use.mask or use.force in mixins, you
> > > > can end up having unsatisfiable deps that repoman will never
> > > > catch. Arguably, desktop profiles relying on having an useflag
> > > > forced on a given package are already semi-broken: they'd be
> > > > better with the useflag default enabled and proper usedeps, so
> > > > the mask/force game doesnt seem really useful for mixins.  
> > > 
> > > That's why if you do such a thing, you would have to declare a
> > > regular profile using this mix-in for repoman to test.
> > > 
> > 
> > still that doesn't account for a 'ihatelennart' mixin masking udev &
> > systemd and a 'ilovelennart' mixin masking udev & eudev and an user
> > enabling them both  
> 
> That's why they can define blockers/conflicts.


well then 'ihateudev' masking udev, 'ihateeudev' masking eudev and
'ihatesystemd' masking systemd; what are the blockers here?


> > why not let such a stupid example be, it is similar to package.mask
> > users can already fill, but I'm pretty sure more subtle breakage
> > will appear
> > 
> > repoman will test n out of 2^n (or n!) possibilities the way you
> > suggest, which is basically nothing when n is big  
> 
> Are you going somewhere in particular with this or just arguing for
> the sake of arguing?

arguing for the sake of arguing are the above examples; this is the
reason why it is useless to argue on this because you're basically
hiding a sat solver inside repoman: it has to answer the question "does
there exist an assignment of mixins that makes that dep unsatisfiable?"
if you want the check to be complete

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Michael Orlitzky]

On 01/23/2017 02:30 PM, Alexis Ballier wrote:
> 
> repoman will test n out of 2^n (or n!) possibilities the way you
> suggest, which is basically nothing when n is big
> 

Corollary: big is basically nothing.

I like it.

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Matthias Maier]

On Mon, Jan 23, 2017, at 13:30 CST, Alexis Ballier  wrote:

> still that doesn't account for a 'ihatelennart' mixin masking udev &
> systemd and a 'ilovelennart' mixin masking udev & eudev and an user
> enabling them both

But that's exactly what is ruled out by the "blocks @group" mechanism in
mgorny's proposal.

Best,
Matthias

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Michał Górny]

On Mon, 23 Jan 2017 20:30:38 +0100
Alexis Ballier  wrote:

> On Mon, 23 Jan 2017 18:37:15 +0100
> Michał Górny  wrote:
> > > For example, if you allow use.mask or use.force in mixins, you can
> > > end up having unsatisfiable deps that repoman will never catch.
> > > Arguably, desktop profiles relying on having an useflag forced on a
> > > given package are already semi-broken: they'd be better with the
> > > useflag default enabled and proper usedeps, so the mask/force game
> > > doesnt seem really useful for mixins.
> > 
> > That's why if you do such a thing, you would have to declare a regular
> > profile using this mix-in for repoman to test.
> >   
> 
> still that doesn't account for a 'ihatelennart' mixin masking udev &
> systemd and a 'ilovelennart' mixin masking udev & eudev and an user
> enabling them both

That's why they can define blockers/conflicts.

> why not let such a stupid example be, it is similar to package.mask
> users can already fill, but I'm pretty sure more subtle breakage will
> appear
> 
> repoman will test n out of 2^n (or n!) possibilities the way you
> suggest, which is basically nothing when n is big

Are you going somewhere in particular with this or just arguing for the
sake of arguing?

-- 
Best regards,
Michał Górny



pgpiNTljQdjgr.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Alexis Ballier]

On Mon, 23 Jan 2017 18:37:15 +0100
Michał Górny  wrote:
> > For example, if you allow use.mask or use.force in mixins, you can
> > end up having unsatisfiable deps that repoman will never catch.
> > Arguably, desktop profiles relying on having an useflag forced on a
> > given package are already semi-broken: they'd be better with the
> > useflag default enabled and proper usedeps, so the mask/force game
> > doesnt seem really useful for mixins.  
> 
> That's why if you do such a thing, you would have to declare a regular
> profile using this mix-in for repoman to test.
> 

still that doesn't account for a 'ihatelennart' mixin masking udev &
systemd and a 'ilovelennart' mixin masking udev & eudev and an user
enabling them both

why not let such a stupid example be, it is similar to package.mask
users can already fill, but I'm pretty sure more subtle breakage will
appear

repoman will test n out of 2^n (or n!) possibilities the way you
suggest, which is basically nothing when n is big

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Rich Freeman]

On Mon, Jan 23, 2017 at 4:23 AM, Michał Górny  wrote:
>
> I've written a short proposal that aims to provide basic infrastructure
> for defining mix-in profiles in Gentoo. I've tried to keep it simple,
> and backwards compatible. The main goal is to be able to start defining
> some mix-ins without having to reinvent the whole profile tree.
>

Would it actually make sense to reinvent more of the profile tree
while we're at it?  So, have a few categories of mixins like kernel,
arch, and some category that covers really invasive stuff like
hardened/libc/etc?

Those might be 1-of-n selections.

Then we could have the fluff that sits on top and just set some rules
about what they can do.

Part of me wonders if some of this could also fit in with the use of
virtuals (think foo-meta virtuals but bigger).  A virtual can of
course pull in USE dependencies, and a lot of other stuff.  We could
have convenience virtuals that all the profiles pull in by default but
which gets stuff like openssh out of @system.

I'm only suggesting the last bit to the extent where we see tie-ins to
improve the initial mix-in implementation.  A lot of that is probably
an expansion in scope, and to that extent I'm not suggesting that it
needs to be addressed.  I just want to think about it broadly at first
to make sure we're not missing something.

-- 
Rich

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[james]

On 01/23/2017 04:23 AM, Michał Górny wrote:

Hi, everyone.

I've written a short proposal that aims to provide basic infrastructure
for defining mix-in profiles in Gentoo. I've tried to keep it simple,
and backwards compatible. The main goal is to be able to start defining
some mix-ins without having to reinvent the whole profile tree.

Most important points:

1. Mix-ins are applied on top of base profile (which works the same as
before),

2. Mix-ins are supported via 'eselect profile'
replacing /etc/portage/make.profile symlink with a directory, without
need for Portage patching (this is how Funtoo does it),

3. Most important mix-ins are used to construct base profiles which
provides both backwards compatibility and proper targets for repoman
(to avoid having to check all possible mix-in combinations).

Complete text:

https://wiki.gentoo.org/wiki/User:MGorny/GLEP:Mix-ins

Please review.


On 01/23/2017 04:30 AM, Michał Górny wrote:
> Hi, everyone.
>
> I've written a short proposal that aims to provide basic infrastructure
> for defining mix-in profiles in Gentoo. I've tried to keep it simple,
> and backwards compatible. The main goal is to be able to start defining
> some mix-ins without having to reinvent the whole profile tree.
>
> Most important points:
>
> 1. Mix-ins are applied on top of base profile (which works the same as
> before),
>
> 2. Mix-ins are supported via 'eselect profile'
> replacing /etc/portage/make.profile symlink with a directory, without
> need for Portage patching (this is how Funtoo does it),
>
> 3. Most important mix-ins are used to construct base profiles which
> provides both backwards compatibility and proper targets for repoman
> (to avoid having to check all possible mix-in combinations).
>
> Complete text:
>
> https://wiki.gentoo.org/wiki/User:MGorny/GLEP:Mix-ins
>
> Please review.
>

Hello Michal,

Excellent work. My cluster development work is (minimal and embedded 
gentoo) systems that are dynamically able to coalesce into various 
cluster configurations, very similar to how 'unikernels' work. [1]


I'm curious how 'mix-in' profiles might be used to build from bare-metal 
up a minimized gentoo cluster, just a skeleton approach using mix-in 
profiles?



[1] http://unikernel.org/

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Alexis Ballier]

On Mon, 23 Jan 2017 18:38:07 +0100
Michał Górny  wrote:

> On Mon, 23 Jan 2017 18:17:58 +0100
> "Paweł Hajdan, Jr."  wrote:
> 
> > Michał: also consider including in the GLEP how eselect profile
> > would look like and behave.  
> 
> Sounds like unnecessary scrutiny.
> 

Is 'eselect profile enable mixinA && eselect profile enable mixinB' the
same as 'eselect profile enable mixinB && eselect profile enable
mixinA' ?

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Alexis Ballier]

On Mon, 23 Jan 2017 18:17:58 +0100
"Paweł Hajdan, Jr."  wrote:

> On 23/01/2017 13:12, Alexis Ballier wrote:
> > For example, if you allow use.mask or use.force in mixins, you can
> > end up having unsatisfiable deps that repoman will never catch.  
> 
> Whoa, that sounds bad. Could you elaborate why we wouldn't be able to
> catch these errors?

Because repoman wont be able to check the 2^{# of mixins} possibilities
to have something broken there.

> > Arguably, desktop profiles relying on having an useflag forced on a
> > given package are already semi-broken: they'd be better with the
> > useflag default enabled and proper usedeps, so the mask/force game
> > doesnt seem really useful for mixins.  
> 
> Could you give examples of such assumptions? I'd agree in general
> usedeps sound like the proper solution.

gentoo-x86/profiles/targets/desktop/package.use.force:

# Ensure shared-mime-info is pulled in by glib, otherwise GNOME, XFCE,and
# numerous gtk-based applications will break, see bug #511894
dev-libs/glib mime

> > It'd also be great to have "rules" ensuring all mixins commute, but
> > I doubt that's easily doable.  
> 
> Could you elaborate more on that, and what the difficulties are?

First you need to define *exactly* what goes into mixins.

Then, you want that a combination of mixins doesnt depend on the order
they are processed. It is left to the reader to show that it is
equivalent to that any two mixins commute, that is, the resulting
profiles with the 'parent' file containing:
mixinA
mixinB
or:
mixinB
mixinA

are the same.

If you define mixins having only default enabled or default disabled
useflags (globally or per package), then commuting is implied
by 'no two mixins handle intersecting sets of (package,useflag)'.

One can also go the simple way: Defining a mixin priority (either
explicitly or by alphabetical order for example), so that e.g. in the
above, mixinA is always processed first.



Without doing something in that regard I think this is too loosely
specified and asks for headaches.

Alexis.

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Michał Górny]

On Mon, 23 Jan 2017 18:17:58 +0100
"Paweł Hajdan, Jr."  wrote:

> Michał: also consider including in the GLEP how eselect profile would
> look like and behave.

Sounds like unnecessary scrutiny.

-- 
Best regards,
Michał Górny



pgpzD7SVfXUw5.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Michał Górny]

On Mon, 23 Jan 2017 13:12:19 +0100
Alexis Ballier  wrote:

> On Mon, 23 Jan 2017 10:23:35 +0100
> Michał Górny  wrote:
> 
> > Hi, everyone.
> > 
> > I've written a short proposal that aims to provide basic
> > infrastructure for defining mix-in profiles in Gentoo. I've tried to
> > keep it simple, and backwards compatible. The main goal is to be able
> > to start defining some mix-ins without having to reinvent the whole
> > profile tree.
> > 
> > Most important points:
> > 
> > 1. Mix-ins are applied on top of base profile (which works the same as
> > before),
> > 
> > 2. Mix-ins are supported via 'eselect profile'
> > replacing /etc/portage/make.profile symlink with a directory, without
> > need for Portage patching (this is how Funtoo does it),
> > 
> > 3. Most important mix-ins are used to construct base profiles which
> > provides both backwards compatibility and proper targets for repoman
> > (to avoid having to check all possible mix-in combinations).
> > 
> > Complete text:
> > 
> > https://wiki.gentoo.org/wiki/User:MGorny/GLEP:Mix-ins
> 
> 
> Dont we need to restrict what is allowed in mixins profiles ?
> It doesn't have to be in the glep, but I think it'd be good to have.

At some point, probably yes. I wanted to start with a GLEP to have
technical basics, then see how it all works out. I don't consider
myself capable of predicting it all right now.

> For example, if you allow use.mask or use.force in mixins, you can end
> up having unsatisfiable deps that repoman will never catch.
> Arguably, desktop profiles relying on having an useflag forced on a
> given package are already semi-broken: they'd be better with the
> useflag default enabled and proper usedeps, so the mask/force game
> doesnt seem really useful for mixins.

That's why if you do such a thing, you would have to declare a regular
profile using this mix-in for repoman to test.

-- 
Best regards,
Michał Górny



pgpfaQgf5gjX8.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Paweł Hajdan , Jr .]

On 23/01/2017 13:12, Alexis Ballier wrote:
> For example, if you allow use.mask or use.force in mixins, you can end
> up having unsatisfiable deps that repoman will never catch.

Whoa, that sounds bad. Could you elaborate why we wouldn't be able to
catch these errors?

> Arguably, desktop profiles relying on having an useflag forced on a
> given package are already semi-broken: they'd be better with the
> useflag default enabled and proper usedeps, so the mask/force game
> doesnt seem really useful for mixins.

Could you give examples of such assumptions? I'd agree in general
usedeps sound like the proper solution.

> It'd also be great to have "rules" ensuring all mixins commute, but I
> doubt that's easily doable.

Could you elaborate more on that, and what the difficulties are?

Michał: also consider including in the GLEP how eselect profile would
look like and behave.

Paweł




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Pre-GLEP for review: mix-in profiles

[Alexis Ballier]

On Mon, 23 Jan 2017 10:23:35 +0100
Michał Górny  wrote:

> Hi, everyone.
> 
> I've written a short proposal that aims to provide basic
> infrastructure for defining mix-in profiles in Gentoo. I've tried to
> keep it simple, and backwards compatible. The main goal is to be able
> to start defining some mix-ins without having to reinvent the whole
> profile tree.
> 
> Most important points:
> 
> 1. Mix-ins are applied on top of base profile (which works the same as
> before),
> 
> 2. Mix-ins are supported via 'eselect profile'
> replacing /etc/portage/make.profile symlink with a directory, without
> need for Portage patching (this is how Funtoo does it),
> 
> 3. Most important mix-ins are used to construct base profiles which
> provides both backwards compatibility and proper targets for repoman
> (to avoid having to check all possible mix-in combinations).
> 
> Complete text:
> 
> https://wiki.gentoo.org/wiki/User:MGorny/GLEP:Mix-ins


Dont we need to restrict what is allowed in mixins profiles ?
It doesn't have to be in the glep, but I think it'd be good to have.


For example, if you allow use.mask or use.force in mixins, you can end
up having unsatisfiable deps that repoman will never catch.
Arguably, desktop profiles relying on having an useflag forced on a
given package are already semi-broken: they'd be better with the
useflag default enabled and proper usedeps, so the mask/force game
doesnt seem really useful for mixins.


It'd also be great to have "rules" ensuring all mixins commute, but I
doubt that's easily doable.

[gentoo-dev] Pre-GLEP for review: mix-in profiles

[Michał Górny]

Hi, everyone.

I've written a short proposal that aims to provide basic infrastructure
for defining mix-in profiles in Gentoo. I've tried to keep it simple,
and backwards compatible. The main goal is to be able to start defining
some mix-ins without having to reinvent the whole profile tree.

Most important points:

1. Mix-ins are applied on top of base profile (which works the same as
before),

2. Mix-ins are supported via 'eselect profile'
replacing /etc/portage/make.profile symlink with a directory, without
need for Portage patching (this is how Funtoo does it),

3. Most important mix-ins are used to construct base profiles which
provides both backwards compatibility and proper targets for repoman
(to avoid having to check all possible mix-in combinations).

Complete text:

https://wiki.gentoo.org/wiki/User:MGorny/GLEP:Mix-ins

Please review.

-- 
Best regards,
Michał Górny



pgp2E8mzaoB24.pgp
Description: OpenPGP digital signature