Re: [gentoo-dev] RFC: Userkit.eclass

[Daniel Campbell]

On 11/28/2016 03:26 PM, M. J. Everitt wrote:
> On 28/11/16 19:39, William L. Thomson Jr. wrote:
>> For now who cares about other OS or distros. If Gentoo gets its house in 
>> order 
>> others may follow.
>>
> At the risk of a huge flame, remind me, who uses Gentoo again?!
> 
Unless something's changed in the past year or two, iirc Sony uses
Gentoo as part of the backend of Gaikai, Google's used it for the base
of ChromeOS... I can't speak for other 'big names', but Gentoo's not
quite as niche as the small, active userbase has most of us believing.

There's also our downstream neighbors: Funtoo, Pentoo, Sabayon,
Calculate, Exherbo, etc

As for communities, lots of places from 4chan to lainchan, various mesh
network users, security-conscious communities, OCD support groups
(kidding), etc.

I'm sure I'm missing some mentions here; this is just off the top of my
head.
-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

On Wednesday, November 30, 2016 4:38:57 PM EST Michael Mol wrote:
> 
> You're asserting that Red Hat and Debian do things differently because
> there's nobody to force them to do things the same way. It can't be because
> there's no reference for them to look at; for sure, the second into market
> could simply have looked at the first. It's probable they did.

I would recommend doing some research on LSB. If Debian gave up deb for RPM, 
then you would see more unification. Ultimately the two had different origins, 
purposes, and thus can only align on some things but not all.

You can only push standards so far. When you have two different opposing ways 
of doing something. A call has to be made as to which to standardize unless 
you do both. It is impossible to please everyone all the time when it comes to 
consensus stuff. People have different ideas and will accomplish the same 
things in different ways.

Look at the whole systemd thing that came out of RedHat and took over the 
Linux world. It damn near fractured Debian, and did create a fork. Most other 
distros including Debian gave in.

Gentoo is one of the few still offering an alternative OpenRC. Much like the 
UID/GID thing I blame Gentoo for OpenRC not being adopted more widely and 
giving room for things like systemd to take over with little alternative for 
most.

This kind of thing will happen again and again. Though if other distros do not 
start leading. It will become more and more a RedHat lead Linux world. Though 
some things go the other way. RedHat seems to have adopted Debians 
alternatives system, which is basically eselect, and related tools *-config.

But we are way off topic to the userkit.eclass and UID/GID. New thread, but I 
am past the discussion. Thanks!

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[Michael Mol]

On Wednesday, November 30, 2016 03:25:21 PM William L. Thomson Jr. wrote:
> On Wednesday, November 30, 2016 3:08:30 PM EST Michael Mol wrote:
> > > IMHO it is something that should be  a part of LSB. If not POSIX in
> > > general. One cannot really change the past or current state of things.
> > > But can make
> > 
> > the future better.
> > 
> > > For now who cares about other OS or distros. If Gentoo gets its house in
> > > order
> > > others may follow.
> > 
> > I will note that it's this point when I first replied; that was the point
> > when you chose to expand the scope outside Gentoo.
> 
> Stop making things into something they are not. Re-read the above I said it
> should be part of official standards. I also said others MAY follow...

Honestly, that sounded to me like advocacy; "a benefit of doing this is that 
others may follow." If that's not the spirit in which it was intended, I 
apologize.

> 
> > > Gentoo cannot force others to do anything.
> > 
> > I didn't say force. I said invite.
> 
> I never typed the word invite. I never mentioned Gentoo being proactive
> about pushing its specific things on others. Please stop making stuff up
> and going way off topic.

As I note above, I interpreted what you said as advocacy.

> 
> > As you noted, Arch appeared to attempt this, and others did not follow.
> 
> Arch themselves never got it squared away. It was just a concept. If Arch
> does not implement it how can others? I hardly consider Arch a leading
> distro like RHEL or Debian, which both have derivatives in wide use,
> Fedora, CentOS and Ubuntu.
> 
> That right there likely covers over 50% of all Linux installs.
> 
> > That's fine. As I pointed out, I only started chiming in when you began
> > advocating exporting Gentoo's list to a broader ecosystem.
> 
> You are reading things I never typed, and coming up with some far fetched
> scenarios. Nothing you are saying is anywhere near what I wrote.

Again, read above. If that's not how it was intended, I apologize.

> 
> > If RHEL and Debian are consistent from one system to the next, obviously
> > it's sensical to use their list. But why don't they use each others? Or am
> > I missing something, and that's exactly what they're doing?
> 
> Going back to my first point about this being part of LSB or POSIX. Because
> it is part of neither RedHat and Debian do things differently.


You're asserting that Red Hat and Debian do things differently because there's 
nobody to force them to do things the same way. It can't be because there's no 
reference for them to look at; for sure, the second into market could simply 
have looked at the first. It's probable they did.

I know Debian starts their non-system UIDs at 1000, while RH, once upon a 
time, started theirs at 500. Why the difference? Dunno. RH came before Debian, 
so I imagine Debian wanted a bit more headroom to work with. Are there static 
UIDs in the 500-999 range on Debian? That would be why RH doesn't use Debian's 
set; they'd have a UID conflict on their hands.

Staring at a CentOS7 live environment in front of me, it looks like RH now 
starts at 1000.

It's probable they could settle on a common spec now, but there would still be 
a great number of legacy systems out there to support., and you've still got a 
very limited namespace to work with.

> 
> Why does RedHat not use deb format over rpm. Why does Debian use deb instead
> of RPM. 

Well, RPM was developed to be a better alternative to the tarball. Debian 
thought the RPM format was lacking, and developed their own spec. For sure, 
nobody likes to do work for no reason. Even hugely disruptive changes have 
motivations behind them.

I'm sorry, was that a rhetorical question? I just realized...

> These are different distros with different approaches. If their
> UID/ GID are the same, its likely per legacy reasons. Though they may be
> looking at each other.
> 
> Debian at this time does not produce a list. The only I found were RedHat
> and Arch, with Archs' being unofficial and never adopted.

I'll note I'm treating the concept of a list as very abstract; if things are 
consistent, then there's de facto a consistent state that could be distilled 
deterministically into a listing.

> 
> > Sure. But if you clone a seed node, does it matter that a second
> > from-scratch install may not have the same mapping?
> 
> Yes if they are to be added to the same fleet or cluster of systems. In that
> event it would likely start a new from scratch base image. But that is
> pretty rare. I do update base images, though rarely do system UID/GID
> change from initial install.

You know, I would expect for a system of that scale, that you'd have 
standardized and preseeded your passwd and group files with your site standard 
enumerations. It would be trivial to do in any Gentoo install; copy your files 
into place before your initial chroot. All of which you should have scripted 
at this point. If you'd like, I'll send you a link to mine; you can use i

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

On Wednesday, November 30, 2016 3:08:30 PM EST Michael Mol wrote:
>
> > IMHO it is something that should be  a part of LSB. If not POSIX in
> > general. One cannot really change the past or current state of things.
> > But can make
> the future better.
> 
> > For now who cares about other OS or distros. If Gentoo gets its house in
> > order
> > others may follow.
> 
> I will note that it's this point when I first replied; that was the point
> when you chose to expand the scope outside Gentoo.

Stop making things into something they are not. Re-read the above I said it 
should be part of official standards. I also said others MAY follow...

> > Gentoo cannot force others to do anything.
> 
> I didn't say force. I said invite.

I never typed the word invite. I never mentioned Gentoo being proactive about 
pushing its specific things on others. Please stop making stuff up and going 
way off topic.
 
 
> As you noted, Arch appeared to attempt this, and others did not follow.

Arch themselves never got it squared away. It was just a concept. If Arch does 
not implement it how can others? I hardly consider Arch a leading distro like 
RHEL or Debian, which both have derivatives in wide use, Fedora, CentOS and 
Ubuntu.

That right there likely covers over 50% of all Linux installs.


> That's fine. As I pointed out, I only started chiming in when you began
> advocating exporting Gentoo's list to a broader ecosystem.

You are reading things I never typed, and coming up with some far fetched 
scenarios. Nothing you are saying is anywhere near what I wrote.

> If RHEL and Debian are consistent from one system to the next, obviously
> it's sensical to use their list. But why don't they use each others? Or am
> I missing something, and that's exactly what they're doing?

Going back to my first point about this being part of LSB or POSIX. Because it 
is part of neither RedHat and Debian do things differently.

Why does RedHat not use deb format over rpm. Why does Debian use deb instead 
of RPM. These are different distros with different approaches. If their UID/
GID are the same, its likely per legacy reasons. Though they may be looking at 
each other.

Debian at this time does not produce a list. The only I found were RedHat and 
Arch, with Archs' being unofficial and never adopted.

> Sure. But if you clone a seed node, does it matter that a second
> from-scratch install may not have the same mapping?

Yes if they are to be added to the same fleet or cluster of systems. In that 
event it would likely start a new from scratch base image. But that is pretty 
rare. I do update base images, though rarely do system UID/GID change from 
initial install.

> If UID/GID are consistent between RH and Debian, then yeah, what you have is
> a de facto standard, and it would be reasonable to conform, if there are
> people who actually have a need for that cross-system mirroring.

If Gentoo does the same, that would make one other and moving all more in the 
direction of a standard.

> > > More daemons will be build that are intended to
> > > run as local users. More software will be pushed into opaque blobs a la
> > > Snap and Flatpack.
> > 
> > I am talking about core system accounts
> 
> Who decides what qualifies as a core system account?

This is pretty silly now and way off topic. I will leave it to others to 
decide. I would prefer to go beyond just system so it is Gentoo wide. Arch was 
not limited to system stuff, like RedHat and Debian.

Really up to Gentoo Developers to decide it all.

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[Michael Mol]

On Wednesday, November 30, 2016 01:41:24 PM William L. Thomson Jr. wrote:
> On Wednesday, November 30, 2016 1:22:07 PM EST Michael Mol wrote:
> > If Gentoo wants to do it internally, that's one thing.
> 
> This list is about Gentoo internal things

Here, let me bring up a bit of recent history from your Message-ID 
, which had a signature of 
iEYEABECAAYFAlg8iAQACgkQTXGypIOqM1A2EgCglmZkNYaJ16qQkSxezTqCtI4/
PwoAnR2dW0XUFZk8QUmgrVwu+3OpRxS+
=tuat, which my client indicated matched the key 
0xC47A576A663995BADF1B54724D71B2A483AA3350, but I don't have your key trusted, 
so whatever:

> I believe the main reason such is the case is a lack of any such list or 
> database for others to adhere to. Once again an area Gentoo could be
> leading. 
> Had Gentoo done this years ago others might have adopted.

> IMHO it is something that should be  a part of LSB. If not POSIX in general. 
> One cannot really change the past or current state of things. But can make 
the future better.

> For now who cares about other OS or distros. If Gentoo gets its house in
> order 
> others may follow.

I will note that it's this point when I first replied; that was the point when 
you chose to expand the scope outside Gentoo.

> 
> > But I would recommend
> > against inviting other distributions to use Gentoo's list, which was
> > something you seemed to be suggesting. Doing so asks that Gentoo shoulder
> > the bureaucratic load from other distributions that want things added to
> > Gentoo's list.
> 
> Gentoo cannot force others to do anything.

I didn't say force. I said invite.

> If Gentoo is leading in a
> direction, others choose to follow or not. Gentoo does not set standards
> that would be up to LSB and/or POSIX.
> 
> My point is Gentoo should do its own thing, lead the way. Ideally others
> follow and it becomes a standard either in LSB or POSIX. Hopefully that will
> clarify my position.

As you noted, Arch appeared to attempt this, and others did not follow.

> 
> > If you want to tie this specifically to Gentoo packaging, that's fine.
> 
> Which is why it is being discussed on a Gentoo development list and not
> others.

That's fine. As I pointed out, I only started chiming in when you began 
advocating exporting Gentoo's list to a broader ecosystem.

[snip]

> > > This is not needless bureaucracy , this is necessary.
> > 
> > Opinion. Why is it necessary? What is it necessary for?
> 
> It is necessary so Gentoo base system installs are consistent from one
> system to the next, Just as RHEL and Debian, and likely others. When
> working with large amounts of installs, You want them all to be the same or
> as close to identical as possible. Thus the rise of Docker, CoreOS, etc.

If RHEL and Debian are consistent from one system to the next, obviously it's 
sensical to use their list. But why don't they use each others? Or am I 
missing something, and that's exactly what they're doing?

> 
> > Oh, I understand the problem, but you haven't explained why your solution
> > is the necessary solution to it, or how you would cope with the plethora
> > of edge cases I brought up. It would seem there are already many
> > established workarounds for the status quo, unstable-UID/GID in a
> > cross-system context.
> My solution is to avoid such issues. I start with a common base image. I try
> to ensure anything else installed beyond that, which adds new users/groups
> is the same. At times I will re-image and use that as well for other
> similar systems. Rather than mess with doing the same install to many and
> trying to sync UID/GID.
> 
> Think cloning rather than installing.

Sure. But if you clone a seed node, does it matter that a second from-scratch 
install may not have the same mapping?

[snip]

> > Less bad if you intend to keep it unique to
> > Gentoo, but the broader you make the scope, the more strain you'll put on
> > the ecosystem as a whole.
> 
> Standards need to exist so there is consistency. In the absence of said
> standard, next best thing you can do is look to what others are doing and do
> the same. Thus I tend to say go with RedHat UID/GID over say Arch, maybe
> even Debian.   But those two likely have larger install bases than most any
> other distro. If the UID/GID are the same between RedHat and Debian, that
> already makes a good deal of systems consistent now.

If UID/GID are consistent between RH and Debian, then yeah, what you have is a 
de facto standard, and it would be reasonable to conform, if there are people 
who actually have a need for that cross-system mirroring.

> 
> > More daemons will be build that are intended to
> > run as local users. More software will be pushed into opaque blobs a la
> > Snap and Flatpack.
> 
> I am talking about core system accounts

Who decides what qualifies as a core system account?

If there's any trend I've been able to clearly observe over the last fifteen 
years, it's the grinding of such boundaries into finer and finer granularity. 
Heck, I think there was a thre

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

On Wednesday, November 30, 2016 1:22:07 PM EST Michael Mol wrote:
>
> 
> If Gentoo wants to do it internally, that's one thing. 

This list is about Gentoo internal things

> But I would recommend
> against inviting other distributions to use Gentoo's list, which was
> something you seemed to be suggesting. Doing so asks that Gentoo shoulder
> the bureaucratic load from other distributions that want things added to
> Gentoo's list.

Gentoo cannot force others to do anything. If Gentoo is leading in a 
direction, others choose to follow or not. Gentoo does not set standards that 
would be up to LSB and/or POSIX.

My point is Gentoo should do its own thing, lead the way. Ideally others 
follow and it becomes a standard either in LSB or POSIX. Hopefully that will 
clarify my position.

> If you want to tie this specifically to Gentoo packaging, that's fine.

Which is why it is being discussed on a Gentoo development list and not 
others.

> Though I'd recommend you put the user and group allocation in the ebuild.
> Then your "list" is trivially generable by parsing portage. Further, you
> can *enforce* these allocations when calculating the dependency tree. If
> you're not enforcing them, what's the point? Is there a benefit without
> said enforcement?

As stated, enewuser/enewgroup would utilize such a list/database directly. In 
addition to repoman so issues are prevented before ebuilds are committed.

> > This is not needless bureaucracy , this is necessary.
> 
> Opinion. Why is it necessary? What is it necessary for?

It is necessary so Gentoo base system installs are consistent from one system 
to the next, Just as RHEL and Debian, and likely others. When working with 
large amounts of installs, You want them all to be the same or as close to 
identical as possible. Thus the rise of Docker, CoreOS, etc.
 
> Oh, I understand the problem, but you haven't explained why your solution is
> the necessary solution to it, or how you would cope with the plethora of
> edge cases I brought up. It would seem there are already many established
> workarounds for the status quo, unstable-UID/GID in a cross-system context.

My solution is to avoid such issues. I start with a common base image. I try 
to ensure anything else installed beyond that, which adds new users/groups is 
the same. At times I will re-image and use that as well for other similar 
systems. Rather than mess with doing the same install to many and trying to 
sync UID/GID.

Think cloning rather than installing.

> But trying to set up a list for everyone to move in lockstep with seems to
> me like a bad way to go.

See my other post, other distros already do this for core system accounts.

> Less bad if you intend to keep it unique to
> Gentoo, but the broader you make the scope, the more strain you'll put on
> the ecosystem as a whole. 

Standards need to exist so there is consistency. In the absence of said 
standard, next best thing you can do is look to what others are doing and do 
the same. Thus I tend to say go with RedHat UID/GID over say Arch, maybe even 
Debian.   But those two likely have larger install bases than most any other 
distro. If the UID/GID are the same between RedHat and Debian, that already 
makes a good deal of systems consistent now.

> More daemons will be build that are intended to
> run as local users. More software will be pushed into opaque blobs a la
> Snap and Flatpack.

I am talking about core system accounts

> As a general rule, the bigger the hassle you make something, the less people
> will want to engage.

When standards exist, others will follow, ideally. When standards do not 
exist, everyone is left to their own way of doing things. IMHO it is less of a 
hassle to comply with standards than all the various ways of doing something.

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[Michael Mol]

On Tuesday, November 29, 2016 04:49:24 PM William L. Thomson Jr. wrote:
> On Tuesday, November 29, 2016 10:40:20 AM EST Michael Mol wrote:
> > Highly detailed lists like that--used as a broad standard--are a bad idea.
> > They represent a single synchronization point that everyone must adhere
> > to.
> 
> That is a statement based on opinion.

Of course. And then I gave examples as to why.

> You say it is a bad idea. I say it is
> necessary and needed. Otherwise wrt to Gentoo ebuilds can stomp on each
> other. Using same GID or UID in more than one ebuild causing problems.
> There has to be something know so others do not use ones others are
> already.

If Gentoo wants to do it internally, that's one thing. But I would recommend 
against inviting other distributions to use Gentoo's list, which was something 
you seemed to be suggesting. Doing so asks that Gentoo shoulder the 
bureaucratic load from other distributions that want things added to Gentoo's 
list.

> 
> > That means that every prospective adjustment to the list requires active
> > maintenance. That means that for every new daemon someone writes, they
> > have
> > to go through an admissions process. For every contentious fork of a
> > project, you risk conflict over who the designated contact for the
> > assignment should be.
> 
> If they package such in Gentoo someone is making a call as to what UID and
> GID should be used. If you think about it from packaging said new daemon in
> Gentoo, it is a MUST.
> 
> If it does not exist, should it be entirely random from the packager
> perspective? What if they use a GID/UID specific to them and not others.
> 
> There has to be some standard some consistency in Gentoo.

If you want to tie this specifically to Gentoo packaging, that's fine. Though 
I'd recommend you put the user and group allocation in the ebuild. Then your 
"list" is trivially generable by parsing portage. Further, you can *enforce* 
these allocations when calculating the dependency tree. If you're not 
enforcing them, what's the point? Is there a benefit without said enforcement?

> 
> > It adds a large bureaucratic load on everyone. Every itch some developer
> > thinks about scratching has to be weighed against engaging with some
> > process- laden entity. Maybe they'll participate, but they likely won't.
> 
> Gentoo shines at bureaucratic load. That may be one of the only things
> Gentoo is really good at, needless bureaucratic loads that just slow things
> down and fracture the community, exherbo, funtoo, and likely others...

I was under the impression that Gentoo was chronically undermanned for even 
the workload it has.

> 
> This is not needless bureaucracy , this is necessary.

Opinion. Why is it necessary? What is it necessary for?

> 
> > Have you watched the IANA ports assignment registry over the years?
> > Consider how many services and tools you've seen that *don't* respect it.
> 
> Yes, how often to ports < 1024 change? Hardly ever Proving the exact
> point why this is needed. People can change them themselves but 99% of the
> time its to some other port > 1024.
> 
> Why is there IANA port assignment registry in the first place? Likely for a
> similar reason.

How relevant even *is* the <1024 distinction any longer? Once upon a time, the 
idea was you had to have special privileges to open those ports. Now, there is 
really no reason for anyone to care; capabilities-oriented permissions 
completely obviated the need, and I can only think of ssh, telnet and ftp and 
as server services that should require special host privileges to 
operate...and that's only because they may need to be able to call setuid().

And because the <1024 port privilege distinction has been so restrictive and 
bureaucratically sloggish, applications adapted to use ports above 1024. 
Games? Sync utilities? Proxy servers? Far more commonly-observed ports are 
above 1024 than below it, and many (most?) don't even get added to IANA's 
list. *That's* why the <1024 ports don't change much; the feature is obsolete, 
and users don't bother.

As an example, I just checked on Syncthing, to see if its three ports were on 
IANA's list. They're not, and I stumbled across a Github issue where the devs 
flatly stated they didn't care.

The IANA ports list is, by and large, obsolete. It became obsolete because it 
was too much a hassle for people to participate in.

> 
> > All of this is why we use identity management tools like LDAP in the first
> > place. Heck, it's why we have passwd and group files for mapping names to
> > ids and didn't simply hardcode system IDs decades ago.
> 
> LDAP typical manages user accounts not system. If the LDAP server is not
> reachable you would make a system completely nonfunctional if it relied on
> LDAP for system accounts.

That's fair. Although I really like how one LDAP alternative operates over 
DNS, permitting local caching. (I can't for the life of me remember the name 
of that system, though.)

> 
> Also needed f

Re: [gentoo-dev] RFC: Userkit.eclass

[james]

On 11/30/2016 10:23 AM, William L. Thomson Jr. wrote:

A couple more links, I should have provided initially as they better support
the argument.

First from Debian, I cannot find a list, but it is clearly mentioned.

"0-99:
Globally allocated by the Debian project, the same on every Debian system"
https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.2.2

This is even better, what Gentoo lacks, and could build upon.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/
Deployment_Guide/s1-users-groups-standard-users.html

Also carries to CentOS of course
https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-users-groups-standard-users.html

Per previous links installing some RPMs that have fixed UID/GID will result in
problems of other things are using it

"The vdsm user however is fixed to a UID of 36 and the kvm group is fixed to a
GID of 36.
If UID 36 or GID 36 is already used by another account on the system then a
conflict will arise during installation of the vdsm and qemu-kvm-rhev
packages."

https://access.redhat.com/documentation/en-US/
Red_Hat_Enterprise_Virtualization/3.5/html/Installation_Guide/sect-
System_Accounts.html


I appreciate all the discussion on uid-gid as it is central to cluster 
provisioning work.


Some Background::


My specific area of development is heterogeneous (hardware) gentoo 
clusters with a specific focus on "uni-kernels' (UK). I define UK as a 
minimized, optimized hardened kernel that are specifically tuned to a 
minimized and optimize framework for a specific problem or specific 
category of problem for High Performance Computing (HPC) needs. In fact 
the need to benchmark and compare a myriad of codes, such as openstack 
on RHEL vs a skinny gentoo solution, on the exact same hardware will 
necessitate provisioning from bare metal up to full stack online and 
thus require numerous boot cycles. uid/gid symmetry would be a keen 
component of to my solutions. One of the challenges I have not worked on 
yet, is a systematic and automated solution for a variety of uid-gid 
differences between the systems I need to test and compare.


I am not certain that an ebuild or PMS level solution will work for 
comparing images(canned solutions from various sources) to a minimized 
and optimized gentoo solution. Furthermore, I'd definitely appreciated 
any advice and templates/profiles/scripts/etc that facilitate the 
automation of uid/gid compatibility for as wide a variety of 
kernels+OS+framework at least within gentoo. Note: for me a 'framework' 
is vary similar to the world-file. On other distros, a framework is the 
sum of additional codes on top of a basic installation of that distro. 
Applicability to other major distros, such as *bunu, RH, debian, and 
arch derivatives would be keenly useful for my research and development 
needs. Furthermore, I believe that docker is just killing the cluster 
competition with uni-kernels and a minimized distro such as Alpine. This
is an embarrassment to Gentoo that docker+alpine is 'killing it' in a 
space that is natural for Gentoo to dominate, imho.



This is a complex issue, as most of what has already been posted to this 
thread are all impactfully true. So flexibility is paramount, imho.  In 
fact if there is a way, I'd suggest that a multitude of scenarios are 
supported to the point that for my work there could easily be hundreds 
of variants. The keyword, 'profiles' comes to mind, but that has 
additional connotations within gentoo. Surely a robust and automated way 
to deal with differences in uid/gid between differing systems (same 
distro or not) would be an excellent project. If this is or is not 
possible, regardless of whether other distros use this capability, it 
would certainly aid folks in migrating other systems
from different distros to gentoo; so that bring enormous value to gentoo 
as a distro.



More specifically::

One thing is for sure, uni-kernels are just killing 'canned cluster' 
solutions for specific types of problems, particular defined by HPC. I
strongly believe that all of that pioneering work on HPC clustering will 
definitely impact routine web/admin/processing venues, eventually.
A given organization will be able to find the optimal images for their 
needs and then easily migrate their needs to a wide variety of 
datacenters for peak or scale-up. Unikernels in a wide variety of forms,
will enable hybrid clusters and ease the migration of business, web and 
other needs between clusters, in a seamless fashion.


A robust and flexible way to automate, orchestrate (overused term I 
know) and provision thousands of  systems is desperately needed, imo,

and a tool to transparently handle uid/gid differences would be keen.

I want to thank you, for introducing this topic and I tremendously 
appreciate all of the comments folks are interjecting, even the terse 
comments from admins that need a way to 'turn off' these features.
Ultimately, CoreOS has an automated provis

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

A couple more links, I should have provided initially as they better support 
the argument.

First from Debian, I cannot find a list, but it is clearly mentioned.

"0-99:
Globally allocated by the Debian project, the same on every Debian system"
https://www.debian.org/doc/debian-policy/ch-opersys.html#s9.2.2

This is even better, what Gentoo lacks, and could build upon.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/
Deployment_Guide/s1-users-groups-standard-users.html

Also carries to CentOS of course
https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-users-groups-standard-users.html

Per previous links installing some RPMs that have fixed UID/GID will result in 
problems of other things are using it

"The vdsm user however is fixed to a UID of 36 and the kvm group is fixed to a 
GID of 36.
If UID 36 or GID 36 is already used by another account on the system then a 
conflict will arise during installation of the vdsm and qemu-kvm-rhev 
packages."

https://access.redhat.com/documentation/en-US/
Red_Hat_Enterprise_Virtualization/3.5/html/Installation_Guide/sect-
System_Accounts.html

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

On Wednesday, November 30, 2016 8:54:42 AM EST Michał Górny wrote:
> On Tue, 29 Nov 2016 18:13:29 -0500
> 
> "William L. Thomson Jr."  wrote:
>>
> > I think you mean enewgroup and enewuser
> 
> FYI, enew* functions handle UID/GID collisions gracefully, and just
> fallback to using next free UID/GID.

I would disagree with such and some what makes specifying a UID/GID pointless 
if it simply will use the next available in the event of a collision. Which 
available likely comes from the default allocation range > 500 or 1000. If 
system and was intended to be below that, not really ideal.
 
> I'm not sure if you're aware that but most of tools doing backups
> actually use usernames/group names. So does new enough tar. So does
> ssh.

tar can map users and groups via file, but why waste the time with such?

> Are you specifically using some obsolete or braindead tools to prove
> your point? If you don't sync UIDs/GIDs properly, then you don't use
> them when moving data across systems. Simple as that.

I start with consistent base images and have the same uid/gid all on all so 
syncing is not needed. Nor do I need to deal with it during restoration.

> The only thing that you could worry about then are missing users/groups
> on the target system. But then, so far none of your talk solved that
> problem.

A problem that should not exist with a proper setup.

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[Michał Górny]

On Tue, 29 Nov 2016 18:13:29 -0500
"William L. Thomson Jr."  wrote:

> On Wednesday, November 30, 2016 12:49:44 AM EST Alan McKinnon wrote:
> > 
> > Why would you end up with duplicated UIDs and GIDs? The only real ways
> > that can happen is
> > - ebuild "edits" passwd and group directly using echo/sed and the like.
> > - ebuild runs useradd|groupadd specifying the uid/gid as arguments  
> 
> I think you mean enewgroup and enewuser

FYI, enew* functions handle UID/GID collisions gracefully, and just
fallback to using next free UID/GID.

> > Who cares what the uid/gid is? There's a range of about 950 to chose
> > from. The way to ensure a filesystem object has the correct owner and
> > group is by using chown/chgrp.  
> 
> See above, any administrator moving files between systems, restoring backups, 
> etc.
> 
> Say you do a fresh install. What if all your UID/GID differ from your backup? 
> HUGE MESS

I'm not sure if you're aware that but most of tools doing backups
actually use usernames/group names. So does new enough tar. So does
ssh.

Are you specifically using some obsolete or braindead tools to prove
your point? If you don't sync UIDs/GIDs properly, then you don't use
them when moving data across systems. Simple as that.

The only thing that you could worry about then are missing users/groups
on the target system. But then, so far none of your talk solved that
problem.

Furthermore, I should add that neither repeating the same argument
thrice, nor adding some random caps and exclamations marks, won't make
it any more valid.

-- 
Best regards,
Michał Górny



pgp7QoTXwfj8C.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

On Wednesday, November 30, 2016 12:49:44 AM EST Alan McKinnon wrote:
> 
> Why would you end up with duplicated UIDs and GIDs? The only real ways
> that can happen is
> - ebuild "edits" passwd and group directly using echo/sed and the like.
> - ebuild runs useradd|groupadd specifying the uid/gid as arguments

I think you mean enewgroup and enewuser

> Both of which are silly. Just use useradd/groupadd without uid/gid
> arguments. The utility will make sure the uid/gids are non-duplicate,
> and ensure they are <1000 or whatever for system accounts

Randomly chosen GID and UID are a problem in the making. If you haven't 
experienced such yet, give yourself time. Moving files between systems, you 
have to chown/chgrp, etc it is NOT fun...

Or worse you mix stuff and give something improper permissions and really mess 
up security...

> How do you intend to MAKE devs follow it? More eternal bike-shedding?

A nifty tool called repoman which could do a quick lookup. As could enewgroup/
enewuser. They could hit the list/database. If something is trying to use 
existing error, etc. Otherwise process to reserve it, etc.
 

> Who cares what the uid/gid is? There's a range of about 950 to chose
> from. The way to ensure a filesystem object has the correct owner and
> group is by using chown/chgrp.

See above, any administrator moving files between systems, restoring backups, 
etc.

Say you do a fresh install. What if all your UID/GID differ from your backup? 
HUGE MESS
 
> Except for a few cases out on left field (like nfs shares - a problem
> that nfs must fix) you don't really care what the uid/gid is, as long as
> it's not duplicated. The thing you care about is the NAME

Not really just cases you haven't run into yet, which can be very common.
 
> > This is not needless bureaucracy , this is necessary.
> 
> This is a joke right?

Not at all, others are clearly not aware of all the potential issues, having 
not experienced them first hand, yet

Work with enough systems, move files around, share lots of stuff, restore 
backups, you will start to see a major need.

> >> Have you watched the IANA ports assignment registry over the years?
> >> Consider how many services and tools you've seen that *don't* respect
> >> it.
> > 
> > Yes, how often to ports < 1024 change? Hardly ever Proving the exact
> > point why this is needed. People can change them themselves but 99% of
> > the time its to some other port > 1024.
> > 
> > Why is there IANA port assignment registry in the first place? Likely for
> > a
> > similar reason.
> 
> It's so that things like browsers, email tools and the like can drop
> 
> : for the most part and be reasonably sure stuffs will still work.
> 
> Of the 65535 +-1 possible port numbers, only the first 1024 are truly
> important, and of those less than about a quarter are in common use
> (wild guess).

Most of the UID/GID I speak of are below 1000. System accounts, daemons, etc. 
Very likely the exact same stuff running on privileged ports  but not all.

> The next 10,000 or so are not standards by any means, just a list of
> stuff that happens to have been seen in the wild. Apps can and do pick
> any old port they feel like - witness the several things that will use
> 5000 out the box. Is this a problem? Not really, as very very few
> machines out there will install two apps both trying to use port 5000 by
> default.

Nor would that ever be with any system. All *nix systems have a reserved UID/
GID range and users stuff starts above that. Some 500, others 1000, etc.


> I have packaged a few things in Gentoo (privately only)

Try doing it for the public, which will end up with thousands of installs.

> , and written
> more shell installers, puppet manifests, ansible playbooks and user
> account deployers than I care to recall; I've never run into this
> problem that I couldn't solve trivially - usually by just knowing the
> username|groupname and looking up the corresponding uid/gid. Really,
> it's just data mapping and we have tools to do the lookup real fast.

Clearly you haven't come across it yet, and likely because experience has 
differed. But I have given you a few examples of how this could happen to 
anyone and why there would be a need.

Say it is a failed mail server, and you need to take the queue/spool to 
another. Same with print, or other jobs... You need them to have the same UID/
GID, or you end up wasting MORE time syncing them to the system they go onto. 
Much easier to ensure all are the same.

This goes for many other things. Lots of data gets owned by system accounts. 
Moving that data from system to system, with different UID/GIDs is a 
nightmare...

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[Alan McKinnon]

On 29/11/2016 23:49, William L. Thomson Jr. wrote:
> On Tuesday, November 29, 2016 10:40:20 AM EST Michael Mol wrote:
>>
>>
>> Highly detailed lists like that--used as a broad standard--are a bad idea.
>> They represent a single synchronization point that everyone must adhere to.
> 
> That is a statement based on opinion. You say it is a bad idea. I say it is 
> necessary and needed. Otherwise wrt to Gentoo ebuilds can stomp on each 
> other. 
> Using same GID or UID in more than one ebuild causing problems. There has to 
> be something know so others do not use ones others are already.

Why would you end up with duplicated UIDs and GIDs? The only real ways
that can happen is
- ebuild "edits" passwd and group directly using echo/sed and the like.
- ebuild runs useradd|groupadd specifying the uid/gid as arguments

Both of which are silly. Just use useradd/groupadd without uid/gid
arguments. The utility will make sure the uid/gids are non-duplicate,
and ensure they are <1000 or whatever for system accounts

> 
>> That means that every prospective adjustment to the list requires active
>> maintenance. That means that for every new daemon someone writes, they have
>> to go through an admissions process. For every contentious fork of a
>> project, you risk conflict over who the designated contact for the
>> assignment should be.
> 
> If they package such in Gentoo someone is making a call as to what UID and 
> GID 
> should be used. If you think about it from packaging said new daemon in 
> Gentoo, it is a MUST.

How do you intend to MAKE devs follow it? More eternal bike-shedding?

> If it does not exist, should it be entirely random from the packager 
> perspective? What if they use a GID/UID specific to them and not others.
> 
> There has to be some standard some consistency in Gentoo.

Who cares what the uid/gid is? There's a range of about 950 to chose
from. The way to ensure a filesystem object has the correct owner and
group is by using chown/chgrp.

Except for a few cases out on left field (like nfs shares - a problem
that nfs must fix) you don't really care what the uid/gid is, as long as
it's not duplicated. The thing you care about is the NAME

> 
>> It adds a large bureaucratic load on everyone. Every itch some developer
>> thinks about scratching has to be weighed against engaging with some
>> process- laden entity. Maybe they'll participate, but they likely won't.
> 
> Gentoo shines at bureaucratic load. That may be one of the only things Gentoo 
> is really good at, needless bureaucratic loads that just slow things down and 
> fracture the community, exherbo, funtoo, and likely others...
> 
> This is not needless bureaucracy , this is necessary.

This is a joke right?

>> Have you watched the IANA ports assignment registry over the years? Consider
>> how many services and tools you've seen that *don't* respect it.
> 
> Yes, how often to ports < 1024 change? Hardly ever Proving the exact 
> point 
> why this is needed. People can change them themselves but 99% of the time its 
> to some other port > 1024.
> 
> Why is there IANA port assignment registry in the first place? Likely for a 
> similar reason.

It's so that things like browsers, email tools and the like can drop
: for the most part and be reasonably sure stuffs will still work.

Of the 65535 +-1 possible port numbers, only the first 1024 are truly
important, and of those less than about a quarter are in common use
(wild guess).

The next 10,000 or so are not standards by any means, just a list of
stuff that happens to have been seen in the wild. Apps can and do pick
any old port they feel like - witness the several things that will use
5000 out the box. Is this a problem? Not really, as very very few
machines out there will install two apps both trying to use port 5000 by
default.

The top 45,000 - well that's a free-for-all. Mostly only used as source
ports used by apps trying to contact other apps, and not by listeners.

When looked at IANA port assignments in this light, it really does seem
to be a case of the minimum necessary bureaucracy to make things mostly
nicely most of the time, and not at all a case of bureaucracy to
standardize things as you seem to be arguing.

> 
>> All of this is why we use identity management tools like LDAP in the first
>> place. Heck, it's why we have passwd and group files for mapping names to
>> ids and didn't simply hardcode system IDs decades ago.
> 
> LDAP typical manages user accounts not system. If the LDAP server is not 
> reachable you would make a system completely nonfunctional if it relied on 
> LDAP for system accounts.
> 
> Also needed from a file sharing stand point of view if sharing parts of a 
> system across others. You need consistent GID/UID mappings or things like NFS 
> will have lots of problems.

NFS is an edge case. Maybe edge is not the best possible adjective here,
but it certainly isn't the one killer app that requires the whole
uid/gid scheme needing to be locked down.

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

On Tuesday, November 29, 2016 10:40:20 AM EST Michael Mol wrote:
>
> 
> Highly detailed lists like that--used as a broad standard--are a bad idea.
> They represent a single synchronization point that everyone must adhere to.

That is a statement based on opinion. You say it is a bad idea. I say it is 
necessary and needed. Otherwise wrt to Gentoo ebuilds can stomp on each other. 
Using same GID or UID in more than one ebuild causing problems. There has to 
be something know so others do not use ones others are already.

> That means that every prospective adjustment to the list requires active
> maintenance. That means that for every new daemon someone writes, they have
> to go through an admissions process. For every contentious fork of a
> project, you risk conflict over who the designated contact for the
> assignment should be.

If they package such in Gentoo someone is making a call as to what UID and GID 
should be used. If you think about it from packaging said new daemon in 
Gentoo, it is a MUST.

If it does not exist, should it be entirely random from the packager 
perspective? What if they use a GID/UID specific to them and not others.

There has to be some standard some consistency in Gentoo.

> It adds a large bureaucratic load on everyone. Every itch some developer
> thinks about scratching has to be weighed against engaging with some
> process- laden entity. Maybe they'll participate, but they likely won't.

Gentoo shines at bureaucratic load. That may be one of the only things Gentoo 
is really good at, needless bureaucratic loads that just slow things down and 
fracture the community, exherbo, funtoo, and likely others...

This is not needless bureaucracy , this is necessary.

> Have you watched the IANA ports assignment registry over the years? Consider
> how many services and tools you've seen that *don't* respect it.

Yes, how often to ports < 1024 change? Hardly ever Proving the exact point 
why this is needed. People can change them themselves but 99% of the time its 
to some other port > 1024.

Why is there IANA port assignment registry in the first place? Likely for a 
similar reason.

> All of this is why we use identity management tools like LDAP in the first
> place. Heck, it's why we have passwd and group files for mapping names to
> ids and didn't simply hardcode system IDs decades ago.

LDAP typical manages user accounts not system. If the LDAP server is not 
reachable you would make a system completely nonfunctional if it relied on 
LDAP for system accounts.

Also needed from a file sharing stand point of view if sharing parts of a 
system across others. You need consistent GID/UID mappings or things like NFS 
will have lots of problems.

Package a few things in Gentoo that need a UID and/or GID and you will start 
to understand the problem from a operating system packager perspective.

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[Michael Mol]

On Monday, November 28, 2016 02:39:48 PM William L. Thomson Jr. wrote:
> On Monday, November 28, 2016 10:42:54 AM EST Alec Warner wrote:
> > Generally speaking as a fellow who maintained thousands of systems (many
> > of
> > which ran various operating systems.)
> > 
> > You cannot rely on all OS vendors to synchronize uid / gid. You cannot
> > even
> > rely on some single vendors to synchronize uid / gids between releases of
> > their own products.
> 
> I believe the main reason such is the case is a lack of any such list or
> database for others to adhere to. Once again an area Gentoo could be
> leading. Had Gentoo done this years ago others might have adopted.
> 
> IMHO it is something that should be  a part of LSB. If not POSIX in general.
> One cannot really change the past or current state of things. But can make
> the future better.

Highly detailed lists like that--used as a broad standard--are a bad idea. 
They represent a single synchronization point that everyone must adhere to. 

That means that every prospective adjustment to the list requires active 
maintenance. That means that for every new daemon someone writes, they have to 
go through an admissions process. For every contentious fork of a project, you 
risk conflict over who the designated contact for the assignment should be.

It adds a large bureaucratic load on everyone. Every itch some developer 
thinks about scratching has to be weighed against engaging with some process-
laden entity. Maybe they'll participate, but they likely won't.

Have you watched the IANA ports assignment registry over the years? Consider 
how many services and tools you've seen that *don't* respect it.

And what is the list managing? A limited namespace, currently only 32 bits, 
but with tools like, say, Samba and sssd reserving large chunks for stable UID 
and GID mapping. One could argue that a stable list could obviate the need for 
some of that mapping, but you've got decades-old existing networks that aren't 
going anywhere, and you'll still need to interface with systems run by people 
who will deliberately run counter to such lists as a security layer, just as 
you interface with systems that run SSH or HTTP on nonstandard ports.

You'll still run into all of these issues and more if you try generalize the 
list to region allocation:

Say you try to assign regions for system daemons vs users, and you're on a 
host that interacts with two other hosts without full mutual trust. You're 
serving up a shared filesystem, and all three involved hosts each have a system 
daemon user and a system normal user with an object on that shared filesystem.

Presented with a directory listing showing the UIDs and GIDs for each object, 
how do you distinguish between the system user from each host? The two hosts 
shouldn't have access to each others' files, even if they use the same UID 
locally, because the two hosts don't trust each other.

That considered, how, then, how do you identify between another host's system 
user and its normal user, inasmuch as you can't let them collide with IDs on 
your own system, but are trying to ensure that their IDs get mapped into the 
correct local range?

That considered, what do you do when the Big List Maintainers add another 
region? How do you cope with another host that uses a newer version of that 
list? An older version? And now that you've upgraded, and the new version of 
the list says you should have mapped something somewhere else, what do you do 
with it? Do you even have enough information to know that an ID you mapped 
last year should have been in that other category?

And while we're talking about allocating ranges, we can start talking about 
limited address space. 32 bits seems like a lot of individual identities, but 
when you're carving it up into multiple masses of identities, you'll find it 
gets very small, very quickly. That's why IPv6 went with 128 bits instead of a 
64 bit address space.

All of this is why we use identity management tools like LDAP in the first 
place. Heck, it's why we have passwd and group files for mapping names to ids 
and didn't simply hardcode system IDs decades ago.

-- 
:wq

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[M. J. Everitt]

On 28/11/16 19:39, William L. Thomson Jr. wrote:
> For now who cares about other OS or distros. If Gentoo gets its house in 
> order 
> others may follow.
>
At the risk of a huge flame, remind me, who uses Gentoo again?!



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

On Monday, November 28, 2016 10:42:54 AM EST Alec Warner wrote:
>
> Generally speaking as a fellow who maintained thousands of systems (many of
> which ran various operating systems.)
> 
> You cannot rely on all OS vendors to synchronize uid / gid. You cannot even
> rely on some single vendors to synchronize uid / gids between releases of
> their own products.

I believe the main reason such is the case is a lack of any such list or 
database for others to adhere to. Once again an area Gentoo could be leading. 
Had Gentoo done this years ago others might have adopted.

IMHO it is something that should be  a part of LSB. If not POSIX in general. 
One cannot really change the past or current state of things. But can make the 
future better.

For now who cares about other OS or distros. If Gentoo gets its house in order 
others may follow.

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[Alec Warner]

On Mon, Nov 28, 2016 at 8:21 AM, William L. Thomson Jr. 
wrote:

> On Friday, November 25, 2016 11:39:15 PM EST Daniel Campbell wrote:
> >
> > I could see a use-case for someone wanting to install a given daemon or
> > server with a specific user and/or group. I'm not sure this is the right
> > approach (nor do I know what is), but I think we have room to think
> > about a solution; ideally one that is dead-simple to implement and
> > doesn't have a ton of edge-cases.
> >
> > What is QA's current policy on user/group creation, btw?
>
> Years ago there was talk/discussion of having some list/database of
> UID/GID[1]
> [2], so that we have consistent assignment. Arch seems to be the only
> distro
> thus far who has produced such a list[1], but seems to be outdated and not
> maintained. Also seems to deviate from some UID/GID numbers RedHat uses for
> example[2]. Arch 78 for KVM group, RedHat uses 36.
>
> While there are many reasons people do not care about UID/GID, and
> arguments
> could be made that it might be better to have them vary on systems and be
> unique. Though some things there are already common UID/GID across distros.
>
> I think in the long run, surely for anyone managing lots of systems. It is
> far
> better to have a consistent standard list of UID/GID including names. Maybe
> other distro's will adopt and become more of a standard.
>

Generally speaking as a fellow who maintained thousands of systems (many of
which ran various operating systems.)

You cannot rely on all OS vendors to synchronize uid / gid. You cannot even
rely on some single vendors to synchronize uid / gids between releases of
their own products. If you build your fleet maintenance with this premise
in mind; most folks I've seen come up with a way to manage it.

Often it means things like:

1) Adding shared accounts to a database and using nsswitch to forward
lookups.
2) Adding configuration management rules to add named accounts to every
machine.
3) Building your fleet such as local uid / gid doesn't matter so much
(often this means the demise of shared filesystems or other bolt-on
authentication / authorization mechanisms.

Typically since most folks building a fleet have to synchronize uid / gid
of actual human users anyway (so people can login / access files / etc) and
so the burden just becomes "give me a list of accounts I should add to my
'syncer' so they are auto-populated on all machines'.

The uids and gids don't matter so much (I can assign them myself, often I
need to inter-operate with other systems where names are already in use,
etc.) But just having a list of "these system accounts are important" is
probably useful on its own.

-A


>
> 1. http://marc.info/?l=gentoo-dev&w=2&r=1&s=Assigning+
> unique+system+uid%2Fgid
> +for+new+&q=b
> 2. http://marc.info/?t=11703419445&r=1&w=2
> 3. https://wiki.archlinux.org/index.php?title=DeveloperWiki:
> UID_/_GID_Database
> 4. https://access.redhat.com/documentation/en-US/
> Red_Hat_Enterprise_Virtualization/3.5/html/Installation_Guide/sect-
> System_Accounts.html
>
> --
> William L. Thomson Jr.
>

Re: [gentoo-dev] RFC: Userkit.eclass

[William L. Thomson Jr.]

On Friday, November 25, 2016 11:39:15 PM EST Daniel Campbell wrote:
>
> I could see a use-case for someone wanting to install a given daemon or
> server with a specific user and/or group. I'm not sure this is the right
> approach (nor do I know what is), but I think we have room to think
> about a solution; ideally one that is dead-simple to implement and
> doesn't have a ton of edge-cases.
> 
> What is QA's current policy on user/group creation, btw?

Years ago there was talk/discussion of having some list/database of UID/GID[1]
[2], so that we have consistent assignment. Arch seems to be the only distro 
thus far who has produced such a list[1], but seems to be outdated and not 
maintained. Also seems to deviate from some UID/GID numbers RedHat uses for 
example[2]. Arch 78 for KVM group, RedHat uses 36.

While there are many reasons people do not care about UID/GID, and arguments 
could be made that it might be better to have them vary on systems and be 
unique. Though some things there are already common UID/GID across distros.

I think in the long run, surely for anyone managing lots of systems. It is far 
better to have a consistent standard list of UID/GID including names. Maybe 
other distro's will adopt and become more of a standard.

1. http://marc.info/?l=gentoo-dev&w=2&r=1&s=Assigning+unique+system+uid%2Fgid
+for+new+&q=b
2. http://marc.info/?t=11703419445&r=1&w=2
3. https://wiki.archlinux.org/index.php?title=DeveloperWiki:UID_/_GID_Database
4. https://access.redhat.com/documentation/en-US/
Red_Hat_Enterprise_Virtualization/3.5/html/Installation_Guide/sect-
System_Accounts.html

-- 
William L. Thomson Jr.


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] RFC: Userkit.eclass

[Daniel Campbell]

On 11/23/2016 01:08 AM, Michał Górny wrote:
> On Wed, 23 Nov 2016 09:44:33 +0100
> Manuel Rüger  wrote:
> 
>> I have not started to write it, but I am considering it and rather want
>> to gather feedback on my idea first.
>> I am aware that https://wiki.gentoo.org/wiki/GLEP:27 exists, but as of
>> right now I haven't seen anyone working on it. The goal of this eclass
>> is to improve user/group handling without touching the PMS.
>>
>> tl;dr: Userkit eclass will improve the user handling by externalizing
>> the configuration to variables that can be set from outside of the ebuild.
>>
>> Userkit.eclass will inherit user.eclass and require bash arrays
>> USERKIT_USER and USERKIT_GROUP for configuration.
>> I will export a pkg_setup with the corresponding setup (basically
>> calling enewuser and enewgroup). It will provide get_user, get_uid,
>> get_group, get_gid and get_home functions.
>> This would allow to do something like "fowners $(get_user):$(get_group)
>> foo".
>>
>> If ${CATEGORY}-${PN}_user and ${CATEGORY}-${PN}_group are set, these
>> will replace the contents of USERKIT_USER and USERKIT_GROUP, allowing
>> the user to fully define everything user/group related.
> 
> How does that all map to multiple users/groups? How does that map to
> USE-conditional users/groups? How does that map to users/groups shared
> between multiple packages?
> 
> Besides, this sounds a lot like games.eclass... will developers be
> required to patch upstream software now to force support for using
> custom users/groups?
> 
>> What happens if the ebuild wants to create multiple users/group?
>> Currently, I want to ignore that case and focus on the 80% ebuilds that
>> can profit from such an eclass.
> 
> Do you have specific numbers? I don't see 80% of ebuilds caring about
> users/groups. I don't see the problem you are trying to fix.
> 
> Is it one of those problems that someone thinks it's awesome to make
> everything declaratory, and add tons of middleware to make the
> declaratory work somehow for the most common use cases?
> 
I think the use-case here is ebuilds that need to create a user and/or
group (www-servers/lighttpd is a good example; alongside pretty much
anything else that needs to run as a separate user and serves
something). In lighttpd's case we don't currently support the ability to
declare which user and group lightty uses; the lighttpd user and
lighttpd group will always be created. Later configuration of users and
groups can be worked with, and iirc we recently patched the initscript
so it handles that use case.

I could see a use-case for someone wanting to install a given daemon or
server with a specific user and/or group. I'm not sure this is the right
approach (nor do I know what is), but I think we have room to think
about a solution; ideally one that is dead-simple to implement and
doesn't have a ton of edge-cases.

What is QA's current policy on user/group creation, btw?
-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Userkit.eclass

[Zac Medico]

On 11/23/2016 12:44 AM, Manuel Rüger wrote:
> My only concerns right now are:
> Where to store those ${CATEGORY}-${PN}_user and ${CATEGORY}-${PN}_group?
> One solution could be to have another eclass named userkit-data.eclass,
> which is empty by default and needs to be forked to an overlay and then
> use the eclass-override setting in repos.conf. Unfortunately this will
> cause a lot of md5-cache rewrites.
> Another solution would be portage/package.env or portage/env.

Just allow for people do define bashrc functions that override the
eclass behavior, and they'll have the flexibility to implement anything
they want, without having to override eclasses.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFC: Userkit.eclass

[Zac Medico]

On 11/23/2016 09:46 AM, Kent Fredric wrote:
> On Wed, 23 Nov 2016 09:44:33 +0100
> Manuel Rüger  wrote:
> 
>> What happens if the ebuild wants to create multiple users/group?
>> Currently, I want to ignore that case and focus on the 80% ebuilds that
>> can profit from such an eclass.
> 
> You can solve that part quite easily really.
> 
> Just deem USERKIT_USER and USERKIT_GROUP to be basenames
> for idenitifiers.
> 
> 
> Then you'd have
> 
>get_user == returns "${USERKIT_USER}"
> 
>get_user "admin" == returns "${USERKIT_USER}-admin"
> 
> Its not a perfect solution, but its better than "we just forget about this"

That's not flexible enough, because we don't have control over the
user/group naming scheme used by upstream. So, we have to support an
unbounded number of arbitrarily named users and groups.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFC: Userkit.eclass

[Kent Fredric]

On Wed, 23 Nov 2016 09:44:33 +0100
Manuel Rüger  wrote:

> What happens if the ebuild wants to create multiple users/group?
> Currently, I want to ignore that case and focus on the 80% ebuilds that
> can profit from such an eclass.

You can solve that part quite easily really.

Just deem USERKIT_USER and USERKIT_GROUP to be basenames
for idenitifiers.


Then you'd have

   get_user == returns "${USERKIT_USER}"

   get_user "admin" == returns "${USERKIT_USER}-admin"

Its not a perfect solution, but its better than "we just forget about this"


pgpvhpKfZVOcb.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Userkit.eclass

[Michał Górny]

On Wed, 23 Nov 2016 10:19:42 +0100
Manuel Rüger  wrote:

> On 23.11.2016 10:08, Michał Górny wrote:
> > On Wed, 23 Nov 2016 09:44:33 +0100
> > Manuel Rüger  wrote:
> >   
> >> I have not started to write it, but I am considering it and rather want
> >> to gather feedback on my idea first.
> >> I am aware that https://wiki.gentoo.org/wiki/GLEP:27 exists, but as of
> >> right now I haven't seen anyone working on it. The goal of this eclass
> >> is to improve user/group handling without touching the PMS.
> >>
> >> tl;dr: Userkit eclass will improve the user handling by externalizing
> >> the configuration to variables that can be set from outside of the ebuild.
> >>
> >> Userkit.eclass will inherit user.eclass and require bash arrays
> >> USERKIT_USER and USERKIT_GROUP for configuration.
> >> I will export a pkg_setup with the corresponding setup (basically
> >> calling enewuser and enewgroup). It will provide get_user, get_uid,
> >> get_group, get_gid and get_home functions.
> >> This would allow to do something like "fowners $(get_user):$(get_group)
> >> foo".
> >>
> >> If ${CATEGORY}-${PN}_user and ${CATEGORY}-${PN}_group are set, these
> >> will replace the contents of USERKIT_USER and USERKIT_GROUP, allowing
> >> the user to fully define everything user/group related.  
> > 
> > How does that all map to multiple users/groups? How does that map to
> > USE-conditional users/groups? How does that map to users/groups shared
> > between multiple packages?
> >   
> simply via calling the function conditional in pkg_setup
> My goal is not to focus on handling multiple users/groups. Synchronizing
> settings between multiple packages is a task of the user, it doesn't
> make any sense to make guesses there. People will come up with wonderful
> symlinked solutions.
> > Besides, this sounds a lot like games.eclass... will developers be
> > required to patch upstream software now to force support for using
> > custom users/groups?  
> I am not aware of any patches that are required. What I care about is
> having predictable uid/gid and home for everything I can configure via
> an ebuild.

Wait a minute. So you're talking about configuring UID/GID (as
in numbers) and not changing usernames? That's the problem when you
jump straight to solutions and don't state the problem -- nobody knows
what you're really up to.

-- 
Best regards,
Michał Górny



pgpuf9E7qGzk5.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Userkit.eclass

[Manuel Rüger]

On 23.11.2016 10:08, Michał Górny wrote:
> On Wed, 23 Nov 2016 09:44:33 +0100
> Manuel Rüger  wrote:
> 
>> I have not started to write it, but I am considering it and rather want
>> to gather feedback on my idea first.
>> I am aware that https://wiki.gentoo.org/wiki/GLEP:27 exists, but as of
>> right now I haven't seen anyone working on it. The goal of this eclass
>> is to improve user/group handling without touching the PMS.
>>
>> tl;dr: Userkit eclass will improve the user handling by externalizing
>> the configuration to variables that can be set from outside of the ebuild.
>>
>> Userkit.eclass will inherit user.eclass and require bash arrays
>> USERKIT_USER and USERKIT_GROUP for configuration.
>> I will export a pkg_setup with the corresponding setup (basically
>> calling enewuser and enewgroup). It will provide get_user, get_uid,
>> get_group, get_gid and get_home functions.
>> This would allow to do something like "fowners $(get_user):$(get_group)
>> foo".
>>
>> If ${CATEGORY}-${PN}_user and ${CATEGORY}-${PN}_group are set, these
>> will replace the contents of USERKIT_USER and USERKIT_GROUP, allowing
>> the user to fully define everything user/group related.
> 
> How does that all map to multiple users/groups? How does that map to
> USE-conditional users/groups? How does that map to users/groups shared
> between multiple packages?
> 
simply via calling the function conditional in pkg_setup
My goal is not to focus on handling multiple users/groups. Synchronizing
settings between multiple packages is a task of the user, it doesn't
make any sense to make guesses there. People will come up with wonderful
symlinked solutions.
> Besides, this sounds a lot like games.eclass... will developers be
> required to patch upstream software now to force support for using
> custom users/groups?
I am not aware of any patches that are required. What I care about is
having predictable uid/gid and home for everything I can configure via
an ebuild.

> 
>> What happens if the ebuild wants to create multiple users/group?
>> Currently, I want to ignore that case and focus on the 80% ebuilds that
>> can profit from such an eclass.
> 
> Do you have specific numbers? I don't see 80% of ebuilds caring about
> users/groups. I don't see the problem you are trying to fix.
> 
Okay let me rephrase that here: "probably more than 80% of the ebuilds
that are calling enewuser/enewgroup" install a single user or a single
group. There will be some cases this eclass is not applicable to, but
that is fine. If this is something we really want to coveras well using
the eclass based approach, we probably could start enumerating the
variable and if those available to what needs to be done. Something like
USERKIT_USER_2. Not sure if we want to do that.

> Is it one of those problems that someone thinks it's awesome to make
> everything declaratory, and add tons of middleware to make the
> declaratory work somehow for the most common use cases?
> 
I don't see "tons of middleware" here.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] RFC: Userkit.eclass

[Michał Górny]

On Wed, 23 Nov 2016 09:44:33 +0100
Manuel Rüger  wrote:

> I have not started to write it, but I am considering it and rather want
> to gather feedback on my idea first.
> I am aware that https://wiki.gentoo.org/wiki/GLEP:27 exists, but as of
> right now I haven't seen anyone working on it. The goal of this eclass
> is to improve user/group handling without touching the PMS.
> 
> tl;dr: Userkit eclass will improve the user handling by externalizing
> the configuration to variables that can be set from outside of the ebuild.
> 
> Userkit.eclass will inherit user.eclass and require bash arrays
> USERKIT_USER and USERKIT_GROUP for configuration.
> I will export a pkg_setup with the corresponding setup (basically
> calling enewuser and enewgroup). It will provide get_user, get_uid,
> get_group, get_gid and get_home functions.
> This would allow to do something like "fowners $(get_user):$(get_group)
> foo".
> 
> If ${CATEGORY}-${PN}_user and ${CATEGORY}-${PN}_group are set, these
> will replace the contents of USERKIT_USER and USERKIT_GROUP, allowing
> the user to fully define everything user/group related.

How does that all map to multiple users/groups? How does that map to
USE-conditional users/groups? How does that map to users/groups shared
between multiple packages?

Besides, this sounds a lot like games.eclass... will developers be
required to patch upstream software now to force support for using
custom users/groups?

> What happens if the ebuild wants to create multiple users/group?
> Currently, I want to ignore that case and focus on the 80% ebuilds that
> can profit from such an eclass.

Do you have specific numbers? I don't see 80% of ebuilds caring about
users/groups. I don't see the problem you are trying to fix.

Is it one of those problems that someone thinks it's awesome to make
everything declaratory, and add tons of middleware to make the
declaratory work somehow for the most common use cases?

-- 
Best regards,
Michał Górny



pgpL0n9VY4BGr.pgp
Description: OpenPGP digital signature

[gentoo-dev] RFC: Userkit.eclass

[Manuel Rüger]

Hi everyone,

I have not started to write it, but I am considering it and rather want
to gather feedback on my idea first.
I am aware that https://wiki.gentoo.org/wiki/GLEP:27 exists, but as of
right now I haven't seen anyone working on it. The goal of this eclass
is to improve user/group handling without touching the PMS.

tl;dr: Userkit eclass will improve the user handling by externalizing
the configuration to variables that can be set from outside of the ebuild.

Userkit.eclass will inherit user.eclass and require bash arrays
USERKIT_USER and USERKIT_GROUP for configuration.
I will export a pkg_setup with the corresponding setup (basically
calling enewuser and enewgroup). It will provide get_user, get_uid,
get_group, get_gid and get_home functions.
This would allow to do something like "fowners $(get_user):$(get_group)
foo".

If ${CATEGORY}-${PN}_user and ${CATEGORY}-${PN}_group are set, these
will replace the contents of USERKIT_USER and USERKIT_GROUP, allowing
the user to fully define everything user/group related.

It will also be possible to enable a switch to that makes the ebuild
fail if those are not set, as you then can set those variables first.
Another one allows to make them nops (which is nice for testing the
ebuild via "ebuild $PN test").

My only concerns right now are:
Where to store those ${CATEGORY}-${PN}_user and ${CATEGORY}-${PN}_group?
One solution could be to have another eclass named userkit-data.eclass,
which is empty by default and needs to be forked to an overlay and then
use the eclass-override setting in repos.conf. Unfortunately this will
cause a lot of md5-cache rewrites.
Another solution would be portage/package.env or portage/env.

What happens if the ebuild wants to create multiple users/group?
Currently, I want to ignore that case and focus on the 80% ebuilds that
can profit from such an eclass.

Any thoughts?

Cheers,

Manuel



signature.asc
Description: OpenPGP digital signature