Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On 8/16/2012 6:26 PM, Rich Freeman wrote:
On Thu, Aug 16, 2012 at 4:05 PM, Michael Mol mike...@gmail.com wrote:
The limited-visibility build feature discussed a week or so ago would
go a long way in detecting unexpressed build dependencies.
[snip]
If portage has the
dependency tree in RAM then you just need to dump all the edb listings
for those packages plus @system and feed those into sandbox.
That just requires reading a bunch of text files and no searching, so it
should be pretty quick.
Portage could hypothetically compile such a list while it crawls the
package dependency tree, but I suspect the cost will not be small as you
predict.
As far as I can tell the relevant calls to
check for read access are already being made in sandbox already, and
obviously they aren't taking forever. We just have to see if the
search gets slow if the access list has tens of thousands of entries
(if it does, that is just a simple matter of optimization, but being
in-RAM I can't see how tens of thousands of entries is going to slow
down a modern CPU even if it is just an unsorted list).
I appreciate your optimism but I think you're underestimating the cost.
Can't speak for others, but my portage db's churn too much for comfort
as is. Once we start multiplying per-package-dependency iteration by
the files-per-package iteration, that's going to be O(a-shit-load).
Of course, where there's a will there's a way. I'd be surprised if some
kind of delayed-evaluation + caching scheme wouldn't suffice, or,
barring that, perhaps it's time to create an indexed-database-based
drop-in replacement for the current portage db code.
I've enclosed some scripts you may find helpful in looking at the
numbers. They are kind-of kludgey (originally intended for
in-house-only use and modified for present purposes) but may help shed
some light, if they aren't too buggy, that is...
dumpworld slices and dices emerge -ep output to provide a list of
atoms in the complete dependency tree of a given list of atoms (add
'@system' to get the complete tree, dumpworld won't do so).
dumpfiles operates only on packages installed in the local system
(non-installed atoms are silently dropped), and requires/assumes that
'emerge -ep world' would not change anything if it is to give accurate
information. It takes a list of atoms, transforms them into the
complete lists of atoms in their dependency tree via dumpworld, merges
the lists together, and finds the number of files associated with each
atom in portage. Any collisions will be counted twice, since it doesn't
keep track. It also doesn't add '@system' unless you do. By default it
emits:
o A list of package atoms and the files owned by each atom (stderr)
o total atoms and files
o average filename length
What is, perhaps, more discouraging than the numbers it reports is how
long it takes to run (note: although I suspect an optimized python
implementation could be made to do this faster by a moderate constant
factor, I'm not sure if the big-oh performance characteristics can be
significantly improved without database structure changes like the ones
mentioned above).
My disturbingly bloated and slow workstation gives these answers (note:
here it's even slower because it's running in an emulator):
greg@fedora64vmw ~ $ time bash -c 'dumpfiles @system 2/dev/null'
TOTAL: 402967 files (in 816 ebuilds, average path length: 66)
real15m33.719s
user13m18.909s
sys 2m8.436s
greg@fedora64vmw ~ $ time bash -c 'dumpfiles chromium 2/dev/null'
TOTAL: 401300 files (in 807 ebuilds, average path length: 66)
real15m28.900s
user13m15.126s
sys 2m8.088s
My workstation is surely an outlier as I have a lot of dependencies
and files due to multilib, split-debug, and USE+=$( a lot ). It's also
got slow hardware Raid6 and the emulator only gives it 2G of ram to work
with. But I'm a real portage user; I'm sure there's other ones out
there, if not many, with similar constraints.
-gmt
#!/bin/bash
if [[ x$(qlist -IC app-portage/portage-utils)x == xx || \
x$(qlist -IC app-portage/gentoolkit)x == xx ]] ; then
echo This utility requires both app-portage/portage-utils 2
echo and app-portage/gentoolkit. Emerge them both and try again. 2
exit 1
fi
declare -a arguments atoms
arguments=( )
atoms=( )
verbose=yes
redic=no
for arg in $@ ; do
case $arg in
-q|--quiet) verbose=no ;;
-r|--redic) redic=yes ;;
*) arguments=( ${arguments[@]} $arg ) ;;
esac
done
[[ ${#arguments[*]} == 0 ]] arguments=( '@world' )
for arg in ${arguments[@]} ; do
if [[ ${arg} == @* ]] ; then
newatoms=( ${arg} )
else
newatoms=( $( qlist -eICv ${arg} | sed 's/^/=/' ) )
fi
newatoms=( $( dumpworld ${newatoms[@]} ) )
result=$?
[[ ${result} != 0 ]] { echo dumpworld failed, giving up. 2 ;
exit
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Wed, Aug 15, 2012 at 3:18 PM, Michael Mol mike...@gmail.com wrote: It also sounds like something like that could be a benefit to shrinking @system. I think the solution to the circular dependency issue isn't to make Portage able to completely bootstrap the whole system, but rather just to make it capable of coping with the issues and knowing when to raise an alarm. Gentoo will always involve extracting a tarball/etc for the initial installation since you always need SOMETHING to start with. You can't even chroot into your install directory without a shell being there, and typing emerge won't go so well if portage isn't actually installed. So, continue to build stages like we do right now - no doubt with hard-coding and such to get around the dependencies. As far as objections to listing gcc and such in every ebuild go, why not? We list all kinds of routine stuff in hundreds of ebuilds so that we can install systems without them. Why not just have a toolchain virtual or something? And since ssh was brought up - this is what happens with hacks like this. When you combine the default install with the minimum deps for everything list you end up with an ssh you can't get rid of without the package.provided hack (which really should be used for stuff that is, well, provided). It would be nice if people who want to build a server with Gentoo but then reduce it to only true RDEPENDS could do so. Obviously they'd have to use binary packages to continue to maintain it (and even then they'd need to keep portage on it), or they'd have to build another one. Actually, the trend in general is towards disposable servers anyway so generating an entire new server every time one thing changes is probably a desirable thing, since you probably want to be able to do it every time you add a server anyway. Rich
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On 08/16/2012 04:59 AM, Rich Freeman wrote: And since ssh was brought up - this is what happens with hacks like this. When you combine the default install with the minimum deps for everything list you end up with an ssh you can't get rid of without the package.provided hack (which really should be used for stuff that is, well, provided). FWIW, instead of using package.provided, you can negate *virtual/ssh like this: mkdir -p /etc/portage/profile echo -*virtual/ssh /etc/portage/profile/packages -- Thanks, Zac
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On 8/16/2012 4:59 AM, Rich Freeman wrote: On Wed, Aug 15, 2012 at 3:18 PM, Michael Mol mike...@gmail.com wrote: It also sounds like something like that could be a benefit to shrinking @system. I think the solution to the circular dependency issue isn't to make Portage able to completely bootstrap the whole system, but rather just to make it capable of coping with the issues and knowing when to raise an alarm. Gentoo will always involve extracting a tarball/etc for the initial installation since you always need SOMETHING to start with. You can't even chroot into your install directory without a shell being there, and typing emerge won't go so well if portage isn't actually installed. So, continue to build stages like we do right now - no doubt with hard-coding and such to get around the dependencies. As far as objections to listing gcc and such in every ebuild go, why not? We list all kinds of routine stuff in hundreds of ebuilds so that we can install systems without them. Why not just have a toolchain virtual or something? And since ssh was brought up - this is what happens with hacks like this. When you combine the default install with the minimum deps for everything list you end up with an ssh you can't get rid of without the package.provided hack (which really should be used for stuff that is, well, provided). It would be nice if people who want to build a server with Gentoo but then reduce it to only true RDEPENDS could do so. Obviously they'd have to use binary packages to continue to maintain it (and even then they'd need to keep portage on it), or they'd have to build another one. Actually, the trend in general is towards disposable servers anyway so generating an entire new server every time one thing changes is probably a desirable thing, since you probably want to be able to do it every time you add a server anyway. tldr: I like, approve and otherwise +1 the idea of somehow paring down or eliminating @system but I think it's going to be fairly challenging, so more discussion on this topic is warranted in my humble non-developer opinion :) -- I really like everything you have to say here. Unfortunately, assumptions of toolchain availability have gotten into the DNA of Gentoo in ways that make it nontrivial -- although probably not rocket science, either -- to implement these ideas. I'd say it's the kind of thing where somebody needs to do the work. I think there is demand for this, but when it comes down to brass tacks, people who really need features like this can just write a script to push some tarballs or files around in a way that's good enough for their purposes. What is the cost/bene for a single sys-admin to do all the work and politics of making this change? However, staying with the cost/bene theme, we have here a kind of externality, as they say in economics, (which is a fancy way, I guess, of saying a bad decision or a raw deal), because, in the aggregate, I think it's pretty clear that the cost/bene favors doing that work. To be clear, I don't have religion about getting rid of @system, per-se, but I do have religion about the stuff Larry the Cow told me when I first visited the Gentoo homepage in 2001, or whenever, which was, basically, that the software I was using had a bunch of frobs that I couldn't touch, because I was running an rpm- or .deb-based system, and that Gentoo was going to let me frob them. It's not a total disaster, even now -- a determined sysadmin can absolutely do this right now with features like prefix, ROOT, binpkg and so forth but /really/ fixing this, so that non-standard/minimal setups just work, would allow Gentoo to effectively address a whole bunch of really practical, real-world use-cases -- use-cases Gentoo is in many aspects uniquely suited to address, due to Larry the Cow's brilliant insights -- yet, by-and-large, due to precisely this @system thing and the package-management decisions that have stemmed from it, for which Gentoo has become unsuitable or impractical. Specifically, I'm talking, here, about managed LAMP servers, big-data clusters, and embedded. I suppose I'm not doing much to fix it by ranting and raving like this however. So see first paragraph :) -gmt
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Thu, Aug 16, 2012 at 3:51 PM, Gregory M. Turner g...@malth.us wrote: On 8/16/2012 4:59 AM, Rich Freeman wrote: [snip] tldr: I like, approve and otherwise +1 the idea of somehow paring down or eliminating @system but I think it's going to be fairly challenging, so more discussion on this topic is warranted in my humble non-developer opinion :) -- I really like everything you have to say here. Unfortunately, assumptions of toolchain availability have gotten into the DNA of Gentoo in ways that make it nontrivial -- although probably not rocket science, either -- to implement these ideas. The limited-visibility build feature discussed a week or so ago would go a long way in detecting unexpressed build dependencies. -- :wq
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Thu, Aug 16, 2012 at 4:05 PM, Michael Mol mike...@gmail.com wrote: The limited-visibility build feature discussed a week or so ago would go a long way in detecting unexpressed build dependencies. I can't say that is a coincidence, but my intent would be to include @system as implicit dependencies, at least until we change that policy (though the morbidly curious could use that as a test in a tinderbox to find packages in @system that are good candidates for removal). I haven't gotten to test it, but after studying sandbox it shouldn't be hard to just hack together a manual test by removing read access to root from the config files and adding in a bazillion files. That should at least let me profile performance/etc. I'm not convinced that there isn't room for improvement, but if it works well as-is then automating this shouldn't be hard at all. If portage has the dependency tree in RAM then you just need to dump all the edb listings for those packages plus @system and feed those into sandbox. That just requires reading a bunch of text files and no searching, so it should be pretty quick. As far as I can tell the relevant calls to check for read access are already being made in sandbox already, and obviously they aren't taking forever. We just have to see if the search gets slow if the access list has tens of thousands of entries (if it does, that is just a simple matter of optimization, but being in-RAM I can't see how tens of thousands of entries is going to slow down a modern CPU even if it is just an unsorted list). Rich
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Thu, Aug 16, 2012 at 9:26 PM, Rich Freeman ri...@gentoo.org wrote:
On Thu, Aug 16, 2012 at 4:05 PM, Michael Mol mike...@gmail.com wrote:
The limited-visibility build feature discussed a week or so ago would
go a long way in detecting unexpressed build dependencies.
I can't say that is a coincidence, but my intent would be to include
@system as implicit dependencies, at least until we change that policy
(though the morbidly curious could use that as a test in a tinderbox
to find packages in @system that are good candidates for removal).
I haven't gotten to test it, but after studying sandbox it shouldn't
be hard to just hack together a manual test by removing read access to
root from the config files and adding in a bazillion files. That
should at least let me profile performance/etc. I'm not convinced
that there isn't room for improvement, but if it works well as-is then
automating this shouldn't be hard at all. If portage has the
dependency tree in RAM then you just need to dump all the edb listings
for those packages plus @system and feed those into sandbox. That
just requires reading a bunch of text files and no searching, so it
should be pretty quick. As far as I can tell the relevant calls to
check for read access are already being made in sandbox already, and
obviously they aren't taking forever. We just have to see if the
search gets slow if the access list has tens of thousands of entries
(if it does, that is just a simple matter of optimization, but being
in-RAM I can't see how tens of thousands of entries is going to slow
down a modern CPU even if it is just an unsorted list).
Yeah, I presumed you'd have @system as a set of implicit dependencies.
The obvious approaches would be to either temporarily remove a package
from @system, tell the portage to ignore a package while doing limited
visibility, or copy @system to a different, temporary set and remove
things piecemeal from there.
That last might make the most sense. --implicit-dependencies ---
defaults to @system. Additional instances append to the set of
implicit dependencies. Use, e.g. -${ATOM} or -@system to override
default include.
--
:wq
[gentoo-dev] Re: Questions about SystemD and OpenRC
Rich Freeman posted on Wed, 15 Aug 2012 06:27:41 -0400 as excerpted: Right now having decent KDE and Gnome support with all the bells and whistles[...] isn't that hard, [It] will likely get harder, which means in practice what we'll probably have is a reasonable compromise which will never be quite as polished in any one direction as it could be, unless the end user does the polishing. Well stated. RE you concerns about OpenRC being in @system. Personally I'm a fan of getting rid of @system entirely except as something used to build install CDs or having some sets for convenience in building systems. It only exists for a few reasons that I can think of: 1. Devs don't want to have ebuilds that capture dependencies on every little thing. A few well-chosen virtuals like shell utilities or whatever might help with this. 2. Things like Prefix rely on the system not installing local copies of libraries in the core system it needs to link to. Careful use of package.provided in profiles might address this. 3. We'd need many more virtuals to handle situations like FreeBSD where people don't what GNU on their systems. Right now if they are system packages they just define system appropriately and ebuilds don't directly pull in the GNU stuff anyway. AFAIK, @system also helps resolve a few nasty circular dependencies. In fact, I believe that's it's primary purpose. As such I'm not sure it's practical (as opposed to possible, cost/benefit simply makes it impractical) to entirely get rid of, but it does occur to me that sets would be an interesting way to go. Define a few sets that together compose @system as we have it today, and basically package.provide them during the bootstrap phase. AFAIK the original stage tarball also contains a minimal installed tree, for similar reasons. I'm not actually sure how they interact. That'd be releng/arch/catalyst territory. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Wed, Aug 15, 2012 at 2:21 PM, Duncan 1i5t5.dun...@cox.net wrote: Rich Freeman posted on Wed, 15 Aug 2012 06:27:41 -0400 as excerpted: Right now having decent KDE and Gnome support with all the bells and whistles[...] isn't that hard, [It] will likely get harder, which means in practice what we'll probably have is a reasonable compromise which will never be quite as polished in any one direction as it could be, unless the end user does the polishing. Well stated. RE you concerns about OpenRC being in @system. Personally I'm a fan of getting rid of @system entirely except as something used to build install CDs or having some sets for convenience in building systems. It only exists for a few reasons that I can think of: 1. Devs don't want to have ebuilds that capture dependencies on every little thing. A few well-chosen virtuals like shell utilities or whatever might help with this. 2. Things like Prefix rely on the system not installing local copies of libraries in the core system it needs to link to. Careful use of package.provided in profiles might address this. 3. We'd need many more virtuals to handle situations like FreeBSD where people don't what GNU on their systems. Right now if they are system packages they just define system appropriately and ebuilds don't directly pull in the GNU stuff anyway. AFAIK, @system also helps resolve a few nasty circular dependencies. In fact, I believe that's it's primary purpose. As such I'm not sure it's practical (as opposed to possible, cost/benefit simply makes it impractical) to entirely get rid of, but it does occur to me that sets would be an interesting way to go. Define a few sets that together compose @system as we have it today, and basically package.provide them during the bootstrap phase. AFAIK the original stage tarball also contains a minimal installed tree, for similar reasons. I'm not actually sure how they interact. That'd be releng/arch/catalyst territory. Just piping up here, as this relates to an idea that's been percolating through my mind for a couple weeks. I've occasionally noticed portage tell me about circular dependencies, where the most straight forward resolution is to emerge some package in the loop twice. The first time, with a USE flag disabled (avahi and gtk are the usual suspects), and the second time with the USE flag enabled. In circumstances where it's necessary to do something like that to reach a final desired system state, I'm not sure I see any problem with portage automatically doing the two-stage emerge. It also sounds like something like that could be a benefit to shrinking @system. As far as things such as libc, where many, many packages require their use, I don't personally see a problem with recommending that ebuilds depend on them. My only other notable experience for Linux distributions is Debian/Ubuntu, and a quick glance shows 16,389 packages expressing explicit dependencies on libc6 in Ubuntu 12.04. -- :wq
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Wed, 15 Aug 2012 15:18:24 -0400 Michael Mol mike...@gmail.com wrote: I've occasionally noticed portage tell me about circular dependencies, where the most straight forward resolution is to emerge some package in the loop twice. The first time, with a USE flag disabled (avahi and gtk are the usual suspects), and the second time with the USE flag enabled. In circumstances where it's necessary to do something like that to reach a final desired system state, I'm not sure I see any problem with portage automatically doing the two-stage emerge. That's going to be rather horrible when your package mangler temporarily turns off acl or turns on build... -- Ciaran McCreesh signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Wed, Aug 15, 2012 at 3:26 PM, Ciaran McCreesh ciaran.mccre...@googlemail.com wrote: On Wed, 15 Aug 2012 15:18:24 -0400 Michael Mol mike...@gmail.com wrote: I've occasionally noticed portage tell me about circular dependencies, where the most straight forward resolution is to emerge some package in the loop twice. The first time, with a USE flag disabled (avahi and gtk are the usual suspects), and the second time with the USE flag enabled. In circumstances where it's necessary to do something like that to reach a final desired system state, I'm not sure I see any problem with portage automatically doing the two-stage emerge. That's going to be rather horrible when your package mangler temporarily turns off acl or turns on build... Fair observation; there would need to be a way either weight benign or dangerous USE/package flags, and search paths with weights outside of a 'safe' range would be discarded. A mechanism like that also offers a means of finding and favoring cheap rebuilds over expensive ones; rebuilding gcc an extra time ought to be disfavored over rebuilding, say, sudo. -- :wq
[gentoo-dev] Re: Questions about SystemD and OpenRC
Peter Stuge posted on Sun, 12 Aug 2012 02:12:38 +0200 as excerpted:
viv...@gmail.com wrote:
First problem udev/SD has is that it can't see all the file system
labels, for some reason it only see sda and sdb so it's able to partly
proceed in the boot sequence, mount / (root) but can't mount anything
else.
What software parses the filesystem labels when you boot with openrc?
(I ask because I never use labels myself.)
Short answer, mount and udev, and the kernel directly when fed that
information for root= on the kernel commandline. Openrc has basically
nothing to do with it. As such, I don't know what systemd's doing, but
if it indeed is a systemd bug, it's obviously doing /something/ rather
different... probably interacting in some unpredicted way with udev now
that they're integrating it.
Slightly more detail, quoting the mount (8) manpage:
It is possible to indicate a block special device using its volume LABEL
or UUID (see the -L and -U options below).
The recommended setup is to use LABEL=label or UUID=uuid tags rather
than /dev/disk/by-{label,uuid} udev symlinks in the /etc/fstab file.
The tags are more readable, robust and portable. The mount(8) command
internally uses udev symlinks, so use the symlinks in /etc/fstab has no
advantage over LABEL=/UUID=. For more details see libblkid(3).
Note that mount(8) uses UUIDs as strings. The UUIDs from command line or
fstab(5) are not converted to internal binary representation. The string
representation of the UUID should be based on lower case characters.
The implication of the as strings, not converted to binary in the last
paragraph is that mount's simply doing a dumb string match against the
appropriate /dev/disk/by-* symlink names and dereferencing the symlink
after which it can proceed as usual if there's a match. It's udev (and
thus now systemd, since they're combining) that's actually doing the
work, exposing the /dev/disk/by-* symlinks with the correct references.
As such, mount basically supports anything found in that location,
including device hardware ID (combined with partition number where
appropriate, listed in /dev/disk/by-id), the already mentioned filesystem
LABEL (/dev/disk/by-label), PARTLABEL (GPT partitions have a partition
label that's similar in function to the filesystem label, but set at gdisk
time, not mkfs time, obviously this doesn't apply to legacy MBR
partitioning), PARTUID, PATH (bus path, tho like traditional /dev/[hs]dX,
this can be dynamic and the bus paths are longer and more obtuse for
humans so there's generally little practical use for this one), and (as
the manpage mentions, partition) UUID.
It is however worth noting that PARTLABEL at least is new enough that at
least as of a couple months ago when I looked at it for my md/raid setup,
while udev passed thru the PARTLABEL for physical devices, it was NOT
doing so for partitioned md/raid devices. (That mobo died and I don't
have md/raid on the new system so I can't verify current kernel 3.6-rc1+
and udev-187-r3 PARTLABEL-on-md/raid behavior.) So while people could
setup their md/raid gpt partitions with partition-labels using gptfdisk
(aka gdisk) or whatever, and I did so, only the ones on the physical
disks showed up in udev, and at the time I was looking at it, I was
running md/raid on virtually everything, so the feature was about useless
for me.
But that does mean that if mdev or whatever creates similar /dev/disk/by-
*, that mount and friends should use it just as they would with udev,
since all they're doing is a dumb string match and symlink deref.
It'a also worth noting that the kernel's root= line can take a number of
these (I'm not sure if it can take them all), including partlabel,
according to information someone posted on the btrfs list (which I was
following at the time, before I decided btrfs was too unstable still and
that I'd wait until next year to try again). I was going to try it, but
lost interest when I found I couldn't use it with fstab for my mdraid-
based gpt partitions anyway. So all I can say is that it was reported to
work on the btrfs list, and no one contradicted it, so...
a) SD does not re-calculate it's deptree/services when exiting from
rescue shell, it still try to start the virtual services as fstab
would have never modified, a reboot is needed
systemctl --system daemon-reload ?
I've not tried systemd yet nor have I yet researched it (I have the
research vaguely scheduled for next year, sometime), but thanks to both
of you for these details. Based on past experience, I expect I'm
collecting the pieces even tho I don't have context to assemble them in,
but as a result, when I /do/ research it and get that context, the pieces
I already have thanks to discussions like this, will fall into place, and
it'll be far easier to get upto speed than it would have been had I tried
going into it cold turkey.
So while I could guess, I'll just shutup and listen
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Sun, Aug 12, 2012 at 5:43 AM, Duncan 1i5t5.dun...@cox.net wrote: Peter Stuge posted on Sun, 12 Aug 2012 02:12:38 +0200 as excerpted: What software parses the filesystem labels when you boot with openrc? (I ask because I never use labels myself.) Short answer, mount and udev, and the kernel directly when fed that information for root= on the kernel commandline. ... It'a also worth noting that the kernel's root= line can take a number of these (I'm not sure if it can take them all), including partlabel, according to information someone posted on the btrfs list I'm pretty sure that this particular part of your post is incorrect, but I'd be happy to be corrected. The kernel cannot interpret a root= parameter that doesn't reference a device node that it can't internally generate (with a devtmpfs/etc). I'm pretty sure that rules out just about anything but a physical device (not even /dev/by-uuid/...). I think that people get this confused because 99% of linux users have an initramfs (and about 2% of Gentoo users it seems), and most initramfs implementations DO interpret the root=parameter. If you specify an initramfs then the kernel actually ignores the root=parameter entirely, mounting the initramfs as root, and passing control to its init. The initramfs is expected to mount root (or not - you could just run the whole system off an initramfs I guess). Most initramfs implementations just parse the root= line on the kernel, although it is worth noting that genkernel's initramfs does not and uses real_root instead. So, I could see how many linux users might think that interpreting a complex root= parameter is a kernel function, when it is really just the fact that they use an initramfs. If somebody is running with root=LABEL=foo or something like that without an initramfs I'll happily stand corrected. Rich
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
Il 12/08/2012 11:43, Duncan ha scritto:
Peter Stuge posted on Sun, 12 Aug 2012 02:12:38 +0200 as excerpted:
viv...@gmail.com wrote:
First problem udev/SD has is that it can't see all the file system
labels, for some reason it only see sda and sdb so it's able to partly
proceed in the boot sequence, mount / (root) but can't mount anything
else.
What software parses the filesystem labels when you boot with openrc?
(I ask because I never use labels myself.)
Short answer, mount and udev, and the kernel directly when fed that
information for root= on the kernel commandline. Openrc has basically
nothing to do with it. As such, I don't know what systemd's doing, but
if it indeed is a systemd bug, it's obviously doing /something/ rather
different... probably interacting in some unpredicted way with udev now
that they're integrating it.
exactly my bet, since openrc with the _same_ (not recompiled) udev does work
Slightly more detail, quoting the mount (8) manpage:
It is possible to indicate a block special device using its volume LABEL
or UUID (see the -L and -U options below).
The recommended setup is to use LABEL=label or UUID=uuid tags rather
than /dev/disk/by-{label,uuid} udev symlinks in the /etc/fstab file.
The tags are more readable, robust and portable. The mount(8) command
internally uses udev symlinks, so use the symlinks in /etc/fstab has no
advantage over LABEL=/UUID=. For more details see libblkid(3).
Note that mount(8) uses UUIDs as strings. The UUIDs from command line or
fstab(5) are not converted to internal binary representation. The string
representation of the UUID should be based on lower case characters.
The implication of the as strings, not converted to binary in the last
paragraph is that mount's simply doing a dumb string match against the
appropriate /dev/disk/by-* symlink names and dereferencing the symlink
after which it can proceed as usual if there's a match. It's udev (and
thus now systemd, since they're combining) that's actually doing the
work, exposing the /dev/disk/by-* symlinks with the correct references.
As such, mount basically supports anything found in that location,
including device hardware ID (combined with partition number where
appropriate, listed in /dev/disk/by-id), the already mentioned filesystem
LABEL (/dev/disk/by-label), PARTLABEL (GPT partitions have a partition
label that's similar in function to the filesystem label, but set at gdisk
time, not mkfs time, obviously this doesn't apply to legacy MBR
partitioning), PARTUID, PATH (bus path, tho like traditional /dev/[hs]dX,
this can be dynamic and the bus paths are longer and more obtuse for
humans so there's generally little practical use for this one), and (as
the manpage mentions, partition) UUID.
It is however worth noting that PARTLABEL at least is new enough that at
least as of a couple months ago when I looked at it for my md/raid setup,
while udev passed thru the PARTLABEL for physical devices, it was NOT
doing so for partitioned md/raid devices. (That mobo died and I don't
have md/raid on the new system so I can't verify current kernel 3.6-rc1+
and udev-187-r3 PARTLABEL-on-md/raid behavior.) So while people could
setup their md/raid gpt partitions with partition-labels using gptfdisk
(aka gdisk) or whatever, and I did so, only the ones on the physical
disks showed up in udev, and at the time I was looking at it, I was
running md/raid on virtually everything, so the feature was about useless
for me.
But that does mean that if mdev or whatever creates similar /dev/disk/by-
*, that mount and friends should use it just as they would with udev,
since all they're doing is a dumb string match and symlink deref.
It'a also worth noting that the kernel's root= line can take a number of
these (I'm not sure if it can take them all), including partlabel,
according to information someone posted on the btrfs list (which I was
following at the time, before I decided btrfs was too unstable still and
that I'd wait until next year to try again). I was going to try it, but
lost interest when I found I couldn't use it with fstab for my mdraid-
based gpt partitions anyway. So all I can say is that it was reported to
work on the btrfs list, and no one contradicted it, so...
a) SD does not re-calculate it's deptree/services when exiting from
rescue shell, it still try to start the virtual services as fstab
would have never modified, a reboot is needed
systemctl --system daemon-reload ?
I've not tried systemd yet nor have I yet researched it (I have the
research vaguely scheduled for next year, sometime), but thanks to both
of you for these details. Based on past experience, I expect I'm
collecting the pieces even tho I don't have context to assemble them in,
but as a result, when I /do/ research it and get that context, the pieces
I already have thanks to discussions like this, will fall into place, and
it'll be far easier to get upto speed than it would have been had I tried
going into it
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
Il 12/08/2012 14:25, Rich Freeman ha scritto: On Sun, Aug 12, 2012 at 5:43 AM, Duncan1i5t5.dun...@cox.net wrote: Peter Stuge posted on Sun, 12 Aug 2012 02:12:38 +0200 as excerpted: What software parses the filesystem labels when you boot with openrc? (I ask because I never use labels myself.) Short answer, mount and udev, and the kernel directly when fed that information for root= on the kernel commandline. ... It'a also worth noting that the kernel's root= line can take a number of these (I'm not sure if it can take them all), including partlabel, according to information someone posted on the btrfs list I'm pretty sure that this particular part of your post is incorrect, but I'd be happy to be corrected. The kernel cannot interpret a root= parameter that doesn't reference a device node that it can't internally generate (with a devtmpfs/etc). I'm pretty sure that rules out just about anything but a physical device (not even /dev/by-uuid/...). I think that people get this confused because 99% of linux users have an initramfs (and about 2% of Gentoo users it seems), and most initramfs implementations DO interpret the root=parameter. If you specify an initramfs then the kernel actually ignores the root=parameter entirely, mounting the initramfs as root, and passing control to its init. The initramfs is expected to mount root (or not - you could just run the whole system off an initramfs I guess). Most initramfs implementations just parse the root= line on the kernel, although it is worth noting that genkernel's initramfs does not and uses real_root instead. So, I could see how many linux users might think that interpreting a complex root= parameter is a kernel function, when it is really just the fact that they use an initramfs. If somebody is running with root=LABEL=foo or something like that without an initramfs I'll happily stand corrected. Rich Yes it's incorrect, a kernel w/out initrd is not able to mount by LABEL, even with automounted /dev experienced few weeks ago.
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
Rich Freeman schrieb: So, I could see how many linux users might think that interpreting a complex root= parameter is a kernel function, when it is really just the fact that they use an initramfs. If somebody is running with root=LABEL=foo or something like that without an initramfs I'll happily stand corrected. If your disk is GPT partitioned, then you can use root=PARTUUID=... without initramfs. Note that PARTUUID is the partition UUID, not the filesystem UUID. Best regards, Chí-Thanh Christopher Nguyễn
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Sun, Aug 12, 2012 at 8:25 AM, Rich Freeman ri...@gentoo.org wrote: I think that people get this confused because 99% of linux users have an initramfs (and about 2% of Gentoo users it seems), and most initramfs implementations DO interpret the root=parameter. If you specify an initramfs then the kernel actually ignores the root=parameter entirely, mounting the initramfs as root, and passing control to its init. The initramfs is expected to mount root (or not - you could just run the whole system off an initramfs I guess). Most initramfs implementations just parse the root= line on the kernel, although it is worth noting that genkernel's initramfs does not and uses real_root instead. Small correction: genkernel's /init script accepts both real_root and root on the kernel command line. If real_root is not specified, the value of root is used. It seems that the real_root option is a remnant of the initrd (not intramfs) code, where root needed to be a ram disk (/dev/ram0). With initramfs, the kernel ignores root, so we are free to use it for specifying the actual root device. So, when you see grub2-mkconfig generating entries with root=..., please do not panic; this works just fine. :-)
[gentoo-dev] Re: Questions about SystemD and OpenRC
Chí-Thanh Christopher Nguyễn posted on Sun, 12 Aug 2012 15:43:02 +0200 as excerpted: Rich Freeman schrieb: So, I could see how many linux users might think that interpreting a complex root= parameter is a kernel function, when it is really just the fact that they use an initramfs. If somebody is running with root=LABEL=foo or something like that without an initramfs I'll happily stand corrected. If your disk is GPT partitioned, then you can use root=PARTUUID=... without initramfs. Note that PARTUUID is the partition UUID, not the filesystem UUID. Thanks. Note that the btrfs list discussion was in the context of gpt partlabels (and that I specifically said I don't know if the kernel takes all such assignments in root= or just that in the discussion), so it's quite possible only partlabels and partuuids are accepted, as these are found in the gpt partition table itself, which the kernel must know how to parse in general, so teaching it to parse and honor these probably wasn't /that/ much more work than teaching it to parse the table but ignore them. And the gpt handling code is new enough, it may have simply been added with/for it, without touching the legacy mbr code. Another variable may be bootloader. Grub2 has legacy linux16 loader support, as well as 32-bit linux support. Between that and the fact that it was designed to handle both BIOS and EFI systems, it's quite possible that the legacy 16-bit linux loader protocol doesn't support these features, while the 32-bit (and presumably 64-bit EFI) kernel loader protocol does. I've no idea what grub-legacy used, but it wouldn't surprise me if it was the legacy 16-bit loader protocol, and that feeding root=PARTID= won't work for it but will with the 32-bit loader grub2 defaults to. I really should try it one of these days and know from personal experience, but in this case, it really /is/ easier to just talk than to do, since trying it requires rebooting, so I can't simply try it in another window while I keep this post open, waiting on the result... -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Fri, 10 Aug 2012 05:04:40 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: Michał Górny posted on Thu, 09 Aug 2012 22:47:38 +0200 as excerpted: On Thu, 09 Aug 2012 22:30:02 +0200 Luca Barbato lu_z...@gentoo.org wrote: On 08/09/2012 09:43 PM, Michał Górny wrote: On Thu, 09 Aug 2012 10:42:15 +0200 Luca Barbato lu_z...@gentoo.org wrote: [W]e could discuss about why reinventing shellscript may not be that sound and other less glaring, horrid and appalling design choices. Yes, exactly. So why does openrc reinvent that horrible shellscript? It is not re-invented, in fact we can use any compatible shell. Or anything else what can be spawned for shell. And a lot more what you won't expect. And guess what, people are actually doing crazy things with it because someone forgot to tell them how a init.d script should work. Sounds interesting. A couple quick links to examples of what you had in mind would be nice. =:^) (Or a bit more description, enough to both get the concept and google with would be good, but links could be quicker if you have them handy, and are less likely to spawn even further afield subthreads.) vdr is a first example which comes to my mind. They workaround program configuration limitations and the init.d scripts become a complex extra-configuration parser + plugin loader. Well, another thing here is that upstream AFAIK is not willing to cooperate to fix their config parsing. 'oldnet' is an another example. I'm not saying it should go; I'm saying it should be a stand-alone executable called from the init.d script. Last time I looked, squid init.d was performing post-inst in start(). Many users may find that beneficial but that's not what init.d scripts should be doing. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On 08/10/2012 09:43 AM, Michał Górny wrote: vdr is a first example which comes to my mind. They workaround program configuration limitations and the init.d scripts become a complex extra-configuration parser + plugin loader. Well, another thing here is that upstream AFAIK is not willing to cooperate to fix their config parsing. 'oldnet' is an another example. I'm not saying it should go; I'm saying it should be a stand-alone executable called from the init.d script. Last time I looked, squid init.d was performing post-inst in start(). Many users may find that beneficial but that's not what init.d scripts should be doing. This discussion is really derailing. Currently most people using certain features from openrc feel quite defensive and mildly uneasy with the current situation since systemd is pretty much swallowing components we used to rely on and force radical changes w/out providing explanations to people that do not care and that are perfectly happy with what they have. Now Canek seems to feel like that I'm willing to kill systemd and make it impossible to use it in Gentoo. That's not the case, I consider systemd a bad idea since it is only geared to solve some specific issues and does that in a way that I consider dangerous for the global usage patterns I'm aware of. systemd ideas are interesting and for a dumb linux desktop the implementation would be ok. systemd doesn't work in many scenarios and when it gets pointed apparently there is a disagreement on that because systemd is perfect like it happens with it gets pointed that pulseaudio has shortcomings (that come from its design) or avahi doesn't seem that stable. I can live with a seldom broken audio subsystem and I can cope with the fact pidgin-bonjour could randomly crash, I'm not happy to be forced to move to something with the same attitude as my init system. The whole point of the debate should be if easier to have systemd split itself in usable components so people with certain focuses could leverage it on linux and replace those on non-linux (apparently not a chance) or have what we currently have and works decently and hopefully be compatible (so have a compatible interface for user-daemons, a compatible dbus interface for the desktop integrations and so on), not which project we should help to kill the others. lu
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On Fri, Aug 10, 2012 at 4:45 AM, Luca Barbato lu_z...@gentoo.org wrote: The whole point of the debate should be if easier to have systemd split itself in usable components so people with certain focuses could leverage it on linux and replace those on non-linux (apparently not a chance) or have what we currently have and works decently and hopefully be compatible (so have a compatible interface for user-daemons, a compatible dbus interface for the desktop integrations and so on), not which project we should help to kill the others. I understand where you're coming from, but keep in mind that upstream the debate is more along the lines of what else to integrate, not what to split apart. There is also little interest in supporting non-linux systems with systemd - I don't think anybody working on it uses anything but linux, and I think one of their goals is to not be held back by features not available elsewhere. That might make it more of a niche solution, but it is a niche that probably includes almost all of the boxes running a typical linux distro (servers, desktops, etc - not toasters, phones, DVRs, etc). Things like Prefix or FreeBSD are a very low in market share. In any case, this list is really the wrong place for such a debate, since nobody who has any power to change anything is listening. The success/failure of systemd will have almost nothing to do with how Gentoo adopts it, it is already moderately well-supported on Gentoo as a non-default, and while there are concerns about how udev/etc is packaged and where a few things are installing their files, for the most part nothing is broken. Some purists are concerned that whatever rc system they're not using is sticking files on their system that don't do anything, and that they can't control this, and that seems to be a fault with all of the options, and most of the packages in the tree that install init scripts. There is also quite a bit of people calling each other's babies ugly, which isn't terribly productive or helpful for the community. Rich
[gentoo-dev] Re: Questions about SystemD and OpenRC
Michał Górny posted on Fri, 10 Aug 2012 09:43:26 +0200 as excerpted: On Fri, 10 Aug 2012 05:04:40 + (UTC) Duncan 1i5t5.dun...@cox.net wrote: Michał Górny posted on Thu, 09 Aug 2012 22:47:38 +0200 as excerpted: Or anything else what can be spawned for shell. And a lot more what you won't expect. And guess what, people are actually doing crazy things with it because someone forgot to tell them how a init.d script should work. Sounds interesting. A couple quick links to examples of what you had in mind would be nice. =:^) vdr is a first example which comes to my mind. They workaround program configuration limitations and the init.d scripts become a complex extra-configuration parser + plugin loader. Well, another thing here is that upstream AFAIK is not willing to cooperate to fix their config parsing. 'oldnet' is an another example. I'm not saying it should go; I'm saying it should be a stand-alone executable called from the init.d script. Last time I looked, squid init.d was performing post-inst in start(). Many users may find that beneficial but that's not what init.d scripts should be doing. Thanks. (As I said my intent wasn't to start a subthread on it, but just to see where you were going with the comment, as it was rather opaque to me as it stood. Oldnet was an especially useful example here given that I run openrc- to more closely follow individual commits, and I've traced and reported a few bugs in oldnet over the years. Now that I know where that comment was going, I'll shutup and contemplate.) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: Questions about SystemD and OpenRC
Michał Górny posted on Thu, 09 Aug 2012 22:47:38 +0200 as excerpted: On Thu, 09 Aug 2012 22:30:02 +0200 Luca Barbato lu_z...@gentoo.org wrote: On 08/09/2012 09:43 PM, Michał Górny wrote: On Thu, 09 Aug 2012 10:42:15 +0200 Luca Barbato lu_z...@gentoo.org wrote: [W]e could discuss about why reinventing shellscript may not be that sound and other less glaring, horrid and appalling design choices. Yes, exactly. So why does openrc reinvent that horrible shellscript? It is not re-invented, in fact we can use any compatible shell. Or anything else what can be spawned for shell. And a lot more what you won't expect. And guess what, people are actually doing crazy things with it because someone forgot to tell them how a init.d script should work. Sounds interesting. A couple quick links to examples of what you had in mind would be nice. =:^) (Or a bit more description, enough to both get the concept and google with would be good, but links could be quicker if you have them handy, and are less likely to spawn even further afield subthreads.) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: Questions about SystemD and OpenRC
Chí-Thanh Christopher Nguyễn posted on Fri, 10 Aug 2012 01:26:53 +0200 as excerpted: Olivier Crête schrieb: Can we also have a desktop that doesn't use X? Yes, through Wayland or DirectFB. aolMe too!/aol Seriously, they're working on it, ubuntu already has a target switch-to date (tho it's predicted to slip, but it /does/ make for faster progress), qt is planning to support it with qt5(point-something) and preliminary support is there already, same with kde frameworks (aka kde5) altho AFAIK it's mostly just kwin/plasma dev experiments ATM, and I believe gnome support may actually be further along than kde, especially with ubuntu pushing it, given that they still use the gnome foundations with unity. That's WAY farther along than any of the previous efforts toward replacing X on (other than embedded/Android) Linux got, and it actually looks like it's going to happen this time. So give it a couple years, but it's coming. Actually, wayland's exactly the thing I had in mind in my five years post as the next likely worldview disrupter. I think X will still be around in five years, but it'll be legacy for many, maybe most. I don't know that the full implications of the switch to wayland can be predicted at this point, /I'm/ certainly not going to attempt it, and it's quite possible that as a result of that shift, systemd will look as wrong solution for the wrong problem as hal ended up looking. But the only way to know is to live thru it and see where the experience takes us. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
[gentoo-dev] Re: Questions about SystemD and OpenRC
Michał Górny posted on Tue, 07 Aug 2012 22:13:21 +0200 as excerpted: Sorry for the confusion caused to you and the others. You need to read it bottom-to-top. I reversed the line order for Sylvain who seems to prefer reading that way. LOL! THAT's what it was! Along the same lines... ...senil emas eht gnolA !saw ti tahw s'TAHT !LOL http://www.textreverse.com/ (While the link I had saved was stale it did mean I still remembered enough about it to plug the idea into google and successfully find it. Link updated. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
On 8 August 2012 22:55, Duncan 1i5t5.dun...@cox.net wrote: LOL! THAT's what it was! Along the same lines... ...senil emas eht gnolA !saw ti tahw s'TAHT !LOL http://www.textreverse.com/ (While the link I had saved was stale it did mean I still remembered enough about it to plug the idea into google and successfully find it. Link updated. =:^) https://duckduckgo.com/?q=reverse+hgual%20doog%20a%20ekil%20sdnuos%20taht Google not required ;D (Also, that textreverse.com link was really slow to load ) -- Kent
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
Kent Fredric wrote: (While the link I had saved was stale it did mean I still remembered enough about it to plug the idea into google and successfully find it. Link updated. =:^) https://duckduckgo.com/?q=reverse+hgual%20doog%20a%20ekil%20sdnuos%20taht Google not required ;D Internet not required: $ rev foobar raboof $ tac 'line1 line2' line2 line1 $ //Peter
[gentoo-dev] Re: Questions about SystemD and OpenRC
Dale posted on Tue, 07 Aug 2012 16:36:30 -0500 as excerpted: What I don't like about the way Walter, mdev, is being treated is this. People say that if you don't like the way udev is going, WRITE CODE. If you are not going to write code, don't complain about udev. Then Walter, I think I got the name right, comes along and comes up with a alternative for udev that seems to work well for the people using it. Then people complain because he is actually stepping up and WRITING CODE. Well, it seems a person can't win on this. FWIW, while this isn't (currently at least) a solution I'd use, I certainly respect the man both for coming up with a solution to his own problems, and more importantly, for sharing it with others. =:^) The more so here, since as he's stated (and much like me), he's reasonable with scripting, etc, but doesn't claim to be a C/C++ coder. I believe there's quite a few list readers who have a similar respect for his efforts. Just because it's not something they'd use personally, doesn't mean they don't respect the idea. I believe at least some of the push-back isn't out of disrespect per se, or even saying it shouldn't be done, it's more a skepticism many within the FLOSS community develop over time, having seen all sorts of projects begun, but few of them actually survive more than a few months, continuing to be maintained and updated over years. Just take a look at sourceforge or github or the like, for the number of half-or quarter- finished projects... FLOSS projects are similar to business startups in that regard. Something like 80% don't survive a year or ever become even close to self- sufficient, but if they do... they're generally around for five. (Tho a difference with FLOSS is that in 5-10 years, the /need/ for the project has often disappeared as well, at least as originally envisioned. By that point many projects that actually survived their first year and got a userbase, have either evolved far enough from the original idea that they're hardly recognizable, or have simply disappeared as no longer necessary or useful. By contrast, a business life cycle, once it gets beyond that first year, tends to be rather longer... So I think a lot of it is more a nice idea, we'll see if it sticks around, more than a disrespect for it or the person behind it, per se. If it's still around and actually useful in a couple years, I expect you'll see a lot more overt respect that simply isn't evident, now. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] Re: Questions about SystemD and OpenRC
Duncan wrote: I believe there's quite a few list readers who have a similar respect for his efforts. I believe so too! I think it's a great effort. It may not fit my use cases, but I don't care about that - even if it is *only* Walter scratching his own itch I agree that it's important to show some love for the work. It doesn't matter right now if the itch will go away or if mdev will rule the Linux desktop one day. It's ongoing work and I think it's important not to dismiss it. //Peter
[gentoo-dev] Re: Questions about SystemD and OpenRC
Peter Stuge posted on Wed, 08 Aug 2012 13:22:54 +0200 as excerpted: Internet not required: $ rev foobar raboof $ tac [...] Thanks. I'd read about those before (at least tac), but they aren't links (stale or not), so I'd forgotten them... Hmm... links to the manpages might solve that! =:^) http://ss64.com/bash/tac.html http://ss64.com/bash/rev.html -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
