Re: [systemd-devel] preset activation on first boot takes too long

2015-05-15 Thread Dimitri John Ledkov
On 14 May 2015 at 18:00, Lennart Poettering lenn...@poettering.net wrote:
 On Thu, 14.05.15 17:13, Dimitri John Ledkov (dimitri.j.led...@intel.com) 
 wrote:

 Heya,

 I'm looking at bootcharts and it seems like first boot preset
 activation takes too much time...

 So at the moment, we iterate all units, then iterate through presets
 until we find a match and act upon it.

 However, most distros have disable * as their last setting, or don't
 use presets at all.

 Furthermore looking at a fully featured system (e.g. my ubuntu laptop):
 * 158 files do not have install section
 * 89 have an install secion

 Also it seems odd to have all of this in the pid one critical path -
 e.g. these things are being parsed before anything happens.

 Thus I wonder if the presets should be moved into e.g. a generator
 that will do the following on first boot only:

 Hmm, generators so far were strictly something that would generate
 output in the generator unit paths we pass to them, and nowhere
 else. These directories would then be lifecycled by the daemon
 runtime, and flushed on daemon reload. The generators would then be
 invoked with new, empty generator unit directories on daemon reload.

 The preset logic otoh creates persistent changes in /etc, that will
 only and exlcusively be run if /etc is unpopulated. They logic is not
 rerun on daemon reload, and nothing is ever flushed.

 I am pretty sure this should stay that way. We shouldn't turn
 generators into more than what they are right now. They should not
 have persistent effect.

 However, I am all open for optimizing the codepaths here. Maybe we can
 cache the preset files in memory or so. And/or we could move the
 preset code into its own binary, which we fork off explicitly before
 running the generators, and then wait for after the generators (so
 that the binary runs in parallel to the generators). Alternatively we
 could even run it as thread from PID 1 in parallel to waiting for the
 generators, as long as it shares zero state with the rest of PID
 1. (That said, I am usually not too fond of threads, and especially
 not in PID 1.)

 * parse .preset files
 * construct list of things to enable
 * enable all the units in that list

 OK, so this would mean caching the preset file, and then doing pretty
 much the same as before? I am fine with that. (But not as a generator,
 as mentioned above)

 This should cut I/O and processing time at first boot by a bit, since
 only the units to be activated will be parsed.

 Hmm? not following here. are you suggesting to use the preset file
 list as base list of units to enable and then search for them in the
 file system without ever enumerating unit files in the fs? This will
 not really work, as the list contains glob expressions, and more
 complicated ones than just disable *. For examples things like
 enable avahi-daemon.* which enables both the socket and the service
 in one step. But it could also be enable foo-*.service, for a
 project foo, that has many different services it wants to enable
 with a single line... And crazy people could even use more complex
 matches...

 Hence, I am pretty sure the list of unit files enumerated from disk
 needs to be used as basis, and then checked against the preset file,
 not the other way round.

 One thing you could use for optimizing: a TODO list item has been for
 a while to change the first-boot preset logic to operate in a purely
 additive way: don't bother with disabling any services then (in most
 cases there will be nothing enabled at that time anyway), but only
 operate on the enable cases. Doing this would allow you to avoid
 loading the unit files for the disable lines, as you you don't care
 for their [Install] section then.

Right. I used the term generator too flamboyant here indeed, as this
optimised preset parsing cannot ever be a generator-spec compliant.

I think optimisations above will work, especially given that we don't
honour aliases ahead of time
(that is bar.service unit with Alias=foo.service is not enabled, with
enable foo.service preset)
This is really good, as I think pre-parsing presets will help a lot here.

-- 
Regards,

Dimitri.
Pura Vida!

https://clearlinux.org
Open Source Technology Center
Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] preset activation on first boot takes too long

2015-05-14 Thread Lennart Poettering
On Thu, 14.05.15 17:13, Dimitri John Ledkov (dimitri.j.led...@intel.com) wrote:

 Heya,
 
 I'm looking at bootcharts and it seems like first boot preset
 activation takes too much time...
 
 So at the moment, we iterate all units, then iterate through presets
 until we find a match and act upon it.
 
 However, most distros have disable * as their last setting, or don't
 use presets at all.
 
 Furthermore looking at a fully featured system (e.g. my ubuntu laptop):
 * 158 files do not have install section
 * 89 have an install secion
 
 Also it seems odd to have all of this in the pid one critical path -
 e.g. these things are being parsed before anything happens.
 
 Thus I wonder if the presets should be moved into e.g. a generator
 that will do the following on first boot only:

Hmm, generators so far were strictly something that would generate
output in the generator unit paths we pass to them, and nowhere
else. These directories would then be lifecycled by the daemon
runtime, and flushed on daemon reload. The generators would then be
invoked with new, empty generator unit directories on daemon reload.

The preset logic otoh creates persistent changes in /etc, that will
only and exlcusively be run if /etc is unpopulated. They logic is not
rerun on daemon reload, and nothing is ever flushed.

I am pretty sure this should stay that way. We shouldn't turn
generators into more than what they are right now. They should not
have persistent effect.

However, I am all open for optimizing the codepaths here. Maybe we can
cache the preset files in memory or so. And/or we could move the
preset code into its own binary, which we fork off explicitly before
running the generators, and then wait for after the generators (so
that the binary runs in parallel to the generators). Alternatively we
could even run it as thread from PID 1 in parallel to waiting for the
generators, as long as it shares zero state with the rest of PID
1. (That said, I am usually not too fond of threads, and especially
not in PID 1.)

 * parse .preset files
 * construct list of things to enable
 * enable all the units in that list

OK, so this would mean caching the preset file, and then doing pretty
much the same as before? I am fine with that. (But not as a generator,
as mentioned above)

 This should cut I/O and processing time at first boot by a bit, since
 only the units to be activated will be parsed.

Hmm? not following here. are you suggesting to use the preset file
list as base list of units to enable and then search for them in the
file system without ever enumerating unit files in the fs? This will
not really work, as the list contains glob expressions, and more
complicated ones than just disable *. For examples things like
enable avahi-daemon.* which enables both the socket and the service
in one step. But it could also be enable foo-*.service, for a
project foo, that has many different services it wants to enable
with a single line... And crazy people could even use more complex
matches...

Hence, I am pretty sure the list of unit files enumerated from disk
needs to be used as basis, and then checked against the preset file,
not the other way round.

One thing you could use for optimizing: a TODO list item has been for
a while to change the first-boot preset logic to operate in a purely
additive way: don't bother with disabling any services then (in most
cases there will be nothing enabled at that time anyway), but only
operate on the enable cases. Doing this would allow you to avoid
loading the unit files for the disable lines, as you you don't care
for their [Install] section then. 

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel