Re: [systemd-devel] Reduce unit-loading time

2015-05-26 Thread Lennart Poettering
On Tue, 26.05.15 09:49, Filipe Brandenburger (filbran...@google.com) wrote:

> Hi,
> 
> On Sun, May 24, 2015 at 8:41 PM, cee1  wrote:
> > I tried ureadahead, but got following error:
> >
> > """write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error
> > while tracing: No such file or directory"""
> >
> > Needs an out-of-tree kernel patch?
> 
> Yes, ureadahead needs an out-of-tree kernel patch to add trace events
> for open(), exec() and uselib() syscalls that take file paths.
> 
> http://bugs.launchpad.net/bugs/462111
> 
> AFAICT, this never went upstream because upstream was discussing a
> generic approach of tracing any system calls, but I believe that never
> really panned out...

systemd-readahead used fanotify() which requires no kernel
patching. fanotify is completely generic and perfect for this
purpose...

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-26 Thread Filipe Brandenburger
Hi,

On Sun, May 24, 2015 at 8:41 PM, cee1  wrote:
> I tried ureadahead, but got following error:
>
> """write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error
> while tracing: No such file or directory"""
>
> Needs an out-of-tree kernel patch?

Yes, ureadahead needs an out-of-tree kernel patch to add trace events
for open(), exec() and uselib() syscalls that take file paths.

http://bugs.launchpad.net/bugs/462111

AFAICT, this never went upstream because upstream was discussing a
generic approach of tracing any system calls, but I believe that never
really panned out...

HTH,
Filipe
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-24 Thread cee1
2015-05-20 1:01 GMT+08:00 Martin Pitt :
> Hey cee1,
>
> cee1 [2015-05-18 23:52 +0800]:
>> At the first glance, I find ureadahead has some difference compared
>> with the readahead once in systemd, IIRC:
>
> Yes, for sure. systemd's was improved quite a bit. ureadahead is
> mostly unmaintained, but it works well enough so we didn't bother to
> put work into it until someone actually complains :-)
>
>> 1. ureadahead.service is in default.target, which means ureadahead
>> starts later than systemd's?
>
> That mostly means that it's not started with e. g. emergency or
> rescue. It still starts early (DefaultDependencies=false).
>
>> 2. The original systemd readahead has "collect" and "replay" two
>> services, and these are done in ureadahead.service?
>
> Yes.
>
>> 3. IIRC, The original systemd readahead uses madvise(); ureadahead
>> uses readahead()
>> 4. The original systemd readahead uses fanotify() to get the list of
>> accessed files; ureadahead use fsnotify
>
> I haven't verified these, but this sounds correct. ureadahead is
> really old, presumably the newer features like fanotify weren't
> available back then.

I tried ureadahead, but got following error:

"""write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error
while tracing: No such file or directory"""

Needs an out-of-tree kernel patch?



-- 
Regards,

- cee1
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-19 Thread Cristian Rodríguez
On Tue, May 19, 2015 at 5:39 PM, Cristian Rodríguez
 wrote:
> On Mon, May 18, 2015 at 7:24 AM, cee1  wrote:
>> 2015-05-17 17:45 GMT+08:00 Martin Pitt :
>>> Hello cee,
>>>
>>> cee1 [2015-05-16  0:46 +0800]:
 Thanks for the suggestion, it was other processes running in parallel
 which presumably consuming lots of IO, after sending SIGSTOP at the
 first (and SIGCONT later), the unit loading time is decreased to
 ~100ms.
>>>
>>> You probably want to use some readahead solution. We found that it
>>> makes a significant improvement on ARM boards with slow MMC cards.
>
> You could also
>
> posix_fadvise(fileno(f), 0, 0, POSIX_FADV_SEQUENTIAL);
> in the bits that load the unit file..the kernel is free to ignore that
> advice however.

This however.. won't be of any help, as the default readhead window is
128kb.. which is way bigger than any unit file.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-19 Thread Cristian Rodríguez
On Mon, May 18, 2015 at 7:24 AM, cee1  wrote:
> 2015-05-17 17:45 GMT+08:00 Martin Pitt :
>> Hello cee,
>>
>> cee1 [2015-05-16  0:46 +0800]:
>>> Thanks for the suggestion, it was other processes running in parallel
>>> which presumably consuming lots of IO, after sending SIGSTOP at the
>>> first (and SIGCONT later), the unit loading time is decreased to
>>> ~100ms.
>>
>> You probably want to use some readahead solution. We found that it
>> makes a significant improvement on ARM boards with slow MMC cards.

You could also

posix_fadvise(fileno(f), 0, 0, POSIX_FADV_SEQUENTIAL);
in the bits that load the unit file..the kernel is free to ignore that
advice however.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-19 Thread Martin Pitt
Hey cee1,

cee1 [2015-05-18 23:52 +0800]:
> At the first glance, I find ureadahead has some difference compared
> with the readahead once in systemd, IIRC:

Yes, for sure. systemd's was improved quite a bit. ureadahead is
mostly unmaintained, but it works well enough so we didn't bother to
put work into it until someone actually complains :-)

> 1. ureadahead.service is in default.target, which means ureadahead
> starts later than systemd's?

That mostly means that it's not started with e. g. emergency or
rescue. It still starts early (DefaultDependencies=false).

> 2. The original systemd readahead has "collect" and "replay" two
> services, and these are done in ureadahead.service?

Yes.

> 3. IIRC, The original systemd readahead uses madvise(); ureadahead
> uses readahead()
> 4. The original systemd readahead uses fanotify() to get the list of
> accessed files; ureadahead use fsnotify

I haven't verified these, but this sounds correct. ureadahead is
really old, presumably the newer features like fanotify weren't
available back then.

> 5. ureadahead has different readahead strategies for ssd and hhd:

Right. These were created by some rather wide-scale measurements back
then.

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-18 Thread Lennart Poettering
On Mon, 18.05.15 18:24, cee1 (fykc...@gmail.com) wrote:

> 2015-05-17 17:45 GMT+08:00 Martin Pitt :
> > Hello cee,
> >
> > cee1 [2015-05-16  0:46 +0800]:
> >> Thanks for the suggestion, it was other processes running in parallel
> >> which presumably consuming lots of IO, after sending SIGSTOP at the
> >> first (and SIGCONT later), the unit loading time is decreased to
> >> ~100ms.
> >
> > You probably want to use some readahead solution. We found that it
> > makes a significant improvement on ARM boards with slow MMC cards.
> >
> > Martin
> > --
> > Martin Pitt| http://www.piware.de
> > Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
> 
> Hey,
> 
> Thanks for the suggestion, IIRC, sequential read is also beneficial
> for flash storage.
> 
> Does the readahead-*.service shipped with systemd work for you?

We removed the readahead logic from systemd a while back, since it had
no maintainer, and nobody wanted to pick it up.

> Question:
> How does systemd schedule two services that can be launched in
> parallel?

It's not defined. systemd will fork things of in an undefined order if
ther are multiple units runnable at the same time.

As mentioned elsewhere, I'd be willing to merge a patchat that changes
this and allows control of what service is forked off first, via some
per-unit Priority= setting or so.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-18 Thread cee1
Hi Martin,

At the first glance, I find ureadahead has some difference compared
with the readahead once in systemd, IIRC:

1. ureadahead.service is in default.target, which means ureadahead
starts later than systemd's?
2. The original systemd readahead has "collect" and "replay" two
services, and these are done in ureadahead.service?
3. IIRC, The original systemd readahead uses madvise(); ureadahead
uses readahead()
4. The original systemd readahead uses fanotify() to get the list of
accessed files; ureadahead use fsnotify
5. ureadahead has different readahead strategies for ssd and hhd:
5.1 For the former, initiate multi-threads to perform readahead, and
they are running at the lowest IO priority.
5.2 For the later, perform readahead for both inode and file content
at a very high CPU/IO priority (and only support extN filesystem ?)


2015-05-18 18:40 GMT+08:00 Martin Pitt :
> Hello cee1,
>
> cee1 [2015-05-18 18:24 +0800]:
>> Does the readahead-*.service shipped with systemd work for you?
>
> systemd dropped the builtin readahead in 217. It's reasonably easy to
> get back by reverting the "drop readahead" patches, but carrying that
> patch in packages is fairly intrusive. In Ubuntu we've had
> "ureadahead" [1] for many years which is "good enough" for things like
> phones or other ARM boards with slow MMC storage, so I just added
> systemd units to that. It's a separate project so that we don't need
> to install ureadahead everywhere, just where/when we actually need it.
>
> Martin
>
> [1] https://launchpad.net/ubuntu/+source/ureadahead
> --
> Martin Pitt| http://www.piware.de
> Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)



-- 
Regards,

- cee1
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-18 Thread Martin Pitt
Hello cee1,

cee1 [2015-05-18 18:24 +0800]:
> Does the readahead-*.service shipped with systemd work for you?

systemd dropped the builtin readahead in 217. It's reasonably easy to
get back by reverting the "drop readahead" patches, but carrying that
patch in packages is fairly intrusive. In Ubuntu we've had
"ureadahead" [1] for many years which is "good enough" for things like
phones or other ARM boards with slow MMC storage, so I just added
systemd units to that. It's a separate project so that we don't need
to install ureadahead everywhere, just where/when we actually need it.

Martin

[1] https://launchpad.net/ubuntu/+source/ureadahead
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-18 Thread cee1
2015-05-17 17:45 GMT+08:00 Martin Pitt :
> Hello cee,
>
> cee1 [2015-05-16  0:46 +0800]:
>> Thanks for the suggestion, it was other processes running in parallel
>> which presumably consuming lots of IO, after sending SIGSTOP at the
>> first (and SIGCONT later), the unit loading time is decreased to
>> ~100ms.
>
> You probably want to use some readahead solution. We found that it
> makes a significant improvement on ARM boards with slow MMC cards.
>
> Martin
> --
> Martin Pitt| http://www.piware.de
> Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

Hey,

Thanks for the suggestion, IIRC, sequential read is also beneficial
for flash storage.

Does the readahead-*.service shipped with systemd work for you?


BTW,  some suggestions and questions :)

Suggestion:
I use the following command to figure out why my service is scheduled
at the time:
"systemctl list-dependencies --after target.service"
and expect it could output "timing info(unit starting time and
started time, etc)" of dependent units.

Question:
How does systemd schedule two services that can be launched in parallel?

I found that,  A and B, if specifies "Before=A", B will start first,
otherwise, B will start at a very late time.


-- 
Regards,

- cee1
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-17 Thread Martin Pitt
Hello cee,

cee1 [2015-05-16  0:46 +0800]:
> Thanks for the suggestion, it was other processes running in parallel
> which presumably consuming lots of IO, after sending SIGSTOP at the
> first (and SIGCONT later), the unit loading time is decreased to
> ~100ms.

You probably want to use some readahead solution. We found that it
makes a significant improvement on ARM boards with slow MMC cards.

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-15 Thread cee1
Hey,

Thanks for the suggestion, it was other processes running in parallel
which presumably consuming lots of IO, after sending SIGSTOP at the
first (and SIGCONT later), the unit loading time is decreased to
~100ms.


2015-05-13 19:38 GMT+08:00 Hoyer, Marko (ADITG/SW2) :
> Hi,
>
>> -Original Message-
>> From: systemd-devel [mailto:systemd-devel-
>> boun...@lists.freedesktop.org] On Behalf Of cee1
>> Sent: Wednesday, May 13, 2015 11:52 AM
>> To: systemd Mailing List
>> Subject: [systemd-devel] Reduce unit-loading time
>>
>> Hi all,
>>
>> We're trying systemd to boot up an ARM board, and find systemd uses
>> more than one second to load units.
>
> This sounds far a bit too long to me. Our systemd comes up in an arm based 
> system in about 200-300ms from executing init up to the first unit being 
> executed.
>
>>
>> Comparing with the init of Android on the same board, it manages to
>> boot the system very fast.
>>
>> We guess following factors are involved:
>> 1. systemd has a much bigger footprint than the init of Android: the
>> latter is static linked, and is about 1xxKB (systemd is about 1.4MB,
>> and is linked with libc/libcap/libpthread, etc)
>>
>> 2. systemd spends quiet a while to read/parse unit files.
>
> This depends on the numbers of units involved in the startup (finally 
> connected in the dependency tree that ends in the default.target root). In 
> our case, a comparable large unit set takes by about 40-60ms, not so long, 
> I'd say.
>
>>
>>
>> Any idea to reduce the unit-loading time?
>> e.g. one-single file contains all units descriptions, or a "compiled
>> version"(containing resolved dependencies, or even the boot up
>> sequence)
>
> Could you provide me some additional information about your system setup?
>
> - Version of systemd
> - Are you starting something in parallel to systemd which might take 
> significant IO?
> - Version of the kernel
> - The kernel ticker frequency
> - The enabled cgroups controllers
>
> The last three points might sound a bit far away, but I've an idea in mind ...
>
> Best regards
>
> Marko Hoyer
> Software Group II (ADITG/SW2)
>
> Tel. +49 5121 49 6948
>



-- 
Regards,

- cee1
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-13 Thread Lennart Poettering
On Wed, 13.05.15 17:51, cee1 (fykc...@gmail.com) wrote:

> Hi all,
> 
> We're trying systemd to boot up an ARM board, and find systemd uses
> more than one second to load units.
> 
> Comparing with the init of Android on the same board, it manages to
> boot the system very fast.
> 
> We guess following factors are involved:
> 1. systemd has a much bigger footprint than the init of Android: the
> latter is static linked, and is about 1xxKB (systemd is about 1.4MB,
> and is linked with libc/libcap/libpthread, etc)
> 
> 2. systemd spends quiet a while to read/parse unit files.
> 
> 
> Any idea to reduce the unit-loading time?
> e.g. one-single file contains all units descriptions, or a "compiled
> version"(containing resolved dependencies, or even the boot up
> sequence)

Well, before starting to optimize things one should always profile
this step to identify what precisely is slow. This starts from
questions like whether this is IO bound or CPU bound to precisely
tracing things down to individual files.

systemd unit loading is not particularly optimized, we distribute
everything into small unit files, which generally isn't the best way
to get quick access times for many file systems. You can improve the
situation with read-ahead for these files, but before you do that you
really need to spend some time to figure out what precisely is slow.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Reduce unit-loading time

2015-05-13 Thread Hoyer, Marko (ADITG/SW2)
Hi,

> -Original Message-
> From: systemd-devel [mailto:systemd-devel-
> boun...@lists.freedesktop.org] On Behalf Of cee1
> Sent: Wednesday, May 13, 2015 11:52 AM
> To: systemd Mailing List
> Subject: [systemd-devel] Reduce unit-loading time
> 
> Hi all,
> 
> We're trying systemd to boot up an ARM board, and find systemd uses
> more than one second to load units.

This sounds far a bit too long to me. Our systemd comes up in an arm based 
system in about 200-300ms from executing init up to the first unit being 
executed.

> 
> Comparing with the init of Android on the same board, it manages to
> boot the system very fast.
> 
> We guess following factors are involved:
> 1. systemd has a much bigger footprint than the init of Android: the
> latter is static linked, and is about 1xxKB (systemd is about 1.4MB,
> and is linked with libc/libcap/libpthread, etc)
> 
> 2. systemd spends quiet a while to read/parse unit files.

This depends on the numbers of units involved in the startup (finally connected 
in the dependency tree that ends in the default.target root). In our case, a 
comparable large unit set takes by about 40-60ms, not so long, I'd say.

> 
> 
> Any idea to reduce the unit-loading time?
> e.g. one-single file contains all units descriptions, or a "compiled
> version"(containing resolved dependencies, or even the boot up
> sequence)

Could you provide me some additional information about your system setup?

- Version of systemd
- Are you starting something in parallel to systemd which might take 
significant IO?
- Version of the kernel
- The kernel ticker frequency
- The enabled cgroups controllers

The last three points might sound a bit far away, but I've an idea in mind ...

Best regards

Marko Hoyer
Software Group II (ADITG/SW2)

Tel. +49 5121 49 6948

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel