Re: [systemd-devel] Reduce unit-loading time
On Tue, 26.05.15 09:49, Filipe Brandenburger (filbran...@google.com) wrote: > Hi, > > On Sun, May 24, 2015 at 8:41 PM, cee1 wrote: > > I tried ureadahead, but got following error: > > > > """write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error > > while tracing: No such file or directory""" > > > > Needs an out-of-tree kernel patch? > > Yes, ureadahead needs an out-of-tree kernel patch to add trace events > for open(), exec() and uselib() syscalls that take file paths. > > http://bugs.launchpad.net/bugs/462111 > > AFAICT, this never went upstream because upstream was discussing a > generic approach of tracing any system calls, but I believe that never > really panned out... systemd-readahead used fanotify() which requires no kernel patching. fanotify is completely generic and perfect for this purpose... Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
Hi, On Sun, May 24, 2015 at 8:41 PM, cee1 wrote: > I tried ureadahead, but got following error: > > """write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error > while tracing: No such file or directory""" > > Needs an out-of-tree kernel patch? Yes, ureadahead needs an out-of-tree kernel patch to add trace events for open(), exec() and uselib() syscalls that take file paths. http://bugs.launchpad.net/bugs/462111 AFAICT, this never went upstream because upstream was discussing a generic approach of tracing any system calls, but I believe that never really panned out... HTH, Filipe ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
2015-05-20 1:01 GMT+08:00 Martin Pitt : > Hey cee1, > > cee1 [2015-05-18 23:52 +0800]: >> At the first glance, I find ureadahead has some difference compared >> with the readahead once in systemd, IIRC: > > Yes, for sure. systemd's was improved quite a bit. ureadahead is > mostly unmaintained, but it works well enough so we didn't bother to > put work into it until someone actually complains :-) > >> 1. ureadahead.service is in default.target, which means ureadahead >> starts later than systemd's? > > That mostly means that it's not started with e. g. emergency or > rescue. It still starts early (DefaultDependencies=false). > >> 2. The original systemd readahead has "collect" and "replay" two >> services, and these are done in ureadahead.service? > > Yes. > >> 3. IIRC, The original systemd readahead uses madvise(); ureadahead >> uses readahead() >> 4. The original systemd readahead uses fanotify() to get the list of >> accessed files; ureadahead use fsnotify > > I haven't verified these, but this sounds correct. ureadahead is > really old, presumably the newer features like fanotify weren't > available back then. I tried ureadahead, but got following error: """write(2, "ureadahead: Error while tracing:"..., 59ureadahead: Error while tracing: No such file or directory""" Needs an out-of-tree kernel patch? -- Regards, - cee1 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
On Tue, May 19, 2015 at 5:39 PM, Cristian Rodríguez wrote: > On Mon, May 18, 2015 at 7:24 AM, cee1 wrote: >> 2015-05-17 17:45 GMT+08:00 Martin Pitt : >>> Hello cee, >>> >>> cee1 [2015-05-16 0:46 +0800]: Thanks for the suggestion, it was other processes running in parallel which presumably consuming lots of IO, after sending SIGSTOP at the first (and SIGCONT later), the unit loading time is decreased to ~100ms. >>> >>> You probably want to use some readahead solution. We found that it >>> makes a significant improvement on ARM boards with slow MMC cards. > > You could also > > posix_fadvise(fileno(f), 0, 0, POSIX_FADV_SEQUENTIAL); > in the bits that load the unit file..the kernel is free to ignore that > advice however. This however.. won't be of any help, as the default readhead window is 128kb.. which is way bigger than any unit file. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
On Mon, May 18, 2015 at 7:24 AM, cee1 wrote: > 2015-05-17 17:45 GMT+08:00 Martin Pitt : >> Hello cee, >> >> cee1 [2015-05-16 0:46 +0800]: >>> Thanks for the suggestion, it was other processes running in parallel >>> which presumably consuming lots of IO, after sending SIGSTOP at the >>> first (and SIGCONT later), the unit loading time is decreased to >>> ~100ms. >> >> You probably want to use some readahead solution. We found that it >> makes a significant improvement on ARM boards with slow MMC cards. You could also posix_fadvise(fileno(f), 0, 0, POSIX_FADV_SEQUENTIAL); in the bits that load the unit file..the kernel is free to ignore that advice however. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
Hey cee1, cee1 [2015-05-18 23:52 +0800]: > At the first glance, I find ureadahead has some difference compared > with the readahead once in systemd, IIRC: Yes, for sure. systemd's was improved quite a bit. ureadahead is mostly unmaintained, but it works well enough so we didn't bother to put work into it until someone actually complains :-) > 1. ureadahead.service is in default.target, which means ureadahead > starts later than systemd's? That mostly means that it's not started with e. g. emergency or rescue. It still starts early (DefaultDependencies=false). > 2. The original systemd readahead has "collect" and "replay" two > services, and these are done in ureadahead.service? Yes. > 3. IIRC, The original systemd readahead uses madvise(); ureadahead > uses readahead() > 4. The original systemd readahead uses fanotify() to get the list of > accessed files; ureadahead use fsnotify I haven't verified these, but this sounds correct. ureadahead is really old, presumably the newer features like fanotify weren't available back then. > 5. ureadahead has different readahead strategies for ssd and hhd: Right. These were created by some rather wide-scale measurements back then. Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
On Mon, 18.05.15 18:24, cee1 (fykc...@gmail.com) wrote: > 2015-05-17 17:45 GMT+08:00 Martin Pitt : > > Hello cee, > > > > cee1 [2015-05-16 0:46 +0800]: > >> Thanks for the suggestion, it was other processes running in parallel > >> which presumably consuming lots of IO, after sending SIGSTOP at the > >> first (and SIGCONT later), the unit loading time is decreased to > >> ~100ms. > > > > You probably want to use some readahead solution. We found that it > > makes a significant improvement on ARM boards with slow MMC cards. > > > > Martin > > -- > > Martin Pitt| http://www.piware.de > > Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) > > Hey, > > Thanks for the suggestion, IIRC, sequential read is also beneficial > for flash storage. > > Does the readahead-*.service shipped with systemd work for you? We removed the readahead logic from systemd a while back, since it had no maintainer, and nobody wanted to pick it up. > Question: > How does systemd schedule two services that can be launched in > parallel? It's not defined. systemd will fork things of in an undefined order if ther are multiple units runnable at the same time. As mentioned elsewhere, I'd be willing to merge a patchat that changes this and allows control of what service is forked off first, via some per-unit Priority= setting or so. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
Hi Martin, At the first glance, I find ureadahead has some difference compared with the readahead once in systemd, IIRC: 1. ureadahead.service is in default.target, which means ureadahead starts later than systemd's? 2. The original systemd readahead has "collect" and "replay" two services, and these are done in ureadahead.service? 3. IIRC, The original systemd readahead uses madvise(); ureadahead uses readahead() 4. The original systemd readahead uses fanotify() to get the list of accessed files; ureadahead use fsnotify 5. ureadahead has different readahead strategies for ssd and hhd: 5.1 For the former, initiate multi-threads to perform readahead, and they are running at the lowest IO priority. 5.2 For the later, perform readahead for both inode and file content at a very high CPU/IO priority (and only support extN filesystem ?) 2015-05-18 18:40 GMT+08:00 Martin Pitt : > Hello cee1, > > cee1 [2015-05-18 18:24 +0800]: >> Does the readahead-*.service shipped with systemd work for you? > > systemd dropped the builtin readahead in 217. It's reasonably easy to > get back by reverting the "drop readahead" patches, but carrying that > patch in packages is fairly intrusive. In Ubuntu we've had > "ureadahead" [1] for many years which is "good enough" for things like > phones or other ARM boards with slow MMC storage, so I just added > systemd units to that. It's a separate project so that we don't need > to install ureadahead everywhere, just where/when we actually need it. > > Martin > > [1] https://launchpad.net/ubuntu/+source/ureadahead > -- > Martin Pitt| http://www.piware.de > Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- Regards, - cee1 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
Hello cee1, cee1 [2015-05-18 18:24 +0800]: > Does the readahead-*.service shipped with systemd work for you? systemd dropped the builtin readahead in 217. It's reasonably easy to get back by reverting the "drop readahead" patches, but carrying that patch in packages is fairly intrusive. In Ubuntu we've had "ureadahead" [1] for many years which is "good enough" for things like phones or other ARM boards with slow MMC storage, so I just added systemd units to that. It's a separate project so that we don't need to install ureadahead everywhere, just where/when we actually need it. Martin [1] https://launchpad.net/ubuntu/+source/ureadahead -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
2015-05-17 17:45 GMT+08:00 Martin Pitt : > Hello cee, > > cee1 [2015-05-16 0:46 +0800]: >> Thanks for the suggestion, it was other processes running in parallel >> which presumably consuming lots of IO, after sending SIGSTOP at the >> first (and SIGCONT later), the unit loading time is decreased to >> ~100ms. > > You probably want to use some readahead solution. We found that it > makes a significant improvement on ARM boards with slow MMC cards. > > Martin > -- > Martin Pitt| http://www.piware.de > Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) Hey, Thanks for the suggestion, IIRC, sequential read is also beneficial for flash storage. Does the readahead-*.service shipped with systemd work for you? BTW, some suggestions and questions :) Suggestion: I use the following command to figure out why my service is scheduled at the time: "systemctl list-dependencies --after target.service" and expect it could output "timing info(unit starting time and started time, etc)" of dependent units. Question: How does systemd schedule two services that can be launched in parallel? I found that, A and B, if specifies "Before=A", B will start first, otherwise, B will start at a very late time. -- Regards, - cee1 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
Hello cee, cee1 [2015-05-16 0:46 +0800]: > Thanks for the suggestion, it was other processes running in parallel > which presumably consuming lots of IO, after sending SIGSTOP at the > first (and SIGCONT later), the unit loading time is decreased to > ~100ms. You probably want to use some readahead solution. We found that it makes a significant improvement on ARM boards with slow MMC cards. Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
Hey, Thanks for the suggestion, it was other processes running in parallel which presumably consuming lots of IO, after sending SIGSTOP at the first (and SIGCONT later), the unit loading time is decreased to ~100ms. 2015-05-13 19:38 GMT+08:00 Hoyer, Marko (ADITG/SW2) : > Hi, > >> -Original Message- >> From: systemd-devel [mailto:systemd-devel- >> boun...@lists.freedesktop.org] On Behalf Of cee1 >> Sent: Wednesday, May 13, 2015 11:52 AM >> To: systemd Mailing List >> Subject: [systemd-devel] Reduce unit-loading time >> >> Hi all, >> >> We're trying systemd to boot up an ARM board, and find systemd uses >> more than one second to load units. > > This sounds far a bit too long to me. Our systemd comes up in an arm based > system in about 200-300ms from executing init up to the first unit being > executed. > >> >> Comparing with the init of Android on the same board, it manages to >> boot the system very fast. >> >> We guess following factors are involved: >> 1. systemd has a much bigger footprint than the init of Android: the >> latter is static linked, and is about 1xxKB (systemd is about 1.4MB, >> and is linked with libc/libcap/libpthread, etc) >> >> 2. systemd spends quiet a while to read/parse unit files. > > This depends on the numbers of units involved in the startup (finally > connected in the dependency tree that ends in the default.target root). In > our case, a comparable large unit set takes by about 40-60ms, not so long, > I'd say. > >> >> >> Any idea to reduce the unit-loading time? >> e.g. one-single file contains all units descriptions, or a "compiled >> version"(containing resolved dependencies, or even the boot up >> sequence) > > Could you provide me some additional information about your system setup? > > - Version of systemd > - Are you starting something in parallel to systemd which might take > significant IO? > - Version of the kernel > - The kernel ticker frequency > - The enabled cgroups controllers > > The last three points might sound a bit far away, but I've an idea in mind ... > > Best regards > > Marko Hoyer > Software Group II (ADITG/SW2) > > Tel. +49 5121 49 6948 > -- Regards, - cee1 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
On Wed, 13.05.15 17:51, cee1 (fykc...@gmail.com) wrote: > Hi all, > > We're trying systemd to boot up an ARM board, and find systemd uses > more than one second to load units. > > Comparing with the init of Android on the same board, it manages to > boot the system very fast. > > We guess following factors are involved: > 1. systemd has a much bigger footprint than the init of Android: the > latter is static linked, and is about 1xxKB (systemd is about 1.4MB, > and is linked with libc/libcap/libpthread, etc) > > 2. systemd spends quiet a while to read/parse unit files. > > > Any idea to reduce the unit-loading time? > e.g. one-single file contains all units descriptions, or a "compiled > version"(containing resolved dependencies, or even the boot up > sequence) Well, before starting to optimize things one should always profile this step to identify what precisely is slow. This starts from questions like whether this is IO bound or CPU bound to precisely tracing things down to individual files. systemd unit loading is not particularly optimized, we distribute everything into small unit files, which generally isn't the best way to get quick access times for many file systems. You can improve the situation with read-ahead for these files, but before you do that you really need to spend some time to figure out what precisely is slow. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Reduce unit-loading time
Hi, > -Original Message- > From: systemd-devel [mailto:systemd-devel- > boun...@lists.freedesktop.org] On Behalf Of cee1 > Sent: Wednesday, May 13, 2015 11:52 AM > To: systemd Mailing List > Subject: [systemd-devel] Reduce unit-loading time > > Hi all, > > We're trying systemd to boot up an ARM board, and find systemd uses > more than one second to load units. This sounds far a bit too long to me. Our systemd comes up in an arm based system in about 200-300ms from executing init up to the first unit being executed. > > Comparing with the init of Android on the same board, it manages to > boot the system very fast. > > We guess following factors are involved: > 1. systemd has a much bigger footprint than the init of Android: the > latter is static linked, and is about 1xxKB (systemd is about 1.4MB, > and is linked with libc/libcap/libpthread, etc) > > 2. systemd spends quiet a while to read/parse unit files. This depends on the numbers of units involved in the startup (finally connected in the dependency tree that ends in the default.target root). In our case, a comparable large unit set takes by about 40-60ms, not so long, I'd say. > > > Any idea to reduce the unit-loading time? > e.g. one-single file contains all units descriptions, or a "compiled > version"(containing resolved dependencies, or even the boot up > sequence) Could you provide me some additional information about your system setup? - Version of systemd - Are you starting something in parallel to systemd which might take significant IO? - Version of the kernel - The kernel ticker frequency - The enabled cgroups controllers The last three points might sound a bit far away, but I've an idea in mind ... Best regards Marko Hoyer Software Group II (ADITG/SW2) Tel. +49 5121 49 6948 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
