Re: [systemd-devel] Grouping services in systemd..
We had a similar problem and we solved it by using grouping for critical services and then using startup cpu shares for services that should be responsive within that group. Even if you use startup cpu shares and create a target for everything you would want to boot, some unnecessary services will get the CPU due to CFS. Them getting CPU is by itself a problem, also they would cause a context switch with memory caches being flushed. To avoid this, we have created a target for our critical services and their dependencies. We had yet another service inside this critical group which told systemd to kick off multi-user.target when critical services were up. We didn't use startup IO shares because we are based on ubifs which back then didn't have IO scheduler support. Hope it gives some inspiration. Umut On Thu, Apr 2, 2020 at 3:21 PM nitish nagesh wrote: > Hi folks, > > We are working on an embedded ARM Cortex A9 based system (aka low CPU). > It runs on a custom linux based operating system which uses systemd. > > We have a bunch of daemons (around ~50+) that come up during boot > simultaneously which slows down the boot significantly as the CPU runs out > of breath. We were thinking of staggering these daemons into 2 groups. The > first group containing "critical" daemons (around 15) so that they finish > faster and make the system usable sooner. Followed by the second group of > daemons. > >Separating daemons into buckets could be done using: > 1) systemd targets: Introduce 2 new targets and classify the > services/daemons into them. Layer these targets during boot. > 2) Cgroups: Create a new systemd slice and put all the "critical" services > into it. Allocate sufficient CPUShares value to the slice so that this > slice gets its due CPU% to finish faster boot. > > Can you please suggest which of the above is a better approach? Respective > pros/cons with each. > > Or if there is a third approach better than the above? > > Thanks in advance, > ___ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Grouping services in systemd..
Hi. (Not sure if it's still pertinent.) On Tue, Apr 07, 2020 at 10:23:22AM +0530, nitish nagesh wrote: > In fact i did try a similar approach of assigning CPUShares to a slice. > Basically i separated these critical services into a new slice & > assigned a CPUShare=8192. > However with this i see it takes more time than before to complete the > boot. Note that if you didn't use cpu controller for anything else before, you introduced whole new grouping of tasks (i.e. imagine all were in -.slice previously), that may affect timing interactions. Furthermore, group scheduling works relative to siblings, i.e. system-netns.slice would be prioritized against siblings in in system.slice only. Finally, when prioritizing certain services you may have taken CPU time from other services that are dependencies (priority inversion). It makes sense to prioritize services on critical path but if there's nothing else to prioritize against, that's just the amount of work that has to be done anyway as others pointed out. > So by setting it to 8192 am I reducing the CPUShare and hence seeing > an increase in time? No, this is a special value that represents unset CPUShare= internally. Such units would still apply 1024 cpu.shares. > The DefaultCPUAccounting also seems to be enabled for the system. Aha, then my point about cpu controller regrouping is moot probably. HTH, Michal signature.asc Description: Digital signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Grouping services in systemd..
Hi, In fact i did try a similar approach of assigning CPUShares to a slice. Basically i separated these critical services into a new slice & assigned a CPUShare=8192. However with this i see it takes more time than before to complete the boot. One related observation was, on my system the default value of CPUShares was not 1024 (as mentioned in the man-pages). Instead its a large number. *# systemctl show system-netns.slice | grep -i cpuCPUUsageNSec=192379569CPUAccounting=yesCPUShares=18446744073709551615StartupCPUShares=18446744073709551615CPUQuotaPerSecUSec=infinity * So by setting it to 8192 am I reducing the CPUShare and hence seeing an increase in time? I tried the same with StartupCPUShares but observations were similar. The DefaultCPUAccounting also seems to be enabled for the system. Am i doing something incorrectly here? Please help. This is the systemd version is use: # systemctl --version systemd 229 +PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN Thanks, On Thu, Apr 2, 2020 at 8:00 PM Lennart Poettering wrote: > On Do, 02.04.20 18:51, nitish nagesh (nagesh.nit...@gmail.com) wrote: > > > Hi folks, > > > > We are working on an embedded ARM Cortex A9 based system (aka low CPU). > > It runs on a custom linux based operating system which uses systemd. > > > > We have a bunch of daemons (around ~50+) that come up during boot > > simultaneously which slows down the boot significantly as the CPU runs > out > > of breath. We were thinking of staggering these daemons into 2 groups. > The > > first group containing "critical" daemons (around 15) so that they finish > > faster and make the system usable sooner. Followed by the second group of > > daemons. > > > >Separating daemons into buckets could be done using: > > 1) systemd targets: Introduce 2 new targets and classify the > > services/daemons into them. Layer these targets during boot. > > 2) Cgroups: Create a new systemd slice and put all the "critical" > services > > into it. Allocate sufficient CPUShares value to the slice so that this > > slice gets its due CPU% to finish faster boot. > > > > Can you please suggest which of the above is a better approach? > Respective > > pros/cons with each. > > > > Or if there is a third approach better than the above? > > > > Thanks in advance, > > StartupCPUWeight= and StartupIOWeight= are probably what you should be > using? Have you played around with that? That passes this problem on > to the CPU/IO schedulers of the kernels. i.e. you still enqueue > everything in parallel, but tell the kernel what to schedule first. > > In recent systemd versions the weights configured this way also affect > the order in which jobs are dispatched by systemd itself if multiple > are runnable at the same time. > > Lennart > > -- > Lennart Poettering, Berlin > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Grouping services in systemd..
On Do, 02.04.20 18:51, nitish nagesh (nagesh.nit...@gmail.com) wrote: > Hi folks, > > We are working on an embedded ARM Cortex A9 based system (aka low CPU). > It runs on a custom linux based operating system which uses systemd. > > We have a bunch of daemons (around ~50+) that come up during boot > simultaneously which slows down the boot significantly as the CPU runs out > of breath. We were thinking of staggering these daemons into 2 groups. The > first group containing "critical" daemons (around 15) so that they finish > faster and make the system usable sooner. Followed by the second group of > daemons. > >Separating daemons into buckets could be done using: > 1) systemd targets: Introduce 2 new targets and classify the > services/daemons into them. Layer these targets during boot. > 2) Cgroups: Create a new systemd slice and put all the "critical" services > into it. Allocate sufficient CPUShares value to the slice so that this > slice gets its due CPU% to finish faster boot. > > Can you please suggest which of the above is a better approach? Respective > pros/cons with each. > > Or if there is a third approach better than the above? > > Thanks in advance, StartupCPUWeight= and StartupIOWeight= are probably what you should be using? Have you played around with that? That passes this problem on to the CPU/IO schedulers of the kernels. i.e. you still enqueue everything in parallel, but tell the kernel what to schedule first. In recent systemd versions the weights configured this way also affect the order in which jobs are dispatched by systemd itself if multiple are runnable at the same time. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Grouping services in systemd..
Am 02.04.20 um 15:21 schrieb nitish nagesh: > We are working on an embedded ARM Cortex A9 based system (aka low > CPU). It runs on a custom linux based operating system which uses systemd. > > We have a bunch of daemons (around ~50+) that come up during boot > simultaneously which slows down the boot significantly as the CPU runs > out of breath. We were thinking of staggering these daemons into 2 > groups. The first group containing "critical" daemons (around 15) so > that they finish faster and make the system usable sooner. Followed by > the second group of daemons. > > Separating daemons into buckets could be done using: > 1) systemd targets: Introduce 2 new targets and classify the > services/daemons into them. Layer these targets during boot. > 2) Cgroups: Create a new systemd slice and put all the "critical" > services into it. Allocate sufficient CPUShares value to the slice so > that this slice gets its due CPU% to finish faster boot. > > Can you please suggest which of the above is a better approach? > Respective pros/cons with each. > > Or if there is a third approach better than the above? with proper After/Before ordering should work too don't apply for "Type=simple" and in that case targets won't help either because every "simple" service prevent waiting for something in the end i doubt slowing things down would solve anything! when stuff is needed it's needed and if not make it socket activated instead fire everything up at boot ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Grouping services in systemd..
Hi folks, We are working on an embedded ARM Cortex A9 based system (aka low CPU). It runs on a custom linux based operating system which uses systemd. We have a bunch of daemons (around ~50+) that come up during boot simultaneously which slows down the boot significantly as the CPU runs out of breath. We were thinking of staggering these daemons into 2 groups. The first group containing "critical" daemons (around 15) so that they finish faster and make the system usable sooner. Followed by the second group of daemons. Separating daemons into buckets could be done using: 1) systemd targets: Introduce 2 new targets and classify the services/daemons into them. Layer these targets during boot. 2) Cgroups: Create a new systemd slice and put all the "critical" services into it. Allocate sufficient CPUShares value to the slice so that this slice gets its due CPU% to finish faster boot. Can you please suggest which of the above is a better approach? Respective pros/cons with each. Or if there is a third approach better than the above? Thanks in advance, ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
