Hi,
Good pointers in this thread already. Here are mine:
* Share sstate mirror and download cache from release builds to
developer topic builds. NFS, web server or rsync before calling bitbake
will work.
* I've added buildhistory and prserv database as extra files to sstate mirror
and use that to initiate new developer topic and release builds. This way
we don't add prserv or buildhistory git trees to critical path in builds
but get the benefits of QA checks, binary package versions, full history etc.
* Don't use virtual machines or clouds to build. Bare metal throw away machines
are much faster and more reliable. We've broken all clouds.
* Use rm_work to reduce disk space usage during builds.
* Tune build machines to bind things into memory and to not flush things to
disk
all the time since bitbake tmp, images etc are anyways going to be tar'ed
as build output. If they fit to page cache in RAM, you can avoid a lot of IO
and
save disks/ssd. Linux kernel vm tuning does this:
$ cat /etc/sysctl.d/99-build_server_fs_ops_to_memory.conf
# fs cache can use 90% of memory before system starts io to disk,
# keep as much as possible in RAM
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 90
# keep stuff for 12h in memory before writing to disk,
# allows reusing data as much as possible between builds
vm.dirty_expire_centisecs = 4320000
vm.dirtytime_expire_seconds = 432000
# allow single process to use 60% of system RAM for file caches, e.g. image
build
vm.dirty_bytes = 0
vm.dirty_ratio = 60
# disable periodic background writes, only write when running out of RAM
vm.dirty_writeback_centisecs = 0
* Finding optimal cost and power combination for build slaves is tricky.
Track CPU, memory, IO and network usage for your project and find out which
one is the bottle neck. For us it was RAM. CPUs are not effectly used by
bitbake
builds except when all hell breaks loose with C++ projects and their
templates.
Lots of CPU time is wasted when running single threaded bitbake tasks and
creating images. Avoiding IO to disk and caching to RAM helps. I've not seen
benefits
of having more than 64 gigs of RAM or more than 32 CPUs (with hyper
threading).
Also project evolve over time and suddenly may start eating more RAM and
triggering
the kernel OOM killer, shivers..
Hope this helps,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#48509): https://lists.yoctoproject.org/g/yocto/message/48509
Mute This Topic: https://lists.yoctoproject.org/mt/71347835/21656
Mute #yocto: https://lists.yoctoproject.org/mk?hashtag=yocto&subid=6691583
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
