On 09/16/2018 04:40 PM, Peter Bergin wrote:
Hi,
during the task do_package_write_rpm I get the error "liblzma: Memory
allocation failed". It happens during packaging of binary RPM packages.
The root cause seems to be the host environment that is used in our
project. We run our builds on a big server with 32 cores and 256GB of
physical RAM but each user has a limit of virtual memory usage to 32GB
(ulimit -v). The packaging in rpm-native has been parallelized in the
commit
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3.
What seems to happen is that rpm-native put up 32 parallel tasks with
'#pragma omp', each task is using liblzma that also put up 32 tasks for
#pragma omp
Tha'ts OpenMP, right? I haven't played with that at all but
it looks like you can limit the number of threads using an
environment variable:
OMP_NUM_THREADS num
https://www.openmp.org/wp-content/uploads/OpenMP3.0-SummarySpec.pdf
Doing that would be a little ugly but for now at least, there doesn't
seem to be that many packages using such a pragma.
Does that work for your case?
the compression work. The memory calculations in liblzma is based on the
amount of physical RAM but as the user is limited by 'ulimit -v' we get
into a OOM situation in liblzma.
Here is the code snippet from rpm-native/build/pack.c where it happens:
#pragma omp parallel
#pragma omp single
// re-declaring task variable is necessary, or older gcc versions
will produce code that segfaults
for (struct binaryPackageTaskData *task = tasks; task != NULL; task
= task->next) {
if (task != tasks)
#pragma omp task
{
task->result = packageBinary(spec, task->pkg, cookie,
cheating, &(task->filename), buildTime, buildHost);
rpmlog(RPMLOG_NOTICE, _("Finished binary package job,
result %d, filename %s\n"), task->result, task->filename);
}
}
Steps to reproduce is to set 'ulimit -v' in your shell to, for example,
1/8 of the amount of physical RAM and then build for example
glibc-locale. I have tested this with rocko. If the '#pragma omp'
statements in code snippet above is removed the problem is solved. But
that not good as the parallel processing speed up the process.
Is the host environment used here with restricted virtual memory
supported by Yocto? If it is, someone that have any suggestion for a
solution on this issue?
This is a little tricky.
From bitbake's point of view, it's almost like you are building
on a 32 core, 32 GB box and runing out of RAM/swap.
Clearly we would not fix a build that OOMs in that case
(it does seem odd that 32 GB isn't enough ...)
Are you sure that there isn't something else going on?
I have a 24 core machine with 64 GB RAM that never comes
close to such a problem (so I haven't paid attention to RAM usage).
../Randy
Best regards,
/Peter
--
# Randy MacLeod
# Wind River Linux
--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto
