Hi Peter, > -----Original Message----- > From: yocto-boun...@yoctoproject.org [mailto:yocto-boun...@yoctoproject.org] > On Behalf Of Peter Bergin > Sent: Sunday, September 16, 2018 1:41 PM > To: yocto@yoctoproject.org > Subject: [yocto] liblzma: memory allocation failed > > Hi, > > during the task do_package_write_rpm I get the error "liblzma: Memory > allocation failed". It happens during packaging of binary RPM packages. > The root cause seems to be the host environment that is used in our > project. We run our builds on a big server with 32 cores and 256GB of > physical RAM but each user has a limit of virtual memory usage to 32GB > (ulimit -v). The packaging in rpm-native has been parallelized in the > commit > http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes- > devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3. > What seems to happen is that rpm-native put up 32 parallel tasks with > '#pragma omp', each task is using liblzma that also put up 32 tasks for > the compression work. The memory calculations in liblzma is based on the > amount of physical RAM but as the user is limited by 'ulimit -v' we get > into a OOM situation in liblzma. > > Here is the code snippet from rpm-native/build/pack.c where it happens: > > #pragma omp parallel > #pragma omp single > // re-declaring task variable is necessary, or older gcc versions will > produce code > that segfaults > for (struct binaryPackageTaskData *task = tasks; task != NULL; task = > task->next) { > if (task != tasks) > #pragma omp task > { > task->result = packageBinary(spec, task->pkg, cookie, cheating, > &(task- > >filename), buildTime, buildHost); > rpmlog(RPMLOG_NOTICE, _("Finished binary package job, result %d, > filename > %s\n"), task->result, task->filename); > } > } > > > Steps to reproduce is to set 'ulimit -v' in your shell to, for example, > 1/8 of the amount of physical RAM and then build for example > glibc-locale. I have tested this with rocko. If the '#pragma omp' > statements in code snippet above is removed the problem is solved. But > that not good as the parallel processing speed up the process. > > Is the host environment used here with restricted virtual memory > supported by Yocto? If it is, someone that have any suggestion for a > solution on this issue? >
We had seen this issue as well and concluded that some settings in the server was causing this. See http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146705.html Thanks, Manju -- _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto