On Thu, Sep 20, 2018 at 1:03 PM Peter Bergin <pe...@berginkonsult.se> wrote: > > On 2018-09-17 10:27, Burton, Ross wrote: > > On Mon, 17 Sep 2018 at 08:13, Peter Bergin <pe...@berginkonsult.se> wrote: > > I'm pretty sure I have narrowed down the root cause to the restriction > of virtual memory and that liblzma base its memory calculations on > physical RAM. > > To prove this I added a printout in rpm-native/rpmio/rpmio.c and the > function lzopen_internal. > > uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options); > rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage); > > > The value of memory_usage is the same regardless of which 'ulimit -v' > value I set. On the host with 256GB of physical RAM and 32GB of virtual > memory, memory_usage is ~5.1GB. On another host with 16GB of physical > RAM I get memory_usage of ~660MB. > > I guess you have not seen this kind of failure if you not have > restricted virutal memory on your host. If you want to try to reproduce > this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake > glibc-locale -c package_write_rpm -f'. > > Wouldn't a solution be to change lzma to look at free memory, not > total physical memory? > > Ross > > I have been in contact with the maintainer of liblzma. There is currently no > way to restrict the memory usage in liblzma during multi threaded > compression. He recommended to adjust the number of threads used during > compression. This kind of check is done for 32-bits but not when running > 64-bits (in rpmio/rpmio.c lzopen_internal). To rewrite liblzma is another > option but I took an easier way out. > > I have come up with a patch > (https://patchwork.openembedded.org/patch/155017/) that solves my problem and > do a restriction of memory usage when the 'ulimit -v' is set. The calculation > is based on the assumption that lzopen_internal is run in parallel with as > many instances as cpu threads as '#pragma omp parallel' is used in > build/pack.c. > > When running test on my machine with 4 cores 16GB of physical RAM and 'ulimit > -v 2097152' (2GB). It works good and the log shows: > > XZ: virtual memory restricted to 2147483648 and per CPU thread 536870912 > XZ: Adjusted the number of threads from 4 to 3 to not exceed the memory usage > limit of 2147483648 bytes > > Didn't get a clear answer if this is something Yocto/OE should support but I > hope my patch solves the issue and it will not affect the normal environments > where 'ulimit -v' is not set. > > /Peter > --
Peter, first of all sorry for butting-in again. My bad I misunderstood the point, Looking properly at it, I see the sources do restrict only #if __WORDSIZE == 32. Again, there is the other way to reduce total memory footprint just using lower preset in mt_options. As far as I see default is #define LZMA_PRESET_DEFAULT UINT32_C(6) so you could tr to modify mt_options.preset to your needs. I think more threads is normally better. Cheers Andrea > _______________________________________________ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto -- _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
