Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
On Wed, 2021-04-07 at 22:15 +0200, k...@aspodata.se wrote: > > yea, it was a try to make c++ programs to behave under changing > compiler and library situations. Seems that some such programs > don't want to be built statically so they break whenever some "random" > lib changes. > That's my best guess as to why your build is using so much more RAM. Every library with a static-libs USE flag will have a bunch of other libraries "copy & pasted" into it, and all of those copies take up their own separate chunk of RAM. Personally, I would disable static-libs. Within Gentoo the flag is mostly there because GNU autotools makes it easy to do -- not because it's a well-thought-out and reliable feature. But if it solves a more- annoying problem for you, then maybe it's worth it =) In that case lowering the -j flag for specific packages when you get an allocation failure is a reasonable workaround.
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
Michael: > On Wed, 2021-04-07 at 12:54 +0200, k...@aspodata.se wrote: > > # emerge -pv1 dev-lang/R > > ... > > [ebuild R] dev-lang/R-4.0.4::gentoo USE="... static-libs ..." > > Thanks, this is really good debugging information. Is that USE=static- > libs a global flag on your system? That may explain why your build is > using more memory than mine. yea, it was a try to make c++ programs to behave under changing compiler and library situations. Seems that some such programs don't want to be built statically so they break whenever some "random" lib changes. Regards, /Karl Hammar
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
On Wed, 2021-04-07 at 12:54 +0200, k...@aspodata.se wrote: > # emerge -pv1 dev-lang/R > ... > [ebuild R] dev-lang/R-4.0.4::gentoo USE="... static-libs ..." Thanks, this is really good debugging information. Is that USE=static- libs a global flag on your system? That may explain why your build is using more memory than mine.
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
Michael: ... > What USE flags does "emerge -pv1 dev-lang/R" show? # emerge -pv1 dev-lang/R ... [ebuild R] dev-lang/R-4.0.4::gentoo USE="X jpeg nls openmp perl png readline static-libs tiff -cairo -doc -icu -java -lapack -minimal (-prefix) -profile -test -tk" 0 KiB > Are you willing to watch "htop" in another window while the dev-lang/R > build fails to see how much memory is free? htop -u portage shows high cpu usage and low memory usage (something like less than 2%). if I use this instead of htop: # grep -m1 ^MemFree /proc/meminfo MemFree: 253332 kB I get for -j1: mean = 397662.00592 min = 196772 max = 762304 std = 161394.91895 I have put up data from some -j4 runs at: http://aspodata.se/tmp/R/ where mem4* are made with ./memwatch.pl > mem4_.log. It's columns are just seconds, MemFree, and MemAvailable, taken from: $ head -3 /proc/meminfo MemTotal:6103628 kB MemFree: 1028012 kB MemAvailable:2326588 kB /// from mem4_.log, I see that the minimum free memory is: run 1 203492 run 2 202392 i.e. about 200MB. build_1.log points the finger to sandbox. So it might be that there i no problem at all with R, it might be that sandbox doesn't work here. I don't know what sandbox requires from the kernel so there might be some syscall missing. I did run 2 as FEATURES="-sandbox -usersandbox" emerge -aqv1 dev-lang/R In build_2.log it just says Segment fault in the vincinity of NEWS.2.rds, NEWS.rds, base.Rdts, and methods.Rdts. The first two is mentioned in doc/Makefile, and the latter two in src/library/Makefile. Going into doc and src/library and makeing thoose four works without any errors. Regards, /Karl Hammar
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
On Mon, 2021-04-05 at 18:29 +0200, k...@aspodata.se wrote: > > So, how much memory does R need to build ? > I have: > > $ free > totalusedfree shared buff/cache > available > Mem: 6103628 1807560 2631444 146376 1664624 > 3107868 > Swap:8788924 670108 8118816 > We have the same amount of RAM, but building dev-lang/R doesn't come close to using it all on my machine, even with MAKEOPTS="-j4"... > > I already have (in make.conf): > EMERGE_DEFAULT_OPTS="--jobs 1 ..." > > This seems to solved the issue: > # cat /etc/portage/env/j1 > MAKEOPTS=-j1 > # cat /etc/portage/package.env/R > dev-lang/R j1 That will work around the issue, and you may have to use that trick for some other packages like firefox/LLVM with "only" 6GB of RAM. But it doesn't explain why I can build dev-lang/R and you can't. What USE flags does "emerge -pv1 dev-lang/R" show? Are you willing to watch "htop" in another window while the dev-lang/R build fails to see how much memory is free?
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
Michael: > On Mon, 2021-04-05 at 16:10 +0200, k...@aspodata.se wrote: > > system call failed: Cannot allocate memory > > Segmentation fault > > Is this a low-memory machine? So, how much memory does R need to build ? I have: $ free totalusedfree shared buff/cache available Mem: 6103628 1807560 2631444 146376 1664624 3107868 Swap:8788924 670108 8118816 /// Somewhere in the log it says that something failed to get a sigaltstack(). Did the R devs anticipate out of memory and sigseg and tried to handle it ? > If so, there's no much you can do here > except set a lower number of jobs in MAKEOPTS for the dev-lang/R build: > > https://wiki.gentoo.org/wiki//etc/portage/package.env I already have (in make.conf): EMERGE_DEFAULT_OPTS="--jobs 1 ..." This seems to solved the issue: # cat /etc/portage/env/j1 MAKEOPTS=-j1 # cat /etc/portage/package.env/R dev-lang/R j1 > But if you don't think you're running out of RAM, we can keep digging. > For example you might have some memory limits set in /etc/limits that > you're hitting. I don't have an /etc/limits, and /etc/security/limits.conf is just comments. # prlimit -d -l -s RESOURCE DESCRIPTION SOFT HARD UNITS DATA max data size unlimited unlimited bytes MEMLOCK max locked-in-memory address space unlimited unlimited bytes STACKmax stack size 8388608 unlimited bytes Regards, /Karl Hammar
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
On Mon, 2021-04-05 at 16:10 +0200, k...@aspodata.se wrote: > system call failed: Cannot allocate memory > Segmentation fault Is this a low-memory machine? If so, there's no much you can do here except set a lower number of jobs in MAKEOPTS for the dev-lang/R build: https://wiki.gentoo.org/wiki//etc/portage/package.env But if you don't think you're running out of RAM, we can keep digging. For example you might have some memory limits set in /etc/limits that you're hitting.
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
> On Mon, 2021-04-05 at 00:44 +0200, k...@aspodata.se wrote: ... > > $ cp -a ... > > $ cd ... > > $ make -j1 > > ... > > $ echo $? > > 0 > > > > It builds without failure in that case. > > > > That's (potentially) good news. Can you now try it with the > CFLAGS/CXXFLAGS and LDFLAGS from your `emerge --info`? If the build > fails in that case, then we know it's one of those flags causing a > problem. It seems -j1 succeeds where -j4 fails, and it might be memory related. /// # make clean # make does fail # make clean # ./configure ... R is now configured for x86_64-pc-linux-gnu Source directory:. Installation directory: /usr/local C compiler: gcc -g -O2 Fortran fixed-form compiler: gfortran -fno-optimize-sibling-calls -g -O2 Default C++ compiler:g++ -std=gnu++11 -g -O2 C++14 compiler: g++ -std=gnu++14 -g -O2 C++17 compiler: g++ -std=gnu++17 -g -O2 C++20 compiler: Fortran free-form compiler: gfortran -fno-optimize-sibling-calls -g -O2 Obj-C compiler: gcc -g -O2 -fobjc-exceptions Interfaces supported:X11, tcltk External libraries: pcre2, readline, curl Additional capabilities: PNG, JPEG, TIFF, NLS, cairo, ICU Options enabled: shared BLAS, R profiling Capabilities skipped: Options not enabled: memory profiling Recommended packages:yes # make ok (=succeeds) adding (as environment variables) CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe" make clean; make ok CHOST="x86_64-pc-linux-gnu" CXXFLAGS="-O2 -pipe" ok FCFLAGS="-O2 -pipe" FFLAGS="-O2 -pipe" ok LDFLAGS="-Wl,-O1 -Wl,--as-needed" ok From /portage/dev-lang/R-4.0.4/work/R-4.0.4/config.log # make distclean # ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --docdir=/usr/share/doc/R-4.0.4 --htmldir=/usr/share/doc/R-4.0.4/html --with-sysroot=/ --libdir=/usr/lib64 --enable-byte-compiled-packages --enable-R-shlib --disable-R-framework --with-blas=-lblas rdocdir=/usr/share/doc/R-4.0.4 --disable-java --enable-nls --enable-openmp --disable-R-profiling --disable-memory-profiling --enable-static --enable-R-static-lib --without-cairo --without-ICU --with-jpeglib --without-lapack --with-recommended-packages --with-libpng --with-readline --with-libtiff --without-tcltk --without-tk-config --without-tcl-config --with-x ... R is now configured for x86_64-pc-linux-gnu Source directory:. Installation directory: /usr C compiler: x86_64-pc-linux-gnu-gcc -g -O2 Fortran fixed-form compiler: x86_64-pc-linux-gnu-gfortran -fno-optimize-sibling-calls -g -O2 Default C++ compiler:x86_64-pc-linux-gnu-g++ -std=gnu++11 -g -O2 C++14 compiler: x86_64-pc-linux-gnu-g++ -std=gnu++14 -g -O2 C++17 compiler: x86_64-pc-linux-gnu-g++ -std=gnu++17 -g -O2 C++20 compiler: Fortran free-form compiler: x86_64-pc-linux-gnu-gfortran -fno-optimize-sibling-calls -g -O2 Obj-C compiler: x86_64-pc-linux-gnu-gcc -g -O2 -fobjc-exceptions Interfaces supported:X11 External libraries: pcre2, readline, BLAS(generic), curl Additional capabilities: PNG, JPEG, TIFF, NLS Options enabled: shared R library Capabilities skipped:cairo, ICU Options not enabled: shared BLAS, R profiling, memory profiling Recommended packages:yes # make ok add CFLAGS="-O2 -pipe" CXXFLAGS="-O2 -pipe" FCFLAGS="-O2 -pipe" FFLAGS="-O2 -pipe" LDFLAGS="-Wl,-O1 -Wl,--as-needed" # make distclean # ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --docdir=/usr/share/doc/R-4.0.4 --htmldir=/usr/share/doc/R-4.0.4/html --with-sysroot=/ --libdir=/usr/lib64 --enable-byte-compiled-packages --enable-R-shlib --disable-R-framework --with-blas=-lblas rdocdir=/usr/share/doc/R-4.0.4 --disable-java --enable-nls --enable-openmp --disable-R-profiling --disable-memory-profiling --enable-static --enable-R-static-lib --without-cairo --without-ICU --with-jpeglib --without-lapack --with-recommended-packages --with-libpng --with-readline --with-libtiff --without-tcltk --without-tk-config --without-tcl-config --with-x CFLAGS="-O2 -pipe" CXXFLAGS="-O2 -pipe" FCFLAGS="-O2 -pipe" FFLAGS="-O2 -pipe" LDFLAGS="-Wl,-O1 -Wl,--as-needed" ... R is now configured for x86_64-pc-linux-gnu Source directory:. Installation directory: /usr C compiler: x86_64-pc-linux-gnu-gcc -O2 -pipe Fortran fixed-form compiler: x86_64-pc-linux-gnu-gfortran -fno-optimize-sibling-calls -O2 -pipe Default C++ compiler:
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
On 4/4/21 6:44 PM, k...@aspodata.se wrote: Jack: On 2021.04.03 07:15, k...@aspodata.se wrote: ... make[1]: *** Waiting for unfinished jobs Try running with -j1 ... I already have: EMERGE_DEFAULT_OPTS="--jobs 1 ... in make.conf. Don't know why it still wants to run in parallel. --jobs=1 tells emerge to only emerge one package at a time. The -j 1 (actually needs to be part of 'MAKEOPTS="-j1 "' is passed in to make to tell it to only compile one piece at a time. Using a higher number for -j sometimes fails if it needs some intermediate result that has not yet been produced. Sometimes that failure is consistently repeatable, and sometimes not.
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
On Mon, 2021-04-05 at 00:44 +0200, k...@aspodata.se wrote: > Michael: > > $ cp -a ... > $ cd ... > $ make -j1 > ... > $ echo $? > 0 > > It builds without failure in that case. > That's (potentially) good news. Can you now try it with the CFLAGS/CXXFLAGS and LDFLAGS from your `emerge --info`? If the build fails in that case, then we know it's one of those flags causing a problem.
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
Michael: > On Sat, 2021-04-03 at 13:15 +0200, k...@aspodata.se wrote: > > Is there a way to run emerge step by step to find out why it fails ? > > > > No easy way. You can `cp -a` the source/build directories out of > /var/tmp/portage and then re-run `make -j`. That should re- > start the build more or less where it failed. $ cp -a ... $ cd ... $ make -j1 ... $ echo $? 0 It builds without failure in that case. Regards, /Karl Hammar
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
Jack: > On 2021.04.03 07:15, k...@aspodata.se wrote: ... > > make[1]: *** Waiting for unfinished jobs > Try running with -j1 ... I already have: EMERGE_DEFAULT_OPTS="--jobs 1 ... in make.conf. Don't know why it still wants to run in parallell. Regards, /Karl Hammar
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
On Sat, 2021-04-03 at 13:15 +0200, k...@aspodata.se wrote: > Is there a way to run emerge step by step to find out why it fails ? > No easy way. You can `cp -a` the source/build directories out of /var/tmp/portage and then re-run `make -j`. That should re- start the build more or less where it failed. Feel free to file a bug with what you discover and CC me on it.
Re: [gentoo-user] emerge failure for dev-lang/R-4.0.4
On 2021.04.03 07:15, k...@aspodata.se wrote:
Is there a way to run emerge step by step to find out why it fails ?
///
I get this:
make[1]: Entering directory
'/Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4/doc'
make[1]: Leaving directory
'/Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4'
help2man: can't get `--version' info from ../bin/R
make[1]: *** [Makefile:46: R.1] Error 255
make[1]: *** Waiting for unfinished jobs
Try running with -j1
Error: package or namespace load failed for ?<80><98>methods?<80><99>:
.onLoad failed in loadNamespace() for 'methods', details:
call: (function (n)
error: lazy-load database
'/Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4/library/methods/R/methods.rdb'
is corrupt
During startup - Warning messages:
1: In (function (n) : internal error -4 in R_decompress1
2: package "methods" in options("defaultPackages") was not found
make[1]: Leaving directory
'/Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4/doc'
but ../bin/R --version works:
# cd /Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4/doc
# ../bin/R --version
WARNING: ignoring environment value of R_HOME
R version 4.0.4 (2021-02-15) -- "Lost Library Book"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
https://www.gnu.org/licenses/.
#
///
I also get segfault:
begin installing recommended package MASS
begin installing recommended package lattice
Segmentation fault
make[2]: *** [Makefile:53: MASS.ts] Error 1
make[2]: *** Waiting for unfinished jobs
* installing *source* package ?<80><98>codetools?<80><99> ...
** package ?<80><98>codetools?<80><99> successfully unpacked and MD5
sums checked
** using non-staged installation
///
Full build log is here:
http://aspodata.se/tmp/build.log
Regards,
/Karl Hammar
[gentoo-user] emerge failure for dev-lang/R-4.0.4
Is there a way to run emerge step by step to find out why it fails ?
///
I get this:
make[1]: Entering directory
'/Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4/doc'
make[1]: Leaving directory
'/Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4'
help2man: can't get `--version' info from ../bin/R
make[1]: *** [Makefile:46: R.1] Error 255
make[1]: *** Waiting for unfinished jobs
Error: package or namespace load failed for ?<80><98>methods?<80><99>:
.onLoad failed in loadNamespace() for 'methods', details:
call: (function (n)
error: lazy-load database
'/Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4/library/methods/R/methods.rdb'
is corrupt
During startup - Warning messages:
1: In (function (n) : internal error -4 in R_decompress1
2: package "methods" in options("defaultPackages") was not found
make[1]: Leaving directory
'/Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4/doc'
but ../bin/R --version works:
# cd /Net/gentoo/tmpdir/portage/dev-lang/R-4.0.4/work/R-4.0.4/doc
# ../bin/R --version
WARNING: ignoring environment value of R_HOME
R version 4.0.4 (2021-02-15) -- "Lost Library Book"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
https://www.gnu.org/licenses/.
#
///
I also get segfault:
begin installing recommended package MASS
begin installing recommended package lattice
Segmentation fault
make[2]: *** [Makefile:53: MASS.ts] Error 1
make[2]: *** Waiting for unfinished jobs
* installing *source* package ?<80><98>codetools?<80><99> ...
** package ?<80><98>codetools?<80><99> successfully unpacked and MD5 sums
checked
** using non-staged installation
///
Full build log is here:
http://aspodata.se/tmp/build.log
Regards,
/Karl Hammar
