Re: [Rd] R-4.2.3 build from source on Windows (w Rtools42) - lto1.exe error
On 5/11/23 18:26, W. D. wrote: Thanks to everybody for the reples. tl;dr: it was a config problem with my environment being polluted by other apps (also via $USERPROFILE/.bashrc). R-4-2-branch builds with LTO "march=native" (Rtools42, also managed to build with "mtune=native", which afaik is inherently included when doing march, but still). Details: I will try to aggregate answers to comments/hints here and document a sequentially processed train of action here including results as some form of documentation for current status and hickups from my setup described in this email chain. @Avraham, et al Since I wasn't sure if and how maybe lib updates I had done before starting to build had played a role I downloaded a fresh copy of the Rtools42 installer from here [https://cran.r-project.org/bin/windows/Rtools/rtools42/files/] current version via `cat /x86_64-w6cat /x86_64-w64-mingw32.static.posix/.version` gave "5355" Did some more "tests": `cd R-4-2-branch/src/gnuwin32` Ran build with"mtune=native" gave same error Ran `make clean` Then using no EOPTS options in MkRules.local with version "5355" -> same error Ran `make clean` --- Then ran an update via `pacman -Syuu` Which updates the following: Packages (55) brotli-1.0.9-8 bsdtar-3.6.2-3 ca-certificates-20230311-1 coreutils-8.32-5 curl-8.0.1-1 dash-0.5.12-1 db-5.3.28-4 diffutils-3.9-1 file-5.44-5 gawk-5.2.1-2 gcc-libs-11.3.0-3 gnupg-2.2.41-1 grep-1~3.0-6 heimdal-libs-7.8.0-3 info-7.0.3-1 less-633-1 libcrypt-2.1-4 libcurl-8.0.1-1 libdb-5.3.28-4 libedit-20221030_3.1-1 libexpat-2.5.0-1 libffi-3.4.4-1 libgcrypt-1.10.2-1 libgnutls-3.8.0-1 libgpg-error-1.47-1 libidn2-2.3.4-2 libksba-1.6.3-1 liblzma-5.4.3-1 libnghttp2-1.52.0-1 libopenssl-3.1.0-2 libpcre-8.45-3 libpsl-0.21.2-1 libreadline-8.2.001-3 libsqlite-3.41.2-3 libssh2-1.10.0-3 libunistring-1.1-2 libxml2-2.10.4-1 libzstd-1.5.5-1 make-4.4.1-1 mpfr-4.2.0.p4-1 msys2-keyring-1~20230316-1 ncurses-6.4-1 openssl-3.1.0-2 patch-2.7.6-2 perl-5.36.0-1 pinentry-1.2.1-1 rebase-4.5.0-4 rsync-3.2.7-2 sed-4.9-1 tcl-8.6.12-3 texinfo-7.0.3-1 texinfo-tex-7.0.3-1 xz-5.4.3-1 zlib-1.2.13-1 zstd-1.5.5-1 But what confuses me a bit is that after that ` cat /x86_64-w6cat /x86_64-w64-mingw32.static.posix/.version` still only gives me "5355"? As will be shown later (below) this was not a breaking issue. Rtools contains multiple separate things. It contains Msys2, which you can update using pacman and which has its own versioning. Msys2 is used for build tools and their dependencies (like make) - so e.g. pcre from above is used by some build tool, not linked to R packages. It also contains a pre-built compiler toolchain and libraries, which live in x86_64-w64-mingw32.static.posix and the .version file refers to that. You can update this part also, but differently. These libraries are linked to R packages, and you will find e.g. pcre there as well. Then Rtools also contains several additional tools used by packages when building/checking. These things are described in the documentation in detail, for R 4.2 (and Rtools42) in https://cran.r-project.org/bin/windows/base/howto-R-4.2.html Important change to my environment for the next attempt! I decided to check my ENV Variables (one more time) after thoroughly reading Tomas comment "maybe there is some config problem on the system" as well as Prof. Ripley's "first build without LTO to isolate the issue" and noticed that the old Rtools40/.../bin folder was also in my WINdows %PATH% ENV Variable. So I started a cleanup initiative from there. Also noticed that the .bashrc file in my %USERPROFILE% folder cluttered up my PATH especially there were `/mingw-64/bin` and `.../Library/bin` and similar entries from a miniconda3 installation amongst others ghosting around in there! Despite gcc --version or make commands not returning anything other than `command not found` I cleaned that .bashrc file up quite a bit. After cleanup I restarted MSYS2 Bash and ran `make distcelan` in the /src/gnuwin32 folder one more time then added miktex, /x86_64-w64-mingw32.static.posix/bin, tar, etc to $PATH or environment again. `echo $PATH` now (ater the extension of PATH var as dscibed in https://cran.r-project.org/bin/windows/base/howto-R-4.2.html) finally looked like this ``` /c/Users/gwd/AppData/Local/Programs/MiKTeX/miktex/bin/x64:/x86_64-w64-mingw32.static.posix/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/progra~1/git/cmd:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl ``` with all other Library, bin or similar (mainly miniconda3) folders gone now. and `which make gcc pdflatex tar` looks like this now (not much different to before, but ...) ``` make is /usr/bin/make make is /bin/make gcc is
Re: [Rd] R-4.2.3 build from source on Windows (w Rtools42) - lto1.exe error
Thanks to everybody for the reples. tl;dr: it was a config problem with my environment being polluted by other apps (also via $USERPROFILE/.bashrc). R-4-2-branch builds with LTO "march=native" (Rtools42, also managed to build with "mtune=native", which afaik is inherently included when doing march, but still). Details: I will try to aggregate answers to comments/hints here and document a sequentially processed train of action here including results as some form of documentation for current status and hickups from my setup described in this email chain. @Avraham, et al Since I wasn't sure if and how maybe lib updates I had done before starting to build had played a role I downloaded a fresh copy of the Rtools42 installer from here [https://cran.r-project.org/bin/windows/Rtools/rtools42/files/] current version via `cat /x86_64-w6cat /x86_64-w64-mingw32.static.posix/.version` gave "5355" Did some more "tests": `cd R-4-2-branch/src/gnuwin32` Ran build with"mtune=native" gave same error Ran `make clean` Then using no EOPTS options in MkRules.local with version "5355" -> same error Ran `make clean` --- Then ran an update via `pacman -Syuu` Which updates the following: Packages (55) brotli-1.0.9-8 bsdtar-3.6.2-3 ca-certificates-20230311-1 coreutils-8.32-5 curl-8.0.1-1 dash-0.5.12-1 db-5.3.28-4 diffutils-3.9-1 file-5.44-5 gawk-5.2.1-2 gcc-libs-11.3.0-3 gnupg-2.2.41-1 grep-1~3.0-6 heimdal-libs-7.8.0-3 info-7.0.3-1 less-633-1 libcrypt-2.1-4 libcurl-8.0.1-1 libdb-5.3.28-4 libedit-20221030_3.1-1 libexpat-2.5.0-1 libffi-3.4.4-1 libgcrypt-1.10.2-1 libgnutls-3.8.0-1 libgpg-error-1.47-1 libidn2-2.3.4-2 libksba-1.6.3-1 liblzma-5.4.3-1 libnghttp2-1.52.0-1 libopenssl-3.1.0-2 libpcre-8.45-3 libpsl-0.21.2-1 libreadline-8.2.001-3 libsqlite-3.41.2-3 libssh2-1.10.0-3 libunistring-1.1-2 libxml2-2.10.4-1 libzstd-1.5.5-1 make-4.4.1-1 mpfr-4.2.0.p4-1 msys2-keyring-1~20230316-1 ncurses-6.4-1 openssl-3.1.0-2 patch-2.7.6-2 perl-5.36.0-1 pinentry-1.2.1-1 rebase-4.5.0-4 rsync-3.2.7-2 sed-4.9-1 tcl-8.6.12-3 texinfo-7.0.3-1 texinfo-tex-7.0.3-1 xz-5.4.3-1 zlib-1.2.13-1 zstd-1.5.5-1 But what confuses me a bit is that after that ` cat /x86_64-w6cat /x86_64-w64-mingw32.static.posix/.version` still only gives me "5355"? As will be shown later (below) this was not a breaking issue. Important change to my environment for the next attempt! I decided to check my ENV Variables (one more time) after thoroughly reading Tomas comment "maybe there is some config problem on the system" as well as Prof. Ripley's "first build without LTO to isolate the issue" and noticed that the old Rtools40/.../bin folder was also in my WINdows %PATH% ENV Variable. So I started a cleanup initiative from there. Also noticed that the .bashrc file in my %USERPROFILE% folder cluttered up my PATH especially there were `/mingw-64/bin` and `.../Library/bin` and similar entries from a miniconda3 installation amongst others ghosting around in there! Despite gcc --version or make commands not returning anything other than `command not found` I cleaned that .bashrc file up quite a bit. After cleanup I restarted MSYS2 Bash and ran `make distcelan` in the /src/gnuwin32 folder one more time then added miktex, /x86_64-w64-mingw32.static.posix/bin, tar, etc to $PATH or environment again. `echo $PATH` now (ater the extension of PATH var as dscibed in https://cran.r-project.org/bin/windows/base/howto-R-4.2.html) finally looked like this ``` /c/Users/gwd/AppData/Local/Programs/MiKTeX/miktex/bin/x64:/x86_64-w64-mingw32.static.posix/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/c/progra~1/git/cmd:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl ``` with all other Library, bin or similar (mainly miniconda3) folders gone now. and `which make gcc pdflatex tar` looks like this now (not much different to before, but ...) ``` make is /usr/bin/make make is /bin/make gcc is /x86_64-w64-mingw32.static.posix/bin/gcc pdflatex is /c/Users/gwd/AppData/Local/Programs/MiKTeX/miktex/bin/x64/pdflatex tar is /usr/bin/tar tar is /bin/tar tar is /c/Windows/System32/tar ``` seems to look "better"! Then I started to build again ... which took me some time - that's why there's this delay in my answer here. .. First w/o any EOPTS -> succeeded! Then `make distclean` And then with march=native` [as well as mtune] -> succeeded! I will give the R-devel version with Rtools43 a shot as well (next) week - if you don't hear back from me -> you can assume that (also) worked correctly with LTO. So ... as is so often the case: "the problem was sitting in front of the PC". Thanks for the assistance and informative hints and sorry for bothering you all basically b/c of my (.bashrc) setup conundrum! Greetings, Walter On Thu, 11 May 2023 at 09:06, Tomas
Re: [Rd] R-4.2.3 build from source on Windows (w Rtools42) - lto1.exe error
On 5/11/23 03:07, Gmail wrote: Windows 11 PRO Version 10.0.22621 Build 22621 Processor: Intel(R) Core(TM) i7-1065G7 "Icelake-client" ``` svn info Path: . Working Copy Root Path: /d/R_DEV/R-4/R42/R-4-2-branch URL: https://svn.r-project.org/R/branches/R-4-2-branch Relative URL: ^/branches/R-4-2-branch Repository Root: https://svn.r-project.org/R Repository UUID: 00db46b3-68df-0310-9c12-caf00c1e9a41 Revision: 84417 Node Kind: directory Schedule: normal Last Changed Author: kalibera Last Changed Rev: 84249 Last Changed Date: 2023-04-13 07:12:24 + (Thu, 13 Apr 2023) ``` Only adaptation done in MkRules.local was adding: `EOPTS = -march=native` - that's why I included the cpu-type info above; running make all recommended fails at/with: ``` gcc -shared -s -static-libgcc -o utils.dll tmp.def init.o io.o size.o sock.o stubs.o utils.o hashtab.o windows/dataentry.o windows/dialogs.o windows/registry.o windows/util.o windows/widgets.o ../../../gnuwin32/dllversion.o -lRgraphapp -lversion -L/x86_64-w64-mingw32.static.posix/lib/x64 -llzma -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib -L../../../../bin/x64 -lR lto1.exe: fatal error: bytecode stream in file 'windows/dataentry.o' generated with LTO version 9.3 instead of the expected 9.4 compilation terminated. lto-wrapper.exe: fatal error: C:\rtools42\x86_64-w64-mingw32.static.posix\bin\gcc.exe returned 1 exit status compilation terminated. C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: error: lto-wrapper failed collect2.exe: error: ld returned 1 exit status cp: cannot stat 'utils.dll': No such file or directory make[4]: *** [Makefile.win:36: shlib] Error 1 make[3]: *** [../../../share/make/basepkg.mk:145: mksrc-win2] Error 1 make[2]: *** [Makefile.win:24: all] Error 2 make[1]: *** [Makefile.win:34: R] Error 1 make: *** [Makefile:18: all] Error 2 ``` Any hints/ideas on how to fix this? I guess I could gcc -c -flto ... windows/dataentry.c -o windows/dataentry.o with the exact path of that fiile ... All of the object files are generated on your system during the build. If dataentry.o is generated using an older version than other object files, maybe there is some configuration problem on the system. It might be worth checking the compilers and linkers from Rtools42 are used, and then running "make distclean", and then trying the build again. I never tried building R with LTO on Windows myself, I don't know if that works even on a system set up according to the documentation for R 4.2 (https://cran.r-project.org/bin/windows/base/howto-R-4.2.html). and it hopefully will fix that but I guess it would make sense to add a Revision to update that LTO version mismatch there, and I don't know yet if this is the only one? Perhaps it is better to use Rtools43 and R-devel if you can, so that if you find some problem in either, it can still be fixed. Greetings, W Best, Tomas __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] R-4.2.3 build from source on Windows (w Rtools42) - lto1.exe error
Initial comments - R 4.2.3 is not current - -flto does not seem to be the default in src/gnuwin32/MkRules. - LTO versions in GCC are tied to the compiler version, and in recent GCC are the same as the compiler version. The recommended toolchain for R 4.2.x is Rtools42 which according to NEWS is based on GCC 10 not 9. - LTO mismatches in my experience are most often seen in incremental builds, so first do an ab initio build. (Not so long ago they were not detected during linking but gave segfaults.) Assuming it is important to use LTO, I would first build without LTO to isolate the issue. And be aware of the following in the NEWS for R 4.3.0: • The Rcomplex definition (in header R_ext/Complex.h) has been extended to prevent possible mis-compilation when interfacing with Fortran (PR#18430). AFAIR that "possible mis-compilation" is most likely using LTO. On 11/05/2023 02:07, Gmail wrote: Windows 11 PRO Version 10.0.22621 Build 22621 Processor: Intel(R) Core(TM) i7-1065G7 "Icelake-client" ``` svn info Path: . Working Copy Root Path: /d/R_DEV/R-4/R42/R-4-2-branch URL: https://svn.r-project.org/R/branches/R-4-2-branch Relative URL: ^/branches/R-4-2-branch Repository Root: https://svn.r-project.org/R Repository UUID: 00db46b3-68df-0310-9c12-caf00c1e9a41 Revision: 84417 Node Kind: directory Schedule: normal Last Changed Author: kalibera Last Changed Rev: 84249 Last Changed Date: 2023-04-13 07:12:24 + (Thu, 13 Apr 2023) ``` Only adaptation done in MkRules.local was adding: `EOPTS = -march=native` - that's why I included the cpu-type info above; running make all recommended fails at/with: ``` gcc -shared -s -static-libgcc -o utils.dll tmp.def init.o io.o size.o sock.o stubs.o utils.o hashtab.o windows/dataentry.o windows/dialogs.o windows/registry.o windows/util.o windows/widgets.o ../../../gnuwin32/dllversion.o -lRgraphapp -lversion -L/x86_64-w64-mingw32.static.posix/lib/x64 -llzma -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib -L../../../../bin/x64 -lR lto1.exe: fatal error: bytecode stream in file 'windows/dataentry.o' generated with LTO version 9.3 instead of the expected 9.4 compilation terminated. lto-wrapper.exe: fatal error: C:\rtools42\x86_64-w64-mingw32.static.posix\bin\gcc.exe returned 1 exit status compilation terminated. C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: error: lto-wrapper failed collect2.exe: error: ld returned 1 exit status cp: cannot stat 'utils.dll': No such file or directory make[4]: *** [Makefile.win:36: shlib] Error 1 make[3]: *** [../../../share/make/basepkg.mk:145: mksrc-win2] Error 1 make[2]: *** [Makefile.win:24: all] Error 2 make[1]: *** [Makefile.win:34: R] Error 1 make: *** [Makefile:18: all] Error 2 ``` Any hints/ideas on how to fix this? I guess I could gcc -c -flto ... windows/dataentry.c -o windows/dataentry.o with the exact path of that fiile ... and it hopefully will fix that but I guess it would make sense to add a Revision to update that LTO version mismatch there, and I don't know yet if this is the only one? Greetings, W __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel -- Brian D. Ripley, rip...@stats.ox.ac.uk Emeritus Professor of Applied Statistics, University of Oxford __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] R-4.2.3 build from source on Windows (w Rtools42) - lto1.exe error
Hi. Are you sure you have the most up-to-date version of the libraries? It should be version 5550. Tomas updates the entire toolchain semi-regularly. If you call “ cat /x86_64-w64-mingw32.static.posix/.version” and it’s not 5550, try updating the libraries as per https://cran.r-project.org/bin/windows/base/howto-R-devel.html. Hope that helps, Avi Sent from my iPhone > On May 10, 2023, at 9:07 PM, Gmail wrote: > > Windows 11 PRO Version 10.0.22621 Build 22621 > Processor: Intel(R) Core(TM) i7-1065G7 "Icelake-client" > ``` > svn info > Path: . > Working Copy Root Path: /d/R_DEV/R-4/R42/R-4-2-branch > URL: https://svn.r-project.org/R/branches/R-4-2-branch > Relative URL: ^/branches/R-4-2-branch > Repository Root: https://svn.r-project.org/R > Repository UUID: 00db46b3-68df-0310-9c12-caf00c1e9a41 > Revision: 84417 > Node Kind: directory > Schedule: normal > Last Changed Author: kalibera > Last Changed Rev: 84249 > Last Changed Date: 2023-04-13 07:12:24 + (Thu, 13 Apr 2023) > ``` > > Only adaptation done in MkRules.local was adding: `EOPTS = -march=native` - > that's why I included the cpu-type info above; > running make all recommended fails at/with: > ``` > gcc -shared -s -static-libgcc -o utils.dll tmp.def init.o io.o size.o sock.o > stubs.o utils.o hashtab.o windows/dataentry.o windows/dialogs.o > windows/registry.o windows/util.o windows/widgets.o > ../../../gnuwin32/dllversion.o -lRgraphapp -lversion > -L/x86_64-w64-mingw32.static.posix/lib/x64 -llzma > -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib/x64 > -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib -L../../../../bin/x64 -lR > > lto1.exe: fatal error: bytecode stream in file 'windows/dataentry.o' > generated with LTO version 9.3 instead of the expected 9.4 > compilation terminated. > > lto-wrapper.exe: fatal error: > C:\rtools42\x86_64-w64-mingw32.static.posix\bin\gcc.exe returned 1 exit status > compilation terminated. > C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: error: lto-wrapper > failed > collect2.exe: error: ld returned 1 exit status > cp: cannot stat 'utils.dll': No such file or directory > make[4]: *** [Makefile.win:36: shlib] Error 1 > make[3]: *** [../../../share/make/basepkg.mk:145: mksrc-win2] Error 1 > make[2]: *** [Makefile.win:24: all] Error 2 > make[1]: *** [Makefile.win:34: R] Error 1 > make: *** [Makefile:18: all] Error 2 > ``` > Any hints/ideas on how to fix this? I guess I could > gcc -c -flto ... windows/dataentry.c -o windows/dataentry.o > with the exact path of that fiile ... > and it hopefully will fix that but I guess it would make sense to add a > Revision to update that LTO version mismatch there, and I don't know yet if > this is the only one? > > Greetings, > W > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] R-4.2.3 build from source on Windows (w Rtools42) - lto1.exe error
Windows 11 PRO Version 10.0.22621 Build 22621 Processor: Intel(R) Core(TM) i7-1065G7 "Icelake-client" ``` svn info Path: . Working Copy Root Path: /d/R_DEV/R-4/R42/R-4-2-branch URL: https://svn.r-project.org/R/branches/R-4-2-branch Relative URL: ^/branches/R-4-2-branch Repository Root: https://svn.r-project.org/R Repository UUID: 00db46b3-68df-0310-9c12-caf00c1e9a41 Revision: 84417 Node Kind: directory Schedule: normal Last Changed Author: kalibera Last Changed Rev: 84249 Last Changed Date: 2023-04-13 07:12:24 + (Thu, 13 Apr 2023) ``` Only adaptation done in MkRules.local was adding: `EOPTS = -march=native` - that's why I included the cpu-type info above; running make all recommended fails at/with: ``` gcc -shared -s -static-libgcc -o utils.dll tmp.def init.o io.o size.o sock.o stubs.o utils.o hashtab.o windows/dataentry.o windows/dialogs.o windows/registry.o windows/util.o windows/widgets.o ../../../gnuwin32/dllversion.o -lRgraphapp -lversion -L/x86_64-w64-mingw32.static.posix/lib/x64 -llzma -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools42/x86_64-w64-mingw32.static.posix/lib -L../../../../bin/x64 -lR lto1.exe: fatal error: bytecode stream in file 'windows/dataentry.o' generated with LTO version 9.3 instead of the expected 9.4 compilation terminated. lto-wrapper.exe: fatal error: C:\rtools42\x86_64-w64-mingw32.static.posix\bin\gcc.exe returned 1 exit status compilation terminated. C:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: error: lto-wrapper failed collect2.exe: error: ld returned 1 exit status cp: cannot stat 'utils.dll': No such file or directory make[4]: *** [Makefile.win:36: shlib] Error 1 make[3]: *** [../../../share/make/basepkg.mk:145: mksrc-win2] Error 1 make[2]: *** [Makefile.win:24: all] Error 2 make[1]: *** [Makefile.win:34: R] Error 1 make: *** [Makefile:18: all] Error 2 ``` Any hints/ideas on how to fix this? I guess I could gcc -c -flto ... windows/dataentry.c -o windows/dataentry.o with the exact path of that fiile ... and it hopefully will fix that but I guess it would make sense to add a Revision to update that LTO version mismatch there, and I don't know yet if this is the only one? Greetings, W __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
