Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
On 10/12/2013 07:43 PM, Anton Gladky wrote:
Thanks all, guys, for recommendations and explanations. I will try to split
some large cpp-files as Dmitrijs mentioned (there are a couple of them,
which are failing reliably on weak archs), remove some optimization options
as Mathieu advised and we will see, whether it helps.
As I have a similar problem with gmic, here is my workaround in debian/rules,
maybe its helpful for you:
BUILDD_MEM := $(shell awk '/^MemTotal:/ {print $$2}' /proc/meminfo)
BUILDD_MEM_OK := $(strip $(shell test $(BUILDD_MEM) -gt 180 echo yes))
BUILDD_SLOW_ARCH := $(strip $(shell dpkg-architecture -qDEB_BUILD_ARCH_CPU | \
grep -q -E '^(arm|mips|sh4|m68k).*' echo yes))
ifneq (yes,$(BUILDD_MEM_OK))
GMIC_CFLAGS = -O0 -g
else
ifneq (yes,$(BUILDD_SLOW_ARCH))
GMIC_CFLAGS = -O3 -g
else
GMIC_CFLAGS = -O1 -g -fno-tree-pre
endif
endif
Good luck,
Bernd
I would also try to build Yade with clang on failing archs, if upper
mentioned actions will not help to solve the problem. Though
openmp-parallelisation will be disabled in this case, ant it is not very
convenient.
Thanks again,
Anton
On 10/12/2013 02:34 PM, Mathieu Malaterre wrote:
Anton,
On Fri, Oct 11, 2013 at 9:00 PM, Anton Gladky gl...@debian.org wrote:
[...]
/usr/bin/c++ -Dyade_EXPORTS -g -O2 -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security
-D_FORTIFY_SOURCE=2 -DYADE_PTR_CAST=static_pointer_cast
-DYADE_CAST=static_cast -fPIC -DYADE_VTK -DYADE_OPENMP -fopenmp
-DYADE_GTS -DQGLVIEWER_FOUND -DYADE_OPENGL -frounding-math
-DYADE_CGAL -DFLOW_ENGINE -frounding-math -DLINSOLV -DFLOW_ENGINE
-DYADE_GL2PS -O3 -DNDEBUG -fPIC -
You are using cmake in Release mode, which is bad IMHO. So having -O2 and
-O3 (in that order) compiles code in -O3, which is way too much AFAIK.
Ref: http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html ... If you
use multiple -O options, with or without level numbers, the last such
option is the one that is effective. ...
See also here why CMAKE_BUILD_TYPE should be required to be set to on
debian: http://bugs.debian.org/701231#12
2cts
- --
Bernd ZeimetzDebian GNU/Linux Developer
http://bzed.dehttp://www.debian.org
GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/
iQIcBAEBCAAGBQJSWtwtAAoJEOs2Fxpv+UNfk/AP/2bOgmDo/AOEamZQorJe/Kns
i0+rsTnEZ69s5DJ0nt1zRMvKCMTNrfkcunxEwhjYEEsAqVHUXKl+hSS0i+s4zPbI
zYOo9DvrrTUrBEz1hqeOVJZxLXJrwS78+Y74sgn9ViBadNaz1nqGOE9/Uux3FDpp
memIwR2b+FRiPNvww3QZy0xG7nVwO91zpLobo1HpVod/XeQ3Xo5huX3LUsMlhGza
BzGPfvsb8m0a5QtNomndWLwakceD35LonqMHqTwzkTCEkFGs0FUes/fhkxey74Wq
PkuZfpplFSApvAFDn+xekHmFnTFdLtXNCneGIUo0S+FytyLniJ6lFSViNURtqFNC
iyrbY1gWKFOzcw42eUVUNlWRPL6fYdjnrnwWMnM6PTYnklY+H6s96RDUVzCOyEDv
z5p2r2RSuFstqUa12XZN7HyoV3XmMDJAm95CLxWL8RW3dUB0antKyKyO71BaJUhW
0q9WFkNnAnDzj5f4fazi9iIP8j+hackG2mro6g3weoraL9p/QX5Aihp6awNdI2UQ
iFSLlr8HTcPc8ucLp7k2zGCDK0xm8DzuVUIIcfYxgvLj7QoMoq5gg1/TiTulqi9y
Oi2L93bOWn+T8APYFbu4iJTG7VzLhR3blkbjv+8EQ/OCAW6n+2GhTbT/75QLBuZ2
1YEkw8Cj4jW+naG1TIgm
=47Fr
-END PGP SIGNATURE-
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
Anton, On Fri, Oct 11, 2013 at 9:00 PM, Anton Gladky gl...@debian.org wrote: [...] /usr/bin/c++ -Dyade_EXPORTS -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -DYADE_PTR_CAST=static_pointer_cast -DYADE_CAST=static_cast -fPIC -DYADE_VTK -DYADE_OPENMP -fopenmp -DYADE_GTS -DQGLVIEWER_FOUND -DYADE_OPENGL -frounding-math -DYADE_CGAL -DFLOW_ENGINE -frounding-math -DLINSOLV -DFLOW_ENGINE -DYADE_GL2PS -O3 -DNDEBUG -fPIC - You are using cmake in Release mode, which is bad IMHO. So having -O2 and -O3 (in that order) compiles code in -O3, which is way too much AFAIK. Ref: http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html ... If you use multiple -O options, with or without level numbers, the last such option is the one that is effective. ... See also here why CMAKE_BUILD_TYPE should be required to be set to on debian: http://bugs.debian.org/701231#12 2cts -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
Thanks all, guys, for recommendations and explanations. I will try to split some large cpp-files as Dmitrijs mentioned (there are a couple of them, which are failing reliably on weak archs), remove some optimization options as Mathieu advised and we will see, whether it helps. I would also try to build Yade with clang on failing archs, if upper mentioned actions will not help to solve the problem. Though openmp-parallelisation will be disabled in this case, ant it is not very convenient. Thanks again, Anton On 10/12/2013 02:34 PM, Mathieu Malaterre wrote: Anton, On Fri, Oct 11, 2013 at 9:00 PM, Anton Gladky gl...@debian.org wrote: [...] /usr/bin/c++ -Dyade_EXPORTS -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -DYADE_PTR_CAST=static_pointer_cast -DYADE_CAST=static_cast -fPIC -DYADE_VTK -DYADE_OPENMP -fopenmp -DYADE_GTS -DQGLVIEWER_FOUND -DYADE_OPENGL -frounding-math -DYADE_CGAL -DFLOW_ENGINE -frounding-math -DLINSOLV -DFLOW_ENGINE -DYADE_GL2PS -O3 -DNDEBUG -fPIC - You are using cmake in Release mode, which is bad IMHO. So having -O2 and -O3 (in that order) compiles code in -O3, which is way too much AFAIK. Ref: http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html ... If you use multiple -O options, with or without level numbers, the last such option is the one that is effective. ... See also here why CMAKE_BUILD_TYPE should be required to be set to on debian: http://bugs.debian.org/701231#12 2cts signature.asc Description: OpenPGP digital signature
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
severity 726009 wishlist retitle 726009 Yade requires too much RAM for building thanks Hello, thanks for bug-report. The problem is, that all build-failures are due to insufficient RAM on build-machines [1]. I do not really know how to fix that except of backlisting of some machines, as was suggested by Julien [2]. The same package builds fine on Launchpad's PPA. It seems, the package builds only on machines, where 4Gb RAM is available. Thus I am reducing the bug's severity and CC-ing -devel with the hope to get dome more ideas how to fix the problem. I am also one of upstream-authors of this software and during last several years we did not add to the code something, what can sufficiently increase RAM-consumption. [1] https://buildd.debian.org/status/package.php?p=yade [2] https://lists.debian.org/debian-science/2013/10/msg7.html Thanks, Anton On 10/11/2013 09:12 AM, Ralf Treinen wrote: Source: yade Version: 1.00.0-2 Severity: serious User: trei...@debian.org Usertags: edos-outdated Hello, yade 1.00.0-2 FTBFS on the i396 autobuilder with message: /usr/bin/c++ -Dyade_EXPORTS -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -DYADE_PTR_CAST=static_pointer_cast -DYADE_CAST=static_cast -fPIC -DYADE_VTK -DYADE_OPENMP -fopenmp -DYADE_GTS -DQGLVIEWER_FOUND -DYADE_OPENGL -frounding-math -DYADE_CGAL -DFLOW_ENGINE -frounding-math -DLINSOLV -DFLOW_ENGINE -DYADE_GL2PS -O3 -DNDEBUG -fPIC -I/usr/lib/pymodules/python2.7/numpy/core/include -I/usr/include/eigen3 -I/usr/include/vtk-5.8 -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include -I/usr/include/qt4/QtDesigner -I/usr/include/qt4/QtDeclarative -I/usr/include/qt4/QtScriptTools -I/usr/include/qt4/QtDBus -I/usr/include/qt4/QtXml -I/usr/include/qt4/QtSql -I/usr/include/qt4/QtOpenGL -I/usr/include/qt4/QtNetwork -I/usr/include/qt4/QtXmlPatterns -I/usr/include/qt4/QtHelp -I/usr/include/qt4/QtUiTools -I/usr/include/qt4/QtTest -I/usr/include/qt4/QtScript -I/usr/include/qt4/QtSvg -I/usr/include/qt4/Qt3Support -I/usr/include/q t4/QtGui -I/usr/include/qt4/QtCore -I/usr/share/qt4/mkspecs/default -I/usr/include/qt4 -I/usr/include/suitesparse -I/usr/include/python2.7 -I/usr/include/i386-linux-gnu/python2.7 -I/«PKGBUILDDIR»/extra/floating_point_utilities_v3 -I/«PKGBUILDDIR»/debian/build-o CMakeFiles/yade.dir/pkg/dem/DomainLimiter.cpp.o -c /«PKGBUILDDIR»/pkg/dem/DomainLimiter.cpp virtual memory exhausted: Cannot allocate memory make[3]: *** [CMakeFiles/yade.dir/pkg/dem/DomainLimiter.cpp.o] Error 1 make[3]: Leaving directory `/«PKGBUILDDIR»/debian/build' make[2]: *** [CMakeFiles/yade.dir/all] Error 2 make[2]: Leaving directory `/«PKGBUILDDIR»/debian/build' make[1]: *** [all] Error 2 make[1]: Leaving directory `/«PKGBUILDDIR»/debian/build' dh_auto_build: make -j1 returned exit code 2 make: *** [build-arch] Error 2 dpkg-buildpackage: error: debian/rules build-arch gave error exit status 2 See https://buildd.debian.org/status/fetch.php?pkg=yadearch=i386ver=1.00.0-2stamp=1380696072 There are also build failures on other architectures. -Ralf. signature.asc Description: OpenPGP digital signature
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
severity 726009 serious thanks This remains a serious bug. Your package, which previously built on multiple architectures, is now failing to build due to memory exhaustion. While in some circumstances it is permissible to remove the old binaries and drop support for an architecture, this remains a serious bug until this has been done. (And anyway, your package won't reach testing in the current state, so is de facto unreleasable.) On Fri, Oct 11, 2013 at 09:00:36PM +0200, Anton Gladky wrote: severity 726009 wishlist retitle 726009 Yade requires too much RAM for building thanks thanks for bug-report. The problem is, that all build-failures are due to insufficient RAM on build-machines [1]. I do not really know how to fix that except of backlisting of some machines, as was suggested by Julien [2]. The same package builds fine on Launchpad's PPA. It seems, the package builds only on machines, where 4Gb RAM is available. This diagnosis is incorrect. The error you are hitting here is not that you are exhausting the available memory on the machine, it's that you're exhausting the *address space* on the machine. Adding more memory to the buildd would have zero effect, because you're on a 32-bit system which has a limit of 4GB of address space anyway. (In practice, I believe this is 3GB for userspace and 1GB for kernel on i386.) The buildd almost certainly has swap already, giving it total available memory in excess of 4GB, but that doesn't help if you have a single process - in this case g++ - that needs more than 3GB all to itself. If this same package version built on Launchpad but is failing to build in Debian unstable, then you should look at differences in toolchain versions between the two. It's possible that Ubuntu has a compiler fix that isn't yet available in unstable; it's equally possible that the successful builds in Launchpad were done with an earlier toolchain, and that there's a more recent regression in g++ memory usage. Either way, it's not the buildd's fault. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
On 11 October 2013 20:32, Steve Langasek vor...@debian.org wrote: severity 726009 serious thanks This remains a serious bug. Your package, which previously built on multiple architectures, is now failing to build due to memory exhaustion. While in some circumstances it is permissible to remove the old binaries and drop support for an architecture, this remains a serious bug until this has been done. (And anyway, your package won't reach testing in the current state, so is de facto unreleasable.) On Fri, Oct 11, 2013 at 09:00:36PM +0200, Anton Gladky wrote: severity 726009 wishlist retitle 726009 Yade requires too much RAM for building thanks thanks for bug-report. The problem is, that all build-failures are due to insufficient RAM on build-machines [1]. I do not really know how to fix that except of backlisting of some machines, as was suggested by Julien [2]. The same package builds fine on Launchpad's PPA. It seems, the package builds only on machines, where 4Gb RAM is available. This diagnosis is incorrect. The error you are hitting here is not that you are exhausting the available memory on the machine, it's that you're exhausting the *address space* on the machine. Adding more memory to the buildd would have zero effect, because you're on a 32-bit system which has a limit of 4GB of address space anyway. (In practice, I believe this is 3GB for userspace and 1GB for kernel on i386.) The buildd almost certainly has swap already, giving it total available memory in excess of 4GB, but that doesn't help if you have a single process - in this case g++ - that needs more than 3GB all to itself. If this same package version built on Launchpad but is failing to build in Debian unstable, then you should look at differences in toolchain versions between the two. It's possible that Ubuntu has a compiler fix that isn't yet available in unstable; it's equally possible that the successful builds in Launchpad were done with an earlier toolchain, and that there's a more recent regression in g++ memory usage. Either way, it's not the buildd's fault. I'm not sure, but launchpad is running 64-bit machines even when compiling for the i386 architecture, and then launchpad supports PAE only and thus can get 4GB of address space. I think debian buildds are also all 64-bit apart from one (or something like that) thus it shouldn't be a problem there. Last time I spoke with Colin about yade FTBFS due to memory exhaustion, the recommendation he gave was to reduce translation units and thus to reduce the compiler memory usage. GCC memory usage can go very large and has regressed since 3.3 when templates are used http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12850 It has been done before for some other packages, but i haven't yet had time to look more into yade. I think that's the best way to go for yade, to address it in the source-code / restructure it to use less memory at compile time. Regards, Dmitrijs. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
On Fri, Oct 11, 2013 at 12:32:27PM -0700, Steve Langasek wrote: severity 726009 serious thanks This remains a serious bug. Your package, which previously built on multiple architectures, is now failing to build due to memory exhaustion. While in some circumstances it is permissible to remove the old binaries and drop support for an architecture, this remains a serious bug until this has been done. (And anyway, your package won't reach testing in the current state, so is de facto unreleasable.) On Fri, Oct 11, 2013 at 09:00:36PM +0200, Anton Gladky wrote: severity 726009 wishlist retitle 726009 Yade requires too much RAM for building thanks thanks for bug-report. The problem is, that all build-failures are due to insufficient RAM on build-machines [1]. I do not really know how to fix that except of backlisting of some machines, as was suggested by Julien [2]. The same package builds fine on Launchpad's PPA. It seems, the package builds only on machines, where 4Gb RAM is available. This diagnosis is incorrect. The error you are hitting here is not that you are exhausting the available memory on the machine, it's that you're exhausting the *address space* on the machine. Adding more memory to the buildd would have zero effect, because you're on a 32-bit system which has a limit of 4GB of address space anyway. (In practice, I believe this is 3GB for userspace and 1GB for kernel on i386.) I've explained this same thing to people before. All i386 buildds are now actually running on an amd64 hosts with an amd64 kernel. All of them have more than 4 GB available except from brahms which only as 2 GB + 512 MB of swap. The buildds it failed on (babin, biber) each have 8 GB of RAM available. With an amd64 kernel, i386 userspace can actually use 4 GB, which is more than it can on a real i386 host. If you fix it on the other buildds and it still fails on brahms I can either ask DSA to give more RAM to it or I can exclude the package there. Kurt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
On Fri, Oct 11, 2013 at 09:55:34PM +0100, Dmitrijs Ledkovs wrote: [...] I'm not sure, but launchpad is running 64-bit machines even when compiling for the i386 architecture, and then launchpad supports PAE only and thus can get 4GB of address space. [...] Which bit of 'Physical Address Extension' do you not understand? Ben. -- Ben Hutchings No political challenge can be met by shopping. - George Monbiot -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
On Fri, Oct 11, 2013 at 09:55:34PM +0100, Dmitrijs Ledkovs wrote: I'm not sure, but launchpad is running 64-bit machines even when compiling for the i386 architecture, and then launchpad supports PAE only and thus can get 4GB of address space. A 32-bit process can still only address 32-bits of memory. PAE only lets you extend past 4GB across *multiple* processes; the 4GB limit still applies to any one 32-bit process. I think debian buildds are also all 64-bit apart from one (or something like that) thus it shouldn't be a problem there. Last time I spoke with Colin about yade FTBFS due to memory exhaustion, the recommendation he gave was to reduce translation units and thus to reduce the compiler memory usage. GCC memory usage can go very large and has regressed since 3.3 when templates are used http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12850 It has been done before for some other packages, but i haven't yet had time to look more into yade. I think that's the best way to go for yade, to address it in the source-code / restructure it to use less memory at compile time. Yes, even if we can pinpoint a specific recent change in the compiler that caused increased memory usage, the most reliable fix for the problem is going to be to just refactor this code. If your compiler is taking anywhere near 3GB of memory to build a single object file, that's not a good thing anyway. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Bug#726009: Insufficient RAM on build-machines (was Bug#726009: yade: FTBFS on i386 (and others))
On 11 October 2013 22:34, Ben Hutchings b...@decadent.org.uk wrote: On Fri, Oct 11, 2013 at 09:55:34PM +0100, Dmitrijs Ledkovs wrote: [...] I'm not sure, but launchpad is running 64-bit machines even when compiling for the i386 architecture, and then launchpad supports PAE only and thus can get 4GB of address space. [...] Which bit of 'Physical Address Extension' do you not understand? ignore me, friday evening. the parts where virtual address space is limitted to 4gb per process none the less. Regards, Dmitrijs. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
