Bug#827953: maxima: Uses too much memory to build
Greetings! Santiago Vila writes: > On Sun, 2 Oct 2016, Camm Maguire wrote: > >> Greetings! I just wanted to keep you abreast of the latest upload. >> I've put in an improvement which I hope will allow better use of the >> ram available without running into the fork/exec faults we had seen in >> some configurations. This does not address the question of 'how much >> memory should we attempt to make use of', but you may find this upload >> more robust nonetheless. GCL_MEM_MULTIPLE defaults back to 1.0 from >> 0.85 without apparent issue. >> >> Feedback appreciated. > > Hello. > > I finally tested maxima_5.38.1-2 today. > Thanks! > On a machine with 6GB RAM and 4 GB swap, memory usage was about > 7500 MB (1500 MB more than the available RAM). > > On a machine with 3GB RAM and 4 GB swap, memory usage was about > 3700 MB (700 MB more than the available RAM). > > I'm still measuring memory usage with Committed_AS, by taking note of > its value during the build and substracting the value before the build > starts. > > In your previous email, you suggest that I don't use Committed_AS > to measure memory usage, but something else. > Because the linux docs say this is an anticipated worst case guess, not the amount of memory in use. > I'm not sure what else I could use instead which is better than > Committed_AS, and I fear that the more memory I have available, the > more memory would maxima use as well, just as it happens with > Committed_AS, so are you sure that using something else other > than Committed_AS would really fix my problem? If you use the variables I output in my post, I think you will find that the build does not swap. > > Do you have a suggestion about which other parameter in /proc/meminfo > would be more suitable than Committed_AS? These were the variables I output in my sample build. > > Maybe this Committed_AS value being higher than the available RAM is > "normal and expected", but the way I see it, it seems part of a > general problem called "overcommit". > Indeed, all unices nowadays do this. > The question would be: How much overcommit should we reasonably expect > from a package when it's being built? > I think it is reasonable that a program make use of the available ram as long as it stays out of swap. Take care, > [ I still have to read your previous email in detail, this is just a > first reply to tell my experience with maxima_5.38.1-2 ] > > Thanks. > > > > -- Camm Maguirec...@maguirefamily.org == "The earth is but one country, and mankind its citizens." -- Baha'u'llah
Bug#827953: maxima: Uses too much memory to build
On Sun, 2 Oct 2016, Camm Maguire wrote: > Greetings! I just wanted to keep you abreast of the latest upload. > I've put in an improvement which I hope will allow better use of the > ram available without running into the fork/exec faults we had seen in > some configurations. This does not address the question of 'how much > memory should we attempt to make use of', but you may find this upload > more robust nonetheless. GCL_MEM_MULTIPLE defaults back to 1.0 from > 0.85 without apparent issue. > > Feedback appreciated. Hello. I finally tested maxima_5.38.1-2 today. On a machine with 6GB RAM and 4 GB swap, memory usage was about 7500 MB (1500 MB more than the available RAM). On a machine with 3GB RAM and 4 GB swap, memory usage was about 3700 MB (700 MB more than the available RAM). I'm still measuring memory usage with Committed_AS, by taking note of its value during the build and substracting the value before the build starts. In your previous email, you suggest that I don't use Committed_AS to measure memory usage, but something else. I'm not sure what else I could use instead which is better than Committed_AS, and I fear that the more memory I have available, the more memory would maxima use as well, just as it happens with Committed_AS, so are you sure that using something else other than Committed_AS would really fix my problem? Do you have a suggestion about which other parameter in /proc/meminfo would be more suitable than Committed_AS? Maybe this Committed_AS value being higher than the available RAM is "normal and expected", but the way I see it, it seems part of a general problem called "overcommit". The question would be: How much overcommit should we reasonably expect from a package when it's being built? [ I still have to read your previous email in detail, this is just a first reply to tell my experience with maxima_5.38.1-2 ] Thanks.
Bug#827953: maxima: Uses too much memory to build
Greetings! I just wanted to keep you abreast of the latest upload. I've put in an improvement which I hope will allow better use of the ram available without running into the fork/exec faults we had seen in some configurations. This does not address the question of 'how much memory should we attempt to make use of', but you may find this upload more robust nonetheless. GCL_MEM_MULTIPLE defaults back to 1.0 from 0.85 without apparent issue. Feedback appreciated. Take care, Santiago Vila writes: > Hello Camm. > > Yes, I received the email that the BTS apparently didn't like. > > Please allow me some time to read it in full and reply the way it > deserves. > > Thanks. > > > > -- Camm Maguirec...@maguirefamily.org == "The earth is but one country, and mankind its citizens." -- Baha'u'llah
Bug#827953: maxima: Uses too much memory to build
Greetings! The BTS seems to have swallowed my reply of yesterday to you. In any case, my results seem to show that the total build avoids swap, and that Committed_AS is misleading relative to other variables in /proc/meminfo for this purpose. Does this close the issue for you? (P.S. if you have the text of my results and can post it to the BTS in a way that takes, that would be great.) Take care, Santiago Vila writes: > Package: maxima > Version: 5.38.0-3 > > Hello Camm. > > The more memory it has the machine on which I build this package, > the more memory it uses during the build, and there does not seem to > be a limit for that. > > Last time it was a machine with 8GB RAM. The file /proc/meminfo > reported a Committed_AS memory of 17823MB (more than 17GB!). > > This is specially bad for me because I rely on /proc/meminfo to know > how much memory the package "really" needs. > > While checking for "dpkg-buildpackage -A" I have measured the memory > required to build around 3400 different source packages in stretch. > If I order them by required memory, these are the last 20: > > [...] > gnutls28 4189 > mapnik 4199 > mia 4279 > gcc-4.9 4287 > gap-guava 4315 > seqan 4376 > yade 4509 > webkit2gtk 4558 > llvm-toolchain-3.8 4673 > aces3 4795 > python2.7 5685 > libbson 5700 > webkitgtk 7015 > acl2 7355 > gap 8268 > gcl 9375 > mame 10363 > gdk-pixbuf 11564 > maxima 17823 > axiom 21190 > > In practice, this means several things: > > * As I don't currently have such a big machine, I have to rent one to > build those packages. > > * As they use so much memory, I can't use a 8GB plan or a 12GB plan, I > have to use a 24GB plan (or whatever is close). > > * I need a bigger machine *only* to build the packages at the end of > the above list. > > I know that you have been fine-tuning the memory usage for these > programs lately, but it still happens that the more memory II have, > the more memory they take, so I still have the problem that I can't > measure how much memory they really need. > > Thanks. > From unknown Mon Sep 26 19:22:45 2016 > X-Loop: ow...@bugs.debian.org > Subject: Bug#827953: maxima: Uses too much memory to build > Reply-To: Camm Maguire , 827...@bugs.debian.org > Resent-From: Camm Maguire > Resent-To: debian-bugs-dist@lists.debian.org > Resent-CC: Camm Maguire > X-Loop: ow...@bugs.debian.org > Resent-Date: Sun, 10 Jul 2016 15:39:06 + > Resent-Message-ID: > Resent-Sender: ow...@bugs.debian.org > X-Debian-PR-Message: followup 827953 > X-Debian-PR-Package: maxima > X-Debian-PR-Keywords: > X-Debian-PR-Source: maxima > Received: via spool by 827953-sub...@bugs.debian.org > id=B827953.146816505024139 > (code B ref 827953); Sun, 10 Jul 2016 15:39:06 + > Received: (at 827953) by bugs.debian.org; 10 Jul 2016 15:37:30 + > X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02 > (2014-02-07) on buxtehude.debian.org > X-Spam-Level: > X-Spam-Status: No, score=-4.4 required=4.0 tests=BAYES_00,HAS_BUG_NUMBER, > HDRS_LCASE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3, > RCVD_IN_MSPIKE_WL,T_MANY_HDRS_LCASE,URIBL_CNKR autolearn=ham > autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02 > X-Spam-Bayes: score:0. Tokens: new, 10; hammy, 150; neutral, 132; spammy, > 0. spammytokens: hammytokens:0.000-+--python27, 0.000-+--python2.7, > 0.000-+--H*f:sk:alpine., 0.000-+--H*u:Gnus, 0.000-+--H*u:linux > Received: from vms173017pub.verizon.net ([206.46.173.17]) > by buxtehude.debian.org with esmtps > (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128) > (Exim 4.84_2) > (envelope-from ) > id 1bMGnS-0006H2-3e > for 827...@bugs.debian.org; Sun, 10 Jul 2016 15:37:30 + > Received: from vz-proxy-m001.mx.aol.com ([64.236.83.14]) > by vms173017.mailsrvcs.net > (Oracle Communications Messaging Server 7.0.5.32.0 64bit (built Jul 16 2014)) > with ESMTPA id <0oa300dg7slhk...@vms173017.mailsrvcs.net> for > 827...@bugs.debian.org; Sun, 10 Jul 2016 09:36:59 -0500 (CDT) > X-CMAE-Score: 0 > X-CMAE-Analysis: v=2.1 cv=J+9Xl1TS c=1 sm=1 tr=0 > a=MJxOpqxZADbEbEImuSX/mw==:117 > a=kj9zAlcOel0A:10 a=cAmyUtKerLwA:10 a=9N09Ue-c:8 > a=ZeoOlCyH8KQX3YoRE1wA:9 > a=CjuIK1q_8ugA:10 > Received: by 173.61.133.218 with SMTP id 18f50746; Sun, 10 Jul 2016 14:36:59 > GMT > Received: from camm by localhost.m.enhanced.com with local (Exim 4.80) > (envelope-from )id 1bMFqR-0006PD-Im; > Sun, > 10 Jul 2016 10:36:31 -0400 > From: Camm Maguire > To: Santiago Vila > Cc: 827...@bugs.debian.org > References: > Date: Sun, 10
Bug#827953: maxima: Uses too much memory to build
Hello Camm. Nice to hear from you again. On Tue, 13 Sep 2016, Camm Maguire wrote: > It appears that all is working as intended, that gc is triggered and > keeping the active heap to about 2.8Gb (or 3.9Gb with relocatable copy > area) of a detected max of 5.3Gb (see core pages, and physical memory). > At the end of this experiment, the rss of this process as viewed from a > separate shell should confirm these figures. So I do not think gcl is > expanding into swap here. Ok, even if it's working as intended, maybe the intended way is still not the optimal way :-), and maybe we might still consider to change the intended way. I'll elaborate on that below. > As for the /proc/meminfo, all I can say is that the build employs many > other programs as well. Might be interesting to correlate this figure > with the various stages of the build. In other words, how much just to > run the debian/rules build target, etc. That's probably the problem. The garbage collection procedure may be very well designed not to take all available memory, but this good care it takes ends up being not enough when we consider all the other processes involved in the build of the package, i.e. the build process as a whole. The current procedure is a little bit "egoistical", so to speak, as it takes almost all the available memory for himself, not leaving enough memory for the other processes. The problem I'd like to see fixed is the amount of memory used by the Debian package build as a whole, for which the garbage collection thing is only a part. If you can fix this problem by fine-tuning GCL, fine, but if not, I'd like you to consider the problem as a whole, not just the garbage collection thing. Maybe this is just a matter of setting GCL_MEMORY_something=somevalue somewhere in debian/rules and nothing less. I really don't know. But I'll better describe what I do in case you want to replicate my results. It's quite simple indeed. When my autobuilder starts building a package (just before it starts), it takes note of the value of Committed_AS in /proc/meminfo. Then the build starts and it keeps taking notes of Committed_AS at every second, then substracts the initial value and stores the difference. At the end I have a text file with the amount of memory that it was committed during the build. I call this "memory profile". (My autobuilder does not do anything else other than building a package at a time, so this is as accurate as it can be with the limited available resources I have). I attach the two memory profiles I have for maxima. I do not remember how much memory these two autobuilders had, because I often change the configuration from time to time, but I'm almost sure that I did not have 17 GB as required by "lutecio" (the second attach). If my records are correct, this was a Linode with only 8 GB of RAM (and maybe a similar amount of swap). So, to summarize: Would you please try observing Committed_AS in /proc/meminfo while the Debian maxima package is building to see if you can reproduce what I report, i.e. that the amount of memory it takes is usually a lot more than the available RAM? Thanks a lot.0 0 40 40 40 40 92 67 67 67 67 67 105 40 40 40 40 40 40 66 66 135 135 63 66 66 104 102 49 66 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 79 78 78 78 78 94 94 94 94 94 94 94 94 94 94 78 78 78 78 78 78 78 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 94 97 97 97 97 97 97 97 97 97 97 97 113 128 83 83 98 93 93 93 93 91 137 138 137 82 69 69 40 61 61 61 61 61 83 59 75 60 61 61 2370 2370 2387 2387 2387 2403 2377 2385 2404 2404 2385 2370 2370 2385 2370 2401 2417 2417 2417 2417 2432 2403 2382 2417 2433 2449 2465 2465 2465 2370 2403 2370 2370 2385 2370 2385 2370 2386 2386 2370 2385 2401 2401 2370 2386 2370 2385 2370 2385 2385 2385 2386 2386 2385 2386 2404 2417 2449 2464 2480 2496 2371 2419 2451 2483 2404 2370 2380 2419 2451 2388 2388 4680 2371 2420 2389 2402 2388 2421 2404 2388 2388 2402 2384 2404 2404 2404 2387 2387 2387 2387 2371 2387 2404 2384 2371 2403 2435 2419 2434 2468 2483 2483 2387 2404 2404 2420 2388 2387 2387 2381 2419 2450 2387 2418 2466 2481 2503 2503 2519 2402 2434 2418 2450 2467 2498 2515 2387 2436 2469 2485 2388 2421 2388 2389 2387 2387 2434 2467 2499 2389 2388 2387 2434 2450 2483 2500 2389 2389 2404 2404 2387 2402 2418 2389 2383 2465 2483 2500 2515 2515 2531 2531 2548 2548 2548 2564 2564 2580 2580 2597 2597 2597 2613 2613 2630 2630 2645 2415 2371 2406 2437 2453 2404 2435 2387 2403 2420 2419 2435 2466 2371 2421 2421 2438 2471 2419 2450 2465 2389 2402 2434 2466 2482 2371 2370 4678 2381 2372 2386 2387 2370 2370 2370 2386 2370 2370 2403 2384 2403 2386 2418 2451 2370 2403 2386 2370 2385 2385 2385 2386 2386 2370 2386 2377 2385 2401 2370 2386 23
Bug#827953: maxima: Uses too much memory to build
Greetings, and thanks so much! It appears that all is working as intended, that gc is triggered and keeping the active heap to about 2.8Gb (or 3.9Gb with relocatable copy area) of a detected max of 5.3Gb (see core pages, and physical memory). At the end of this experiment, the rss of this process as viewed from a separate shell should confirm these figures. So I do not think gcl is expanding into swap here. As for the /proc/meminfo, all I can say is that the build employs many other programs as well. Might be interesting to correlate this figure with the various stages of the build. In other words, how much just to run the debian/rules build target, etc. Santiago Vila writes: > On Mon, 25 Jul 2016, Camm Maguire wrote: > >> If this is happening, then it is indeed a bug. The intent is to use >> physical ram only by default unless the application itself insists on >> more. Going into swap by default obviously defeats the performance goal >> of expanding the memory anyway. You can look into this by >> >> gcl >> >(room t) >> >> and >> >> maxima >> (..) :lisp (room t) >> (..) :lisp (setq si::*notify-gbc* t) >> (..) run_testsuite(); >> (..) :lisp (room t) >> >> If you can post the results for this on the 4/4 machine you describe >> above I can see if we have a problem. > > Sorry for the late reply. > > I attach the results for maxima, on a machine with 6GB RAM and 4GB swap. > > I don't know how to interpret the results. In addition to the test you > requested, when I do this > > grep "Committed_AS:" /proc/meminfo > > I get this before running maxima: > > Committed_AS: 254276 kB > > and this after the test finished, before exiting maxima: > > Committed_AS:6878284 kB > > I guess this does still not explain why Committed_AS: grows so much > when I'm actually trying to build the Debian package. > > Thanks. > > (sid)buildd@skywalker1:~$ maxima > > Maxima 5.38.0 http://maxima.sourceforge.net > using Lisp GNU Common Lisp (GCL) GCL 2.6.12 > Distributed under the GNU Public License. See the file COPYING. > Dedicated to the memory of William Schelter. > The function bug_report() provides bug reporting information. > (%i1) (%i1) :lisp (room t) > > 1828/182888.8% CONS FIXNUM SHORT-FLOAT LONG-FLOAT > BIT-VECTOR PATHNAME SPICE > 220/220 99.8% ARRAY CHARACTER PACKAGE HASH-TABLE VECTOR > RANDOM-STATE CCLOSURE CLOSURE > 67/67 56.7% STRING BIGNUM RATIO COMPLEX > 399/399 98.8% STRUCTURE >1/1 65.2% STREAM > 46/46 99.4% CFUN CFDATA > 592/592 99.8% SFUN SYMBOL READTABLE GFUN VFUN AFUN > > 6400/6400 contiguous (30 blocks) > 1173337 hole > 216/216 99.9% relocatable > > 3153 pages for cells > > 9769 total pages in core > 9769 current core maximum pages >216 pages reserved for gc >3519614 pages available for adding to core > 35556 pages reserved for core exhaustion > >3565155 maximum pages > > > Key: > > WS: words per struct > UP: allocated pages > MP: maximum pages > FI: fraction of cells in use on allocated pages > GC: number of gc triggers allocating this type > > word size:64 bits > page size:4096 bytes > heap start: 0xE81000 > heap max :0x368365000 > shared library start: 0x0 > cstack start: 0x0 > cstack mark offset: 0 bytes > cstack direction: downward > cstack alignment: 32 bytes > cstack max: 16001 bytes > immfix start: 0x8000 > immfix size: 4611686018427387904 fixnums > physical memory: 1299534 pages > (%i1) (%i1) :lisp (setq si::*notify-gbc* t) > > T > (%i1) (%i1) run_testsuite(); > Running tests in rtest_rules: 103/103 tests passed > Running tests in rtestnset: 597/597 tests passed > Running tests in rtest1: 180/180 tests passed (not counting 1 expected errors) > Running tests in rtest1a: 27/27 tests passed > Running tests in rtest2: 144/144 tests passed (not counting 2 expected errors) > Running tests in rtest4: 93/93 tests passed > Running tests in rtest5: > ** Problem 78 *** > Input: > describe(sin) > > > Result: > > error-catch > > This differed from the expected result: > true > start address -T 0x28ee010 start address -T 0x291e650 start address -T > 0x2925ce0 start address -T 0x292a1a0 start address -T 0x292e660 start address > -T 0x2936f20 start address -T 0x293c010 start address -T 0x293fb70 > 79/80 tests passed > > The following 1 problem failed: (78) > Running tests in rtest6: 39/39 tests passed > Running tests in rtest6a: 62/62 tests passed > Running tests in rtest6b: 16/16 tests passed > Running tests in rtest7: 85/85 tests passed > Running tests in rtest9: 89/89 tests passed > Running tests in rtest9a: 76/76 tests passed > Running tests in rtest10: 62/62 tests passed (not counting 2 ex
Bug#827953: maxima: Uses too much memory to build
On Mon, 25 Jul 2016, Camm Maguire wrote: > If this is happening, then it is indeed a bug. The intent is to use > physical ram only by default unless the application itself insists on > more. Going into swap by default obviously defeats the performance goal > of expanding the memory anyway. You can look into this by > > gcl > >(room t) > > and > > maxima > (..) :lisp (room t) > (..) :lisp (setq si::*notify-gbc* t) > (..) run_testsuite(); > (..) :lisp (room t) > > If you can post the results for this on the 4/4 machine you describe > above I can see if we have a problem. Sorry for the late reply. I attach the results for maxima, on a machine with 6GB RAM and 4GB swap. I don't know how to interpret the results. In addition to the test you requested, when I do this grep "Committed_AS:" /proc/meminfo I get this before running maxima: Committed_AS: 254276 kB and this after the test finished, before exiting maxima: Committed_AS:6878284 kB I guess this does still not explain why Committed_AS: grows so much when I'm actually trying to build the Debian package. Thanks.(sid)buildd@skywalker1:~$ maxima Maxima 5.38.0 http://maxima.sourceforge.net using Lisp GNU Common Lisp (GCL) GCL 2.6.12 Distributed under the GNU Public License. See the file COPYING. Dedicated to the memory of William Schelter. The function bug_report() provides bug reporting information. (%i1) (%i1) :lisp (room t) 1828/182888.8% CONS FIXNUM SHORT-FLOAT LONG-FLOAT BIT-VECTOR PATHNAME SPICE 220/220 99.8% ARRAY CHARACTER PACKAGE HASH-TABLE VECTOR RANDOM-STATE CCLOSURE CLOSURE 67/67 56.7% STRING BIGNUM RATIO COMPLEX 399/399 98.8% STRUCTURE 1/1 65.2% STREAM 46/46 99.4% CFUN CFDATA 592/592 99.8% SFUN SYMBOL READTABLE GFUN VFUN AFUN 6400/6400 contiguous (30 blocks) 1173337 hole 216/216 99.9% relocatable 3153 pages for cells 9769 total pages in core 9769 current core maximum pages 216 pages reserved for gc 3519614 pages available for adding to core 35556 pages reserved for core exhaustion 3565155 maximum pages Key: WS: words per struct UP: allocated pages MP: maximum pages FI: fraction of cells in use on allocated pages GC: number of gc triggers allocating this type word size:64 bits page size:4096 bytes heap start: 0xE81000 heap max :0x368365000 shared library start: 0x0 cstack start: 0x0 cstack mark offset: 0 bytes cstack direction: downward cstack alignment: 32 bytes cstack max: 16001 bytes immfix start: 0x8000 immfix size: 4611686018427387904 fixnums physical memory: 1299534 pages (%i1) (%i1) :lisp (setq si::*notify-gbc* t) T (%i1) (%i1) run_testsuite(); Running tests in rtest_rules: 103/103 tests passed Running tests in rtestnset: 597/597 tests passed Running tests in rtest1: 180/180 tests passed (not counting 1 expected errors) Running tests in rtest1a: 27/27 tests passed Running tests in rtest2: 144/144 tests passed (not counting 2 expected errors) Running tests in rtest4: 93/93 tests passed Running tests in rtest5: ** Problem 78 *** Input: describe(sin) Result: error-catch This differed from the expected result: true start address -T 0x28ee010 start address -T 0x291e650 start address -T 0x2925ce0 start address -T 0x292a1a0 start address -T 0x292e660 start address -T 0x2936f20 start address -T 0x293c010 start address -T 0x293fb70 79/80 tests passed The following 1 problem failed: (78) Running tests in rtest6: 39/39 tests passed Running tests in rtest6a: 62/62 tests passed Running tests in rtest6b: 16/16 tests passed Running tests in rtest7: 85/85 tests passed Running tests in rtest9: 89/89 tests passed Running tests in rtest9a: 76/76 tests passed Running tests in rtest10: 62/62 tests passed (not counting 2 expected errors) Running tests in rtest11: 181/181 tests passed Running tests in rtest13: 23/23 tests passed Running tests in rtest13s: 17/17 tests passed Running tests in rtest14: [GC for 234547 CONS pages..(T=25).GC finished] [GC for 188813 RELOCATABLE-BLOCKS pages..(T=19).GC finished] [GC for 140149 STRUCTURE pages..(T=21).GC finished] 418/418 tests passed Running tests in rtest15: [GC for 234547 CONS pages..(T=18).GC finished] 334/334 tests passed Running tests in rtest16: 540/540 tests passed Running tests in rtestode: 90/90 tests passed Running tests in rtestode_zp: 30/30 tests passed Running tests in rtest3: [GC for 235957 CONS pages..(T=22).GC finished] 149/149 tests passed Running tests in rtest8: 164/164 tests passed Running tests in rtest12: 79/79 tests passed (not counting 2 expected errors) Running tests in rexamples: 137/137 tests passed Running tests in rtesthyp: [GC for 16574 SFUN pag
Bug#827953: maxima: Uses too much memory to build
Greetings, and thanks for your report! Santiago Vila writes: > On Sun, 10 Jul 2016, Camm Maguire wrote: > >> Greetings, and thanks so much for your report! More reply later, but >> just want to point out for now that the memory used for the build >> depends on the memory available at runtime. These packages will build >> just fine on machines with small memories. The memory usage algorithm >> is a performance booster only. > > Hello Camm. > > There are at least two problems with the current approach: > > * One of them (which I already pointed out) is that I can't measure how > much this program really needs by looking at memory usage. I can do > that for more than 3300 different packages in stretch, but I can't for > maxima and axiom, which is a pity. OK, this might be an inconvenience from an 'package builder memory measurer' but is not a critical issue, right? > > * The other problem is that not only maxima takes as much RAM as I > have, it seems to use all available SWAP as well! This would explain > the effect "the more memory I have, the more memory it takes". Let's > suppose that I build with 4 GB RAM and 4 GB SWAP. Then I measure > that maxima takes 8GB and next time I will need 8 GB RAM to build. > If this is happening, then it is indeed a bug. The intent is to use physical ram only by default unless the application itself insists on more. Going into swap by default obviously defeats the performance goal of expanding the memory anyway. You can look into this by gcl >(room t) and maxima (..) :lisp (room t) (..) :lisp (setq si::*notify-gbc* t) (..) run_testsuite(); (..) :lisp (room t) If you can post the results for this on the 4/4 machine you describe above I can see if we have a problem. The available memory should be 8G, garbage collection should start at 2G, become very heavy by 0.9*4G, beyond which the system should not expand unless the memory cannot be reclaimed by garbage collection. Take care, -- Camm Maguirec...@maguirefamily.org == "The earth is but one country, and mankind its citizens." -- Baha'u'llah
Bug#827953: maxima: Uses too much memory to build
On Sun, 10 Jul 2016, Camm Maguire wrote: > Greetings, and thanks so much for your report! More reply later, but > just want to point out for now that the memory used for the build > depends on the memory available at runtime. These packages will build > just fine on machines with small memories. The memory usage algorithm > is a performance booster only. Hello Camm. There are at least two problems with the current approach: * One of them (which I already pointed out) is that I can't measure how much this program really needs by looking at memory usage. I can do that for more than 3300 different packages in stretch, but I can't for maxima and axiom, which is a pity. * The other problem is that not only maxima takes as much RAM as I have, it seems to use all available SWAP as well! This would explain the effect "the more memory I have, the more memory it takes". Let's suppose that I build with 4 GB RAM and 4 GB SWAP. Then I measure that maxima takes 8GB and next time I will need 8 GB RAM to build. If maxima is going to take all the memory I have, whatever the amount, it should be the available RAM only, not available RAM + available SWAP. If it's not able to do that and it only takes the total memory in account, meybe it should assume that at least half of that is SWAP and avoid using it. Currently, it seems to assume that all the memory is available to use, which is not realistic for people using SWAP as a safety net. Thanks.
Bug#827953: maxima: Uses too much memory to build
Greetings, and thanks so much for your report! More reply later, but just want to point out for now that the memory used for the build depends on the memory available at runtime. These packages will build just fine on machines with small memories. The memory usage algorithm is a performance booster only. Take care, Santiago Vila writes: > Package: maxima > Version: 5.38.0-3 > > Hello Camm. > > The more memory it has the machine on which I build this package, > the more memory it uses during the build, and there does not seem to > be a limit for that. > > Last time it was a machine with 8GB RAM. The file /proc/meminfo > reported a Committed_AS memory of 17823MB (more than 17GB!). > > This is specially bad for me because I rely on /proc/meminfo to know > how much memory the package "really" needs. > > While checking for "dpkg-buildpackage -A" I have measured the memory > required to build around 3400 different source packages in stretch. > If I order them by required memory, these are the last 20: > > [...] > gnutls28 4189 > mapnik 4199 > mia 4279 > gcc-4.9 4287 > gap-guava 4315 > seqan 4376 > yade 4509 > webkit2gtk 4558 > llvm-toolchain-3.8 4673 > aces3 4795 > python2.7 5685 > libbson 5700 > webkitgtk 7015 > acl2 7355 > gap 8268 > gcl 9375 > mame 10363 > gdk-pixbuf 11564 > maxima 17823 > axiom 21190 > > In practice, this means several things: > > * As I don't currently have such a big machine, I have to rent one to > build those packages. > > * As they use so much memory, I can't use a 8GB plan or a 12GB plan, I > have to use a 24GB plan (or whatever is close). > > * I need a bigger machine *only* to build the packages at the end of > the above list. > > I know that you have been fine-tuning the memory usage for these > programs lately, but it still happens that the more memory II have, > the more memory they take, so I still have the problem that I can't > measure how much memory they really need. > > Thanks. > > > > -- Camm Maguirec...@maguirefamily.org == "The earth is but one country, and mankind its citizens." -- Baha'u'llah
Bug#827953: maxima: Uses too much memory to build
Package: maxima Version: 5.38.0-3 Hello Camm. The more memory it has the machine on which I build this package, the more memory it uses during the build, and there does not seem to be a limit for that. Last time it was a machine with 8GB RAM. The file /proc/meminfo reported a Committed_AS memory of 17823MB (more than 17GB!). This is specially bad for me because I rely on /proc/meminfo to know how much memory the package "really" needs. While checking for "dpkg-buildpackage -A" I have measured the memory required to build around 3400 different source packages in stretch. If I order them by required memory, these are the last 20: [...] gnutls28 4189 mapnik 4199 mia 4279 gcc-4.9 4287 gap-guava 4315 seqan 4376 yade 4509 webkit2gtk 4558 llvm-toolchain-3.8 4673 aces3 4795 python2.7 5685 libbson 5700 webkitgtk 7015 acl2 7355 gap 8268 gcl 9375 mame 10363 gdk-pixbuf 11564 maxima 17823 axiom 21190 In practice, this means several things: * As I don't currently have such a big machine, I have to rent one to build those packages. * As they use so much memory, I can't use a 8GB plan or a 12GB plan, I have to use a 24GB plan (or whatever is close). * I need a bigger machine *only* to build the packages at the end of the above list. I know that you have been fine-tuning the memory usage for these programs lately, but it still happens that the more memory II have, the more memory they take, so I still have the problem that I can't measure how much memory they really need. Thanks.