Bug#827953: maxima: Uses too much memory to build

[Camm Maguire]

Greetings!

Santiago Vila  writes:

> On Sun, 2 Oct 2016, Camm Maguire wrote:
>
>> Greetings!  I just wanted to keep you abreast of the latest upload.
>> I've put in an improvement which I hope will allow better use of the
>> ram available without running into the fork/exec faults we had seen in
>> some configurations.  This does not address the question of 'how much
>> memory should we attempt to make use of', but you may find this upload
>> more robust nonetheless.  GCL_MEM_MULTIPLE defaults back to 1.0 from
>> 0.85 without apparent issue.
>> 
>> Feedback appreciated.
>
> Hello.
>
> I finally tested maxima_5.38.1-2 today.
>

Thanks!

> On a machine with 6GB RAM and 4 GB swap, memory usage was about
> 7500 MB (1500 MB more than the available RAM).
>
> On a machine with 3GB RAM and 4 GB swap, memory usage was about
> 3700 MB (700 MB more than the available RAM).
>
> I'm still measuring memory usage with Committed_AS, by taking note of
> its value during the build and substracting the value before the build
> starts.
>
> In your previous email, you suggest that I don't use Committed_AS
> to measure memory usage, but something else.
>

Because the linux docs say this is an anticipated worst case guess, not
the amount of memory in use.

> I'm not sure what else I could use instead which is better than
> Committed_AS, and I fear that the more memory I have available, the
> more memory would maxima use as well, just as it happens with
> Committed_AS, so are you sure that using something else other
> than Committed_AS would really fix my problem?

If you use the variables I output in my post, I think you will find that
the build does not swap.

>
> Do you have a suggestion about which other parameter in /proc/meminfo
> would be more suitable than Committed_AS?

These were the variables I output in my sample build.

>
> Maybe this Committed_AS value being higher than the available RAM is
> "normal and expected", but the way I see it, it seems part of a
> general problem called "overcommit".
>

Indeed, all unices nowadays do this.

> The question would be: How much overcommit should we reasonably expect
> from a package when it's being built?
>

I think it is reasonable that a program make use of the available ram as
long as it stays out of swap.

Take care,

> [ I still have to read your previous email in detail, this is just a
>   first reply to tell my experience with maxima_5.38.1-2 ]
>
> Thanks.
>
>
>
>

-- 
Camm Maguirec...@maguirefamily.org
==
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Bug#827953: maxima: Uses too much memory to build

[Santiago Vila]

On Sun, 2 Oct 2016, Camm Maguire wrote:

> Greetings!  I just wanted to keep you abreast of the latest upload.
> I've put in an improvement which I hope will allow better use of the
> ram available without running into the fork/exec faults we had seen in
> some configurations.  This does not address the question of 'how much
> memory should we attempt to make use of', but you may find this upload
> more robust nonetheless.  GCL_MEM_MULTIPLE defaults back to 1.0 from
> 0.85 without apparent issue.
> 
> Feedback appreciated.

Hello.

I finally tested maxima_5.38.1-2 today.

On a machine with 6GB RAM and 4 GB swap, memory usage was about
7500 MB (1500 MB more than the available RAM).

On a machine with 3GB RAM and 4 GB swap, memory usage was about
3700 MB (700 MB more than the available RAM).

I'm still measuring memory usage with Committed_AS, by taking note of
its value during the build and substracting the value before the build
starts.

In your previous email, you suggest that I don't use Committed_AS
to measure memory usage, but something else.

I'm not sure what else I could use instead which is better than
Committed_AS, and I fear that the more memory I have available, the
more memory would maxima use as well, just as it happens with
Committed_AS, so are you sure that using something else other
than Committed_AS would really fix my problem?

Do you have a suggestion about which other parameter in /proc/meminfo
would be more suitable than Committed_AS?

Maybe this Committed_AS value being higher than the available RAM is
"normal and expected", but the way I see it, it seems part of a
general problem called "overcommit".

The question would be: How much overcommit should we reasonably expect
from a package when it's being built?

[ I still have to read your previous email in detail, this is just a
  first reply to tell my experience with maxima_5.38.1-2 ]

Thanks.

Bug#827953: maxima: Uses too much memory to build

[Camm Maguire]

Greetings!  I just wanted to keep you abreast of the latest upload.
I've put in an improvement which I hope will allow better use of the
ram available without running into the fork/exec faults we had seen in
some configurations.  This does not address the question of 'how much
memory should we attempt to make use of', but you may find this upload
more robust nonetheless.  GCL_MEM_MULTIPLE defaults back to 1.0 from
0.85 without apparent issue.

Feedback appreciated.

Take care,

Santiago Vila  writes:

> Hello Camm.
>
> Yes, I received the email that the BTS apparently didn't like.
>
> Please allow me some time to read it in full and reply the way it
> deserves.
>
> Thanks.
>
>
>
>

-- 
Camm Maguirec...@maguirefamily.org
==
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Bug#827953: maxima: Uses too much memory to build

[Camm Maguire]

Greetings!  The BTS seems to have swallowed my reply of yesterday to
you.  In any case, my results seem to show that the total build avoids
swap, and that Committed_AS is misleading relative to other variables in
/proc/meminfo for this purpose.

Does this close the issue for you?

(P.S. if you have the text of my results and can post it to the BTS in a
way that takes, that would be great.)

Take care,

Santiago Vila <sanv...@unex.es> writes:

> Package: maxima
> Version: 5.38.0-3
>
> Hello Camm.
>
> The more memory it has the machine on which I build this package,
> the more memory it uses during the build, and there does not seem to
> be a limit for that.
>
> Last time it was a machine with 8GB RAM. The file /proc/meminfo
> reported a Committed_AS memory of 17823MB (more than 17GB!).
>
> This is specially bad for me because I rely on /proc/meminfo to know
> how much memory the package "really" needs.
>
> While checking for "dpkg-buildpackage -A" I have measured the memory
> required to build around 3400 different source packages in stretch.
> If I order them by required memory, these are the last 20:
>
> [...]
> gnutls28 4189
> mapnik 4199
> mia 4279
> gcc-4.9 4287
> gap-guava 4315
> seqan 4376
> yade 4509
> webkit2gtk 4558
> llvm-toolchain-3.8 4673
> aces3 4795
> python2.7 5685
> libbson 5700
> webkitgtk 7015
> acl2 7355
> gap 8268
> gcl 9375
> mame 10363
> gdk-pixbuf 11564
> maxima 17823
> axiom 21190
>
> In practice, this means several things:
>
> * As I don't currently have such a big machine, I have to rent one to
> build those packages.
>
> * As they use so much memory, I can't use a 8GB plan or a 12GB plan, I
> have to use a 24GB plan (or whatever is close).
>
> * I need a bigger machine *only* to build the packages at the end of
> the above list.
>
> I know that you have been fine-tuning the memory usage for these
> programs lately, but it still happens that the more memory II have,
> the more memory they take, so I still have the problem that I can't
> measure how much memory they really need.
>
> Thanks.
> From unknown Mon Sep 26 19:22:45 2016
> X-Loop: ow...@bugs.debian.org
> Subject: Bug#827953: maxima: Uses too much memory to build
> Reply-To: Camm Maguire <c...@maguirefamily.org>, 827...@bugs.debian.org
> Resent-From: Camm Maguire <c...@maguirefamily.org>
> Resent-To: debian-bugs-dist@lists.debian.org
> Resent-CC: Camm Maguire <c...@debian.org>
> X-Loop: ow...@bugs.debian.org
> Resent-Date: Sun, 10 Jul 2016 15:39:06 +
> Resent-Message-ID: <handler.827953.b827953.146816505024...@bugs.debian.org>
> Resent-Sender: ow...@bugs.debian.org
> X-Debian-PR-Message: followup 827953
> X-Debian-PR-Package: maxima
> X-Debian-PR-Keywords: 
> X-Debian-PR-Source: maxima
> Received: via spool by 827953-sub...@bugs.debian.org 
> id=B827953.146816505024139
>   (code B ref 827953); Sun, 10 Jul 2016 15:39:06 +
> Received: (at 827953) by bugs.debian.org; 10 Jul 2016 15:37:30 +
> X-Spam-Checker-Version: SpamAssassin 3.4.0-bugs.debian.org_2005_01_02
>   (2014-02-07) on buxtehude.debian.org
> X-Spam-Level: 
> X-Spam-Status: No, score=-4.4 required=4.0 tests=BAYES_00,HAS_BUG_NUMBER,
>   HDRS_LCASE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,
>   RCVD_IN_MSPIKE_WL,T_MANY_HDRS_LCASE,URIBL_CNKR autolearn=ham
>   autolearn_force=no version=3.4.0-bugs.debian.org_2005_01_02
> X-Spam-Bayes: score:0. Tokens: new, 10; hammy, 150; neutral, 132; spammy,
>   0. spammytokens: hammytokens:0.000-+--python27, 0.000-+--python2.7,
>   0.000-+--H*f:sk:alpine., 0.000-+--H*u:Gnus, 0.000-+--H*u:linux
> Received: from vms173017pub.verizon.net ([206.46.173.17])
>   by buxtehude.debian.org with esmtps 
> (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128)
>   (Exim 4.84_2)
>   (envelope-from <c...@maguirefamily.org>)
>   id 1bMGnS-0006H2-3e
>   for 827...@bugs.debian.org; Sun, 10 Jul 2016 15:37:30 +
> Received: from vz-proxy-m001.mx.aol.com ([64.236.83.14])
>  by vms173017.mailsrvcs.net
>  (Oracle Communications Messaging Server 7.0.5.32.0 64bit (built Jul 16 2014))
>  with ESMTPA id <0oa300dg7slhk...@vms173017.mailsrvcs.net> for
>  827...@bugs.debian.org; Sun, 10 Jul 2016 09:36:59 -0500 (CDT)
> X-CMAE-Score: 0
> X-CMAE-Analysis: v=2.1 cv=J+9Xl1TS c=1 sm=1 tr=0  
> a=MJxOpqxZADbEbEImuSX/mw==:117
>  a=kj9zAlcOel0A:10 a=cAmyUtKerLwA:10  a=9N09Ue-c:8 
> a=ZeoOlCyH8KQX3YoRE1wA:9
>  a=CjuIK1q_8ugA:10
> Received: by 173.61.133.218 with SMTP id 18f50746; Sun, 10 Jul 2016 14:36:59 
> GMT
> Received: from camm by localhost.m.enhanced.com with local (Exim 4.80)
>   (envelope-from <

Bug#827953: maxima: Uses too much memory to build

[Santiago Vila]

Hello Camm.

Nice to hear from you again.

On Tue, 13 Sep 2016, Camm Maguire wrote:

> It appears that all is working as intended, that gc is triggered and
> keeping the active heap to about 2.8Gb (or 3.9Gb with relocatable copy
> area) of a detected max of 5.3Gb (see core pages, and physical memory).
> At the end of this experiment, the rss of this process as viewed from a
> separate shell should confirm these figures.  So I do not think gcl is
> expanding into swap here.

Ok, even if it's working as intended, maybe the intended way is still
not the optimal way :-), and maybe we might still consider to change the
intended way. I'll elaborate on that below.

> As for the /proc/meminfo, all I can say is that the build employs many
> other programs as well.  Might be interesting to correlate this figure
> with the various stages of the build.  In other words, how much just to
> run the debian/rules build target, etc.

That's probably the problem.

The garbage collection procedure may be very well designed not to take
all available memory, but this good care it takes ends up being not enough
when we consider all the other processes involved in the build of the
package, i.e. the build process as a whole.

The current procedure is a little bit "egoistical", so to speak, as it
takes almost all the available memory for himself, not leaving enough memory for
the other processes.

The problem I'd like to see fixed is the amount of memory used by the
Debian package build as a whole, for which the garbage collection thing is
only a part.

If you can fix this problem by fine-tuning GCL, fine, but if not,
I'd like you to consider the problem as a whole, not just the garbage
collection thing.

Maybe this is just a matter of setting

GCL_MEMORY_something=somevalue

somewhere in debian/rules and nothing less. I really don't know.


But I'll better describe what I do in case you want to replicate my
results. It's quite simple indeed.

When my autobuilder starts building a package (just before it starts),
it takes note of the value of Committed_AS in /proc/meminfo.

Then the build starts and it keeps taking notes of Committed_AS at every
second, then substracts the initial value and stores the difference.

At the end I have a text file with the amount of memory that it was
committed during the build. I call this "memory profile".

(My autobuilder does not do anything else other than building a
package at a time, so this is as accurate as it can be with the
limited available resources I have).

I attach the two memory profiles I have for maxima.

I do not remember how much memory these two autobuilders had, because
I often change the configuration from time to time, but I'm almost
sure that I did not have 17 GB as required by "lutecio" (the second
attach). If my records are correct, this was a Linode with only
8 GB of RAM (and maybe a similar amount of swap).


So, to summarize: Would you please try observing Committed_AS in
/proc/meminfo while the Debian maxima package is building to see if
you can reproduce what I report, i.e. that the amount of memory it
takes is usually a lot more than the available RAM?

Thanks a lot.0
0
40
40
40
40
92
67
67
67
67
67
105
40
40
40
40
40
40
66
66
135
135
63
66
66
104
102
49
66
79
79
79
79
79
79
79
79
79
79
79
79
79
79
79
79
79
79
79
79
78
78
78
78
94
94
94
94
94
94
94
94
94
94
78
78
78
78
78
78
78
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
94
97
97
97
97
97
97
97
97
97
97
97
113
128
83
83
98
93
93
93
93
91
137
138
137
82
69
69
40
61
61
61
61
61
83
59
75
60
61
61
2370
2370
2387
2387
2387
2403
2377
2385
2404
2404
2385
2370
2370
2385
2370
2401
2417
2417
2417
2417
2432
2403
2382
2417
2433
2449
2465
2465
2465
2370
2403
2370
2370
2385
2370
2385
2370
2386
2386
2370
2385
2401
2401
2370
2386
2370
2385
2370
2385
2385
2385
2386
2386
2385
2386
2404
2417
2449
2464
2480
2496
2371
2419
2451
2483
2404
2370
2380
2419
2451
2388
2388
4680
2371
2420
2389
2402
2388
2421
2404
2388
2388
2402
2384
2404
2404
2404
2387
2387
2387
2387
2371
2387
2404
2384
2371
2403
2435
2419
2434
2468
2483
2483
2387
2404
2404
2420
2388
2387
2387
2381
2419
2450
2387
2418
2466
2481
2503
2503
2519
2402
2434
2418
2450
2467
2498
2515
2387
2436
2469
2485
2388
2421
2388
2389
2387
2387
2434
2467
2499
2389
2388
2387
2434
2450
2483
2500
2389
2389
2404
2404
2387
2402
2418
2389
2383
2465
2483
2500
2515
2515
2531
2531
2548
2548
2548
2564
2564
2580
2580
2597
2597
2597
2613
2613
2630
2630
2645
2415
2371
2406
2437
2453
2404
2435
2387
2403
2420
2419
2435
2466
2371
2421
2421
2438
2471
2419
2450
2465
2389
2402
2434
2466
2482
2371
2370
4678
2381
2372
2386
2387
2370
2370
2370
2386
2370
2370
2403
2384
2403
2386
2418
2451
2370
2403
2386
2370
2385
2385
2385
2386
2386
2370
2386
2377
2385
2401
2370
2386

Bug#827953: maxima: Uses too much memory to build

[Camm Maguire]

Greetings, and thanks so much!

It appears that all is working as intended, that gc is triggered and
keeping the active heap to about 2.8Gb (or 3.9Gb with relocatable copy
area) of a detected max of 5.3Gb (see core pages, and physical memory).
At the end of this experiment, the rss of this process as viewed from a
separate shell should confirm these figures.  So I do not think gcl is
expanding into swap here.

As for the /proc/meminfo, all I can say is that the build employs many
other programs as well.  Might be interesting to correlate this figure
with the various stages of the build.  In other words, how much just to
run the debian/rules build target, etc.

Santiago Vila  writes:

> On Mon, 25 Jul 2016, Camm Maguire wrote:
>
>> If this is happening, then it is indeed a bug.  The intent is to use
>> physical ram only by default unless the application itself insists on
>> more.  Going into swap by default obviously defeats the performance goal
>> of expanding the memory anyway.  You can look into this by
>> 
>> gcl
>> >(room t)
>> 
>> and
>> 
>> maxima
>> (..) :lisp (room t)
>> (..) :lisp (setq si::*notify-gbc* t)
>> (..) run_testsuite();
>> (..) :lisp (room t)
>> 
>> If you can post the results for this on the 4/4 machine you describe
>> above I can see if we have a problem.
>
> Sorry for the late reply.
>
> I attach the results for maxima, on a machine with 6GB RAM and 4GB swap.
>
> I don't know how to interpret the results. In addition to the test you
> requested, when I do this
>
> grep "Committed_AS:" /proc/meminfo
>
> I get this before running maxima:
>
> Committed_AS: 254276 kB
>
> and this after the test finished, before exiting maxima:
>
> Committed_AS:6878284 kB
>
> I guess this does still not explain why Committed_AS: grows so much
> when I'm actually trying to build the Debian package.
>
> Thanks.
>
> (sid)buildd@skywalker1:~$ maxima
>
> Maxima 5.38.0 http://maxima.sourceforge.net
> using Lisp GNU Common Lisp (GCL) GCL 2.6.12
> Distributed under the GNU Public License. See the file COPYING.
> Dedicated to the memory of William Schelter.
> The function bug_report() provides bug reporting information.
> (%i1) (%i1) :lisp (room t)
> 
> 1828/182888.8% CONS FIXNUM SHORT-FLOAT LONG-FLOAT 
> BIT-VECTOR PATHNAME SPICE
>  220/220 99.8% ARRAY CHARACTER PACKAGE HASH-TABLE VECTOR 
> RANDOM-STATE CCLOSURE CLOSURE
>   67/67  56.7% STRING BIGNUM RATIO COMPLEX
>  399/399 98.8% STRUCTURE
>1/1   65.2% STREAM
>   46/46  99.4% CFUN CFDATA
>  592/592 99.8% SFUN SYMBOL READTABLE GFUN VFUN AFUN
>
> 6400/6400  contiguous (30 blocks)
>  1173337   hole
>  216/216 99.9% relocatable
>
>   3153 pages for cells
>
>   9769 total pages in core
>   9769 current core maximum pages
>216 pages reserved for gc
>3519614 pages available for adding to core
>  35556 pages reserved for core exhaustion
>
>3565155 maximum pages
>
>
> Key:
>
> WS: words per struct
> UP: allocated pages
> MP: maximum pages
> FI: fraction of cells in use on allocated pages
> GC: number of gc triggers allocating this type
>
> word size:64 bits
> page size:4096 bytes
> heap start:   0xE81000
> heap max :0x368365000
> shared library start: 0x0
> cstack start: 0x0
> cstack mark offset:   0 bytes
> cstack direction: downward
> cstack alignment: 32 bytes
> cstack max:   16001 bytes
> immfix start: 0x8000
> immfix size:  4611686018427387904 fixnums
> physical memory:  1299534 pages
> (%i1) (%i1) :lisp (setq si::*notify-gbc* t)
> 
> T
> (%i1) (%i1) run_testsuite();
> Running tests in rtest_rules: 103/103 tests passed
> Running tests in rtestnset: 597/597 tests passed
> Running tests in rtest1: 180/180 tests passed (not counting 1 expected errors)
> Running tests in rtest1a: 27/27 tests passed
> Running tests in rtest2: 144/144 tests passed (not counting 2 expected errors)
> Running tests in rtest4: 93/93 tests passed
> Running tests in rtest5: 
> ** Problem 78 ***
> Input:
> describe(sin)
>
>
> Result:
>
> error-catch
>
> This differed from the expected result:
> true
> start address -T 0x28ee010 start address -T 0x291e650 start address -T 
> 0x2925ce0 start address -T 0x292a1a0 start address -T 0x292e660 start address 
> -T 0x2936f20 start address -T 0x293c010 start address -T 0x293fb70 
> 79/80 tests passed
>
> The following 1 problem failed: (78)
> Running tests in rtest6: 39/39 tests passed
> Running tests in rtest6a: 62/62 tests passed
> Running tests in rtest6b: 16/16 tests passed
> Running tests in rtest7: 85/85 tests passed
> Running tests in rtest9: 89/89 tests passed
> Running tests in rtest9a: 76/76 tests passed
> Running tests in rtest10: 62/62 tests passed 

Bug#827953: maxima: Uses too much memory to build

[Santiago Vila]

On Mon, 25 Jul 2016, Camm Maguire wrote:

> If this is happening, then it is indeed a bug.  The intent is to use
> physical ram only by default unless the application itself insists on
> more.  Going into swap by default obviously defeats the performance goal
> of expanding the memory anyway.  You can look into this by
> 
> gcl
> >(room t)
> 
> and
> 
> maxima
> (..) :lisp (room t)
> (..) :lisp (setq si::*notify-gbc* t)
> (..) run_testsuite();
> (..) :lisp (room t)
> 
> If you can post the results for this on the 4/4 machine you describe
> above I can see if we have a problem.

Sorry for the late reply.

I attach the results for maxima, on a machine with 6GB RAM and 4GB swap.

I don't know how to interpret the results. In addition to the test you
requested, when I do this

grep "Committed_AS:" /proc/meminfo

I get this before running maxima:

Committed_AS: 254276 kB

and this after the test finished, before exiting maxima:

Committed_AS:6878284 kB

I guess this does still not explain why Committed_AS: grows so much
when I'm actually trying to build the Debian package.

Thanks.(sid)buildd@skywalker1:~$ maxima

Maxima 5.38.0 http://maxima.sourceforge.net
using Lisp GNU Common Lisp (GCL) GCL 2.6.12
Distributed under the GNU Public License. See the file COPYING.
Dedicated to the memory of William Schelter.
The function bug_report() provides bug reporting information.
(%i1) 
(%i1) :lisp (room t)


1828/182888.8% CONS FIXNUM SHORT-FLOAT LONG-FLOAT 
BIT-VECTOR PATHNAME SPICE
 220/220 99.8% ARRAY CHARACTER PACKAGE HASH-TABLE VECTOR 
RANDOM-STATE CCLOSURE CLOSURE
  67/67  56.7% STRING BIGNUM RATIO COMPLEX
 399/399 98.8% STRUCTURE
   1/1   65.2% STREAM
  46/46  99.4% CFUN CFDATA
 592/592 99.8% SFUN SYMBOL READTABLE GFUN VFUN AFUN

6400/6400  contiguous (30 blocks)
 1173337   hole
 216/216 99.9% relocatable

  3153 pages for cells

  9769 total pages in core
  9769 current core maximum pages
   216 pages reserved for gc
   3519614 pages available for adding to core
 35556 pages reserved for core exhaustion

   3565155 maximum pages


Key:

WS: words per struct
UP: allocated pages
MP: maximum pages
FI: fraction of cells in use on allocated pages
GC: number of gc triggers allocating this type

word size:64 bits
page size:4096 bytes
heap start:   0xE81000
heap max :0x368365000
shared library start: 0x0
cstack start: 0x0
cstack mark offset:   0 bytes
cstack direction: downward
cstack alignment: 32 bytes
cstack max:   16001 bytes
immfix start: 0x8000
immfix size:  4611686018427387904 fixnums
physical memory:  1299534 pages
(%i1) 
(%i1) :lisp (setq si::*notify-gbc* t)


T
(%i1) 
(%i1) run_testsuite();

Running tests in rtest_rules: 103/103 tests passed
Running tests in rtestnset: 597/597 tests passed
Running tests in rtest1: 180/180 tests passed (not counting 1 expected errors)
Running tests in rtest1a: 27/27 tests passed
Running tests in rtest2: 144/144 tests passed (not counting 2 expected errors)
Running tests in rtest4: 93/93 tests passed
Running tests in rtest5: 
** Problem 78 ***
Input:
describe(sin)


Result:

error-catch

This differed from the expected result:
true
start address -T 0x28ee010 start address -T 0x291e650 start address -T 
0x2925ce0 start address -T 0x292a1a0 start address -T 0x292e660 start address 
-T 0x2936f20 start address -T 0x293c010 start address -T 0x293fb70 
79/80 tests passed

The following 1 problem failed: (78)
Running tests in rtest6: 39/39 tests passed
Running tests in rtest6a: 62/62 tests passed
Running tests in rtest6b: 16/16 tests passed
Running tests in rtest7: 85/85 tests passed
Running tests in rtest9: 89/89 tests passed
Running tests in rtest9a: 76/76 tests passed
Running tests in rtest10: 62/62 tests passed (not counting 2 expected errors)
Running tests in rtest11: 181/181 tests passed
Running tests in rtest13: 23/23 tests passed
Running tests in rtest13s: 17/17 tests passed
Running tests in rtest14: [GC for 234547 CONS pages..(T=25).GC finished]
[GC for 188813 RELOCATABLE-BLOCKS pages..(T=19).GC finished]
[GC for 140149 STRUCTURE pages..(T=21).GC finished]
418/418 tests passed
Running tests in rtest15: [GC for 234547 CONS pages..(T=18).GC finished]
334/334 tests passed
Running tests in rtest16: 540/540 tests passed
Running tests in rtestode: 90/90 tests passed
Running tests in rtestode_zp: 30/30 tests passed
Running tests in rtest3: [GC for 235957 CONS pages..(T=22).GC finished]
149/149 tests passed
Running tests in rtest8: 164/164 tests passed
Running tests in rtest12: 79/79 tests passed (not counting 2 expected errors)
Running tests in rexamples: 137/137 tests passed
Running tests in rtesthyp: [GC for 16574 SFUN 

Bug#827953: maxima: Uses too much memory to build

[Camm Maguire]

Greetings, and thanks for your report!

Santiago Vila  writes:

> On Sun, 10 Jul 2016, Camm Maguire wrote:
>
>> Greetings, and thanks so much for your report!  More reply later, but
>> just want to point out for now that the memory used for the build
>> depends on the memory available at runtime.  These packages will build
>> just fine on machines with small memories.  The memory usage algorithm
>> is a performance booster only.
>
> Hello Camm.
>
> There are at least two problems with the current approach:
>
> * One of them (which I already pointed out) is that I can't measure how
> much this program really needs by looking at memory usage. I can do
> that for more than 3300 different packages in stretch, but I can't for
> maxima and axiom, which is a pity.

OK, this might be an inconvenience from an 'package builder memory
measurer' but is not a critical issue, right?

>
> * The other problem is that not only maxima takes as much RAM as I
> have, it seems to use all available SWAP as well! This would explain
> the effect "the more memory I have, the more memory it takes". Let's
> suppose that I build with 4 GB RAM and 4 GB SWAP. Then I measure
> that maxima takes 8GB and next time I will need 8 GB RAM to build.
>

If this is happening, then it is indeed a bug.  The intent is to use
physical ram only by default unless the application itself insists on
more.  Going into swap by default obviously defeats the performance goal
of expanding the memory anyway.  You can look into this by

gcl
>(room t)

and

maxima
(..) :lisp (room t)
(..) :lisp (setq si::*notify-gbc* t)
(..) run_testsuite();
(..) :lisp (room t)

If you can post the results for this on the 4/4 machine you describe
above I can see if we have a problem.

The available memory should be 8G, garbage collection should start at
2G, become very heavy by 0.9*4G, beyond which the system should not
expand unless the memory cannot be reclaimed by garbage collection.

Take care,
-- 
Camm Maguirec...@maguirefamily.org
==
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Bug#827953: maxima: Uses too much memory to build

[Santiago Vila]

On Sun, 10 Jul 2016, Camm Maguire wrote:

> Greetings, and thanks so much for your report!  More reply later, but
> just want to point out for now that the memory used for the build
> depends on the memory available at runtime.  These packages will build
> just fine on machines with small memories.  The memory usage algorithm
> is a performance booster only.

Hello Camm.

There are at least two problems with the current approach:

* One of them (which I already pointed out) is that I can't measure how
much this program really needs by looking at memory usage. I can do
that for more than 3300 different packages in stretch, but I can't for
maxima and axiom, which is a pity.

* The other problem is that not only maxima takes as much RAM as I
have, it seems to use all available SWAP as well! This would explain
the effect "the more memory I have, the more memory it takes". Let's
suppose that I build with 4 GB RAM and 4 GB SWAP. Then I measure
that maxima takes 8GB and next time I will need 8 GB RAM to build.

If maxima is going to take all the memory I have, whatever the amount,
it should be the available RAM only, not available RAM + available SWAP.

If it's not able to do that and it only takes the total memory in
account, meybe it should assume that at least half of that is SWAP and
avoid using it.

Currently, it seems to assume that all the memory is available to use,
which is not realistic for people using SWAP as a safety net.

Thanks.

Bug#827953: maxima: Uses too much memory to build

[Camm Maguire]

Greetings, and thanks so much for your report!  More reply later, but
just want to point out for now that the memory used for the build
depends on the memory available at runtime.  These packages will build
just fine on machines with small memories.  The memory usage algorithm
is a performance booster only.
 
Take care,

Santiago Vila  writes:

> Package: maxima
> Version: 5.38.0-3
>
> Hello Camm.
>
> The more memory it has the machine on which I build this package,
> the more memory it uses during the build, and there does not seem to
> be a limit for that.
>
> Last time it was a machine with 8GB RAM. The file /proc/meminfo
> reported a Committed_AS memory of 17823MB (more than 17GB!).
>
> This is specially bad for me because I rely on /proc/meminfo to know
> how much memory the package "really" needs.
>
> While checking for "dpkg-buildpackage -A" I have measured the memory
> required to build around 3400 different source packages in stretch.
> If I order them by required memory, these are the last 20:
>
> [...]
> gnutls28 4189
> mapnik 4199
> mia 4279
> gcc-4.9 4287
> gap-guava 4315
> seqan 4376
> yade 4509
> webkit2gtk 4558
> llvm-toolchain-3.8 4673
> aces3 4795
> python2.7 5685
> libbson 5700
> webkitgtk 7015
> acl2 7355
> gap 8268
> gcl 9375
> mame 10363
> gdk-pixbuf 11564
> maxima 17823
> axiom 21190
>
> In practice, this means several things:
>
> * As I don't currently have such a big machine, I have to rent one to
> build those packages.
>
> * As they use so much memory, I can't use a 8GB plan or a 12GB plan, I
> have to use a 24GB plan (or whatever is close).
>
> * I need a bigger machine *only* to build the packages at the end of
> the above list.
>
> I know that you have been fine-tuning the memory usage for these
> programs lately, but it still happens that the more memory II have,
> the more memory they take, so I still have the problem that I can't
> measure how much memory they really need.
>
> Thanks.
>
>
>
>

-- 
Camm Maguirec...@maguirefamily.org
==
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah

Bug#827953: maxima: Uses too much memory to build

[Santiago Vila]

Package: maxima
Version: 5.38.0-3

Hello Camm.

The more memory it has the machine on which I build this package,
the more memory it uses during the build, and there does not seem to
be a limit for that.

Last time it was a machine with 8GB RAM. The file /proc/meminfo
reported a Committed_AS memory of 17823MB (more than 17GB!).

This is specially bad for me because I rely on /proc/meminfo to know
how much memory the package "really" needs.

While checking for "dpkg-buildpackage -A" I have measured the memory
required to build around 3400 different source packages in stretch.
If I order them by required memory, these are the last 20:

[...]
gnutls28 4189
mapnik 4199
mia 4279
gcc-4.9 4287
gap-guava 4315
seqan 4376
yade 4509
webkit2gtk 4558
llvm-toolchain-3.8 4673
aces3 4795
python2.7 5685
libbson 5700
webkitgtk 7015
acl2 7355
gap 8268
gcl 9375
mame 10363
gdk-pixbuf 11564
maxima 17823
axiom 21190

In practice, this means several things:

* As I don't currently have such a big machine, I have to rent one to
build those packages.

* As they use so much memory, I can't use a 8GB plan or a 12GB plan, I
have to use a 24GB plan (or whatever is close).

* I need a bigger machine *only* to build the packages at the end of
the above list.

I know that you have been fine-tuning the memory usage for these
programs lately, but it still happens that the more memory II have,
the more memory they take, so I still have the problem that I can't
measure how much memory they really need.

Thanks.