Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
Hi, Kjetil S. Matheussen [EMAIL PROTECTED] writes: and for guile I think you use -rpath? : Yes. No. But I have ran a series of benchmarks now: Thanks for doing this! To sum up, BGC is always slower but usually more memory-efficient than GGC. In Guile + libgc, you could try adding: GC_set_free_space_divisor (2); right before `GC_expand_hp ()' in `libguile/gc.c'. AIUI, this would trade memory for CPU time. Ideally, we'd need a better benchmarking framework, where each benchmark is run tens of times and where both the average and variance are reported. Unfortunately, I don't have much time to work on it right now. Thanks, Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
On Mon, 4 Jun 2007, Ludovic Courtès wrote: Hi, Kjetil Svalastog Matheussen [EMAIL PROTECTED] writes: [2] tla my-default-archive [EMAIL PROTECTED] tla get guile-core--boehm-gc So that's the one you've been using and referring to as Guile + Boehm GC? Glad to hear it! ;-) Sorry I didn't write that in my first mail. Did you make sure to compile libgc the right way, so that locking in with a multi-threaded libgc doesn't hurt performance[0]? I have tried with and without both --enable-threads=posix and --enable-parallel-mark. No difference, it seems to work perfectly anyway. The main thing that needs to be done before we can consider this solution now is to compare both memory usage _and_ execution time of the two Guiles. Yes, but for some kinds of software, like programs with custom gui's, sound processing programs, interactive graphical programs, interactivety is exclusively more important than those two. I consider the freeze that the guile's built-in gc cause to be its biggest problem. In fact, I don't think execution time is an important factor here at all. Don't get me wrong, of course execution time is important in general, but a change in execution time of a gc benchmark program within the factor of two sounds like an insignificant difference compared to what is gained in interactivity. For instance, with the default settings, Guile + Boehm GC is slightly faster than Guile when running `gcbench.scm', but it might consume more memory. This needs to be analyzed. Reports as to how this impacts SND are also more than welcome, of course! ;-) I'm going to run Guile + Boehm GC + SND for a while now and report back if anything unusual happens. I might even release a special version of SND with guile + the Boehm GC included to the public in the near future, because of the huge advantages it has. This might generate a lot of feedback. And if no one reports back, then its a good sign that it works well. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
On Mon, 4 Jun 2007, Kjetil S. Matheussen wrote: The main thing that needs to be done before we can consider this solution now is to compare both memory usage _and_ execution time of the two Guiles. Yes, but for some kinds of software, like programs with custom gui's, sound processing programs, interactive graphical programs, interactivety is exclusively more important than those two. I consider the freeze that the guile's built-in gc cause to be its biggest problem. Actually, I haven't used guile for other interactive stuff than snd. And snd is huge. It could be that the boehm gc works a lot lot better for really large programs than guile's gc. I'll try to run the gc benchmark program inside snd (with lots of scheme files loaded), it might give some interesting results. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
Hi, Kjetil S. Matheussen [EMAIL PROTECTED] writes: Yes, but for some kinds of software, like programs with custom gui's, sound processing programs, interactive graphical programs, interactivety is exclusively more important than those two. I consider the freeze that the guile's built-in gc cause to be its biggest problem. In fact, I don't think execution time is an important factor here at all. Don't get me wrong, of course execution time is important in general, but a change in execution time of a gc benchmark program within the factor of two sounds like an insignificant difference compared to what is gained in interactivity. Right, interactivity may be an important metric for some classes of applications, albeit one that can sometimes be hard to capture. That said, I guess a large fraction of Guile users will be even more concerned about memory consumption and execution time. I'm going to run Guile + Boehm GC + SND for a while now and report back if anything unusual happens. I might even release a special version of SND with guile + the Boehm GC included to the public in the near future, because of the huge advantages it has. This might generate a lot of feedback. And if no one reports back, then its a good sign that it works well. Alright, I'll wait for bug reports. ;-) Actually, I haven't used guile for other interactive stuff than snd. And snd is huge. It could be that the boehm gc works a lot lot better for really large programs than guile's gc. I'll try to run the gc benchmark program inside snd (with lots of scheme files loaded), it might give some interesting results. Cool, let us know how it goes. I'll probably sync my `boehm-gc' branch with HEAD one of these days, too (I think it doesn't lack a lot of upstream changes anyway). Thanks! Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
On Mon, 4 Jun 2007, Ludovic Courtès wrote: Actually, I haven't used guile for other interactive stuff than snd. And snd is huge. It could be that the boehm gc works a lot lot better for really large programs than guile's gc. I'll try to run the gc benchmark program inside snd (with lots of scheme files loaded), it might give some interesting results. Cool, let us know how it goes. I used the gc benchmark program posted by Han-Wen: http://www.mail-archive.com/guile-devel@gnu.org/msg01077.html libgc (v6.8) was compiled with the --enable-threads=posix only. For normal guile I used 1.8.1, default gentoo build. (-O2) For GC guile, I built from your tree. (also -O2) Snd is snd-ls V0.9.7.12. My machine is an XP2800, single processor. Running the benchmark program directly in guile gives no difference. Both spent about 50 seconds running the test. Inside snd is another matter: [1] [2] [3] Guile gc 96mb 176mb54s. Boehm gc 99mb 107mb 118s. [1] Memory before running test reported by top. [2] Memory after running test reported by top [3] Time to run test in seconds. All tests performed twice to ensure nothing special happened in one of them. The numbers here are from last test, and are about the same as from the run. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
Hi, Kjetil S. Matheussen [EMAIL PROTECTED] writes: libgc (v6.8) was compiled with the --enable-threads=posix only. So `THREAD_LOCAL_ALLOC' was defined in your libgc build, right? Running the benchmark program directly in guile gives no difference. Both spent about 50 seconds running the test. Directly in Guile means that you just run: $ ./pre-inst-guile gcbench.scm Is that correct? Inside snd is another matter: [1] [2] [3] Guile gc 96mb 176mb54s. Boehm gc 99mb 107mb 118s. [1] Memory before running test reported by top. [2] Memory after running test reported by top [3] Time to run test in seconds. What does inside snd mean exactly? Is snd multi-threaded? Does it use `scm_without_guile ()' for instance (e.g., when a thread blocks for I/O)? Does the second run of `gcbench.scm' within Guile alone (no snd) show similar performance behavior in the libgc case? I.e.: $ guile-boehm --no-debug guile (load gcbench.scm) ;;; takes 50s. to complete guile (load gcbench.scm) ;;; takes 118s. to complete For libgc, there are a few environment variables that might be influential, e.g., `GC_MAXIMUM_HEAP_SIZE' (see `README.environment' from libgc). Likewise for Guile's GC, but they're undocumented AFAIK (grep for `scm_getenv_int' in the `libguile' directory). Thanks, Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
On Mon, 4 Jun 2007, Ludovic Courtès wrote: Hi, Kjetil S. Matheussen [EMAIL PROTECTED] writes: libgc (v6.8) was compiled with the --enable-threads=posix only. So `THREAD_LOCAL_ALLOC' was defined in your libgc build, right? Yes: -DTHREAD_LOCAL_ALLOC=1 Running the benchmark program directly in guile gives no difference. Both spent about 50 seconds running the test. Directly in Guile means that you just run: $ ./pre-inst-guile gcbench.scm Is that correct? I have to compile snd against libguile and its header files, so I have two guile installed in two different directories, and set LD_LIBRARY_PATH to the right place before running. LD_LIBRARY_PATH is set automatically in the script that starts snd, and for guile I think you use -rpath? : ldd -r /usr/bin/guile linux-gate.so.1 = (0xe000) libguile.so.17 = /usr/lib/libguile.so.17 (0xb7ed2000) ldd -r /home/kjetil/site/bin/guile linux-gate.so.1 = (0xe000) libguile.so.18 = /home/kjetil/site/lib/libguile.so.18 (0xb7ed8000) libgc.so.1 = /home/kjetil/site/lib/libgc.so.1 (0xb7ea8000) Inside snd is another matter: [1] [2] [3] Guile gc 96mb 176mb54s. Boehm gc 99mb 107mb 118s. [1] Memory before running test reported by top. [2] Memory after running test reported by top [3] Time to run test in seconds. What does inside snd mean exactly? Snd has its own REPLs which interacts with libguile. Is snd multi-threaded? Does it use `scm_without_guile ()' for instance (e.g., when a thread blocks for I/O)? No, not when snd is being run standalone. Does the second run of `gcbench.scm' within Guile alone (no snd) show similar performance behavior in the libgc case? I.e.: $ guile-boehm --no-debug guile (load gcbench.scm) ;;; takes 50s. to complete guile (load gcbench.scm) ;;; takes 118s. to complete No. But I have ran a series of benchmarks now: (benchmark 20) in Guile: Guile: 4.4mb / 277s / 149mb Boehm: 4.4mb / 243s / 148mb (benchmark 20) in Snd: Guile: 96mb / 336s / 216mb Boehm: 99mb / 442s / 257mb [1] (benchmark) (benchmark), in Snd: Guile: 96mb / 58s / 174mb / 59s / 174mb Boehm: 99mb / 105s / 108mb / 99s / 116mb - (benchmark) (benchmark), in Guile: Guile: 4.4mb / 50s / 78mb / 50s / 80mb Boehm: 5.2mb / 74s / 55mb / 71s / 63mb [2] [1] Memory usage flipped over from 182.2 to 256.6 right before the benchmark was finished. [2] I ran another time, and got 75s. It seems like my previous result of 50s might have been ran with the wrong program. For libgc, there are a few environment variables that might be influential, e.g., `GC_MAXIMUM_HEAP_SIZE' (see `README.environment' from libgc). Likewise for Guile's GC, but they're undocumented AFAIK (grep for `scm_getenv_int' in the `libguile' directory). Any tip about what to try? I did some tweaking with it 7 years ago when I ported it to beos and worked on the amigaos port, but haven't touched it since. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
On Mon, 4 Jun 2007, Kjetil S. Matheussen wrote: (benchmark 20) in Guile: Guile: 4.4mb / 277s / 149mb Boehm: 4.4mb / 243s / 148mb Unfortunately, that one was probably wrong. Here is the retest result with the Boehm GC guile: Boehm: 5.2mb / 380s / 164mb ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
(CC guile-dev) On Sun, 3 Jun 2007, Bill Schottstaedt wrote: Thanks! I'll merge your changes into my versions this morning. I saw some discussion about the Boehm GC, but I didn't try it -- let me know how it goes! I have it running now, and it feels much better. (gc) still triggers a glip in the sound though, but thats to be expected. But the good things are (with snd_conffils.scm loaded): * I don't hear any glips in the sound using a dac-size of 64. Can't remember that ever working before. Even a dac-size of 1 almost works. * When changing the range, there is no sign of the graphics now and then lagging behind. I have never seen that before. However, I need to run a test with the normal guile 1.9 as well. It might not be because of the Boehm GC, but I think so. I'll report back in case it wasn't the new GC. I'm thinking about releasing a new version of snd-ls with embedded guile + boehm gc patch. It'll be a monster package, but snd feels so nice now that I might do it. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
On Sun, 3 Jun 2007, Kjetil Svalastog Matheussen wrote: (CC guile-dev) On Sun, 3 Jun 2007, Bill Schottstaedt wrote: Thanks! I'll merge your changes into my versions this morning. I saw some discussion about the Boehm GC, but I didn't try it -- let me know how it goes! I have it running now, and it feels much better. (gc) still triggers a glip in the sound though, but thats to be expected. I have tried Snd with Guile 1.9 with and without the Boehm GC on the same two computers (amd xp3000 and intel mac MP 1.3gz) using the same version of Snd (8.4/snd-ls 0.9.7.12). And my experiences are as follows: But the good things are (with snd_conffils.scm loaded): * I don't hear any glips in the sound using a dac-size of 64. Can't remember that ever working before. Even a dac-size of 1 almost works. This is the same behaviour for both versions of guile. On the MP machine, there is never any glips at all, on the other there are some. * When changing the range, there is no sign of the graphics now and then lagging behind. I have never seen that before. Large difference. Without the Boehm GC, graphic freezes for about 0.5 seconds each 4th seconds. Very frustrating, just as with guile 1.8.1, but better than 1.6.x though. With the Boehm GC, on the other hand: no freeze at all. Conclusion: Guile with the Boehm GC seems to improve interactivety for one guile programs by a large amount, somehow. It would be nice if the Boehm GC was used in guile 1.8 as well, and enabled by default. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Experience with guile + Boehm GC in SND (Re: New versions of rt-compiler.scm, rt-examples.scm and rt.tex
On Sun, 3 Jun 2007, Kjetil Svalastog Matheussen wrote: On Sun, 3 Jun 2007, Kjetil Svalastog Matheussen wrote: (CC guile-dev) On Sun, 3 Jun 2007, Bill Schottstaedt wrote: Thanks! I'll merge your changes into my versions this morning. I saw some discussion about the Boehm GC, but I didn't try it -- let me know how it goes! I have it running now, and it feels much better. (gc) still triggers a glip in the sound though, but thats to be expected. I have tried Snd with Guile 1.9 with and without the Boehm GC on the same two computers (amd xp3000 and intel mac MP 1.3gz) using the same version of Snd (8.4/snd-ls 0.9.7.12). And my experiences are as follows: But the good things are (with snd_conffils.scm loaded): * I don't hear any glips in the sound using a dac-size of 64. Can't remember that ever working before. Even a dac-size of 1 almost works. This is the same behaviour for both versions of guile. On the MP machine, there is never any glips at all, on the other there are some. * When changing the range, there is no sign of the graphics now and then lagging behind. I have never seen that before. Large difference. Without the Boehm GC, graphic freezes for about 0.5 seconds each 4th seconds. Very frustrating, just as with guile 1.8.1, but better than 1.6.x though. With the Boehm GC, on the other hand: no freeze at all. Here's another example of the difference between Guile's own built-in GC [1] and the Boehm GC [2]: http://www.notam02.no/~kjetism/jack_capture_01.ogg http://www.notam02.no/~kjetism/jack_capture_02.ogg These are two soundfiles where Snd is used as a realtime software synthesizer receiving midi from an external program [3]. One of the files is produced using Guile with Guile's own built-in GC and the other is produced using Guile with the Boehm GC. I won't tell which is which, but I personally think its pretty obvious that one of them has a lot more steady rhythm than the other. The one I think has a lot more steady rhythm is produced with the Boehm GC. This also gives some interesting opportunities. Earlier I considered scheduling sound from scheme as mostly useless because guile's garbage collector often decides to take long brakes. But with the Boehm GC, it actually sounds pretty descent. Scheduling from scheme is also a lot more convenient than the current situation where you must calculate start and end points in the realtime code to get proper timing, so this is great progress. [1] ftp://ftp.dt.e-technik.uni-dortmund.de/pub/guile/snapshots/guile-core.unstable-20070429.tar.gz [2] tla my-default-archive [EMAIL PROTECTED] tla get guile-core--boehm-gc [3] (begin (load rt-examples.scm) (start-synth /hom/kjetism/Blub_mono16.wav 60) (system aplaymidi --port=rt-midi /hom/kjetism/bwv539.mid )) ;;http://snd.cvs.sourceforge.net/snd/cvs-snd/rt-examples.scm?revision=1.16view=markup ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Hi, I'm finally getting back to this (sorry for the delay!). Han-Wen Nienhuys [EMAIL PROTECTED] writes: I've patched it a bit to use GC_typed alloc for tagged data. It probably doesn't make much of a difference, since 90% of the data is regular cells, but see http://www.xs4all.nl/~hanwen/public/software/guile-bgc.patch With the tree benchmark (included in patch, I get 54 secs (Guile 1.8) vs. 1:25 (typed BGC). I forgot to measure regular BGC, though. Thanks for the patch. I tried it out but it doesn't apparently yield any performance improvement (see below). Note that BGC has ALL_INTERIOR_POINTERS switched on by default nowadays, which means that it may scan too much. You could try switching that off. Good idea. I tried this (patch available in my Arch branch) and it does indeed yield a slight improvement. Here's the summary of the various performance measurements, each time running the whole `gcbench.scm' that you provided: * Guile 1.8 52 sec. (+0%) * GBGC w/ tagged allocs 127 (+40%) * GBGC w/o tagged allocs 118 (+33%) * GBGC + no interior pointers 113 (+29%) * GBGC + no interior ptrs + `GC_expand_hp'113 (+29%) In the last case, `GC_expand_hp ()' is systematically used to increase the initial heap size at startup time; this may have a positive impact on short-lived programs, but doesn't have any impact here, understandably. I guess we need other bright ideas. ;-) Thanks, Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
[EMAIL PROTECTED] (Ludovic Courtès) writes: Yes, `autogen.sh' makes a lot of assumptions about what your file system should contain. Personally, I no longer use it: I use the more reliable (and less intrusive) `autoreconf -i'. I think we should either remove that script or simply have it contain that `autoreconf' invocation. Perhaps, but we'd need more discussion of the other things that the script does first. Given that you can choose just to use autoreconf if you want to, I don't think the discussion's worth it right now. Regards, Neil ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Hi, Han-Wen Nienhuys [EMAIL PROTECTED] writes: Ludovic Courtès schreef: $ tla register-archive http://www.laas.fr/~lcourtes/software/arch-2005/ $ tla get [EMAIL PROTECTED]/guile-core--boehm-gc--1.9 gcc -DHAVE_CONFIG_H -I. -I. -I.. -I.. -I.. -g -O2 -Wall -Wmissing-prototypes -Werror -MT libguile_la-gc.lo -MD -MP -MF .deps/libguile_la-gc.Tpo -c gc.c -fPIC -DPIC -o .libs/libguile_la-gc.o cc1: warnings being treated as errors gc.c:230: warning: 't_before_gc' defined but not used make[2]: *** [libguile_la-gc.lo] Error 1 Yeah, thanks. `-Werror' is too harsh. ;-) Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Ludovic Courtès schreef: Hi Han-Wen, Han-Wen Nienhuys [EMAIL PROTECTED] writes: Ludovic Courtès schreef: But please, relax about performance, we still haven't run any meaningful benchmark. ;-) btw, is there any news on this patch? I'm quite keen on dumping LilyPond's Rational class in favor of Scheme rationals, but adding gc mark functions for that is just too much work; BGC would be a nice soluiton to this. Well, no real news. My not-so-meaningful benchmarks (running a loop[*]) reproductively show that GBGC is noticeably slower than real Guile (1.5 to 2 times slower). On larger samples the difference gets smaller. With a 128mb footprint (GC tree benchmark), the difference is that BGC is just 23% slower, -- Han-Wen Nienhuys - [EMAIL PROTECTED] - http://www.xs4all.nl/~hanwen ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Ludovic Courtès schreef: Well, no real news. My not-so-meaningful benchmarks (running a loop[*]) reproductively show that GBGC is noticeably slower than real Guile (1.5 to 2 times slower). with some final tweaks I got it to 26% (large test) and 32% (small test) slower. -- Han-Wen Nienhuys - [EMAIL PROTECTED] - http://www.xs4all.nl/~hanwen ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Ludovic Courtès schreef: But please, relax about performance, we still haven't run any meaningful benchmark. ;-) btw, is there any news on this patch? I'm quite keen on dumping LilyPond's Rational class in favor of Scheme rationals, but adding gc mark functions for that is just too much work; BGC would be a nice soluiton to this. -- Han-Wen Nienhuys - [EMAIL PROTECTED] - http://www.xs4all.nl/~hanwen ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
On Wed, 2006-05-31 at 10:49 +0200, Ludovic Courtès wrote: You can get it from my Arch archive: $ tla register-archive http://www.laas.fr/~lcourtes/software/arch-2005 $ tla get -A [EMAIL PROTECTED] guile-core--boehm-gc--1.9 \ guile-core--bgc A tarball and a diff against current CVS HEAD are also available: http://www.laas.fr/~lcourtes/software/arch-2005/guile-core/guile-core--boehm-gc/guile-core--boehm-gc--1.9/patch-35/guile-core--boehm-gc--1.9--patch-35.tar.gz http://www.laas.fr/~lcourtes/software/guile/guile-core--bgc--patch-35.diff.gz SHA1 sums: 6b6867dc637a24d7f9f25f99fb7d81c8a54b9190 guile-core--bgc--patch-35.diff.gz 4a54c2489dbca7e5415be788c3d1fdc9884584f9 guile-core--boehm-gc--1.9--patch-35.tar.gz Your patch touches a lot of files and reverts them to older versions. Could you resync with cvs head, and send another patch with only the real changes? If you don't have time I can do it. Has anyone actually tried testing the performance of a real guile program with the patch yet? -- http://unknownlamer.org Jabber: [EMAIL PROTECTED] AIM:unknownlamer IRC:[EMAIL PROTECTED] Real Men use Free Software; Open Source is for tools 443E 4F1A E213 7C54 A306 E328 7601 A1F0 F403 574B signature.asc Description: This is a digitally signed message part ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
On 6/2/06, Neil Jerram [EMAIL PROTECTED] wrote: Perhaps we aren't yet running a truly representative set of benchmarks though? One should try to run some larger application with a large set of slowly changing data structures, where the benefits of a generational collector gets a chance to shine through. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Hi, [EMAIL PROTECTED] (Han-Wen Nienhuys) writes: Boehm is generational, AFAIK. Virtually everyone uses BGC. GCJ, MzScheme, BigLoog, GNU Obj-C, etc. Yeah, and it's actively maintained and actively used. The mailing list is active as well and Hans Boehm has been very helpful answering my questions. Also, it's ported to a wide range of platforms, it's highly tuned, it's designed to live with and take advantage of preemptive multithreading, etc. It's the result of a decade of design and implementation work, as can be seen from http://www.hpl.hp.com/personal/Hans_Boehm/gc/ . I believe we would never have sufficient manpower to compete with it (and it seems that most other language runtime implementors arrived to the same conclusion). Thanks, Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Hi, Neil Jerram [EMAIL PROTECTED] writes: Fascinating! Assuming we can resolve the details you have listed, what are the other high-level pros/cons, apart from performance? Does this mean we would discard all Guile's own GC code? Also, is Boehm GC as sophisticated as the generational GC ideas that people have talked over the last year about adding Guile, or can we expect future Boehm GC development to cover this? As Han-Wen said, BGC is generational. More details are available from: http://www.hpl.hp.com/personal/Hans_Boehm/gc/gcdescr.html For Guile, the main advantage is maintainability: using BGC significantly reduces code complexity (i.e., it removes the hardest-to-read parts of the code, it disentangles GC code from functional code), it may improve portability, and it has the potential to improve performance --- although that potential hasn't shown up yet. ;-) From a technical viewpoint, it is much, much easier to use, and avoids many common programming errors: unless otherwise specified, every register and every piece of memory is scanned for pointers, not only the heap (however, only heap regions allocated via the GC allocation routines are scanned). Also, it periodically clears the unused part of the stack, which may help avoid certain leaks (as the eval-stack leak described in [0]). Thanks, Ludovic. [0] http://lists.gnu.org/archive/html/guile-devel/2005-11/msg00047.html ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Hi, [EMAIL PROTECTED] (Han-Wen Nienhuys) writes: The per object GC stats are a hack of mine, and although I would be sad to see it go (it makes debugging memory leaks easier), I think getting BGC is worth it. I don't see the point of the general GC stats. I think I've never ever used it. Same for me: I find `gc-live-object-stats' more helpful than `gc-stats'. But as I said, even if we no longer have it in-core, we should be able to partially emulate it using guardians et al. Isn't it is possible to get other lowlevel statistics, or a list of live objects from BGC? That doesn't seem to be the case, but I need to investigate it further. Anyway, even if we can have such information, it will be lower-level than `gc-stats', so not very useful I'm afraid. there is a more useful benchmark at http://www.xs4all.nl/~hanwen/public/software/GCBench.scheme It's part of a suite by Hans Boehm of GC benchmarks, and I think you can find more Scheme transliterations on the net. Thanks for the pointer! I'll look into it. Thanks, Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
Hi, Mikael Djurfeldt [EMAIL PROTECTED] writes: Yet, as long as the current GC is more efficient (as measured by performance tests), there is no reason to switch, right? Well, it's still unclear whether the current GC is more efficient, and how much more if it is. Furthermore, the GBGC code is a few weeks old so it may be possible to tune it a bit more. IMO, although I'm quite concerned with performance, I don't think it should be the only criterion: maintainability and portability are important as well. The fact that Bigloo uses BGC also tends to reassure me: if Guile can someday perform as bad as Bigloo compiled code (or simply, as bad as its interpreter), then I'll be very happy. ;-) Thanks, Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
[EMAIL PROTECTED] (Han-Wen Nienhuys) writes: Boehm is generational, AFAIK. Virtually everyone uses BGC. GCJ, MzScheme, BigLoog, GNU Obj-C, etc. See, http://www.hpl.hp.com/personal/Hans_Boehm/gc/ for a longer list. Thanks; lots of interesting stuff there. By using BGC, you potentially loose a bit of performance, since BGC isn't specialized for LISP/Scheme, but you get a system that is much better overall researched and tuned. I like the better research and the maintainability angles, but I don't see how we can say that BGC is better tuned than Guile's own GC, if current data shows it performing less well. Perhaps we aren't yet running a truly representative set of benchmarks though? Regards, Neil ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
[EMAIL PROTECTED] (Ludovic Courtès) writes: unless otherwise specified, every register and every piece of memory is scanned for pointers, not only the heap (however, only heap regions allocated via the GC allocation routines are scanned). I wonder if this is the main cause of BGC not performing as well as Guile's own GC? Guile's own GC knows, for example, that there's no point scanning the characters of a string, or the elements of a uniform vector, and that SMOB memory looks after itself. Does BGC give us any way to feed this kind of information in, so that BGC could avoid such scans in the same way Guile's GC does? Regards, Neil ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Guile + Boehm GC: First Remarks
Hello, Some time ago, I started porting Guile to Boehm GC, mostly in order to see whether this was feasible without loss of functionality and while remaining as compatible as possible with the current Guile. The good news is that I got to a working, almost as feature-full, and mostly-compatible, version of Guile that uses BGC. Below are details about the almost and mostly qualifiers, as well as pointers to the code. ;-) Feedback is welcome! * Memory Allocation In the current prototype, I changed the semantics of `scm_gc_malloc ()' so that it basically does a `GC_MALLOC ()'. This means that the data returned by `scm_gc_malloc ()' will: 1. be scanned for pointers (Scheme references); 2. automatically be garbage-collected. I added a variant, `scm_gc_malloc_pointerless ()' which corresponds to `GC_MALLOC_ATOMIC ()', i.e., the returned memory area will automatically be garbage-collected but it will not be scanned for pointers. This is useful, e.g., for stringbufs. This is different from the current semantics of `scm_gc_malloc ()' where it just [i]nforms the GC that the memory [...] can potentially be freed during a GC. However, it is (almost) a backward-compatible change: code that currently allocates memory using `scm_gc_malloc ()' is expected to free it explicitly with `scm_gc_free ()' and this will just work fine. Unfortunately, the combination `scm_gc_unregister_collectable_memory' + `free', which is supposed to work now, will no longer work. Hopefully, this is a rare enough idiom and we can forget about it (?). These two `scm_gc_malloc' functions are very convenient since Guile-aware object implementations (e.g., SMOBs) can use them to allocate memory associated to the Scheme object and they no longer need to explicitly mark/free it (see below). * Marking For SMOBs, use of a custom mark procedure is still supported, but should be useless in most cases: if `scm_gc_malloc ()' is used to allocate the memory associated to a SMOB, then this memory will automatically get marked and will be freed when needed. * Finalization Guile basically offers three finalization mechanisms: guardians, the free procedure for SMOB types, and the free procedure for structs. All of them are implemented using libgc's finalizers, and they can coexist peacefully (e.g., a SMOB whose type defines a free procedure can also be guarded by one or more guardians). However, the implementation assumes that nobody besides guardians, SMOBs, and structs directly use libgc's finalizers (which is a reasonable assumption IMO). * Weak Stuff Guile has two weak data types: weak hash tables and weak vectors. (Actually, it also has a third one, weak alist vectors, which is very similar to weak hash tables and thus should vanish --- that's the topic of another discussion) Both of them are implemented using libgc's disappearing links. More precisely, weak vectors are implemented as regular vectors with a specific memory allocation method. Weak hash tables are implemented as non-weak vectors of weak alists. A weak alist is defined as a list of weak pairs (weak-car pairs, weak-cdr pairs, or doubly-weak pairs). libgc's disappearing links work by setting weak pointers to NULL when the objects they point to becomes unreachable. Because of this, an extra level of indirection had to be added for weak vectors and weak pairs: we need to make sure that user code never sees `PTR2SCM (NULL)'. As a consequence, weak vectors can only be accessed via `scm_c_vector_ref ()' (I did not add the extra test to `SCM_SIMPLE_VECTOR_REF ()' so that regular vectors don't suffer from this), and weak hash tables can only be accessed via the authorized functions. Unfortunately, there are pieces of code in Guile that break this encapsulation. In particular, `lookup_interned_symbol ()' does this, as well as `environments.c' (but we don't care about this one since it's unused). I believe the best fix here would be to actually allow for read access to hash tables with no function-call overhead, e.g., by exporting part of the API as inlines, and also by exporting a lower-level API (allowing for access to the raw buckets and bucket vectors). Again, that's the topic for another discussion. ;-) * GC Hooks Only `before-gc-hook' and `after-gc-hook' (which are run before/after any `scm_i_gc ()' call) are honored. The others just cannot be honored because they are too low-level. * GC Stats I haven't thought much about this, but it looks is quite problematic. We probably can't provide the level of details of `gc-stats'. And we cannot either provide per-object-type information as currently provided by `gc-live-object-stats' (this would require registering a finalizer for each and every object in order to update the object count). OTOH, per-object-type information, when needed, can be obtained by writing
Re: Guile + Boehm GC: First Remarks
In article [EMAIL PROTECTED], Ludovic Courtès [EMAIL PROTECTED] wrote: * GC Stats I haven't thought much about this, but it looks is quite problematic. We probably can't provide the level of details of `gc-stats'. And we cannot either provide per-object-type information as currently provided by `gc-live-object-stats' (this would require registering a finalizer for each and every object in order to update the object count). OTOH, per-object-type information, when needed, can be obtained by writing code that makes use of one of the finalization mechanism. For instance, one can wrap a type constructor, or a class `initialize' method, so that every new instance is added to a guardian; then, said guardian can be periodically queried to update the instance counter. I haven't done any serious performance measurement, but just to give an idea, the test suite roughly takes as long to run with the current Guile and with GBGC. However, GBGC seems to be slightly slower when running programs that allocate and discard lots of cells: Hi, The per object GC stats are a hack of mine, and although I would be sad to see it go (it makes debugging memory leaks easier), I think getting BGC is worth it. I don't see the point of the general GC stats. I think I've never ever used it. Isn't it is possible to get other lowlevel statistics, or a list of live objects from BGC? $ time ../pre-inst-guile -c (use-modules (srfi srfi-1)) (fold (lambda (x y) (list x)) '() (iota 30)) (exit 0) real0m3.550s user0m3.383s sys 0m0.162s $ time guile-1.8.0 -c (use-modules (srfi srfi-1)) (fold (lambda (x y) (list x)) '() (iota 30)) (exit 0) real0m2.738s user0m2.650s sys 0m0.059s there is a more useful benchmark at http://www.xs4all.nl/~hanwen/public/software/GCBench.scheme It's part of a suite by Hans Boehm of GC benchmarks, and I think you can find more Scheme transliterations on the net. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC: First Remarks
In article [EMAIL PROTECTED], Neil Jerram [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] (Ludovic Courtès) writes: Sorry for the long mail and if you have any comment, don't hesitate! Fascinating! Assuming we can resolve the details you have listed, what are the other high-level pros/cons, apart from performance? Does this mean we would discard all Guile's own GC code? Also, is Boehm GC as sophisticated as the generational GC ideas that people have talked over the last year about adding Guile, or can we expect future Boehm GC development to cover this? Is Boehm GC an active project? What else uses it? Boehm is generational, AFAIK. Virtually everyone uses BGC. GCJ, MzScheme, BigLoog, GNU Obj-C, etc. See, http://www.hpl.hp.com/personal/Hans_Boehm/gc/ for a longer list. By using BGC, you potentially loose a bit of performance, since BGC isn't specialized for LISP/Scheme, but you get a system that is much better overall researched and tuned. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Guile + Boehm GC
Hi all, Integration of Boehm's GC within Guile was discussed at length in the past, especially back in 2000. Apparently, Han-Wen even got to the point of having an preliminary implementation of Guile that uses it[0]. Unfortunately the URL mentioned there is no longer valid. Han-Wen: Could you summarize your conclusions and maybe make your code available as well? More generally, do people (still) believe that Guile should use BGC? Back then, there was apparently some consensus that it would be a Good Thing. Thanks, Ludovic. [0] http://sources.redhat.com/ml/guile/2000-07/msg00252.html ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC
[EMAIL PROTECTED] (Ludovic =?iso-8859-1?Q?Court=E8s?=) writes: Hi all, Integration of Boehm's GC within Guile was discussed at length in the past, especially back in 2000. Apparently, Han-Wen even got to the point of having an preliminary implementation of Guile that uses it[0]. Unfortunately the URL mentioned there is no longer valid. Han-Wen: Could you summarize your conclusions and maybe make your code available as well? More generally, do people (still) believe that Guile should use BGC? Back then, there was apparently some consensus that it would be a Good Thing. I looked at this about 2 years ago. My goodness it was complicated. I thought that there was a good argument for overhauling the objects and the GC together. I thought it would be particularly interesting if glib objects were used as guile's objects. This wouldn't make overhauling the GC any easier (except by removing cruft) but it would be a worthwhile exercise in reduction IMHO. Nic Ferrier ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC
Hi Han-Wen, [EMAIL PROTECTED] (Han-Wen Nienhuys) writes: it's here, http://www.xs4all.nl/~hanwen/public/guile/guile-boehm.gz note that this is a patch for the old GC code, before I took a stab at rewriting. The good news is that it looks quite simple! The main changes are located here (diffstat): libguile/gc.c | 854 +- libguile/gc.h | 57 -- However, it looks like you didn't look at weak hash tables back then. My impression is that porting them would be the most difficult task. It was a quick hack, and performance was a little worse, but I think it could be just as performant as GGC. Note that many interpreters (eg. MzScheme) use BGC by default. Also, measuring performance of GC is hard, since there is a space/time tradeoff, so you should really measure the tradeoff, something I haven't done. Yes. But GGC's performance is terrible IMO (see my posts a few months ago), and also GGC is used only within Guile. So I'm confident that BGC may work better, and even if it doesn't, chances are that it may improve over time given the number of users it has had. I think it would be good to move to BGC. BGC is used by many other interpreters, and receives more attention wrt performance and bugfixing than GGC. Dropping code is almost always good Agreed. BTW, I ran across Judy trees today, http://judy.sourceforge.net/ would it be worthwhile to try replacing resizing hash tables in Guile with Judy trees? I expect that it would incur a significant speedup. I was not aware of this project but I'm not so sure it would have a significant impact. Currently, Guile spends a huge amount of time managing memory and improving this should be priority #1 I think. Thanks, Ludovic. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
Re: Guile + Boehm GC
In article [EMAIL PROTECTED], Ludovic Courtès [EMAIL PROTECTED] wrote: http://www.xs4all.nl/~hanwen/public/guile/guile-boehm.gz note that this is a patch for the old GC code, before I took a stab at rewriting. The good news is that it looks quite simple! The main changes are located here (diffstat): libguile/gc.c | 854 +- libguile/gc.h | 57 -- However, it looks like you didn't look at weak hash tables back then. My impression is that porting them would be the most difficult task. Yes, correct. It was a quick hack. BTW, I ran across Judy trees today, http://judy.sourceforge.net/ would it be worthwhile to try replacing resizing hash tables in Guile with Judy trees? I expect that it would incur a significant speedup. I was not aware of this project but I'm not so sure it would have a significant impact. Currently, Guile spends a huge amount of time managing memory and improving this should be priority #1 I think. Quite possibly, but LilyPond is my main application, and Lily also spends a lot of time doing various kinds of hashtable lookups. ___ Guile-devel mailing list Guile-devel@gnu.org http://lists.gnu.org/mailman/listinfo/guile-devel
