Ludovic Courtès skribis:
> It goes OK until building
> /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
> consumes memory until it gets OOM-killed.
The culprit is easily found (here on just the 22K packages of ‘guix’, so
less than half of what you get with ‘guix-cran’):
--8<---cut here---start->8---
$ guix repl
GNU Guile 3.0.9
Copyright (C) 1995-2023 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.
Enter `,help' for help.
scheme@(guix-user)> ,m(gnu packages)
scheme@(gnu packages)> ,use(statprof)
scheme@(gnu packages)> (gcprof (lambda () (generate-package-cache "/tmp/t")))
% cumulative self
time seconds seconds procedure
50.00 13.77 13.77 ice-9/boot-9.scm:2498:4
6.94 1.91 1.91 display
5.56 1.53 1.53 object->string
2.78 0.76 0.76 reverse
2.78 0.76 0.76 gensym
2.78 0.76 0.76 ice-9/boot-9.scm:2217:0:%load-announce
1.39 6.88 0.38 gnu/packages.scm:388:4:expand-cache
1.39 0.76 0.38 srfi/srfi-1.scm:1028:0:lset-intersection
1.39 0.38 0.38 number->string
1.39 0.38 0.38 gnu/packages/crates-io.scm:17783:2:arguments
1.39 0.38 0.38 open-output-string
1.39 0.38 0.38 gnu/packages/crates-io.scm:18450:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:36208:2:arguments
1.39 0.38 0.38 guix/packages.scm:1320:8:mproc
1.39 0.38 0.38 gnu/packages/crates-io.scm:25101:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:29972:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:883:2:arguments
1.39 0.38 0.38 ice-9/vlist.scm:449:0:vhash-cons
1.39 0.38 0.38 append
1.39 0.38 0.38 string-append
1.39 0.38 0.38 system/vm/assembler.scm:2175:8
1.39 0.38 0.38
guix/build/syscalls.scm:1168:0:read-dirent-header/linux
1.39 0.38 0.38 srfi/srfi-1.scm:1033:17
1.39 0.38 0.38 gnu/packages/crates-io.scm:9336:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:64009:2:arguments
1.39 0.38 0.38 guix/build-system/cargo.scm:229:4
1.39 0.38 0.38 ice-9/boot-9.scm:2759:0:module-make-local-var!
0.00 7328.64 0.00 system/vm/assembler.scm:1258:0:intern-constant
0.00 1423.97 0.00 ice-9/boot-9.scm:220:5:map1
0.00201.90 0.00 ice-9/threads.scm:388:4
[...]
---
Sample count: 72
Total time: 27.531145665 seconds (14.056589389 seconds in GC)
--8<---cut here---end--->8---
The assembler allocates a huge constant table in ‘intern-constant’
(‘generate-package-cache’ produces one literal list containing one small
vector per package, each of which contains strings and other constants).
The attached file emulates what ‘generate-package-cache’ does. Each
vector in the list looks like this:
#(0 "xyzxyz-0" "1.0.0" (gnu packages xyz0) xyzxyz-0 ("out") #t #f
"gnu/packages/xyz0" 0 1)
The program terminates with roughly a 250 MiB heap for 22K entries—about
10 MiB per vector.
The heap profile on completion looks like this:
--8<---cut here---start->8---
% type selfavg obj size
57.5 bytevector 8,205,632 455868.4
23.9 vector 3,413,857 549.3
7.7 struct 1,093,84040.8
5.3 pair 753,37616.0
2.7 stringbuf 385,85651.1
1.1 symbol151,42432.0
0.4 program59,42447.3
0.3 string 49,15232.0
0.2 heap-number23,52032.0
0.2 variable 21,69630.3
0.1 smob 21,05632.0
0.1 vm-continuation20,86432.0
0.1 atomic-box 20,73632.0
0.1 weak-table 19,58431.8
0.1 unknown19,04032.0
0.0 pointer 2,00016.0
0.0 hash-table89632.0
0.0 primitive 14416.0
0.0 weak-vector9616.0
0.0 frame 9632.0
0.0 primitive-generic 6432.0
0.0 keyword1616.0
sampled heap: 13.60166 MiB (heap size: 233.16016 MiB)
--8<---cut here---end--->8---
It’s likely that the ‘buf’
