Hello Guilers! The ‘wip-linker-assembler-memory-consumption’ branch I recently pushed aims to reduce the allocation rate and memory consumption of the linker and assembler, as well as the number of bytevector copies.
In 3.0.8, the linker and assembler keep roughly two copies of the final ELF file in memory: each <linker-object> contains a ‘bv’ field with its wire representation, and then ‘link-elf’ allocates a bytevector to contain the whole ELF file and copies each <linker-object> bytevector back into that one. When you’re running ‘guild compile’, that big bytevector is immediately passed to ‘put-bytevector’ to write it to disk. The branch replaces the ‘bv’ field of <linker-object> with ‘size’ and ‘writer’, the latter being a procedure that takes a bytevector and writes to it. At this point, ‘link-elf’ still allocates one bytevector for each linker object. Eventually we could rewrite those “writer” procedures to use binary I/O primitives instead of expecting a bytevector to write to, and that way we wouldn’t need those temporary bytevectors. It’s a bit tedious to do though. One big source of memory consumption that remains is the assembler: its ‘buf’ field contains an in-memory copy of the ‘.rtl-text’ section, which is often a significant portion of the ELF file. Addressing that would be more difficult. On the compilation at -O1 of a big file (‘gnu/packages/python-xyz.scm’ in Guix), I see a 5% reduction of the heap size as measured at the end of the whole compilation process. There are other factors before the linking step that contribute to the peak heap size, which probably explains why the impact is not higher. The current set of commits looks like this: cdf9a0ac4 * DRAFT linker: Do not store entire ELF in memory when writing to a file. 048651993 * linker: Linker object writer takes a single argument. 2ac4e5a3d * linker, assembler: Avoid intermediate bytevectors. 8c8d4bd37 * DRAFT Add 'bytevector-slice'. 805275882 * linker: Separate effectful part of 'add-elf-objects'. 8194e8a6e * assembler: Separate effectful part of 'link-docstrs'. 0a5768add * assembler: Separate effectful part of 'link-frame-maps'. f6fb95db8 * assembler: Separate effectful part of 'link-procprops'. 099cf8f30 * assembler: Separate effectful part of 'link-dynamic-section'. 43dd2f184 * assembler: Separate effectful part of 'link-symtab'. ad4487091 * assembler: Separate 'process-relocs' from 'patch-relocs!'. There’s no regression. The main issue worth discussing is the last commit, which changes ‘link-elf’ to optionally return a procedure instead of a bytevector. We also need to discuss ‘bytevector-slice’, but that can be done separately. Andy, what do you think of this approach? Cheers, Ludo’.
