Hi Pablo, Happy new year! Sorry for my late response. The new year started with lots of exciting things. Really happy to see yours!
On Wed, 2025-12-24 at 18:39 +0000, Pablo Galindo Salgado wrote: > This is my first contribution to elfutils so wanted to check if the > approach is okay before sending a formal patch. > > Following up on Mark's reply to Bogdan back in July 2024 > (https://sourceware.org/pipermail/elfutils-devel/2024q3/007266.html) > about serving .dwo files from debuginfod. I implemented this and have > it working locally. Is a significant gap in the protocol right now. > More and more people are compiling with -gsplit-dwarf to speed up > builds and reduce binary sizes, but then they loose the ability to > fetch debug info automatically via debuginfod. Been interested in this > for a while and once it lands I'm planing to contribute the > corresponding GDB support to consume it. Nice. GDB does seem to support split-dwarf (single file, .dwo and .dwp) I believe. So the corresponding support in GDB would be to use the debuginfod-client interface to fetch it? > For the .dwo vs .dwp question Mark raised, I went with serving the > whole .dwp file since libdw already knows how to parse the index and > extract what it needs. Duplicating that logic server-side would be > complex and clients requesting one DWO ID from a .dwp will likely want > others from the same package anyway. Yes, that would be my assumption too. > What I implemented: > > On the server side, the scanner now recognizes .dwo and .dwp files > during traversal, iterates through their compilation units, extracts > the DWO IDs, and stores the mappings in a new table. The HTTP handler > serves requests to /dwoid/<DWOID>/debuginfo by looking up the file and > returning it. Do you support single-file split-dwarf? Where the split DWARF is in the .o file together with the skeleton sections? What if there are both separate .dwo files and a .dwp containing the same ID? Is one preferred over the other? Using /dwoid/ instead of overloading /buildid/ seems a good choice. In theory you could overload /buildid/ which we do for alt-files (where we use the ID from the .gnu_debugaltlink as if it was a "normal" build- id). But this might be slightly confusing. Although you can kind of see the dwp file as if it was another auxialary debug/dwarf file. > On the client side, added debuginfod_find_dwo() wich works similar to > the other find functions. Takes the DWO ID bytes, converts to hex, > queries the servers, and caches the result. Also added debuginfod-find > dwoid <DWOID> for manual testing. How does the caching work? When you get a dwp file do you create symlinks for all dwoids in it under ~/.cache/debuginfod_client/? Or is libdw supposed to register/cache them? > The trickier part was making this automatic. In libdw, added > dwarf_set_dwo_lookup() wich lets you register a callback that gets > invoked when the existing local lookups fail (same directory, > DW_AT_comp_dir, etc). The callback receives the DWO ID and returns a > file descriptor. This way libdw doesn't need to know anything about > debuginfod, just calls the callback if one is registered. I would have to see the code to really understand how that works. > Then in libdwfl, when loading DWARF I set up this callback to point to > a wrapper that calls debuginfod_find_dwo(). So from the user > perspective everything just works transparently. Open a skeleton > binary with libdwfl and the split DWARF gets fetched automatically if > needed. Sound useful, but have to see the code to really understand. > > This is how it looks: > > debuginfod/debuginfod-client.c | 382 > +++++++++++++++++++++++++++++++++++++++ > debuginfod/debuginfod-find.c | 16 +- > debuginfod/debuginfod.cxx | 356 ++++++++++++++++++++++++++++++++++-- > debuginfod/debuginfod.h.in | 25 +++ > debuginfod/libdebuginfod.map | 4 + > libdw/Makefile.am | 3 +- > libdw/dwarf_set_dwo_lookup.c | 49 +++++ > libdw/libdw.h | 19 ++ > libdw/libdw.map | 1 + > libdw/libdwP.h | 7 + > libdw/libdw_find_split_unit.c | 91 ++++++---- > libdwfl/debuginfod-client.c | 20 +- > libdwfl/dwfl_module_getdwarf.c | 16 ++ > libdwfl/libdwflP.h | 2 + > tests/Makefile.am | 7 +- > tests/debuginfod_dwoid_find.c | 138 ++++++++++++++ > tests/run-debuginfod-dwoid.sh | 322 +++++++++++++++++++++++++++++++++ > > The test script run-debuginfod-dwoid.sh covers: > > - DWP file indexing for both DWARF 5 and DWARF 4 packages (each > containing multiple DWO IDs) > - Individual .dwo file indexing for both DWARF 5 and DWARF 4 > - Error handling: 404 for non-existent DWO ID, rejection of malformed > hex strings > - The debuginfod-find dwoid client command > - Cache directory structure verification > - End-to-end libdwfl callback integration using skeleton binaries Sounds you really though about testing. Thanks! > The C test program debuginfod_dwoid_find.c opens a skeleton binary via > dwfl_report_offline(), calls dwfl_module_getdwarf() wich triggers the > callback, and verifies that the split DIE gets resolved even though > the .dwo files aren't present locally, only on the server. I have > tested DWARF 4 and DWARF 5 on aarch64 and x86_64 (64 bit). I think I don't fully understand the resolving part when the dwo isn't available locally. But again I assume it makes sense when seeing the code. > Splitting this into smaller patches is somewhat hard because > everything is tangled together. The server needs the endpoint, the > client needs the API, libdw needs the callback mechanism, libdwfl > needs to wire it all up, and the tests need all of it working. Understood. > Does this approach make sense? Happy to send the actual patches if so. Yes, it makes sense to me. Looking forward to the actual patches to better understand the details. Thanks, Mark
