Re: Porting pahole from dwarf_next_unit() to dwarf_get_units()
Hi Dimitri, Sorry, this arrived before my vacation and then the new year happened. On Tue, Dec 05, 2023 at 01:03:01PM +, Dimitri John Ledkov wrote: > Currently pahole warns and does nothing upon hitting > DW_TAG_skeleton_unit as implemented at > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf > > In elfutils, a while back a new API got added that aids with discovery > and processing of such tags - > https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade > > It seems to me if pahole is ported from using dwarf_next_unit() to > instead use dwarf_get_units() native support can be added for > split-dwarf (dwo) files. > > I am trying to write such a port, but it is proving to be very > difficult. I am entirely unfamiliar with neither pahole nor libdw nor > the dwarf file format. Thus it is very confusing when both pahole and > dwarf library use very similar type names and structs. For example > libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct. > > What are the differences between dwarf_nextcu(), dwarf_next_unit(), > dwarf_get_units() and when should one use each one of them? (or nest > them?) The dwarf_nextcu was the original way to iterate over the CUs from .debug_info. Then dwarf_next_unit was added when type units could come from a .debug_types section. Both functions use and return offsets to iterate through the section and then get the CU DIE using dwarf_offdie (or dwarf_offdie_types). This requires the user to know beforehand where to DIE data is stored (in the .debug_info or .debug_types section). For type units one also needs to use the type offset to create the actual type DIE. In DWARF5 DIEs can come from even more data locations. And there are also skeleton units which require the user to find the associated split compile unit DIE (which would come from a different file). The new dwarf_get_units function simplifies iterating over the units in a DWARF file. It doesn't require the user to know where the DIE data is stored, it will automagically iterate over all know data sources (sections) returning the Dwarf_CU and the associated Dwarf_Die if requested. If the user requests to know the associated "subdie" it will also be resolved. A subdie is either a type DIE for a type unit or a split unit DIE for a skeleton unit. The same (and some more) info about DWARF_CUs can also be gotten through the dwarf_cu_info function. You should either use dwarf_nextcu or dwarf_next_unit with dwarf_offdie to get the (top-level) DIE. Or use dwarf_get_units and possibly dwarf_cu_info. In general you shouldn't mix them. Hope this helps and let me know if you need more info. Cheers, Mark
Re: Porting pahole from dwarf_next_unit() to dwarf_get_units()
Em Tue, Dec 05, 2023 at 04:11:06PM +, Dimitri John Ledkov escreveu: > I > > On Tue, 5 Dec 2023, 15:47 Arnaldo Carvalho de Melo, wrote: > > > > Em Tue, Dec 05, 2023 at 01:03:01PM +, Dimitri John Ledkov escreveu: > > > Currently pahole warns and does nothing upon hitting > > > DW_TAG_skeleton_unit as implemented at > > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf > > > > > > In elfutils, a while back a new API got added that aids with discovery > > > and processing of such tags - > > > https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade > > > > > > It seems to me if pahole is ported from using dwarf_next_unit() to > > > instead use dwarf_get_units() native support can be added for > > > split-dwarf (dwo) files. > > > > That would be awesome! > > > > > I am trying to write such a port, but it is proving to be very > > > difficult. > > > > I did some work on supporting split-dwarf months ago, but got > > sidetracked with other work, BTF related and then the code bitrotted, I > > have to go back looking at it to swap back the details into my brain: > > > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=alt_dwarf > > > > The patches after: > > > > 45c044860c2abce7 dwarf_loader: Sync with LINUX_ELFNOTE_LTO_INFO macro from > > kernel > > > > Are the ones to suport alt dwarf. > > I will read into those thanks. Hopefully what you've been work ends up less convoluted by using new elfutils functions, but it may be useful to help understand how the internal pahole code deals with Dwarf offsets to reduce them to 32 bits for conversion to CTF/BTF (where it is not really that important as libbpf does this work). > > > I am entirely unfamiliar with neither pahole nor libdw nor > > > the dwarf file format. Thus it is very confusing when both pahole and > > > dwarf library use very similar type names and structs. For example > > > libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct. > > > > > What are the differences between dwarf_nextcu(), dwarf_next_unit(), > > > dwarf_get_units() and when should one use each one of them? (or nest > > > them?) > > > > > Is a port of > > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/dwarf_loader.c?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf > > > to use dwarf_get_units() a right approach and would be welcomed? > > > > Yes, we need to support DWARF5 fully. > ack > > > Is anyone else interested in providing any help, or guidance? > > > > I'm interested, and I think if Mark could help it would be great as > > well. > > I have something that sort of works, but then like aboarts with > invalid free's on exit - which the purist in me cares, but not sure if > it is of practical value or not. > And eu-readelf code also mentions that it deliberary leaks memory, > because life is hard. Off course ;-) Frees on exit are interesting to try to evaluate if things that are not just frees on exit are leaking, when we're sure that is the case, don't free on exit, as its just overhead. > I will try to address or warn about memory leaks to see if stuff > works, and post and RFC. Great, thanks a lot for working on this! - Arnaldo
Re: Porting pahole from dwarf_next_unit() to dwarf_get_units()
I On Tue, 5 Dec 2023, 15:47 Arnaldo Carvalho de Melo, wrote: > > Em Tue, Dec 05, 2023 at 01:03:01PM +, Dimitri John Ledkov escreveu: > > Currently pahole warns and does nothing upon hitting > > DW_TAG_skeleton_unit as implemented at > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf > > > > In elfutils, a while back a new API got added that aids with discovery > > and processing of such tags - > > https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade > > > > It seems to me if pahole is ported from using dwarf_next_unit() to > > instead use dwarf_get_units() native support can be added for > > split-dwarf (dwo) files. > > That would be awesome! > > > I am trying to write such a port, but it is proving to be very > > difficult. > > I did some work on supporting split-dwarf months ago, but got > sidetracked with other work, BTF related and then the code bitrotted, I > have to go back looking at it to swap back the details into my brain: > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=alt_dwarf > > The patches after: > > 45c044860c2abce7 dwarf_loader: Sync with LINUX_ELFNOTE_LTO_INFO macro from > kernel > > Are the ones to suport alt dwarf. I will read into those thanks. > > > I am entirely unfamiliar with neither pahole nor libdw nor > > the dwarf file format. Thus it is very confusing when both pahole and > > dwarf library use very similar type names and structs. For example > > libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct. > > > What are the differences between dwarf_nextcu(), dwarf_next_unit(), > > dwarf_get_units() and when should one use each one of them? (or nest > > them?) > > > Is a port of > > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/dwarf_loader.c?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf > > to use dwarf_get_units() a right approach and would be welcomed? > > Yes, we need to support DWARF5 fully. ack > > > Is anyone else interested in providing any help, or guidance? > > I'm interested, and I think if Mark could help it would be great as > well. I have something that sort of works, but then like aboarts with invalid free's on exit - which the purist in me cares, but not sure if it is of practical value or not. And eu-readelf code also mentions that it deliberary leaks memory, because life is hard. I will try to address or warn about memory leaks to see if stuff works, and post and RFC. Regards, Dimitri.
Re: Porting pahole from dwarf_next_unit() to dwarf_get_units()
Em Tue, Dec 05, 2023 at 01:03:01PM +, Dimitri John Ledkov escreveu: > Currently pahole warns and does nothing upon hitting > DW_TAG_skeleton_unit as implemented at > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf > > In elfutils, a while back a new API got added that aids with discovery > and processing of such tags - > https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade > > It seems to me if pahole is ported from using dwarf_next_unit() to > instead use dwarf_get_units() native support can be added for > split-dwarf (dwo) files. That would be awesome! > I am trying to write such a port, but it is proving to be very > difficult. I did some work on supporting split-dwarf months ago, but got sidetracked with other work, BTF related and then the code bitrotted, I have to go back looking at it to swap back the details into my brain: https://git.kernel.org/pub/scm/devel/pahole/pahole.git/log/?h=alt_dwarf The patches after: 45c044860c2abce7 dwarf_loader: Sync with LINUX_ELFNOTE_LTO_INFO macro from kernel Are the ones to suport alt dwarf. > I am entirely unfamiliar with neither pahole nor libdw nor > the dwarf file format. Thus it is very confusing when both pahole and > dwarf library use very similar type names and structs. For example > libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct. > What are the differences between dwarf_nextcu(), dwarf_next_unit(), > dwarf_get_units() and when should one use each one of them? (or nest > them?) > Is a port of > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/dwarf_loader.c?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf > to use dwarf_get_units() a right approach and would be welcomed? Yes, we need to support DWARF5 fully. > Is anyone else interested in providing any help, or guidance? I'm interested, and I think if Mark could help it would be great as well. - ARnaldo
Porting pahole from dwarf_next_unit() to dwarf_get_units()
Currently pahole warns and does nothing upon hitting DW_TAG_skeleton_unit as implemented at https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf In elfutils, a while back a new API got added that aids with discovery and processing of such tags - https://sourceware.org/git/?p=elfutils.git;a=commitdiff;h=79f0e623dcde4b042bb72f636a2211d67d5c0ade It seems to me if pahole is ported from using dwarf_next_unit() to instead use dwarf_get_units() native support can be added for split-dwarf (dwo) files. I am trying to write such a port, but it is proving to be very difficult. I am entirely unfamiliar with neither pahole nor libdw nor the dwarf file format. Thus it is very confusing when both pahole and dwarf library use very similar type names and structs. For example libdw has struct Dwarf_CU and pahole has unrelated dwarf_cu struct. What are the differences between dwarf_nextcu(), dwarf_next_unit(), dwarf_get_units() and when should one use each one of them? (or nest them?) Is a port of https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/dwarf_loader.c?id=0135ccd632796ab3aff65b7c99b374c4682c2bcf to use dwarf_get_units() a right approach and would be welcomed? Is anyone else interested in providing any help, or guidance? -- Dimitri
