On 3 Jan 2023, at 9:46, Adrian Moreno wrote:
<SNIP>
>>>>>>> + dbg_file = "/usr/lib/debug/.build-id/{}/{}.debug".format(
>>>>>>> + match.group(1)[:2], match.group(1)[2:])
>>>>>>> +
>>>>>
>>>>> I don't think this part is working for me:
>>>>>> dnf list installed | grep openvs
>>>>> openvswitch.x86_64 2.17.0-6.fc37
>>>>> @fedora
>>>>> openvswitch-debuginfo.x86_64 2.17.0-6.fc37
>>>>> @fedora-debuginfo
>>>>> openvswitch-debugsource.x86_64 2.17.0-6.fc37
>>>>> @fedora-debuginfo
>>>>> openvswitch-dpdk.x86_64 2.17.0-6.fc37
>>>>> @fedora
>>>>>
>>>>>> readelf -n --debug-dump=follow-links /usr/sbin/ovs-vswitchd | grep Build
>>>>> Build ID: 249d375f6905c28d75d0b2792f3508c4c0869a30
>>>>>
>>>>>> ls /usr/lib/debug/24/9d375f6905c28d75d0b2792f3508c4c0869a30
>>>>> ls: cannot access
>>>>> '/usr/lib/debug/24/9d375f6905c28d75d0b2792f3508c4c0869a30': No such file
>>>>> or directory
>>>>>
>>>>> It seems that file is not provided by fedora's openvswitch-debuginfo
>>>>> package at all :/.
>>>>
>>>> This is odd, does GDB work? As at least the debug info for the executable
>>>> should exist.
>>>>
>>>>> Not even using debuginfod:
>>>>>> debuginfod-find debuginfo 249d375f6905c28d75d0b2792f3508c4c0869a30
>>>>> /home/amorenoz/.cache/debuginfod_client/249d375f6905c28d75d0b2792f3508c4c0869a30/debuginfo
>>>>>> ~/src/pahole/build/pahole -C udpif --compile
>>>>>> /home/amorenoz/.cache/debuginfod_client/249d375f6905c28d75d0b2792f3508c4c0869a30/debuginfo
>>>>> pahole: type 'udpif' not found
>>>>>
>>>>> Looking at which file we should have looked in:
>>>>>
>>>>>> for file in $(rpm -ql openvswitch-debuginfo | grep "\.debug"); do
>>>>>> ~/src/pahole/build/pahole -C udpif --compile $file 2>&1 | grep -qv "not
>>>>>> found" && echo "Found symbols in $file" ; done
>>>>> Found symbols in
>>>>> /usr/lib/debug/.build-id/bb/f8d3ddfda867379e9188c1afbfe430bcd5ae65.debug
>>>>> Found symbols in
>>>>> /usr/lib/debug/usr/lib64/libofproto-2.17.so.0.0.0-2.17.0-6.fc37.x86_64.debug
>>>>>
>>>>> ... which is actually the same file:
>>>>>> realpath
>>>>>> /usr/lib/debug/.build-id/bb/f8d3ddfda867379e9188c1afbfe430bcd5ae65.debug
>>>>> /usr/lib/debug/usr/lib64/libofproto-2.17.so.0.0.0-2.17.0-6.fc37.x86_64.debug
>>>>>
>>>>> I'm not an expert in readelf but at first sight the way I found to obtain
>>>>> the definitions from the binary is using ldd to list the libraries and
>>>>> then use debufinfod-find to obtain the debuginfo from the library and
>>>>> then use pahole.
>>>>
>>>> I tested with RHEL and local build. I think it was following file links,
>>>> to open the correct dwz debug info. Maybe Fedora is not dwz, and needs the
>>>> manual linking. Will dig into this after my PTO.
>>>>
>>>
>>>
>>> I did a bit more testing. Both rhel and fedora dev builds seem to use dwz.
>>> However, one have much more symbols!
>>>
>>> (RHEL)
>>>> readelf -wi /usr/lib/debug/.dwz/openvswitch2.17-2.17.0-58.el9fdp.x86_64 |
>>>> wc -l
>>> 835532
>>>
>>> (F37)
>>>> readelf -wi /usr/lib/debug/.dwz/openvswitch-3.0.90-1.fc37.x86_64 | wc -l
>>> 68717
>>>
>>> TBH, I only see one big difference: RHEL statically links ovs-libs while in
>>> fedora they are shared libraries:
>>>
>>> (RHEL)
>>>> readelf -d /usr/sbin/ovs-vswitchd | grep NEEDED
>>> 0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
>>> 0x0000000000000001 (NEEDED) Shared library: [libnuma.so.1]
>>> 0x0000000000000001 (NEEDED) Shared library: [libarchive.so.13]
>>> 0x0000000000000001 (NEEDED) Shared library: [libmlx5.so.1]
>>> 0x0000000000000001 (NEEDED) Shared library: [libibverbs.so.1]
>>> 0x0000000000000001 (NEEDED) Shared library: [libssl.so.3]
>>> 0x0000000000000001 (NEEDED) Shared library: [libcrypto.so.3]
>>> 0x0000000000000001 (NEEDED) Shared library: [libcap-ng.so.0]
>>> 0x0000000000000001 (NEEDED) Shared library: [libunbound.so.8]
>>> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>>> 0x0000000000000001 (NEEDED) Shared library:
>>> [ld-linux-x86-64.so.2]
>>>
>>> (F37)
>>>> readelf -d /usr/sbin/ovs-vswitchd | grep NEEDED
>>> 0x0000000000000001 (NEEDED) Shared library:
>>> [libofproto-3.0.so.0]
>>> 0x0000000000000001 (NEEDED) Shared library:
>>> [libopenvswitch-3.0.so.0]
>>> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>>>
>>> So I belive this method of finding symbols only really works if you point
>>> to the binary or shared library that actually defined the symbol.
>>>
>>> I verified that, tweaking the existing openvswitch-fedora.in to use
>>> "--enable-static", pahole is able to find the symbol in ovs-vswitchd's
>>> .debug file.
>>>
>>> But that still leave us with another question. Why can't we find the right
>>> debug file using the Build ID?
>>>
>>> It seems that in this case there are two debuginfo files, one for dpdk
>>> build and one for non-dpdk build:
>>>
>>>> rpm -ql openvswitch-debuginfo | grep ovs-vswitchd
>>> /usr/lib/debug/usr/sbin/ovs-vswitchd.nodpdk-2.17.0-6.fc37.x86_64.debug
>>>
>>> Ideally, this would have the same Build ID as the actual binary but that's
>>> not the case. It matches the one of /usr/sbin/ovs-vswitchd.nodpdk but not
>>> the one of /usr/sbin/ovs-vswitchd.
>>>
>>> Not sure what's the difference between both binaries. Adding Timothy, maybe
>>> he can explain why we have 3 binaries in fedora :-).
>>
>> I guess all three binaries should have there build ID included in package
>> but will leave this for Timothy to figure out ;)
>>
>>> Anyway, so even if we use debuginfod-find to get the right debuginfo file:
>>>
>>>> pahole -C udpif --compile $(debuginfod-find debuginfo
>>>> /usr/sbin/ovs-vswitchd)
>>> pahole: type 'udpif' not found
>>>
>>> ...but that's because of the shared libraries issue explained before.
>>>
>>> So, AFAICS, a combination of reading dependencies from readelf, looking up
>>> files using ldd and querying debuginfod (using the path itself, no need to
>>> look for the BuildID) should work for all cases.
>>
>> I’m sending out a patch soon to use ldd and readelf.
>>
>
> Another way to avoid using ldd would be to read the process's maps...
Yes, but for now I guess I’ll stick with ldd, as the v6 is out.
>> Not convinced we should query external sources using debuginfod-find for
>> now. Maybe we can do this when we extract this function to a separate
>> library to be used by other usdt scripts, and add a nob to allow external
>> resolution of debug info.
>>
>
> Fair enough. Adding it as a strong dependency might be too much. Some ideas:
> - We could allow the users to provide the path to the debug info file so they
> can run ./reval_monitor.py --debug $(debuginfod-find ...)
> - We could add the debuginfo cache path to the places we look into. That way
> we could just document that debuginfod-find can be used to download the
> debuginfo (as an alternative to distro-package).
All good ideas, I’m planning to do another patch later to extract common code
from the existing scripts into a library. Will add all of this then.
>>>>> Next, I tried using a self-built development package using "make
>>>>> rpm-fedora"
>>>>>> readelf -n --debug-dump=follow-links /usr/sbin/ovs-vswitchd | grep
>>>>>> Build
>>>>> Build ID: 1a01718fe736dc89c4470645ff5f69f24e6c8728
>>>>>
>>>>> This time the file exists (I guess no DPDK build?) in the debuginfo
>>>>> package but pahole did not find the symbol.
>>>>>
>>>>>> ~/src/pahole/build/pahole --compile -C udpif
>>>>>> /usr/lib/debug/.build-id/1a/01718fe736dc89c4470645ff5f69f24e6c8728.debug
>>>>>
<SNIP>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
