Jiri Daněk created DISPATCH-1734:
------------------------------------
Summary: Dynamic tracing of qdrouterd
Key: DISPATCH-1734
URL: https://issues.apache.org/jira/browse/DISPATCH-1734
Project: Qpid Dispatch
Issue Type: Wish
Affects Versions: 1.12.0
Reporter: Jiri Daněk
There are essentially four ways of tracing a C program. As an example, consider
tracing of all pthread lock and unlock calls, for purposes of either
performance monitoring, enforcing lock invariants in tests ("Python lock can be
taken only when all other locks were dropped"), or general debugging.
# Wrap all calls to lock acquire and lock release functions, always use the
wrapper instead of locking directly. This is the approach in PROTON-2144 for
Proton memory allocations.
# LD_PRELOAD an instrumented version of pthreads library (instead of
introducing it in your source code)
# Do some runtime magic from within the program, such as Global Offset Library
modification, https://github.com/lonnywang/glibcmock
# Use Linux tracing mechanism on top of uprobes, which means BPF/bpftrace or
systemtap.
Out of these possibilities, I like BPF best. Systemtap is hard to install on
anything which is not RHEL and it does not use BPF by default (but there is an
option to use BPF). On the other hand, bpftrace is hard to install on RHEL.
There is another BPF frontend, ply, which might work better on RHEL, but I
never tried it.
Overview of Linux tracing. The layering there is that uprobes is a mechanism in
Linux kernel, which all the other tools can use. BPF is a mechanism to upload
and run tracing programs in the kernel in a safe manner (uses bytecode and a
JIT). BCC is a compiler for BPF programs, it uses a C-like programming
language. Then bpftrace and ply are frontends for bcc, which use a higher
level, awk-like language. Systemtap is originally its own thing, which uses
uprobes and such directly (it's a kernel module), but it can also run on top of
BPF. It also uses an awk-like language.
BPF can be installed on RHEL 7 or 8,
https://www.redhat.com/en/blog/introduction-ebpf-red-hat-enterprise-linux-7.
That suggests to me that BPF is the way to go, for portability.
I once tried bpftrace with dispatch, and it worked ok. For portability, it is
better to use BPF (through bcc). Or maybe ply, if it can be made to work
everywhere.
{noformat}
#! /usr/bin/env stap
probe
process("/home/jdanek/Work/repos/qpid-dispatch/build/install/lib/qpid-dispatch/libqpid-dispatch.so").function("qdr_link_flow")
{
printf ("%s, %d -> %s (credit: %d)\n", thread_indent(0), pid(),
probefunc(), $credit)
}
{noformat}
Started like this
{noformat}
[root]$ stap -v ~jdanek/projects/tracing/qdr.stp
Pass 1: parsed user script and 473 library scripts using
235004virt/84616res/5448shr/79592data kb, in 200usr/30sys/234real ms.
Pass 2: analyzed script: 1 probe, 16 functions, 5 embeds, 2 globals using
239736virt/90884res/6868shr/84324data kb, in 60usr/270sys/330real ms.
Pass 3: using cached
/root/.systemtap/cache/7f/stap_7fa00865d178089513198a99ba194c43_6947.c
Pass 4: using cached
/root/.systemtap/cache/7f/stap_7fa00865d178089513198a99ba194c43_6947.ko
Pass 5: starting run.
[...]
{noformat}
I saw e.g. the following in the output
{noformat}
0 qdrouterd(15783):, 15781 -> qdr_link_flow (credit: 9)
0 qdrouterd(15786):, 15786 -> qdr_link_flow (credit: 225)
0 qdrouterd(15786):, 15786 -> qdr_link_flow (credit: 225)
0 qdrouterd(15784):, 15781 -> qdr_link_flow (credit: 7)
0 qdrouterd(15784):, 15781 -> qdr_link_flow (credit: 9)
0 qdrouterd(15784):, 15781 -> qdr_link_flow (credit: 0)
0 qdrouterd(15784):, 15781 -> qdr_link_flow (credit: -9)
0 qdrouterd(15794):, 15791 -> qdr_link_flow (credit: 130)
0 qdrouterd(15784):, 15781 -> qdr_link_flow (credit: 0)
0 qdrouterd(15786):, 15786 -> qdr_link_flow (credit: 213)
0 qdrouterd(15784):, 15781 -> qdr_link_flow (credit: -9)
0 qdrouterd(15799):, 15796 -> qdr_link_flow (credit: 249)
0 qdrouterd(15784):, 15781 -> qdr_link_flow (credit: 0)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
