> On Oct 18, 2021, at 2:32 PM, Mateusz Piotrowski <[email protected]> wrote:
>
> Hello everyone,
>
> I would like to measure the overhead / performance impact that of tracing
> with DTrace has on a system under load. I'm looking for the right way to do
> it. Obviously, I should not be using DTrace to measure that in this
> situation. I could use other stats(7) tools but it does not seem. Measuring
> the impact indirectly sounds like a good idea (e.g., by looking at the
> throughput between two machines in a test lab as they transfer bytes from A
> to B in two scenarios: with active tracing and without). Are there any other
> options? Do you have any insights to share regarding measuring DTrace
> overhead?
>
> I'd be grateful for all suggestions and pointers.
>
I have done this for several sub-systems when I had to — for work — show the
progress of bpftrace against DTrace and formulate plans for sampling strategies
based on observed overhead.
Running in one shell:
dd if=/dev/urandom of=/tmp/trashfile bs=1 count=5000000
And another [Linux] shell:
bpftrace -e 'tracepoint:syscalls:sys_enter_read { }’
This resulted in a 23% performance hit on the dd while bpftrace was running. We
ran the test 10x and on multiple pieces of hardware, under multiple Operating
Systems, and tried our best to control the testing environment parameters.
The hit on FreeBSD was barely noticeable. I think I calculated it as less than
6% at its worst.
By the way, did you know you can run DTrace on CentOS?
cd /etc/yum.repos.d
wget http://public-yum.oracle.com/public-yum-ol7.repo
yum install -y kernel-uek{,-devel} dtrace-utils
# EFI
cp /boot/efi/EFI/centos/grub.cfg{,.`date +%s`.bak}
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
# BIOS
grubby --set-default-index 0
reboot
ASIDE: And it’s actually fun to use and quite powerful. Back to the question
at-hand though …
I have done similar things with “nc” in place of “dd”
I also suggest checking out “dpv” in FreeBSD base for rate monitoring with and
with-out tracing active to gain insights on overhead.
—
Devin
