Hi all,
Here included is a conversation I had on the kvm mailing list regarding how to link traces from host and guests when tracing virtual environments. Following Stefan's last suggestions, here is a suggestion on what can be done in lttng-modules, it's not intrusive and does not require modifications to the kernel. Let me know what you think: * If we want to trace guests _and_ host and linek them together, we require to set the VM's -uuid parameter. I think virsh does that by default anyway. - We can then obtain this uuid in lttng via dmi_get_system_info(DMI_PRODUCT_UUID). We can add it to a statedump event, or in the 'env' section of the metadata (which might be easier to manager and does not require activating an additional event). * Guest statedump: on host, we can access the guests by parsing the struct dentry* kvm_debugfs_dir for entries with the <pid>-xx format. We can then parse the /proc/<pid>/cmdline to obtain the uuid and associate it to the guest. * Guest lifetime during the trace: We can add a lttng-module that adds a udev rule to intercept VM creation/destruction. We can trace those events, adding the pid/uuid in the fields. What do you think of this approach? Thanks, Geneviève -------- Forwarded Message -------- Subject: Re: Shared value between host and guests Date: Fri, 26 Jan 2018 14:03:48 +0000 From: Stefan Hajnoczi <[email protected]> To: Geneviève Bastien <[email protected]> CC: [email protected] On Thu, Jan 25, 2018 at 10:44:12AM -0500, Geneviève Bastien wrote: > On 2018-01-25 04:29 AM, Stefan Hajnoczi wrote: > > On Wed, Jan 24, 2018 at 10:56:26AM -0500, Geneviève Bastien wrote: > >> Thanks for the hint, I didn't know about those. But the uevents are only > >> for the host right? There is no specific value in debugfs for guest. > > I'm not sure I understand the question. > > > > Uevent are emitted on the host. Given the PID field, it should be > > possible to correlate them to a specific guest (e.g. by looking at > > information from the user or VM management tools, or simply by looking > > at /proc/$PID/cmdline for the QEMU -name argument that can be used to > > identify guests on the host side). > > > > Stefan > With the information from the host and those uevents, we can indeed > easily figure out in our traces which processes are associated with a > guest and get its command line and a lot more information. > > The missing piece is in the guest trace. For example, we have lttng > traces taken on the guest and the host. We may have multiple guests and > multiple hosts to trace. It's easy to know which traces are from hosts > from the events, but which are from guests? And which guest trace goes > with which of the kvm processes on which host? The guest trace contains > no information that can be linked to the host. > > I was thinking that the guest could do a hypercall to the host after > bootup to share some unique information, for instance its bootid, that > the host could store somewhere. That information would be available in > the traces so that we can easily associate the guest with its host > process and the states of its virtual CPUs to that of the corresponding > threads, and much more. > > I hope this describes our need a little better. The QEMU -uuid <uuid> option makes a UUID available to the guest via SMBIOS and fw_cfg on x86. Inside the guest you can print it like this: # dmidecode -s system-uuid 01020304-0506-0708-090A-0B0C0D0E0F10 Maybe you can base the guest trace filename off the UUID: guest-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat On the host you can either find the UUIDs in the libvirt domain XML: # virsh dump my-domain <domain ...> <uuid>0102030405060708090a0b0c0d0e0f10</uuid> ... Or you can use the kvm.ko uevent to find the QEMU PID and then check /proc/$PID/cmdline for the uuid. The UUID can also be fetched via the query-uuid QMP command if you don't want to search /proc/$PID/cmdline for -uuid <uuid>. Maybe you can base the host trace filename off the UUID too: host-01020304-0506-0708-090A-0B0C0D0E0F10-trace-001.dat Then the ask of correlating traces becomes pretty easy for post-processing scripts since they can look at the filenames. Stefan
_______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
