Hi Michael, On 23/01/2020 08:57, Michael Hinton wrote: > Ralf, Henning, > > > Thanks for the quick response, and sorry for the delay. > > > Here’s my setup: I’ve got a 6-core Intel x86-64 CPU running Kubuntu > 19.10. I have an inmate that is given a single core and runs a > single-threaded workload. For comparison, I also run the same workload > in Linux under Jailhouse.
What CPU exactly? > > > For a SHA3 workload with the same 20 MiB input, the root Linux cell (and > no inmate running) takes about 2 seconds, while the inmate (and an idle > root cell) takes about 2.8 seconds. That is a worrisome discrepancy, and > I need to understand why it’s 0.8 s slower. What about CPU power features? cpufreq? turbo boost? ... > > > This is the SHA3 workload: > https://github.com/hintron/jailhouse/blob/76e6d446ca682f73679616a0f3df8ac79f4a1cde/inmates/lib/mgh-sha3.c#L185-L208 > > > This is the Linux wrapper for the SHA3 workload: > https://github.com/hintron/jailhouse/blob/76e6d446ca682f73679616a0f3df8ac79f4a1cde/mgh/workloads/src/sha3-512.c#L166-L168 > > > This is the inmate program calling the SHA3 workload: > https://github.com/hintron/jailhouse/blob/76e6d446ca682f73679616a0f3df8ac79f4a1cde/inmates/demos/x86/mgh-demo.c#L370-L379 SHA3 is only computationally 'expensive', right? So it's neither memory intensive nor should it trap. > > > You can see that the inmate and the Linux wrapper both execute the same > function, sha3_mgh(). It's the same C code. > > > The other workloads I run are intentionally more memory intensive. They > see a much worse slowdown. For my CSB workload, the root cell takes only > 0.05 s for a 20 MiB input, while the inmate takes 1.48 s (30x > difference). And for my Random Access workload, the root cell takes 0.08 > s while the inmate takes 3.29 s for a 20 MiB input (40x difference). Now this sounds pretty much like what I once had: too little caches for the inmate. BTW: For a sound comparison, you would need to take care to have a comparable initial hardware state: E.g., you need to take care that workloads in root-cell and non-root inmate are both either uncached or cached when starting the code. But that's fine tuning, and won't explain a 40x difference. > > > Here are the root and inmate cell configs, respectively: > > https://github.com/hintron/jailhouse/blob/76e6d446ca682f73679616a0f3df8ac79f4a1cde/configs/x86/bazooka-root.c I recommend to deactivate hyperthreading. If your inmate just gets one sibling, the other one will still belong to Linux, which could, in case of utilisation, steals a lot of power. So either disable HT or assign both siblings to the inmate. > > https://github.com/hintron/jailhouse/blob/76e6d446ca682f73679616a0f3df8ac79f4a1cde/configs/x86/bazooka-inmate.c > > > I did do some modifications to Jailhouse with VMX and the preemption > timer, but any slowdown that I may have inadvertently introduced should > apply equally to the inmate and root cell. > > > It’s possible that I am measuring the duration of the inmate > incorrectly. But the number of vmexits I measure for the inmate and root > seem to roughly correspond with the duration. I also made sure to avoid Yeah, I would also expect that: Your code only utilises memory + CPU, almost no I/O. > tsc_read_ns() by instead recording the TSC cycles and deriving the > duration by dividing by 3,700,000,000 (the unchanging TSC frequency of > my processor). Without this, the time recorded would overflow after > something like 1.2 seconds. > > > I'm wondering if something else is causing unexpected delays: using > IVSHMEM, memory mapping extra memory pages and using it to hold my > input, printing to a virtual console in addition to a serial console, > disabling hardware p-states, turbo boost in the root cell, maybe the > workload code is being compiled to different instructions for the inmate > vs. Linux, etc. The latter one: You definitely need to check that. If your Linux compiler generates (e.g.) AVX code and your inmate. You could also try to link the same library object to your target binaries -- the build system is your friend. > > > Sorry for all the detail, but I am grasping at straws at this point. Any > ideas at what I could look into are appreciated. Benchmarking is fun. Especially getting the hardware under control. :-) Ralf > > > Thanks, > > Michael > > On Monday, January 20, 2020 at 6:46:32 AM UTC-7, Henning Schild wrote: > > On Sun, 19 Jan 2020 23:45:46 -0800 > Michael Hinton <[email protected] <javascript:>> wrote: > > > Hello, > > > > I have found that running code in an inmate is a lot slower than > > running that same code in the root cell on my x86 machine. I am not > > sure why. > > Can you elaborate on "code" and "a lot"? Maybe roughly tell us what > your testcase does and how severe your slowdown is. Synthetic > microbenchmark to measure context switching ? > > As Ralf already said, anything causing "exits" can be subject to > slowdown. But that should be roughly the same for the root cell or any > non-root cell, is it truly the "same" code? > > And of cause anything accessing shared resources can be slowed down by > the sharing. Caches/buses ... but i would not expect "a lot". > > regards, > Henning > > -- > You received this message because you are subscribed to the Google > Groups "Jailhouse" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jailhouse-dev/b258dc63-26a9-4eff-852a-23d72d2e3258%40googlegroups.com > <https://groups.google.com/d/msgid/jailhouse-dev/b258dc63-26a9-4eff-852a-23d72d2e3258%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/1c6fd98d-5f05-eb49-5780-273387eae195%40oth-regensburg.de.
