On Mar 14, 2007, at 6:17 AM, Christian Ehrhardt wrote:

Jimi Xenidis wrote:
Christian, nice summary.
One question I have is does Xen allow the domain to extract domain oprofile information as linxu would without Xen, or does xen allow for some transport that eases the collection?
Short Answer: "some transport "- Shared buffers (Xen/Guests) + xen virtual interrupts which are received in the linux oprofile driver for xen Long Answer: Every explanation would be almost identical to http:// xenoprof.sourceforge.net/xenoprof_2.0.txt so I recommend to read this because section one explains the background mechanisms.

Ahh, ok.
So active domains collect their own information, and Xen collects the information for the passive domains? My thought was that a hypervisor could ease the burden of data collection by providing a communication buffer and collect the information "out of body", especially information about low-level operations, but I guess they did not do that.

Anyway.. Lets stick with oprofiling the domains only.
There really very little Linux work to do here.  We need:
 1. An hcall that turn the performance monitor on for the domain
2. Save and restore the relevant registers for any domain that is has it turned on.
 3. Turn it off for domains that have it disabled.

So you suggest to freeze counters while hypervisor or other domains are running via "Performance Monitor Mark" MSR(PMM) and "Freeze counters while Mark" MMCR0(FCM0/1) - combined with keeping the interrupt handler in the linux code ?

Again, sticking with oprofiling "active" domains only.
Yes, so MMCR0[FCH]=1 (freeze counters while MSR[HV]=1) always.
Then whenever we perform a domain switch (context_switch()) we:
  if (prev->domain->perf) {
    save the MMCRs and counters
  } else if (prev->regs->msr & MSR_PMM || MMCR0[FC] == 0) {
    warning("bad domain!")

  MMCR0[FC]=1 // turn off all counters in case a domain is being bad
  if (next->domain->perf) {
    restore the MMCRs and counters

of course we can be a little smarter with the on/of logic.
NOTE: the MSR[PMM] bit will get save and restores with the other MSR bits, so no special handling is required.

->Is it allowed to direct an interrupt directly to a linux guest in xen (If there would be too much latency between the irq and the read of the samples we may block a lot of occurring perf interrupts before we are resetting MMCR0(PMAE))? Otherwise it would end up very similar to the xeonoprof approach that handles the interrupt in xen which put the data to a shared buffer and then passing the information about "new data" to the appropriate domain with a virtual interrupt.

Ahh, the point is, there is no way to direct performance interrupts to Xen while the domain is running, all performance interrupts go directly to the domain, so really the only thing you can/should virtualize is MMCR0 thru a single hcall.

NOTE: When Xen is running MSR[HV]=1 and when a domain is running MSR [HV]=0. So if an interrupt (like the performance interrupts) does not effect the value of MSR[HV] then the interrupt will not cause the processor to switch from domain to Xen.

There is a table in in book 3 "Interrupt Definitions" that describes how each interrupt effects the MSR (and yes we run with LPES[1]=1)

->Is there a single or at least limited number of transition points between domains (e.g. in the scheduler) and xen where we could place a hook to change the MSR(PMM) as we need it or will this state transition need to be spread all over the code?

yes, context_switch() should be the only place.

I also must add that this would not solve the described issue that another domain could write to the performance monitor registers and interfere a profiling session.

Since you are performing save and restore of these registers a domain can only hurt its self. You do bring up an excellent point in that the domain can mess with MMCR0, so in that case we would have to make sure that MMCR0[FCH]=1 whenever we enter the HV in exceptions.S (See the EXCEPTION_HEAD macro, and beware, if you make it to big you will get assembler errors and we'll have to cook up some magic).

If we decide that this issue is negligible for the moment we could also continue the xenoprof oriented approach which would provide us with the profiling of multiple domains and xen itself.

I believe that the work of profiling active domains is easy, maybe 2-3 weeks worth of work since the domains collect their own information, the only LinuxPPC work would be to add any Xen specific events. Depending on your expertise you can contact other members to help you with the assembly in exceptions.S (that file is a PITA).

I can't really estimate if it would be a worthwhile task to implement this Domain only profiling and then switching/extending it, do you think it is that much easier to implement this in the first place ?

Yes, because we know it works. Don't forget, LinuxPPC already runs on a hypervisor, most of this stuff has been done so we get a lot of stuff for free (including lots of bad stuff, but this is good stuff).

My hope is that this experience would allow you to then profile passive domains, which will be disappointing because the profiling will be limited to events that cause you to enter Xen and will probably have nothing to do with the counters. This step might not make sense at all but we should explore it.

Then the next step would be to profile Xen, which can also be staged into chewable pieces which should probably take about 3 weeks to get some profiling and then iteratively get closer and closer to covering more and more events.

I thought we should think/decide about the privilege issue I described. Especially in view of the fact that this applies to both approaches.

I think we nailed and described the privilege issues, do you have more in mind?

you can see the hcall being setup here:
arch/powerpc/platforms/pseries/setup.c pSeries_setup_arch 322 ppc_md.enable_pmcs = pseries_lpar_enable_pmcs;

Here is the spec:

Lets try and go deep with this and then think about how to oprofile Xen itself.

On Mar 13, 2007, at 1:11 PM, Christian Ehrhardt wrote:

Hi Folks,
I analyzed the oprofile/xenoprof code and tried to do a simple minded powerpc mapping in the last two weeks. As it come up in a phone call on Monday I overlooked some possible issues arising out of the simple mapping of xenoprof to the power architecture. In this mail I briefly describe some backgrounds as well as my considerations so far. I'm not sure If I got all power and x86 specifics in the right way so feel free to correct me - I'm open to any comments and ideas - I hope together we reach a realizable plan if and how this could be implemented.

-- Background I - oprofile basic principles --
Oprofile is a common profiling tool used in the linux world. It consists of two layers. First the kernel space driver that contains a generic infrastructure and management part as well as a architecture dependent part that handles the hardware specific tasks. The second part is the userspace component that controls the kernel part and computes the output to different reports.

-- Background II - xenoprof approach --
To use oprofile (http://oprofile.sourceforge.net/about/) in the xen environment it was extended to xenoprof (http:// xenoprof.sourceforge.net/) which adds a third layer in the xen hypervisor. The linux kernel space driver supports now a new “architecture” that repesents xen. This implementation uses a hypercall instead of hardware specific code. The data that is usually reported by interrupts is now reported to xen by the hardware. Xen distinguish some parameters and reports the data chunk to the profiling domain via the virtual interrupt event notification provided by xen. This gets more complex with multiple domains etc. For more read the docs on xenoprof web page. The hardware specific code that once was in the oprofile kernel drivers is now located (adapted to the new environment) in the xen source where the new hypercalls are mapped to the real hardware.

-- Mapping xenoprof to Power - simple approach --
This approach tries to use as much of the initial xenoprof architecture by trying to map the power implementation to the technically x86 oriented xenoprof architecture. This would ease the implementation but spawn some risks I try to list here (The list is not complete, there may be more not yet realized issues). The basic principle of those profiling implementations is a performance counter (real time, cycles, special events, ... ) that triggers an interrupt. This interrupt then tries to save information about the current point of execution in its interrupt handler. The oprofile implementation for power works in a similar scheme so I thought this should be the easiest way.

-- Possible issues and their background --
Please take a look at this graphic before/while reading the following details (https://ltc.linux.ibm.com/wiki/XenPPC/ profilingdiscussion) – it might also be useful to have a PowerISA doc to read about special registers and bits (http:// www.power.org/news/articles/new_brand/#isa). The setting of the used hardware elements in the x86 implementation needs ring0 afaik and the Dom kernel runs in ring1, because of that it can't interfere the nmi programming done by xen in ring0. In the power architecture there are three privilege levels and the linux kernel usually runs in the second level. Afaik the Dom linux kernel does also run in this level in the xen-ppc implementation, because of that we could set performance monitor registers up in the right way in xen but could not really be sure that a Dom kernel does not change the related registers without “asking” the hypervisor.
-> is there a way still unknown to me to protect those registers?

-- Other possible approaches --
After consulting the current Power ISA documents again I found some points that may allow other implementations of profiling in xen. a) Because the Dom Kernel seem to be able to setup the performance profiling without invoking the hypervisor it could be possible to let a domain just do the profiling on their own. But there are other issues in this way too e.g. In which way would samples of other domains occur and would this be a security breach?

b) The Power architecture provides a very potent performance monitor with features that allow the freezing of the counters e.g. Freeze them while the execution is in hypervisor mode MSR_HVPR =0b10. But such features would only help to distinguish vertically in the graphics referenced above. Only the hypervisor is in a position to differ horizontally between different domains.

I'm planning to move the illustration I used to the public wiki after the first round of review and keep the planned design up to date there.
More but not yet mature thought&ideas about that in mind,
Grüsse / regards,
Christian Ehrhardt

IBM Linux Technology Center, Open Virtualization
+49 7031/16-3385

IBM Deutschland Entwicklung GmbH
Vorsitzender des Aufsichtsrats: Johann Weihen
Geschäftsführung: Herbert Kircher
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

Xen-ppc-devel mailing list


Grüsse / regards, Christian Ehrhardt

IBM Linux Technology Center, Open Virtualization
+49 7031/16-3385

IBM Deutschland Entwicklung GmbH
Vorsitzender des Aufsichtsrats: Johann Weihen Geschäftsführung: Herbert Kircher Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

Xen-ppc-devel mailing list

Reply via email to