Kevin,
On Fri, Mar 23, 2007 at 10:18:34AM -0500, Kevin Corry wrote:
> Hi Stephane,
>
> Sorry I've been MIA for a while. :( I got pulled off the Perfmon work I was
> doing last summer to do some work on Cell benchmarking and tools. But now I'm
> back, and Carl Love and I will be working on getting Perfmon working on Cell.
> Carl has been working on porting oprofile to Cell, and I've been working with
> tools for the Cell performance counters. I've been maintaining an
> IBM-internal driver that someone around here wrote for accessing the hardware
> counters, as well as a corresponding user-space tool. But now I finally get
> to ditch the driver and port the tool to use the Perfmon interface instead.
> Yea!
>
Glad to see you're back at it.
> So I thought I'd start out by asking if anyone has done any work on porting
> Perfmon to Cell. I recall that Phil made some comments last fall about
> working on Cell support, but I don't remember seeing anything else about it.
> So if anyone is already working on this, let us know and we'll lend them a
> hand. And since Cell is a sub-platform of powerpc, it's possible we'll be
> able to help get the POWER4/5/6 support finished off as well (although Cell
> is our first priority right now). And Carl has quite a bit of experience with
> the PMUs on those systems.
>
Nobody so far has come forward to start porting to Cell.
> I've also got some questions about issues that might come up during this
> porting effort.
>
> 1) Perfmon seems to have an implicit assumption that a PMU's counters are a
> fixed width. Specifically, the "pfm_pmu_config" structure has
> a "counter_width" field that applies to the whole PMU. However, Cell provides
> four 32-bit counters, and each of those can independently be configured as
> two 16-bit counters. So I'm curious if it will be possible to support this
> capability within Perfmon, especially regarding the 64-bit counter
> virtualization. Do you know of any other platforms that have variable-width
> counters?
>
Yes, today the code assumes all counters have the same width.
Variable width counters usually means you have a variable number
of counters, e.g., 4 32-bits or 2 64-bit counters. Te perfmon interface
always exposes counters as 64-bit no matter how they are implemented. Thus
it is not so obvious why one would want to use 32 vs 64-bit counters. Of course,
the 64-bit emulation is based upon the counter overflow interrupt. The narrower
the counter the more interrupts you get. Then I see a problem with naming if
you
allow the user to set this up. Tkaing the example above, with 4 counters you'd
have PMD0-PMD3, but when operating in 64-bit you'd have PMD0-PMD1 or PMD0/PMD2.
When writing PMD0, you have to actually write two registers, I guess. I think
this one could be hidden in the arch code.
So overall, I think this could be made to work but will require some
modifications.
And I'd like to understand better the advantage of this (besides fewer overflow
interrupts).
> 2) Each Cell processor includes SMT support (symmetric multi-threading,
> basically the same as Intel hyperthreading), but only has one PMU.
> Unfortunately, the PMU can't be shared between two threads running
> simultaneously on the same CPU (unlike Intel Pentium4, which splits up the
> counters between the two threads when HT is enabled). Are there any
> mechanisms yet to prevent scheduling two threads that both want to use the
> PMU on the same physical CPU? This problem will also exist on POWER4/5/6.
>
Ouch! yet another HT problem. Well, you could modify the affinity mask of
the thread but as I said many times on this list: this ia a very bad idea.
Many workloads depends on being evenly distributed across cores/threads.
In fact, I believe more and more will adjust their affinity manually for
performance reasons. An important point is that affinity is never for
correctness but only for performance. Yet in our context, we want to measure
performance, so we'd better make sure the thread runs where it is supposed
to. I believe perfmon cannot modify the affinity of a thread. Even if it did,
there would be no guarantee, the thread could not adjust it later on, so the
kernel would have to prevent modifications. Looks like we have 3 solutions:
- perfmon does not work when HT is on
- only allows one monitored thread in HT across the entire system
- create artificial blind spots when a monitored thread migrates to
to core where there is already a monitored thread. This is likely
to cause problems if the blind spot is not reported to the tool.
> 3) In your announcement in February of the latest Perfmon kernel patches you
> noted that the code will not currently build on powerpc due to the new TIF
> flags in inclue/asm-*/thread_info.h. Has this issue been solved yet? I don't
> personally know much about powerpc assembly (I tried hacking around on that
> section of code a bit, but had no luck), but I can almost certainly track
> down some people here at IBM to help if we still need to find a fix.
>
As for the compilation problem, thanks to Phil it is now fixed and the
modification will be released in my next patch.
> 4) After the announcement in February, there was some discussion about
> submitting Perfmon again to lkml. I was just wondering how close you are to
> posting the code. If there's anything you need help with before Perfmon can
> be submitted (code review, testing, cleanup, reorg, etc.), I'm happy to lend
> a hand. Carl and I are definitely eager to see Perfmon get included in
> mainline.
>
Yes, I will submit what we have for the 2.6.22 timeframe. The code has been
reviewed
on lkml already and I made many changes to satisfy the reviewers. There is never
enough review, so I would certainly appreciate your feedback on the existing
codebase.
> Thanks a lot for any info you can provide. I hope to get a little more active
> again on this list, and we'll keep you updated as we get a bit further along
> with the Cell support.
>
Thanks and welcome back.
--
-Stephane
_______________________________________________
perfmon mailing list
[email protected]
http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/
