Re: [perfmon2] Kernel support for user level rdpmc?
On Thu, Sep 1, 2011 at 1:07 AM, Corey Ashford wrote: > On 08/25/2011 07:19 AM, stephane eranian wrote: >> Hi, >> >> Sorry for late reply. >> >> The current support for mmaped count is broken on perf_event x86. >> It simply does not work. I think it only works on PPC at this point. > > Just as an aside, you can access the counter registers from user space > on Power (aka PPC) machines, but because the kernel is free to schedule > the events onto whatever counters that meet the resource constraints, > it's not at all clear which hardware counter to read from user space, > and in fact, with event rotation, the counter being used can change from > one system tick till the next. > > If you program a single event, you can be guaranteed that it won't move > around, but you still will have to guess or somehow determine which > hardware counter is being used by the kernel. > Yes, and that's why they have this 'lock' field in there.It's not really a lock but rather a generation counter. You need to read it before you attempt to read and you need to check it when you're done reading. If the two values don't match then the counter changed and you need to retry. And changes means it may have moved to a different counter. But the key problem here is the time scaling. In case you are multiplex you need to be able to retrieve time_enabled and time_running to scale the count. But that's not exposed, thus it does not work as soon as you have multiplexing. Well, unless you only care about deltas and not the absolute values. > - Corey > > -- > Special Offer -- Download ArcSight Logger for FREE! > Finally, a world-class log management solution at an even better > price-free! And you'll get a free "Love Thy Logs" t-shirt when you > download Logger. Secure your free ArcSight Logger TODAY! > http://p.sf.net/sfu/arcsisghtdev2dev > ___ > perfmon2-devel mailing list > perfmon2-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/perfmon2-devel > -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ perfmon2-devel mailing list perfmon2-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
Re: [perfmon2] Kernel support for user level rdpmc?
On 01/09/2011 1:55 AM, stephane eranian wrote: > On Thu, Sep 1, 2011 at 1:07 AM, Corey Ashford > wrote: >> On 08/25/2011 07:19 AM, stephane eranian wrote: >>> Hi, >>> >>> Sorry for late reply. >>> >>> The current support for mmaped count is broken on perf_event x86. >>> It simply does not work. I think it only works on PPC at this point. >> Just as an aside, you can access the counter registers from user space >> on Power (aka PPC) machines, but because the kernel is free to schedule >> the events onto whatever counters that meet the resource constraints, >> it's not at all clear which hardware counter to read from user space, >> and in fact, with event rotation, the counter being used can change from >> one system tick till the next. >> >> If you program a single event, you can be guaranteed that it won't move >> around, but you still will have to guess or somehow determine which >> hardware counter is being used by the kernel. >> > Yes, and that's why they have this 'lock' field in there.It's not really a > lock > but rather a generation counter. You need to read it before you attempt to > read and you need to check it when you're done reading. If the two values > don't match then the counter changed and you need to retry. And changes > means it may have moved to a different counter. This protocol is actually documented pretty well in , too. Read the lock, read the index, read hw counter[index-1], read lock again to verify. > But the key problem here is the time scaling. In case you are multiplex > you need to be able to retrieve time_enabled and time_running to scale > the count. But that's not exposed, thus it does not work as soon as you > have multiplexing. Well, unless you only care about deltas and not the > absolute values. Doesn't perf_event_mmap_page expose both those, also protected by the generation counter? Or are you saying the kernel doesn't actually update those fields right now? Ryan -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ perfmon2-devel mailing list perfmon2-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
Re: [perfmon2] Kernel support for user level rdpmc?
On Thu, Sep 1, 2011 at 3:06 PM, Ryan Johnson wrote: > On 01/09/2011 1:55 AM, stephane eranian wrote: >> On Thu, Sep 1, 2011 at 1:07 AM, Corey Ashford >> wrote: >>> On 08/25/2011 07:19 AM, stephane eranian wrote: Hi, Sorry for late reply. The current support for mmaped count is broken on perf_event x86. It simply does not work. I think it only works on PPC at this point. >>> Just as an aside, you can access the counter registers from user space >>> on Power (aka PPC) machines, but because the kernel is free to schedule >>> the events onto whatever counters that meet the resource constraints, >>> it's not at all clear which hardware counter to read from user space, >>> and in fact, with event rotation, the counter being used can change from >>> one system tick till the next. >>> >>> If you program a single event, you can be guaranteed that it won't move >>> around, but you still will have to guess or somehow determine which >>> hardware counter is being used by the kernel. >>> >> Yes, and that's why they have this 'lock' field in there.It's not really a >> lock >> but rather a generation counter. You need to read it before you attempt to >> read and you need to check it when you're done reading. If the two values >> don't match then the counter changed and you need to retry. And changes >> means it may have moved to a different counter. > This protocol is actually documented pretty well in > , too. Read the lock, read the index, read hw > counter[index-1], read lock again to verify. > >> But the key problem here is the time scaling. In case you are multiplex >> you need to be able to retrieve time_enabled and time_running to scale >> the count. But that's not exposed, thus it does not work as soon as you >> have multiplexing. Well, unless you only care about deltas and not the >> absolute values. > Doesn't perf_event_mmap_page expose both those, also protected by the > generation counter? Or are you saying the kernel doesn't actually update > those fields right now? > Yes, it does. I am not sure they're updated correctly, though. I have not tried that in a very long time. > Ryan > > > -- > Special Offer -- Download ArcSight Logger for FREE! > Finally, a world-class log management solution at an even better > price-free! And you'll get a free "Love Thy Logs" t-shirt when you > download Logger. Secure your free ArcSight Logger TODAY! > http://p.sf.net/sfu/arcsisghtdev2dev > ___ > perfmon2-devel mailing list > perfmon2-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/perfmon2-devel > -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ perfmon2-devel mailing list perfmon2-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
Re: [perfmon2] Kernel support for user level rdpmc?
On Thu, Sep 1, 2011 at 3:29 PM, stephane eranian wrote: > On Thu, Sep 1, 2011 at 3:06 PM, Ryan Johnson > wrote: >> On 01/09/2011 1:55 AM, stephane eranian wrote: >>> On Thu, Sep 1, 2011 at 1:07 AM, Corey Ashford >>> wrote: On 08/25/2011 07:19 AM, stephane eranian wrote: > Hi, > > Sorry for late reply. > > The current support for mmaped count is broken on perf_event x86. > It simply does not work. I think it only works on PPC at this point. Just as an aside, you can access the counter registers from user space on Power (aka PPC) machines, but because the kernel is free to schedule the events onto whatever counters that meet the resource constraints, it's not at all clear which hardware counter to read from user space, and in fact, with event rotation, the counter being used can change from one system tick till the next. If you program a single event, you can be guaranteed that it won't move around, but you still will have to guess or somehow determine which hardware counter is being used by the kernel. >>> Yes, and that's why they have this 'lock' field in there.It's not really a >>> lock >>> but rather a generation counter. You need to read it before you attempt to >>> read and you need to check it when you're done reading. If the two values >>> don't match then the counter changed and you need to retry. And changes >>> means it may have moved to a different counter. >> This protocol is actually documented pretty well in >> , too. Read the lock, read the index, read hw >> counter[index-1], read lock again to verify. >> >>> But the key problem here is the time scaling. In case you are multiplex >>> you need to be able to retrieve time_enabled and time_running to scale >>> the count. But that's not exposed, thus it does not work as soon as you >>> have multiplexing. Well, unless you only care about deltas and not the >>> absolute values. >> Doesn't perf_event_mmap_page expose both those, also protected by the >> generation counter? Or are you saying the kernel doesn't actually update >> those fields right now? >> > Yes, it does. I am not sure they're updated correctly, though. > I have not tried that in a very long time. > Did you manage to make libpfm4's self_count program work correctly? Even by just looking at the raw count coming out of rdpmc? I think there are issues with hdr->offset, i.e., the 64-bit sw-maintained base for the counter. >> Ryan >> >> >> -- >> Special Offer -- Download ArcSight Logger for FREE! >> Finally, a world-class log management solution at an even better >> price-free! And you'll get a free "Love Thy Logs" t-shirt when you >> download Logger. Secure your free ArcSight Logger TODAY! >> http://p.sf.net/sfu/arcsisghtdev2dev >> ___ >> perfmon2-devel mailing list >> perfmon2-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/perfmon2-devel >> > -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ perfmon2-devel mailing list perfmon2-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
Re: [perfmon2] Kernel support for user level rdpmc?
On 01/09/2011 6:45 AM, stephane eranian wrote:
> On Thu, Sep 1, 2011 at 3:29 PM, stephane eranian
> wrote:
>> On Thu, Sep 1, 2011 at 3:06 PM, Ryan Johnson
>> wrote:
>>> On 01/09/2011 1:55 AM, stephane eranian wrote:
On Thu, Sep 1, 2011 at 1:07 AM, Corey Ashford
wrote:
> On 08/25/2011 07:19 AM, stephane eranian wrote:
>> Hi,
>>
>> Sorry for late reply.
>>
>> The current support for mmaped count is broken on perf_event x86.
>> It simply does not work. I think it only works on PPC at this point.
> Just as an aside, you can access the counter registers from user space
> on Power (aka PPC) machines, but because the kernel is free to schedule
> the events onto whatever counters that meet the resource constraints,
> it's not at all clear which hardware counter to read from user space,
> and in fact, with event rotation, the counter being used can change from
> one system tick till the next.
>
> If you program a single event, you can be guaranteed that it won't move
> around, but you still will have to guess or somehow determine which
> hardware counter is being used by the kernel.
>
Yes, and that's why they have this 'lock' field in there.It's not really a
lock
but rather a generation counter. You need to read it before you attempt to
read and you need to check it when you're done reading. If the two values
don't match then the counter changed and you need to retry. And changes
means it may have moved to a different counter.
>>> This protocol is actually documented pretty well in
>>> , too. Read the lock, read the index, read hw
>>> counter[index-1], read lock again to verify.
>>>
But the key problem here is the time scaling. In case you are multiplex
you need to be able to retrieve time_enabled and time_running to scale
the count. But that's not exposed, thus it does not work as soon as you
have multiplexing. Well, unless you only care about deltas and not the
absolute values.
>>> Doesn't perf_event_mmap_page expose both those, also protected by the
>>> generation counter? Or are you saying the kernel doesn't actually update
>>> those fields right now?
>>>
>> Yes, it does. I am not sure they're updated correctly, though.
>> I have not tried that in a very long time.
>>
> Did you manage to make libpfm4's self_count program work correctly?
> Even by just looking at the raw count coming out of rdpmc?
>
> I think there are issues with hdr->offset, i.e., the 64-bit sw-maintained
> base for the counter.
I only did limited testing because things took priority the last couple
of weeks, but I'll be back into it in the next couple of weeks.
Meanwhile, here's what I know:
The machine is a Westmere EX (which is why I can't just use an older
kernel+perfctr) running kernel 2.6.38. I've got the cvs head for papi,
wired up with git version 9fc1bc1e of libpfm4. self_count seg faults by
default because rdpmc is privileged, and papi's unit tests cause the
machine to hard-lock (have to use the hypervisor to reboot). One
definite culprit is ctests/overflow_allcounters, but I haven't done a
bisection search in 2.6.38 to see if there are any others. I upgraded to
kernel 2.6.39, ctests/overflow_allcounters is the only unit test
failure, but it "only" hard-locks the perf events infrastructure rather
than the whole machine. The unit tests's process hangs with 0% cpu util
and becomes unkillable, and any later process attempting to use perf
events suffers the same fate. The mmap+rdpmc support is apparently
disabled in 2.6.39, in that index=0 for all time. The self_count test
runs without errors and reports monotonically increasing values, but I
never attempted to verify that the starting count was meaningful.
For now I've rolled back to 2.6.38, since the later version is a step
backwards for my needs. With the kernel module I mentioned before,
user-level rdpmc seems to stay enabled indefinitely and self_count runs
without errors. I've extended the test slightly to run fib with
n={30,35,40}, to track which counter number it used directly (if any),
and to report the deltas between measurements. Here's the output I get:
> $ ./self_count
> raw=0xcd73 offset=0x0, ena=36278 run=36278 idx=-1 direct=0
>52595 PERF_COUNT_HW_CPU_CYCLES (delta=cd73)
> raw=0x811d738b offset=0x7fff, ena=36278 run=36278 idx=0 direct=1
> 281474995417994 PERF_COUNT_HW_CPU_CYCLES (delta= 1011ca617)
> raw=0x8d588633 offset=0x7fff, ena=36278 run=36278 idx=0 direct=1
> 281475200615986 PERF_COUNT_HW_CPU_CYCLES (delta= c3b12a8)
> raw=0x94aa8789 offset=0xfffe, ena=36278 run=36278 idx=0 direct=1
> 281477470914439 PERF_COUNT_HW_CPU_CYCLES (delta=87520155)
> raw=0x95c33ede offset=0xfffe, ena=36278 run=36278 idx=0 direct=1
> 281477489311452 PERF_COUNT_HW_CPU_CYCLES (delta= 118b755)
> raw=0x
Re: [perfmon2] Kernel support for user level rdpmc?
On Thu, 1 Sep 2011, Ryan Johnson wrote: > definite culprit is ctests/overflow_allcounters, but I haven't done a > bisection search in 2.6.38 to see if there are any others. That's a bug in the kernel that will be fixed in 3.1 (and has been backported to the stable trees). The current PAPI CVS has a patch that disables the cause of the test (overflow on software counts) from overflow_allcounters so that people with unpatched kernels won't have issues running the PAPI tests. Vince -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ perfmon2-devel mailing list perfmon2-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
[perfmon2] Help on make the Pfmon-3.9
Hi guys, I have one problem on making pfmon-3.9. I know this question is kind of low level, but I have tried a lot of ways and no result. The system I am using is Ubuntu (2.6.38 kernel), and I have already installed libpfm-3.10 successfully. The information for this failure is: make[1]: Entering directory `/home/junjie/tools/PMU/pfmon-3.9/pfmon' cc -g -ggdb -Wall -Werror -D_REENTRANT -I/home/junjie/tools/PMU/libpfm3.10/include -DCONFIG_PFMON_X86_64 -DPFMON_DEBUG -DDATADIR=\"/home/junjie/tools/PMU/pfmon3.9/share/pfmon\" -I. -I/usr/include/libelf -D_GNU_SOURCE -DPFMON_DEBUG -g -c pfmon_symbols.c cc1: warnings being treated as errors pfmon_symbols.c: In function pfmon_gather_module_symbols? pfmon_symbols.c:720:3: error: implicit declaration of function tat? make[1]: *** [pfmon_symbols.o] Error 1 make[1]: Leaving directory `/home/junjie/tools/PMU/pfmon-3.9/pfmon' make: *** [all] Error 2 Could anyone give me some hint or solution for this? I have modified the config.mk according to the README. Thanks! Best Junjie-- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free "Love Thy Logs" t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev___ perfmon2-devel mailing list perfmon2-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
