Ok, I have some more data about this lock-up problem. I turned on perfmon
debugging and saw that the last thing that perfmon did was to call
down_write() from pfm_smp_buf_space_release.212. That code attempts to
acquire a lock, so I decided to turn on lock debugging in the kernel, and
got this output when I ran the test case:
perfmon: pfm_smpl_buf_space_release.212: CPU2 [3318]: doing down_write
=============================================
[ INFO: possible recursive locking detected ]
2.6.28-rc6-pfm2-09445-g4fca1a2-dirty #12
---------------------------------------------
task_smpl/3318 is trying to acquire lock:
(&mm->mmap_sem){----}, at: [<c0000000003037d8>]
.pfm_smpl_buf_space_release+0xa0/0x180
but task is already holding lock:
(&mm->mmap_sem){----}, at: [<c000000000102b34>] .sys_munmap+0x54/0xa0
other info that might help us debug this:
1 lock held by task_smpl/3318:
#0: (&mm->mmap_sem){----}, at: [<c000000000102b34>]
.sys_munmap+0x54/0xa0
stack backtrace:
Call Trace:
[c00000000ca77380] [c000000000012254] .show_stack+0x94/0x198 (unreliable)
[c00000000ca77430] [c000000000012380] .dump_stack+0x28/0x3c
[c00000000ca774b0] [c0000000000a14f0] .validate_chain+0x690/0xdc0
[c00000000ca77570] [c0000000000a2404] .__lock_acquire+0x7e4/0x8bc
[c00000000ca77670] [c0000000000a2588] .lock_acquire+0xac/0xf8
[c00000000ca77740] [c0000000005cb630] .down_write+0x64/0xbc
[c00000000ca777d0] [c0000000003037d8]
.pfm_smpl_buf_space_release+0xa0/0x180
[c00000000ca77870] [c00000000030d464] .pfm_smpl_buf_free+0x8c/0x104
[c00000000ca77900] [c00000000030f2a0] .pfm_free_context+0x40/0xc8
[c00000000ca77990] [c000000000307d5c] .__pfm_close+0x2f8/0x33c
[c00000000ca77a60] [c000000000308af8] .pfm_close+0x98/0xb4
[c00000000ca77af0] [c00000000012b56c] .__fput+0x16c/0x258
[c00000000ca77ba0] [c00000000012baa4] .fput+0x50/0x68
[c00000000ca77c30] [c0000000001003c4] .remove_vma+0x90/0xf8
[c00000000ca77cc0] [c0000000001015d8] .do_munmap+0x30c/0x358
[c00000000ca77d90] [c000000000102b48] .sys_munmap+0x68/0xa0
[c00000000ca77e30] [c0000000000084d4] syscall_exit+0x0/0x40
Does this ring any bells with you?
Thanks,
- Corey
"stephane eranian" <eran...@googlemail.com> wrote on 01/07/2009 12:03:24
PM:
> Corey,
>
> I was expecting success with the program below if /tmp/foo exists.
>
> The perfmon code that handles all of this is generic, so there must be a
> race condition somewhere which is only exposed on Power.
>
> On Wed, Jan 7, 2009 at 8:02 PM, Corey J Ashford <cjash...@us.ibm.com>
wrote:
> > Thanks for the reply, Stephane. I tried the test case you suggested:
> >
> > main() {
> > int fd;
> > void *addr;
> >
> > fd = open ("/tmp/foo", O_RDONLY);
> > printf("fd = %d\n", fd);
> > addr = mmap(NULL, 10, PROT_READ, MAP_PRIVATE, fd, 0);
> > printf("addr = %p\n", addr);
> > if (close(fd)) {
> > printf("close failed\n");
> > }
> > if (munmap(addr, 10)) {
> > printf("munmap failed\n");
> > }
> > }
> >
> > and it worked fine. So apparently there is a problem related to
> > munmap'ing a perfmon fd on Power. This will need more investigation,
> > obviously.
> >
> > - Corey
> >
> > "stephane eranian" <eran...@googlemail.com> wrote on 01/06/2009
10:28:41
> > PM:
> >
> >> Corey,
> >>
> >> On Wed, Jan 7, 2009 at 3:24 AM, Corey J Ashford <cjash...@us.ibm.com>
> > wrote:
> >> >
> >> > Hello,
> >> >
> >> > I'd appreciate it if someone on this mailing list could try out the
> > libpfm
> >> > example: task_smpl and see if it runs correctly for you on any
other
> >> > architecture besides Power.
> >> >
> >> > When I run it on my Power5-based machine here, I get a system hang
> > that
> >> > occurs when the munmap call is made. Looking at the code in the
> > example, I
> >> > reversed the order of the close and munmap... so that the memory is
> > unmapped
> >> > before the fd is closed, and this allows the test to run to
completion
> >> > without error and causes no hang. I also tried commenting out the
> > call to
> >> > pfm_start, to cut perfmon out of the loop for the most part, and
the
> >> > behavior still reproduces - the system hangs unless I reverse those
> > two
> >> > calls.
> >> >
> >> > When the system hangs like this, if I get it to go into Xmon, none
of
> > the
> >> > CPU stacks are interesting. They all appear to be idle.
> >> >
> >> > I run the test as follows:
> >> >
> >> > ./task_smpl /bin/sleep 3
> >> >
> >>
> >> This test runs fine on my x86-64 system (Core 2). The order of the
> >> close() vs munmap()
> >> should not matter. The calls can be made in any order. The perfmon
> >> context is destroyed
> >> when the last reference to the file descriptor disappears, mmap
counts
> >> as 1. If you do close()
> >> followed by munmap(), the perfmon context is destroyed as part of the
> >> munmap(). This sequence
> >> should not hang for you. What happens if you do a similar sequence
but
> >> just with a regular file:
> >> fd = open("/tmp/foo);
> >> addr = mmap(fd);
> >> close(fd);
> >> munmap(addr);
> >>
> >> The test runs to completion on both x86-64 and ia64:
> >>
> >> $ task_smpl /bin/sleep 3
> >> sycall base 295
> >> major version 2
> >> minor version 82
> >> [FIXED_CTRL(pmc16)=0xaa pmi0=1 en0=0x2 pmi1=1 en1=0x2 pmi2=1 en2=0x0]
> >> INSTRUCTIONS_RETIRED UNHALTED_CORE_CYCLES
> >> [FIXED_CTR0(pmd16)]
> >> [FIXED_CTR1(pmd17)]
> >> programming 1 PMCS and 2 PMDS
> >> buffer mapped @0x7f999029b000
> >> hdr_cur_offs=128 version=1.0
> >> task terminated
> >> entry 0 PID:32691 TID:32691 CPU:2 LAST_VAL:100000 IIP:0x7f66702246c2
> >> PMD16 :0x0000000000004130
> >> entry 1 PID:32691 TID:32691 CPU:2 LAST_VAL:100213 IIP:0x7f6670227560
> >> PMD16 :0x000000000000ef70
> >> entry 2 PID:32691 TID:32691 CPU:2 LAST_VAL:100060 IIP:0x7f6670233e52
> >> PMD16 :0x000000000000f384
> >> entry 3 PID:32691 TID:32691 CPU:2 LAST_VAL:100155
IIP:0xffffffff805c9e6f
> >> PMD16 :0x00000000000104fe
> >> 4 samples (4 in partial buffer) collected in 0 buffer overflows
> >> real 0h00m03.001s user 0h00m00.000s sys 0h00m00.001s
> >> $
> >
> >
------------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It is the best place to buy or sell services for
just about anything Open Source.
http://p.sf.net/sfu/Xq1LFB
_______________________________________________
perfmon2-devel mailing list
perfmon2-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
