I've been able to replicate what seems to be a race condition based bug.
I can run many traces without a crash, and sometimes it will crash on
the first attempt.
I've attached a log of the dmesg output of what is happening. Please
let me know if there is anything else I can provide that might help
diagnose.
----------------------------------------
From: Manu Awasthi <manu.awas...@gmail.com>
Date: Tue, Nov 3, 2009 at 10:31 AM
Subject: kernel panic with monitoring DRAM events
To: perfmon2-devel@lists.sourceforge.net
Hi all,
I have been measuring memory events for the parsec benchmark suite on a
dual socket, quad-core opteron machine with pfmlib version 3.9, kernel
pfmon version 2.82 and kernel version 2.6.29.6 . This is what I use as
my commandline option:
>> pfmon --with-header --outfile=test1 --verbose -u --switch-timeout=100
-eDRAM_ACCESSES_PAGE:HIT,DRAM_ACCESSES_PAGE:MISS,DRAM_ACCESSES_PAGE:CONFLICT,DRAM_ACCESSES_PAGE:ALL
-eDRAM_ACCESSES_PAGE:DCT1_PAGE_HIT,DRAM_ACCESSES_PAGE:DCT1_PAGE_MISS,DRAM_ACCESSES_PAGE:DCT1_PAGE_CONFLICT,DRAM_ACCESSES_PAGE:ALL
$PASEC_COMMAND
The problem is, sometimes, over different runs of the same
(multi-threaded) benchmark, the kernel panics and the machine freezes
up. has anybody ever experienced something of this sort before? Or is
there something that I am doing wrong? Is there a better way to measure
these stats (system-wide monitoring?)?
Any help is appreciated.
Thanks,
Manu
-----------------------------
David W Nellans
dnell...@cs.utah.edu
[ 214.111050] BUG: using smp_processor_id() in preemptible [00000000]
code: pfmon/2344
[ 214.111070] caller is pfm_arch_get_pmu_module_name+0x1a/0xe8
[ 214.111077] Pid: 2344, comm: pfmon Not tainted 2.6.29.6 #1
[ 214.111082] Call Trace:
[ 214.111104] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 214.111113] [<ffffffff80227863>]
pfm_arch_get_pmu_module_name+0x1a/0xe8
[ 214.111126] [<ffffffff803c756f>] pfm_pmu_conf_get+0x4b/0x117
[ 214.111138] [<ffffffff803cbf0a>] __pfm_create_context+0x2b/0x26d
[ 214.111145] [<ffffffff803c8897>] sys_pfm_create_context+0x8e/0xaa
[ 214.111155] [<ffffffff802bb45a>] ? vfs_write+0xce/0x106
[ 214.111163] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
[ 214.111169] BUG: using smp_processor_id() in preemptible [00000000]
code: pfmon/2344
[ 214.111177] caller is pfm_arch_get_pmu_module_name+0x93/0xe8
[ 214.111182] Pid: 2344, comm: pfmon Not tainted 2.6.29.6 #1
[ 214.111186] Call Trace:
[ 214.111191] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 214.111201] [<ffffffff802278dc>]
pfm_arch_get_pmu_module_name+0x93/0xe8
[ 214.111209] [<ffffffff803c756f>] pfm_pmu_conf_get+0x4b/0x117
[ 214.111219] [<ffffffff803cbf0a>] __pfm_create_context+0x2b/0x26d
[ 214.111228] [<ffffffff803c8897>] sys_pfm_create_context+0x8e/0xaa
[ 214.111237] [<ffffffff802bb45a>] ? vfs_write+0xce/0x106
[ 214.111246] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
[ 214.142267] BUG: using smp_processor_id() in preemptible [00000000]
code: modprobe/2346
[ 214.142281] caller is pfm_amd64_probe_pmu+0x22/0x3f7 [perfmon_amd64]
[ 214.142287] Pid: 2346, comm: modprobe Not tainted 2.6.29.6 #1
[ 214.142292] Call Trace:
[ 214.142304] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 214.142316] [<ffffffffa02b5727>] pfm_amd64_probe_pmu+0x22/0x3f7
[perfmon_amd64]
[ 214.142328] [<ffffffff80280a32>] ?
tracepoint_update_probe_range+0x92/0x9b
[ 214.142342] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.142350] [<ffffffff803c770a>] pfm_pmu_register+0x5f/0x301
[ 214.142359] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.142367] [<ffffffffa02ba010>] pfm_amd64_pmu_init_module+0x10/0x12
[perfmon_amd64]
[ 214.142375] [<ffffffff80209056>] _stext+0x56/0x12b
[ 214.142383] [<ffffffff80251987>] ? up_read+0x9/0xb
[ 214.142391] [<ffffffff802523e8>] ?
__blocking_notifier_call_chain+0x56/0x62
[ 214.142403] [<ffffffff80261741>] sys_init_module+0xa5/0x1c8
[ 214.142409] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
[ 214.142419] BUG: using smp_processor_id() in preemptible [00000000]
code: modprobe/2346
[ 214.142428] caller is pfm_amd64_probe_pmu+0x4a/0x3f7 [perfmon_amd64]
[ 214.142450] Pid: 2346, comm: modprobe Not tainted 2.6.29.6 #1
[ 214.142457] Call Trace:
[ 214.142463] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 214.142475] [<ffffffffa02b574f>] pfm_amd64_probe_pmu+0x4a/0x3f7
[perfmon_amd64]
[ 214.142487] [<ffffffff80280a32>] ?
tracepoint_update_probe_range+0x92/0x9b
[ 214.142498] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.142511] [<ffffffff803c770a>] pfm_pmu_register+0x5f/0x301
[ 214.142521] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.142535] [<ffffffffa02ba010>] pfm_amd64_pmu_init_module+0x10/0x12
[perfmon_amd64]
[ 214.142542] [<ffffffff80209056>] _stext+0x56/0x12b
[ 214.142551] [<ffffffff80251987>] ? up_read+0x9/0xb
[ 214.142561] [<ffffffff802523e8>] ?
__blocking_notifier_call_chain+0x56/0x62
[ 214.142573] [<ffffffff80261741>] sys_init_module+0xa5/0x1c8
[ 214.142585] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
[ 214.142593] BUG: using smp_processor_id() in preemptible [00000000]
code: modprobe/2346
[ 214.142601] caller is pfm_amd64_probe_pmu+0xae/0x3f7 [perfmon_amd64]
[ 214.142609] Pid: 2346, comm: modprobe Not tainted 2.6.29.6 #1
[ 214.142612] Call Trace:
[ 214.142617] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 214.142631] [<ffffffffa02b57b3>] pfm_amd64_probe_pmu+0xae/0x3f7
[perfmon_amd64]
[ 214.142643] [<ffffffff80280a32>] ?
tracepoint_update_probe_range+0x92/0x9b
[ 214.142654] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.142667] [<ffffffff803c770a>] pfm_pmu_register+0x5f/0x301
[ 214.142678] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.142689] [<ffffffffa02ba010>] pfm_amd64_pmu_init_module+0x10/0x12
[perfmon_amd64]
[ 214.142701] [<ffffffff80209056>] _stext+0x56/0x12b
[ 214.142706] [<ffffffff80251987>] ? up_read+0x9/0xb
[ 214.142715] [<ffffffff802523e8>] ?
__blocking_notifier_call_chain+0x56/0x62
[ 214.142726] [<ffffffff80261741>] sys_init_module+0xa5/0x1c8
[ 214.142735] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
[ 214.142744] BUG: using smp_processor_id() in preemptible [00000000]
code: modprobe/2346
[ 214.142752] caller is pfm_amd64_probe_pmu+0xef/0x3f7 [perfmon_amd64]
[ 214.142758] Pid: 2346, comm: modprobe Not tainted 2.6.29.6 #1
[ 214.142761] Call Trace:
[ 214.142766] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 214.142778] [<ffffffffa02b57f4>] pfm_amd64_probe_pmu+0xef/0x3f7
[perfmon_amd64]
[ 214.142790] [<ffffffff80280a32>] ?
tracepoint_update_probe_range+0x92/0x9b
[ 214.142801] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.142814] [<ffffffff803c770a>] pfm_pmu_register+0x5f/0x301
[ 214.142824] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.142838] [<ffffffffa02ba010>] pfm_amd64_pmu_init_module+0x10/0x12
[perfmon_amd64]
[ 214.142851] [<ffffffff80209056>] _stext+0x56/0x12b
[ 214.142860] [<ffffffff80251987>] ? up_read+0x9/0xb
[ 214.142871] [<ffffffff802523e8>] ?
__blocking_notifier_call_chain+0x56/0x62
[ 214.142891] [<ffffffff80261741>] sys_init_module+0xa5/0x1c8
[ 214.142904] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
[ 214.142916] found family=16 IBSCTL=0x0
[ 214.148340] BUG: using smp_processor_id() in preemptible [00000000]
code: modprobe/2346
[ 214.148357] caller is pfm_amd64_probe_pmu+0x13a/0x3f7 [perfmon_amd64]
[ 214.148365] Pid: 2346, comm: modprobe Not tainted 2.6.29.6 #1
[ 214.148370] Call Trace:
[ 214.148386] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 214.148397] [<ffffffffa02b583f>] pfm_amd64_probe_pmu+0x13a/0x3f7
[perfmon_amd64]
[ 214.148409] [<ffffffff80280a32>] ?
tracepoint_update_probe_range+0x92/0x9b
[ 214.148425] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.148436] [<ffffffff803c770a>] pfm_pmu_register+0x5f/0x301
[ 214.148445] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.148453] [<ffffffffa02ba010>] pfm_amd64_pmu_init_module+0x10/0x12
[perfmon_amd64]
[ 214.148465] [<ffffffff80209056>] _stext+0x56/0x12b
[ 214.148473] [<ffffffff80251987>] ? up_read+0x9/0xb
[ 214.148480] [<ffffffff802523e8>] ?
__blocking_notifier_call_chain+0x56/0x62
[ 214.148491] [<ffffffff80261741>] sys_init_module+0xa5/0x1c8
[ 214.148499] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
[ 214.148508] NorthBridge socket-level event access control: enabled
[ 214.148534] Failed to initialize extended interrupts for IBS
[ 214.148538] Unable to use IBS
[ 214.148542] BUG: using smp_processor_id() in preemptible [00000000]
code: modprobe/2346
[ 214.148551] caller is pfm_amd64_probe_pmu+0x3a3/0x3f7 [perfmon_amd64]
[ 214.148556] Pid: 2346, comm: modprobe Not tainted 2.6.29.6 #1
[ 214.148560] Call Trace:
[ 214.148566] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 214.148575] [<ffffffffa02b5aa8>] pfm_amd64_probe_pmu+0x3a3/0x3f7
[perfmon_amd64]
[ 214.148582] [<ffffffff80280a32>] ?
tracepoint_update_probe_range+0x92/0x9b
[ 214.148591] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.148603] [<ffffffff803c770a>] pfm_pmu_register+0x5f/0x301
[ 214.148611] [<ffffffffa02ba000>] ?
pfm_amd64_pmu_init_module+0x0/0x12 [perfmon_amd64]
[ 214.148620] [<ffffffffa02ba010>] pfm_amd64_pmu_init_module+0x10/0x12
[perfmon_amd64]
[ 214.148626] [<ffffffff80209056>] _stext+0x56/0x12b
[ 214.148632] [<ffffffff80251987>] ? up_read+0x9/0xb
[ 214.148638] [<ffffffff802523e8>] ?
__blocking_notifier_call_chain+0x56/0x62
[ 214.148646] [<ffffffff80261741>] sys_init_module+0xa5/0x1c8
[ 214.148654] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
[ 214.148763] AMD64 PMU installed
[ 543.442145] 3w-9xxx: scsi0: AEN: INFO (0x04:0x0029): Verify
started:unit=0.
[ 571.776021] BUG: using smp_processor_id() in preemptible [00000000]
code: facesim/2355
[ 571.776035] caller is __pfm_exit_thread+0xaf/0xe8
[ 571.776041] Pid: 2355, comm: facesim Not tainted 2.6.29.6 #1
[ 571.776044] Call Trace:
[ 571.776053] [<ffffffff803d6956>] debug_smp_processor_id+0xca/0xe0
[ 571.776058] [<ffffffff803cb99f>] __pfm_exit_thread+0xaf/0xe8
[ 571.776064] [<ffffffff8020a5a4>] exit_thread+0x127/0x130
[ 571.776071] [<ffffffff8023e980>] do_exit+0x235/0x755
[ 571.776076] [<ffffffff802a1273>] ? do_munmap+0x2f1/0x313
[ 571.776082] [<ffffffff805df544>] ? _spin_lock_irqsave+0x18/0x34
[ 571.776086] [<ffffffff8023ef13>] do_group_exit+0x73/0xa0
[ 571.776090] [<ffffffff8023ef52>] sys_exit_group+0x12/0x16
[ 571.776094] [<ffffffff8020b79b>] system_call_fastpath+0x16/0x1b
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
perfmon2-devel mailing list
perfmon2-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/perfmon2-devel
