Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-07 Thread Sachin Sant 

Steven Rostedt wrote:

On Tue, 2009-10-06 at 07:20 +1100, Benjamin Herrenschmidt wrote:
  

On Mon, 2009-10-05 at 09:25 -0400, Steven Rostedt wrote:


Sachin, can you give me more details on how you built that kernel ? (or
give them again in case I missed them the first time around :-), ie,
what toolchain, options, etc... or even better, give me remote access to
the build host ?
  

Ok, got access and had a quick look... seems to be a toolchain problem
to me. I'll investigate more tomorrow.


Hi Ben,

Any more word on this issue? 
  

Didn't you fix it using a TOC access ?

Unless I'm confusing things, I think the problem is the usage
of LOAD_REG_IMMEDIATE which generates relocs that we don't support
when CONFIG_RELOCATABLE is set.

I've merged a patch that post-processes the kernel now, to check
for such relocs so at least you should be warned at build time.



I thought we had two issues. One was the use of the relocs that did
cause issues. But then there was still crashes reported after that.
IIRC. I'm still suffering jetlag, so my memory is not that fresh about
  

As Ben suggested, i changed LOAD_REG_IMMEDIATE() to LOAD_REG_ADDR()
as follows.

-   LOAD_REG_IMMEDIATE(r4,ftrace_return_to_handler)
+   LOAD_REG_ADDR(r4,ftrace_return_to_handler)

With this change compile time warnings about bad relocations
related to ftrace are gone.

Before the change :
WARNING: 6 bad relocations
c0008f1a R_PPC64_ADDR16_HIGHEST  __ksymtab+0x00742110
c0008f1e R_PPC64_ADDR16_HIGHER  __ksymtab+0x00742110
c0008f26 R_PPC64_ADDR16_HI  __ksymtab+0x00742110
c0008f2a R_PPC64_ADDR16_LO  __ksymtab+0x00742110
c085e118 R_PPC64_ADDR64__crc_per_cpu__softirq_work_list
c08662d0 R_PPC64_ADDR64__crc_simple_prepare_write

After the change :
WARNING: 2 bad relocations
c085e118 R_PPC64_ADDR64__crc_per_cpu__softirq_work_list
c08662d0 R_PPC64_ADDR64__crc_simple_prepare_write

But i still run into oops while using ftrace function_graph.

Thanks
-Sachin



--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

diff -Naurp a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
--- a/arch/powerpc/kernel/entry_64.S	2009-10-06 15:31:29.0 +0530
+++ b/arch/powerpc/kernel/entry_64.S	2009-10-06 15:34:00.0 +0530
@@ -1038,7 +1038,7 @@ _GLOBAL(mod_return_to_handler)
 	 * We are in a module using the module's TOC.
 	 * Switch to our TOC to run inside the core kernel.
 	 */
-	LOAD_REG_IMMEDIATE(r4,ftrace_return_to_handler)
+	LOAD_REG_ADDR(r4,ftrace_return_to_handler)
 	ld	r2, 8(r4)
 
 	bl	.ftrace_return_to_handler
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-07 Thread Benjamin Herrenschmidt 
On Wed, 2009-10-07 at 14:26 +0530, Sachin Sant wrote:

 As Ben suggested, i changed LOAD_REG_IMMEDIATE() to LOAD_REG_ADDR()
 as follows.
 
 -   LOAD_REG_IMMEDIATE(r4,ftrace_return_to_handler)
 +   LOAD_REG_ADDR(r4,ftrace_return_to_handler)
 
 With this change compile time warnings about bad relocations
 related to ftrace are gone.

You also need to make sure r2 is properly loaded with the kernel TOC
pointer, something like ld r2,PACAKTOC(r13) before the LOAD_REG_ADDR.

Cheers,
Ben.

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-05 Thread Benjamin Herrenschmidt 
On Mon, 2009-10-05 at 09:25 -0400, Steven Rostedt wrote:
   Sachin, can you give me more details on how you built that kernel ? (or
   give them again in case I missed them the first time around :-), ie,
   what toolchain, options, etc... or even better, give me remote access to
   the build host ?
  
  Ok, got access and had a quick look... seems to be a toolchain problem
  to me. I'll investigate more tomorrow.
 
 Hi Ben,
 
 Any more word on this issue? 

Didn't you fix it using a TOC access ?

Unless I'm confusing things, I think the problem is the usage
of LOAD_REG_IMMEDIATE which generates relocs that we don't support
when CONFIG_RELOCATABLE is set.

I've merged a patch that post-processes the kernel now, to check
for such relocs so at least you should be warned at build time.

Cheers,
Ben.

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-10-05 Thread Steven Rostedt 
On Tue, 2009-10-06 at 07:20 +1100, Benjamin Herrenschmidt wrote:
 On Mon, 2009-10-05 at 09:25 -0400, Steven Rostedt wrote:
Sachin, can you give me more details on how you built that kernel ? (or
give them again in case I missed them the first time around :-), ie,
what toolchain, options, etc... or even better, give me remote access to
the build host ?
   
   Ok, got access and had a quick look... seems to be a toolchain problem
   to me. I'll investigate more tomorrow.
  
  Hi Ben,
  
  Any more word on this issue? 
 
 Didn't you fix it using a TOC access ?
 
 Unless I'm confusing things, I think the problem is the usage
 of LOAD_REG_IMMEDIATE which generates relocs that we don't support
 when CONFIG_RELOCATABLE is set.
 
 I've merged a patch that post-processes the kernel now, to check
 for such relocs so at least you should be warned at build time.

I thought we had two issues. One was the use of the relocs that did
cause issues. But then there was still crashes reported after that.
IIRC. I'm still suffering jetlag, so my memory is not that fresh about
this.

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-14 Thread Benjamin Herrenschmidt 
On Mon, 2009-09-14 at 06:25 +1000, Benjamin Herrenschmidt wrote:
   .../...
   
Something is totally messed up here.
   
   Could it be that we don't handle R_PPC64_ADDR16_* relocs in
   arch/powerpc/kernel/modules/module_64.c ?
   
   Sachin, do you see a bunch of Unknown ADD relocation in your dmesg ?
  
  Ben,
  
  The thing is, this is kernel proper. This code is in entry_64.S not in
  the module code.
 
 Argh... indeed.
 
 Sachin, can you give me more details on how you built that kernel ? (or
 give them again in case I missed them the first time around :-), ie,
 what toolchain, options, etc... or even better, give me remote access to
 the build host ?

Ok, got access and had a quick look... seems to be a toolchain problem
to me. I'll investigate more tomorrow.

Cheers,
Ben.


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-13 Thread Steven Rostedt 
On Sun, 2009-09-13 at 14:37 +1000, Benjamin Herrenschmidt wrote:
 On Sun, 2009-09-13 at 00:07 -0400, Steven Rostedt wrote:
 982: R_PPC64_ADDR16_HIGHEST
  ftrace_return_to_handler
   984:   60 84 00 00 ori r4,r4,0
  986: R_PPC64_ADDR16_HIGHER
  ftrace_return_to_handler
   988:   78 84 07 c6 rldicr  r4,r4,32,31
   98c:   64 84 00 00 orisr4,r4,0
  98e: R_PPC64_ADDR16_HI
  ftrace_return_to_handler
   990:   60 84 00 00 ori r4,r4,0
  992: R_PPC64_ADDR16_LO
  ftrace_return_to_handler
   994:   e8 44 00 08 ld  r2,8(r4)
   998:   48 00 00 01 bl  998 .mod_return_to_handler+0x30
  998:
  R_PPC64_REL24  .ftrace_return_to_handler
   99c:   60 00 00 00 nop
   9a0:   7c 68 03 a6 mtlrr3
 
 .../...
 
  Something is totally messed up here.
 
 Could it be that we don't handle R_PPC64_ADDR16_* relocs in
 arch/powerpc/kernel/modules/module_64.c ?
 
 Sachin, do you see a bunch of Unknown ADD relocation in your dmesg ?

Ben,

The thing is, this is kernel proper. This code is in entry_64.S not in
the module code.

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-13 Thread Benjamin Herrenschmidt 

  .../...
  
   Something is totally messed up here.
  
  Could it be that we don't handle R_PPC64_ADDR16_* relocs in
  arch/powerpc/kernel/modules/module_64.c ?
  
  Sachin, do you see a bunch of Unknown ADD relocation in your dmesg ?
 
 Ben,
 
 The thing is, this is kernel proper. This code is in entry_64.S not in
 the module code.

Argh... indeed.

Sachin, can you give me more details on how you built that kernel ? (or
give them again in case I missed them the first time around :-), ie,
what toolchain, options, etc... or even better, give me remote access to
the build host ?

Cheers,
Ben.

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-12 Thread Steven Rostedt 
On Wed, 2009-09-09 at 11:57 +0530, Sachin Sant wrote:
 Steven Rostedt wrote:
  I'm going through old email, and I found this. Do you still see this
  error. I don't recall seeing it myself.

 I can still recreate this with 31-rc9. When i enable tracing
 with function_graph i notice the following oops. This happens
 only once. Later if i try to enable/disable tracing i don't
 get this oops message. This behavior is observed only with
 function_graph. Other tracers work fine.
 
 Oops: Kernel access of bad area, sig: 11 [#1]
 SMP NR_CPUS=1024 NUMA pSeries
 Modules linked in: ipv6 fuse loop dm_mod sr_mod ehea ibmveth sg cdrom sd_mod 
 crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
 NIP: c0008f30 LR: c0008f04 CTR: 800f6d68
 REGS: c0003e98f560 TRAP: 0300   Not tainted  (2.6.31-rc9)
 MSR: 80009032 EE,ME,IR,DR  CR: 24000422  XER: 0020
 DAR: 0008, DSISR: 4000
 TASK = c0003e953b20[2483] 'irqbalance' THREAD: c0003e98c000 CPU: 1
 GPR00: c0008f04 c0003e98f7e0 d117ed38 
 GPR04:  6600 10bf 
 GPR08:  80010021bb40 00ff 80010021bb60
 GPR12: 0002 c1032800  effdff68
 GPR16: 0fffa39fd6a0 0fffa39e6c38 c0003ebe9c38 f000
 GPR20: c0002a6cf980 c0003e98fdf8 c0003e98fba8 0fffa174
 GPR24: f000 80010300 ffe0 0009
 GPR28: c0003db4 0002 d117da78 c0003e98f850
 NIP [c0008f30] .mod_return_to_handler+0x2c/0x64
 LR [c0008f04] .mod_return_to_handler+0x0/0x64
 Call Trace:
 [c0003e98f7e0] [c0002a6cf980] 0xc0002a6cf980 (unreliable)
 [c0003e98f850] [c0008f04] .mod_return_to_handler+0x0/0x64
 [c0003e98f900] [c0008f04] .mod_return_to_handler+0x0/0x64
 [c0003e98f9a0] [c0008f04] .mod_return_to_handler+0x0/0x64
 [c0003e98fa30] [c0008ed0] .return_to_handler+0x0/0x34 
 (.bad_page_fault+0xc8/0xe8)
 [c0003e98fb30] [c0008ed0] .return_to_handler+0x0/0x34 
 (handle_page_fault+0x3c/0x5c)
 [c0003e98fc20] [c0008ed0] .return_to_handler+0x0/0x34 
 (.ehea_h_query_ehea_port+0x74/0x9c [ehea])
 [c0003e98fcd0] [c0008ed0] .return_to_handler+0x0/0x34 
 (.ehea_get_stats+0xa0/0x1d0 [ehea])
 [c0003e98fd80] [c0008ed0] .return_to_handler+0x0/0x34 
 (.dev_get_stats+0x50/0xec)
 [c0003e98fe30] [c0008ed0] .return_to_handler+0x0/0x34 
 (.dev_seq_show+0x5c/0x140)
 Instruction dump:
 4e800020 f881ffe0 f861ffe8 f841fff0 fbe1fff8 7c3f0b78 f821ff91 3c80
 6084 788407c6 6484 6084 e8440008 48126375 6000 7c6803a6
 ---[ end trace bb43efc994aed790 ]---

I'm looking at your back dump and this really bothers me. I did a
objdump -dr arch/powerpc/kernel/entry_64.o and this is what I have:

0968 .mod_return_to_handler:
 968:   f8 81 ff e0 std r4,-32(r1)
 96c:   f8 61 ff e8 std r3,-24(r1)
 970:   f8 41 ff f0 std r2,-16(r1)
 974:   fb e1 ff f8 std r31,-8(r1)
 978:   7c 3f 0b 78 mr  r31,r1
 97c:   f8 21 ff 91 stdur1,-112(r1)
 980:   3c 80 00 00 lis r4,0
982: R_PPC64_ADDR16_HIGHEST ftrace_return_to_handler
 984:   60 84 00 00 ori r4,r4,0
986: R_PPC64_ADDR16_HIGHER  ftrace_return_to_handler
 988:   78 84 07 c6 rldicr  r4,r4,32,31
 98c:   64 84 00 00 orisr4,r4,0
98e: R_PPC64_ADDR16_HI  ftrace_return_to_handler
 990:   60 84 00 00 ori r4,r4,0
992: R_PPC64_ADDR16_LO  ftrace_return_to_handler
 994:   e8 44 00 08 ld  r2,8(r4)
 998:   48 00 00 01 bl  998 .mod_return_to_handler+0x30
998: R_PPC64_REL24  .ftrace_return_to_handler
 99c:   60 00 00 00 nop
 9a0:   7c 68 03 a6 mtlrr3


The bug happened at mod_return_to_handler+0x2c which is 994 above. Your
reg dump shows r4 is 0, and worse yet, looking at the code:

4e800020 f881ffe0 f861ffe8 f841fff0 fbe1fff8 7c3f0b78 f821ff91 3c80
6084 788407c6 6484 6084 e8440008 48126375 6000
7c6803a6

The 6484 6084 shows that the linker never resolved the address
to ftrace_return_to_handle??

Something is totally messed up here.

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-12 Thread Benjamin Herrenschmidt 
On Sun, 2009-09-13 at 00:07 -0400, Steven Rostedt wrote:
982: R_PPC64_ADDR16_HIGHEST
 ftrace_return_to_handler
  984:   60 84 00 00 ori r4,r4,0
 986: R_PPC64_ADDR16_HIGHER
 ftrace_return_to_handler
  988:   78 84 07 c6 rldicr  r4,r4,32,31
  98c:   64 84 00 00 orisr4,r4,0
 98e: R_PPC64_ADDR16_HI
 ftrace_return_to_handler
  990:   60 84 00 00 ori r4,r4,0
 992: R_PPC64_ADDR16_LO
 ftrace_return_to_handler
  994:   e8 44 00 08 ld  r2,8(r4)
  998:   48 00 00 01 bl  998 .mod_return_to_handler+0x30
 998:
 R_PPC64_REL24  .ftrace_return_to_handler
  99c:   60 00 00 00 nop
  9a0:   7c 68 03 a6 mtlrr3

.../...

 Something is totally messed up here.

Could it be that we don't handle R_PPC64_ADDR16_* relocs in
arch/powerpc/kernel/modules/module_64.c ?

Sachin, do you see a bunch of Unknown ADD relocation in your dmesg ?

Ben.


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-11 Thread Steven Rostedt 
On Thu, 2009-09-10 at 11:02 +0530, Sachin Sant wrote:
 Steven Rostedt wrote:
  Ah, seems the bug happens to be in the module handling. Does the call
  back always have .mod_return_to_handler?

 Yes. Every time it ends up in .mod_return_to_handler

Hmm, I still can not reproduce it, and I've confirmed that I
hit .mod_return_to_handler too.

Could you apply the below patch. This wont fix anything, but it will at
least make the trace back show the real functions that were called.

Thanks,

-- Steve

diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 892a9f2..a2e1b15 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1014,9 +1014,13 @@ void show_stack(struct task_struct *tsk, unsigned long 
*stack)
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
int curr_frame = current-curr_ret_stack;
extern void return_to_handler(void);
-   unsigned long addr = (unsigned long)return_to_handler;
+   unsigned long rth = (unsigned long)return_to_handler;
+   unsigned long mrth = -1;
 #ifdef CONFIG_PPC64
-   addr = *(unsigned long*)addr;
+   extern void mod_return_to_handler(void);
+   rth = *(unsigned long *)rth;
+   mrth = (unsigned long)mod_return_to_handler;
+   mrth = *(unsigned long *)mrth;
 #endif
 #endif
 
@@ -1042,7 +1046,7 @@ void show_stack(struct task_struct *tsk, unsigned long 
*stack)
if (!firstframe || ip != lr) {
printk([REG] [REG] %pS, sp, ip, (void *)ip);
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
-   if (ip == addr  curr_frame = 0) {
+   if ((ip == rth || ip == mrth)  curr_frame = 0) {
printk( (%pS),
   (void 
*)current-ret_stack[curr_frame].ret);
curr_frame--;


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-10 Thread Steven Rostedt 
On Thu, 2009-09-10 at 11:02 +0530, Sachin Sant wrote:
 Steven Rostedt wrote:
  Ah, seems the bug happens to be in the module handling. Does the call
  back always have .mod_return_to_handler?

 Yes. Every time it ends up in .mod_return_to_handler

BTW, do you have CONFIG_IRQSTACK set?

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-10 Thread Sachin Sant 

Steven Rostedt wrote:

On Thu, 2009-09-10 at 11:02 +0530, Sachin Sant wrote:
  

Steven Rostedt wrote:


Ah, seems the bug happens to be in the module handling. Does the call
back always have .mod_return_to_handler?
  
  

Yes. Every time it ends up in .mod_return_to_handler



BTW, do you have CONFIG_IRQSTACK set?
  

Yes. That option is enabled.

CONFIG_DEBUGGER=y
CONFIG_IRQSTACKS=y
# CONFIG_VIRQ_DEBUG is not set

thanks
-Sachin

--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-09 Thread Sachin Sant 

Steven Rostedt wrote:

I'm going through old email, and I found this. Do you still see this
error. I don't recall seeing it myself.
  

I can still recreate this with 31-rc9. When i enable tracing
with function_graph i notice the following oops. This happens
only once. Later if i try to enable/disable tracing i don't
get this oops message. This behavior is observed only with
function_graph. Other tracers work fine.

Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=1024 NUMA pSeries
Modules linked in: ipv6 fuse loop dm_mod sr_mod ehea ibmveth sg cdrom sd_mod 
crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
NIP: c0008f30 LR: c0008f04 CTR: 800f6d68
REGS: c0003e98f560 TRAP: 0300   Not tainted  (2.6.31-rc9)
MSR: 80009032 EE,ME,IR,DR  CR: 24000422  XER: 0020
DAR: 0008, DSISR: 4000
TASK = c0003e953b20[2483] 'irqbalance' THREAD: c0003e98c000 CPU: 1
GPR00: c0008f04 c0003e98f7e0 d117ed38 
GPR04:  6600 10bf 
GPR08:  80010021bb40 00ff 80010021bb60
GPR12: 0002 c1032800  effdff68
GPR16: 0fffa39fd6a0 0fffa39e6c38 c0003ebe9c38 f000
GPR20: c0002a6cf980 c0003e98fdf8 c0003e98fba8 0fffa174
GPR24: f000 80010300 ffe0 0009
GPR28: c0003db4 0002 d117da78 c0003e98f850
NIP [c0008f30] .mod_return_to_handler+0x2c/0x64
LR [c0008f04] .mod_return_to_handler+0x0/0x64
Call Trace:
[c0003e98f7e0] [c0002a6cf980] 0xc0002a6cf980 (unreliable)
[c0003e98f850] [c0008f04] .mod_return_to_handler+0x0/0x64
[c0003e98f900] [c0008f04] .mod_return_to_handler+0x0/0x64
[c0003e98f9a0] [c0008f04] .mod_return_to_handler+0x0/0x64
[c0003e98fa30] [c0008ed0] .return_to_handler+0x0/0x34 
(.bad_page_fault+0xc8/0xe8)
[c0003e98fb30] [c0008ed0] .return_to_handler+0x0/0x34 
(handle_page_fault+0x3c/0x5c)
[c0003e98fc20] [c0008ed0] .return_to_handler+0x0/0x34 
(.ehea_h_query_ehea_port+0x74/0x9c [ehea])
[c0003e98fcd0] [c0008ed0] .return_to_handler+0x0/0x34 
(.ehea_get_stats+0xa0/0x1d0 [ehea])
[c0003e98fd80] [c0008ed0] .return_to_handler+0x0/0x34 
(.dev_get_stats+0x50/0xec)
[c0003e98fe30] [c0008ed0] .return_to_handler+0x0/0x34 
(.dev_seq_show+0x5c/0x140)
Instruction dump:
4e800020 f881ffe0 f861ffe8 f841fff0 fbe1fff8 7c3f0b78 f821ff91 3c80
6084 788407c6 6484 6084 e8440008 48126375 6000 7c6803a6
---[ end trace bb43efc994aed790 ]---

function_graph traces are recorded and can be retrieved using
/sys/kernel/debug/tracing/trace.

1)   3.936 us|}
1)   |.release_console_sem() {
1)   0.594 us|  ._spin_lock_irqsave();
1)   0.560 us|  ._call_console_drivers();
1)   0.580 us|  ._call_console_drivers();
1)   0.582 us|  ._spin_lock_irqsave();
1)   |  .up() {
1)   0.592 us|._spin_lock_irqsave();
1)   0.556 us|._spin_unlock_irqrestore();
1)   2.842 us|  }
1)   0.588 us|  ._spin_unlock_irqrestore();
1)   9.750 us|}
1) + 75.274 us   |  }
1)   |  .die() {
1)   |.oops_enter() {

Thanks
-Sachin

--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-09 Thread Steven Rostedt 
On Wed, 2009-09-09 at 11:57 +0530, Sachin Sant wrote:
 Steven Rostedt wrote:
  I'm going through old email, and I found this. Do you still see this
  error. I don't recall seeing it myself.

 I can still recreate this with 31-rc9. When i enable tracing
 with function_graph i notice the following oops. This happens
 only once. Later if i try to enable/disable tracing i don't
 get this oops message. This behavior is observed only with
 function_graph. Other tracers work fine.
 
 Oops: Kernel access of bad area, sig: 11 [#1]
 SMP NR_CPUS=1024 NUMA pSeries
 Modules linked in: ipv6 fuse loop dm_mod sr_mod ehea ibmveth sg cdrom sd_mod 
 crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod
 NIP: c0008f30 LR: c0008f04 CTR: 800f6d68
 REGS: c0003e98f560 TRAP: 0300   Not tainted  (2.6.31-rc9)
 MSR: 80009032 EE,ME,IR,DR  CR: 24000422  XER: 0020
 DAR: 0008, DSISR: 4000
 TASK = c0003e953b20[2483] 'irqbalance' THREAD: c0003e98c000 CPU: 1
 GPR00: c0008f04 c0003e98f7e0 d117ed38 
 GPR04:  6600 10bf 
 GPR08:  80010021bb40 00ff 80010021bb60
 GPR12: 0002 c1032800  effdff68
 GPR16: 0fffa39fd6a0 0fffa39e6c38 c0003ebe9c38 f000
 GPR20: c0002a6cf980 c0003e98fdf8 c0003e98fba8 0fffa174
 GPR24: f000 80010300 ffe0 0009
 GPR28: c0003db4 0002 d117da78 c0003e98f850
 NIP [c0008f30] .mod_return_to_handler+0x2c/0x64
 LR [c0008f04] .mod_return_to_handler+0x0/0x64
 Call Trace:
 [c0003e98f7e0] [c0002a6cf980] 0xc0002a6cf980 (unreliable)
 [c0003e98f850] [c0008f04] .mod_return_to_handler+0x0/0x64
 [c0003e98f900] [c0008f04] .mod_return_to_handler+0x0/0x64
 [c0003e98f9a0] [c0008f04] .mod_return_to_handler+0x0/0x64

Ah, seems the bug happens to be in the module handling. Does the call
back always have .mod_return_to_handler?

This doesn't surprise me any. The module code is a bit harry, and
function graph does some crazy crap with it.

-- Steve

 [c0003e98fa30] [c0008ed0] .return_to_handler+0x0/0x34 
 (.bad_page_fault+0xc8/0xe8)
 [c0003e98fb30] [c0008ed0] .return_to_handler+0x0/0x34 
 (handle_page_fault+0x3c/0x5c)
 [c0003e98fc20] [c0008ed0] .return_to_handler+0x0/0x34 
 (.ehea_h_query_ehea_port+0x74/0x9c [ehea])
 [c0003e98fcd0] [c0008ed0] .return_to_handler+0x0/0x34 
 (.ehea_get_stats+0xa0/0x1d0 [ehea])
 [c0003e98fd80] [c0008ed0] .return_to_handler+0x0/0x34 
 (.dev_get_stats+0x50/0xec)
 [c0003e98fe30] [c0008ed0] .return_to_handler+0x0/0x34 
 (.dev_seq_show+0x5c/0x140)
 Instruction dump:
 4e800020 f881ffe0 f861ffe8 f841fff0 fbe1fff8 7c3f0b78 f821ff91 3c80
 6084 788407c6 6484 6084 e8440008 48126375 6000 7c6803a6
 ---[ end trace bb43efc994aed790 ]---
 
 function_graph traces are recorded and can be retrieved using
 /sys/kernel/debug/tracing/trace.
 
 1)   3.936 us|}
 1)   |.release_console_sem() {
 1)   0.594 us|  ._spin_lock_irqsave();
 1)   0.560 us|  ._call_console_drivers();
 1)   0.580 us|  ._call_console_drivers();
 1)   0.582 us|  ._spin_lock_irqsave();
 1)   |  .up() {
 1)   0.592 us|._spin_lock_irqsave();
 1)   0.556 us|._spin_unlock_irqrestore();
 1)   2.842 us|  }
 1)   0.588 us|  ._spin_unlock_irqrestore();
 1)   9.750 us|}
 1) + 75.274 us   |  }
 1)   |  .die() {
 1)   |.oops_enter() {
 
 Thanks
 -Sachin
 

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-09 Thread Sachin Sant 

Steven Rostedt wrote:

Ah, seems the bug happens to be in the module handling. Does the call
back always have .mod_return_to_handler?
  

Yes. Every time it ends up in .mod_return_to_handler

Thanks
-Sachin


This doesn't surprise me any. The module code is a bit harry, and
function graph does some crazy crap with it.

-- Steve
  



--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-09-08 Thread Steven Rostedt 
On Mon, 2009-08-03 at 16:10 +0530, Sachin Sant wrote:
 Steven Rostedt wrote:
  Thanks,
 
  I've seen issues with my PPC box and function graph, but the bugs were
  also caused by other changes. I'll boot up my PPC64 box and see if
  I see the same issues you have.

 Hi Steven,
 
 I can still recreate this issue with 2.6.31-rc5. Let me know
 if i can provide any information to find a solution for this.

Hi Sachin,

I'm going through old email, and I found this. Do you still see this
error. I don't recall seeing it myself.

Thanks,

-- Steve


___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-08-03 Thread Sachin Sant 

Steven Rostedt wrote:

Thanks,

I've seen issues with my PPC box and function graph, but the bugs were
also caused by other changes. I'll boot up my PPC64 box and see if
I see the same issues you have.
  

Hi Steven,

I can still recreate this issue with 2.6.31-rc5. Let me know
if i can provide any information to find a solution for this.

Thanks
-Sachin


--

-
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
-

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [FTRACE] Enabling function_graph causes OOPS

2009-07-14 Thread Steven Rostedt 

On Tue, 14 Jul 2009, Sachin Sant wrote:

 While enabling function_graph tracer on a Power6 box, machine
 crashed with following trace. Kernel version is 2.6.31-rc3.

Thanks,

I've seen issues with my PPC box and function graph, but the bugs were
also caused by other changes. I'll boot up my PPC64 box and see if
I see the same issues you have.

-- Steve

 
 :/debug/tracing # echo function_graph  current_tracer :/debug/tracing # cat
 current_tracer function_graph
 :/debug/tracing # echo 1  tracing_enabled
 
 cpu 0x0: Vector: 300 (Data Access) at [c0003eb86de0]
pc: c0008f30: .mod_return_to_handler+0x2c/0x64
lr: c0008f04: .mod_return_to_handler+0x0/0x64
sp: c0003eb87060
   msr: 80009032
   dar: 8
 dsisr: 4000
  current = 0xc0003e83c080
  paca= 0xc0ff2400
pid   = 2700, comm = sshd
 enter ? for help
 [c0003eb870d0] c0008f04 .mod_return_to_handler+0x0/0x64
 [c0003eb871a0] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87290] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87330] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb873e0] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87470] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87500] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87640] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87730] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87830] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb878d0] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87a00] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87b30] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87cd0] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87d80] c0008ed0 .return_to_handler+0x0/0x34
 [c0003eb87e30] c0008ed0 .return_to_handler+0x0/0x34
 --- Exception: c01 (System Call) at 0fffb640a8f8
 SP (fffe5b9b960) is in userspace
 0:mon di c0008f04
 c0008f04  f881ffe0  std r4,-32(r1)
 c0008f08  f861ffe8  std r3,-24(r1)
 c0008f0c  f841fff0  std r2,-16(r1)
 c0008f10  fbe1fff8  std r31,-8(r1)
 c0008f14  7c3f0b78  mr  r31,r1
 c0008f18  f821ff91  stdur1,-112(r1)
 c0008f1c  3c80  lis r4,0
 c0008f20  6084  ori r4,r4,0
 c0008f24  788407c6  rldicr  r4,r4,32,31
 c0008f28  6484  orisr4,r4,0
 c0008f2c  6084  ori r4,r4,0
 c0008f30  e8440008  ld  r2,8(r4)  ^ PC points to this
 ^^^
 c0008f34  48123431  bl  c012c364#
 .ftrace_return_to_handler+0x0/0x110
 c0008f38  6000  nop
 c0008f3c  7c6803a6  mtlrr3
 c0008f40  e821  ld  r1,0(r1)
 0:xmon r
 R00 = c0008f04   R16 = 0fffb741d6a0
 R01 = c0003ea4b7e0   R17 = 0fffb7406c38
 R02 = d10bec80   R18 = c0003de41838
 R03 =    R19 = f000
 R04 =    R20 = c0003cfc8c80
 R05 = 6600   R21 = c0003ea4bdf8
 R06 = 10bf   R22 = c0003ea4bba8
 R07 =    R23 = 0fff8eb6
 R08 =    R24 = f000
 R09 = 80010021c740   R25 = 80010300
 R10 = 00ff   R26 = ffe0
 R11 = 80010021c760   R27 = 0009
 R12 = 0002   R28 = c0006f85
 R13 = c0ff2400   R29 = 0002
 R14 =    R30 = d10bd9c8
 R15 = effdff68   R31 = c0003ea4b850
 pc  = c0008f30 .mod_return_to_handler+0x2c/0x64
 lr  = c0008f04 .mod_return_to_handler+0x0/0x64
 msr = 80009032   cr  = 24000442
 ctr = 800f6d68   xer = 0020   trap =  300
 dar = 0008   dsisr = 4000
 0:xmon
 
 Following tracers are supported by the kernel.
 :/debug/tracing # cat available_tracers function_graph function sched_switch
 nop
 
 Other tracers function and sched_switch work fine. Having problem
 only with function_graph.
 
 Have attached the .config.
 
 Thanks
 -Sachin
 
 
 -- 
 
 -
 Sachin Sant
 IBM Linux Technology Center
 India Systems and Technology Labs
 Bangalore, India
 -
 
 
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev