Re: Kernel crashing and log buffers...
OK - so after a bit more digging into things (and a poke by Mike) - I think most of the stuff already exists. Normal run time issues: -- - MTD_OOPS - on 2007-06-28 this was added to Linus's tree: http://lkml.org/lkml/2007/6/18/234 CONFIG_MTD_OOPS tristate Log panic/oops to an MTD buffer depends on MTD help This enables panic and oops messages to be logged to a circular buffer in a flash partition where it can be read back at some later point. To use, add console=ttyMTDx to the kernel command line, where x is the MTD device number to use. Note - this currently _only_ does things when oops_in_progress is set, but should be easy to extend to all - if that what people want. - mtd devices can be backed by RAM. CONFIG_MTD_RAM - mtd devices can be backed by flash (if you want things to be persistant across power cycles)... early boot issues: -- - CONFIG_EARLY_PRINTK - early printk just defined a console - and is supported by: alpha, blackfin, microblaze, mips, powerpc, sh, x86 - it's pretty trivial to support a memory based buffer - some archs already support it. I think this only leaves Wolfgang's desire for memory buffers from the bootloader to get (somehow) into the kernel's log buffer for syslog processing... Anyone else agree? -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote: OK - so after a bit more digging into things (and a poke by Mike) - I think most of the stuff already exists. Normal run time issues: -- - MTD_OOPS - on 2007-06-28 this was added to Linus's tree: http://lkml.org/lkml/2007/6/18/234 CONFIG_MTD_OOPS tristate Log panic/oops to an MTD buffer ... early boot issues: -- - CONFIG_EARLY_PRINTK - early printk just defined a console - and is supported by: alpha, blackfin, microblaze, mips, powerpc, sh, x86 - it's pretty trivial to support a memory based buffer - some archs already support it. I think this only leaves Wolfgang's desire for memory buffers from the bootloader to get (somehow) into the kernel's log buffer for syslog processing... Anyone else agree? Almost. A couple of us also want memory for flight data record FDR data for doing continuous logging. This would, ideally, be either uncached or cached in such a way that data is guaranteed to be written to memory in the event of a watchdog timer-induced system reset. David VL -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
On Fri, Jun 26, 2009 at 13:42, David VomLehn wrote: On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote: OK - so after a bit more digging into things (and a poke by Mike) - I think most of the stuff already exists. Normal run time issues: -- - MTD_OOPS - on 2007-06-28 this was added to Linus's tree: http://lkml.org/lkml/2007/6/18/234 CONFIG_MTD_OOPS tristate Log panic/oops to an MTD buffer ... early boot issues: -- - CONFIG_EARLY_PRINTK - early printk just defined a console - and is supported by: alpha, blackfin, microblaze, mips, powerpc, sh, x86 - it's pretty trivial to support a memory based buffer - some archs already support it. I think this only leaves Wolfgang's desire for memory buffers from the bootloader to get (somehow) into the kernel's log buffer for syslog processing... Anyone else agree? Almost. A couple of us also want memory for flight data record FDR data for doing continuous logging. This would, ideally, be either uncached or cached in such a way that data is guaranteed to be written to memory in the event of a watchdog timer-induced system reset. can none of the existing mtd devices provide uncached ram ? otherwise, create a mtd map that simply calls data flush after every write. then the existing mtd stack can provide what you want. -mike -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
On Fri 26 Jun 2009 13:42, David VomLehn pondered: On Fri, Jun 26, 2009 at 10:39:50AM -0400, Robin Getz wrote: OK - so after a bit more digging into things (and a poke by Mike) - I think most of the stuff already exists. Normal run time issues: -- - MTD_OOPS - on 2007-06-28 this was added to Linus's tree: http://lkml.org/lkml/2007/6/18/234 CONFIG_MTD_OOPS tristate Log panic/oops to an MTD buffer ... early boot issues: -- - CONFIG_EARLY_PRINTK - early printk just defined a console - and is supported by: alpha, blackfin, microblaze, mips, powerpc, sh, x86 - it's pretty trivial to support a memory based buffer - some archs already support it. I think this only leaves Wolfgang's desire for memory buffers from the bootloader to get (somehow) into the kernel's log buffer for syslog processing... Anyone else agree? Almost. A couple of us also want memory for flight data record FDR data for doing continuous logging. This would, ideally, be either uncached or cached in such a way that data is guaranteed to be written to memory in the event of a watchdog timer-induced system reset. Many of the early_printk implementations accept a keep option (which does not remove the console/buffer - and keeps it around forever) - I'm adding that to the Blackfin implementation as we speak - but it is already there on sh. I expect that we could add something to the MTD_OOPS, as to allways write as well (console=ttyMTDx,log_level) or something like that if the backing needed to be flash. Would that satisfy things? -Robin -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
On Sat 13 Jun 2009 14:59, Wolfgang Denk pondered: Dear Russell King, In message 20090613102642.gb7...@flint.arm.linux.org.uk you wrote: The other way I've seen people read out crash messages is using a debugger to dump the kernel's log buffer directly. That seems to work as well as any other method, and has the advantage that it doesn't require any kernel modifications. This works well in the lab during hardware bringup or BSP development. But we are also interested in a solution that allows to get more or less automatic access to the log buffer content after a crash - when you have several ten thousand systems in the field, such a feature can save you a lot of money. in this case - is it true that providing the kernel crash log buffer to the next running kernel - is actually more important than just giving it to the bootloader? (Since the bootloader may not have the facilities to email/post/netcat it to a developer - like the next running kernel might?) In this case - the Bootloader being able to read the buffer, understand that there was a crash, and be able to pass the buffer (with the crash message) into the next running kernel - where that kernel can package it up and do anything it wants to it - would be a requirement? -Robin -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
Dear Russell King, In message 20090613102642.gb7...@flint.arm.linux.org.uk you wrote: The other way I've seen people read out crash messages is using a debugger to dump the kernel's log buffer directly. That seems to work as well as any other method, and has the advantage that it doesn't require any kernel modifications. This works well in the lab during hardware bringup or BSP development. But we are also interested in a solution that allows to get more or less automatic access to the log buffer content after a crash - when you have several ten thousand systems in the field, such a feature can save you a lot of money. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote: On 17 Oct 2007, after much discussion and debate, Mike added add two new functions for reading the kernel log buffer (from kernel space). http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0b15d04af3dd996035d8fa81fc849d049171f9c3 The intention was for them to be used by recovery/dump/debug code so the kernel log can be easily retrieved/parsed by the bootloader (or another kernel) in a crash scenario. I was going to push the arch specific recovery/dump/debug code that uses them upstream (yeah, it has been a little while - but anyway...) it was removed since then by Roel Kluin ... 21 Oct 2008: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=acff181d3574244e651913df77332e897b88bff4 Before I ask Andrew to add it back, I thought I would make sure it was still a useful function, and did everything everyone wanted - and wasn't deemed unnecessary by a feature/function that I wasn't aware of - like the next thing... I saw the patch Grant sent recently - Add Alternative Log Buffer Support for printk Messages (in Nov2008 to the embedded list, and Jan 2009 to lkml) - but I couldn't find any followups - and it doesn't seem to be in Linus's tree. http://thread.gmane.org/gmane.linux.kernel.embedded/1358/focus=1373 http://lkml.org/lkml/2009/1/21/250 I can see the desire on Wolfgang Grant's part - for not needing the copy from/to - (you never have to worry about crashing nicely - the kernel panics, but you still need to copy memory around - potentially causing all kinds of secondary issues - and masking the real reason the crash occurred). But for the majority of the case - the copy from/to would work much better than what we have in mainstream today... I would be interested in Paul and Russell - how have you solved this issue? (Or do your kernel's never crash? :) Our kernel does crash, so we have to do this. I had submitted a patch a while ago that tweaks emit_log_char, but this was bounced, reasonably, because it would be interferring with the great majority of people who are not interested in capturing log output. The suggestion was made that we do this with a console driver. I've recently got a version working, as a module, but I've only started testing of the version integrated into the kernel tree. Still, post early, post often, so I've appended a version of the patch here to see if this seems to be the right direction. Regardless whether my particular patch seems like the right way to do this, we do need to solve this problem for the embedded world. It is one of the few areas where we have needs not shared by the rest of the kernel community. -Robin CUT HERE Provide functions for capturing console output for storage. The primary user is likely to be embedded systems that don't have the storage for core dumps but do have a need to log kernel panic information for later evaluation. It offers two main areas of functionality: o It can maintain a circular log of console output so that kernel log messages written before panic() was called can be retrieved to be added to the failure log. o A function can be registered to store output from printk() in a persistent location, such as a reserved location in RAM. Then, printk() can be used either directly, to print state information, or indirectly, through standard functions like dump_stack() and show_regs(). During normal operation, we use the circular logging. When we crash, almost the first thing we do is to switch to storing output. This goes in a memory buffer that is preserved over reboots. We then write a detailed crash report using printk() and functions that use printk(). We retrieve the last n lines of the log before the crash and print it, so that gets captured in the log, too. Signed-off-by: David VomLehn dvoml...@cisco.com --- drivers/char/Kconfig | 14 +++ drivers/char/Makefile |2 + drivers/char/conslogger.c | 233 include/linux/conslogger.h | 52 ++ 4 files changed, 301 insertions(+), 0 deletions(-) diff --git a/drivers/char/Kconfig b/drivers/char/Kconfig index 735bbe2..882ee57 100644 --- a/drivers/char/Kconfig +++ b/drivers/char/Kconfig @@ -684,6 +684,20 @@ config HVCS which will also be compiled when this driver is built as a module. +config CONSLOGGER + tristate Pseudo-console for capturing console output + depends on PRINTK + default n + help + This contains a pseudo-console to record and divert kernel console + output, which is probably of most used to embedded systems. When + a system crashes, it can divert printk output for logging information + about the failure in some persistent
Re: Kernel crashing and log buffers...
On Thu, Jun 11, 2009 at 02:52:38PM -0400, Robin Getz wrote: On Thu 11 Jun 2009 13:53, David VomLehn pondered: On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote: ... Our kernel does crash, so we have to do this. I had submitted a patch a while ago that tweaks emit_log_char, but this was bounced, reasonably, because it would be interferring with the great majority of people who are not interested in capturing log output. The suggestion was made that we do this with a console driver. I've recently got a version working, as a module, but I've only started testing of the version integrated into the kernel tree. Still, post early, post often, so I've appended a version of the patch here to see if this seems to be the right direction. Yeah - it looks like n people have done this n ways. I think even Andrew M had mentioned that he had done something in a past embedded life. It ain't hard, but I think we need a common implementation to go into the kernel. I think what you have would do what I need it to do - I'm not so sure about what Wolfgang/Grant had in mind (- since they wanted also to dump buffers from the bootloader into the console for syslog processing)... This is definitely different what what Wolfgang/Grant had in mind. Regardless whether my particular patch seems like the right way to do this, we do need to solve this problem for the embedded world. It is one of the few areas where we have needs not shared by the rest of the kernel community. I'm not so sure - googling for things last night led me to this: http://www.faqs.org/patents/app/20090044051 EXTRACTING LOG AND TRACE BUFFERS IN THE EVENT OF SYSTEM CRASHES Hariprasad V. Nellitheertha - Linux Technology Center, India Software Labs, IBM India. So - I think as Paul stated in a previous conversation - the high availability needs of servers are similar to the high availability needs of embedded... This is also different as it involves continuous logging of system state rather than producing crash logs. The continuous logging is definitely of wider interest. I did some work a couple of years ago to see what it would take to specify the buffer for LTT FDR data, though this work was dropped as lower priority, and I never got back to it. -Robin David -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
On Thu, Jun 11, 2009 at 11:22:57AM -0700, Tim Bird wrote: David VomLehn wrote: Our kernel does crash, so we have to do this. I had submitted a patch a while ago that tweaks emit_log_char, but this was bounced, reasonably, because it would be interferring with the great majority of people who are not interested in capturing log output. The suggestion was made that we do this with a console driver. I've recently got a version working, as a module, but I've only started testing of the version integrated into the kernel tree. Still, post early, post often, so I've appended a version of the patch here to see if this seems to be the right direction. Regardless whether my particular patch seems like the right way to do this, we do need to solve this problem for the embedded world. It is one of the few areas where we have needs not shared by the rest of the kernel community. In general, I like this approach to solve this problem. I spend most of my time in the lab with prototype boards that use a serial console, so my terminal history has always been my crashdump repository. ;-) This approach it unobtrusive to the rest of the log buffer system. I never did like the patches that introduced other ways to hack into the log buffer. The printk/log buffer code path is already convoluted enough, and I didn't like the idea of adding additional hooks. this approach leverages an already existing hook. The one thing that is missing over some other approaches is that log output captured by this method will be filtered, just like all of the other consoles, by the printk priority. It might be useful to have the ability to specify the priority level for each console. So, you could have only KERN_CRIT and above going to your serial console, but configure the logging console to capture everything down to KERN_WARNING. This is a refinement, though, and I didn't want to make things overly complex without a good feeling that it was necessary. I don't have much comment on the code specifics, but I have one nit with the config help... ... + help + This contains a pseudo-console to record and divert kernel console + output, which is probably of most used to embedded systems. When should be most use (no 'd') Thanks for picking the nit! Tim Bird David VomLehn -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
On 6/11/09 11:52 AM, Robin Getz wrote: On Thu 11 Jun 2009 13:53, David VomLehn pondered: On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote: I would be interested in Paul and Russell - how have you solved this issue? (Or do your kernel's never crash? :) Our kernel does crash, so we have to do this. I had submitted a patch a while ago that tweaks emit_log_char, but this was bounced, reasonably, because it would be interferring with the great majority of people who are not interested in capturing log output. The suggestion was made that we do this with a console driver. I've recently got a version working, as a module, but I've only started testing of the version integrated into the kernel tree. Still, post early, post often, so I've appended a version of the patch here to see if this seems to be the right direction. Yeah - it looks like n people have done this n ways. I think even Andrew M had mentioned that he had done something in a past embedded life. I think what you have would do what I need it to do - I'm not so sure about what Wolfgang/Grant had in mind (- since they wanted also to dump buffers from the bootloader into the console for syslog processing)... Regrettably, I have not had the time to follow-up on this since my last commentary in March: http://lkml.org/lkml/2009/3/31/293 However, from what I can glean of the discussion, you are correct that this does not precisely cover what Wolfgang (originally) or, later, I was working towards in that neither boot loader logs nor round-trip log buffering (bootloader to kernel and back again) are covered. However, the value and what's being proposed here and what I and others proposed--at least in the embedded kingdom--seems high and the features and implementation needn't be mutually exclusive. Regards, Grant -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
On Thu, Jun 11, 2009 at 13:53, David VomLehn wrote: On Wed, Jun 10, 2009 at 09:26:40PM -0400, Robin Getz wrote: On 17 Oct 2007, after much discussion and debate, Mike added add two new functions for reading the kernel log buffer (from kernel space). http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=0b15d04af3dd996035d8fa81fc849d049171f9c3 The intention was for them to be used by recovery/dump/debug code so the kernel log can be easily retrieved/parsed by the bootloader (or another kernel) in a crash scenario. I was going to push the arch specific recovery/dump/debug code that uses them upstream (yeah, it has been a little while - but anyway...) it was removed since then by Roel Kluin ... 21 Oct 2008: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=acff181d3574244e651913df77332e897b88bff4 Before I ask Andrew to add it back, I thought I would make sure it was still a useful function, and did everything everyone wanted - and wasn't deemed unnecessary by a feature/function that I wasn't aware of - like the next thing... I saw the patch Grant sent recently - Add Alternative Log Buffer Support for printk Messages (in Nov2008 to the embedded list, and Jan 2009 to lkml) - but I couldn't find any followups - and it doesn't seem to be in Linus's tree. http://thread.gmane.org/gmane.linux.kernel.embedded/1358/focus=1373 http://lkml.org/lkml/2009/1/21/250 I can see the desire on Wolfgang Grant's part - for not needing the copy from/to - (you never have to worry about crashing nicely - the kernel panics, but you still need to copy memory around - potentially causing all kinds of secondary issues - and masking the real reason the crash occurred). But for the majority of the case - the copy from/to would work much better than what we have in mainstream today... I would be interested in Paul and Russell - how have you solved this issue? (Or do your kernel's never crash? :) Our kernel does crash, so we have to do this. I had submitted a patch a while ago that tweaks emit_log_char, but this was bounced, reasonably, because it would be interferring with the great majority of people who are not interested in capturing log output. The suggestion was made that we do this with a console driver. I've recently got a version working, as a module, but I've only started testing of the version integrated into the kernel tree. Still, post early, post often, so I've appended a version of the patch here to see if this seems to be the right direction. i guess there's two distinct crash types: early and late. your console driver seems like it should be able to handle the late variety ok (i.e. the kernel is up and running and at some point, it blows up). but it doesnt address the early crashes which is what we're currently looking for (i.e. kernel crash occurs before the serial console is up and running). i think extending this driver to include early printk support should address that ? -mike -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crashing and log buffers...
Dear David VomLehn, In message 20090612003302.ga23...@cuplxvomd02.corp.sa.net you wrote: However, the value and what's being proposed here and what I and others proposed--at least in the embedded kingdom--seems high and the features and implementation needn't be mutually exclusive. I agree. In this case, we have three different approaches to three different problems (bootloader buffers, continuous state logging, and in-kernel capture of logging). All are problems that need to be solved. Personally, I'd love to have the continuous state logging, though I don't need the bootloader buffer part. Question is: do we need three different solutions, or can we come up with one solution that covers all? Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de If it ain't broke, don't fix it. - Bert Lantz -- To unsubscribe from this list: send the line unsubscribe linux-embedded in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
