Re: Automated submission of kernel panic reports: sysutils/panicmail
from Mark Felder: Question that arises is how does the system know where to send the email, and through what SMTP server, especially if panicmail_autosubmit=YES. Every computer on the planet has the capability of being able to send email directly without an SMTP server. The only question is if the receiving end is willing to accept it, or discard it as spam. Mail server at the receiving end might reject the message, or one's ISP might block it. I use mail/mpop and mail/msmtp rather than messing with sendmail or postfix; have multiple email accounts and inboxes. Does it provide a compatible /usr/sbin/sendmail binary? If so, it will just work^TM. msmtp -a account-name -t message-with-headers from Colin Percival: Don't you get daily run output and security run output emails? I didn't think of these messages, all contained within the same computer. These messages are constructed like email, but don't go through mail servers. Tom ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
On Monday, November 04, 2013 4:29:27 pm Colin Percival wrote: On 11/04/13 04:49, Alfred Perlstein wrote: Colin, have you had a few minutes to check out the crash reporting facilities in FreeNAS? Yes. The reason I ask is that: 1) we would like to share code. 2) we have this running for a few months now and have a huge corpus of information. 3) we are building a nice UI (screenshots attached) over it, we have a couple of thousands of lines of code we can share for this. Once I have a useful number of panics collected, I was hoping to take the best pieces from FreeNAS's processing, from the SoC project, and from the processing I've been doing of automatic panic reports from EC2 instances. We send a minimal set of information: kernel stack trace, ddb buffer and hardware. Just enough to get some very, very handy stuff. I'm currently sending the dump header and what I get from kgdb 'bt'. If I find that I'm missing something important, I can always add it to a new version of the panicmail port. ;-) One of my previous employers maintained a database of panics and I added ways to recognize known panics and tag them. I ended up relying a lot on stack trace details from specific OS versions to mark a panic as an instance of a specific bug. Also, you may have very different stack traces even on the same build version for a single bug. In the case of my employer we had a constrained set of kernel configs and specific build versions to work with. It might be harder to correctly match panics in the wild what with patched trees and random kernel configs. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
On 11/05/13 09:27, John Baldwin wrote: One of my previous employers maintained a database of panics and I added ways to recognize known panics and tag them. I ended up relying a lot on stack trace details from specific OS versions to mark a panic as an instance of a specific bug. Also, you may have very different stack traces even on the same build version for a single bug. In the case of my employer we had a constrained set of kernel configs and specific build versions to work with. It might be harder to correctly match panics in the wild what with patched trees and random kernel configs. Right, I'm sure there will be panics I can't match up against anything else -- but this is fine. If I get enough panic reports, I can still get useful data out even if some of them aren't immediately usable. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Automated submission of kernel panic reports: sysutils/panicmail
Hi all, After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have now added sysutils/panicmail to the FreeBSD ports tree. If you install this and add panicmail_enable=YES to your /etc/rc.conf, a panic report will be generated and sent to root@ for you to review and submit (via email). You can skip the reviewing step and submit panics automatically by setting panicmail_autosubmit=YES. The panics submitted are encrypted to an RSA key which I hold in order to keep them secure in transit; and I intend to keep the raw panic reports confidential except to the minimum extent necessary for other developers to help me process the incoming reports. If I receive enough panic reports to be useful, I hope to provide developers with aggregate statistics. This may include: * regular email reports listing the top panics, to help guide developers towards the most fertile areas for stability improvements; * email to specific developers alerting them to recurring panics in code they maintain (especially if it becomes clear that the panic has been recently introduced); and * guidance to re@ and secteam@ about how often a particular panic occurs if an errata notice is being considered as well as other yet-to-be-imagined reports of a similarly aggregate and anonymized nature. So please install the sysutils/panicmail port and enable it in rc.conf! This all depends on getting useful data, and I can't do that without your help. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
Hi, On 4 Nov 2013, at 10:41, Colin Percival wrote: Hi all, After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have now added sysutils/panicmail to the FreeBSD ports tree. [etc] Nice. Is this applicable to all supported branches? -- Bob Bishop r...@gid.co.uk ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
On 11/04/13 02:47, Bob Bishop wrote: On 4 Nov 2013, at 10:41, Colin Percival wrote: After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have now added sysutils/panicmail to the FreeBSD ports tree. [etc] Nice. Is this applicable to all supported branches? Yes... the code should work all the way back to 5.0 (it's an rc.d script), although I doubt ports infrastructure will allow you to install anything from today's ports tree on a system running FreeBSD 5.0. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
On 11/4/13, 2:41 AM, Colin Percival wrote: Hi all, After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have now added sysutils/panicmail to the FreeBSD ports tree. If you install this and add panicmail_enable=YES to your /etc/rc.conf, a panic report will be generated and sent to root@ for you to review and submit (via email). You can skip the reviewing step and submit panics automatically by setting panicmail_autosubmit=YES. The panics submitted are encrypted to an RSA key which I hold in order to keep them secure in transit; and I intend to keep the raw panic reports confidential except to the minimum extent necessary for other developers to help me process the incoming reports. If I receive enough panic reports to be useful, I hope to provide developers with aggregate statistics. This may include: * regular email reports listing the top panics, to help guide developers towards the most fertile areas for stability improvements; * email to specific developers alerting them to recurring panics in code they maintain (especially if it becomes clear that the panic has been recently introduced); and * guidance to re@ and secteam@ about how often a particular panic occurs if an errata notice is being considered as well as other yet-to-be-imagined reports of a similarly aggregate and anonymized nature. So please install the sysutils/panicmail port and enable it in rc.conf! This all depends on getting useful data, and I can't do that without your help. Colin, have you had a few minutes to check out the crash reporting facilities in FreeNAS? The reason I ask is that: 1) we would like to share code. 2) we have this running for a few months now and have a huge corpus of information. 3) we are building a nice UI (screenshots attached) over it, we have a couple of thousands of lines of code we can share for this. Our scripts can be found here: 1) A startup script that sends us the crashes on system start: https://github.com/freenas/freenas/blob/master/nanobsd/Files/etc/rc.d/ix_textdump 2) A script to submit data at boot OR from command line that sends more comprehensive system information ixdiagnose: https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/ixdiagnose 3) A very simple script to upload that report: https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/crashuploader We send a minimal set of information: kernel stack trace, ddb buffer and hardware. Just enough to get some very, very handy stuff. I can share with you offline the crash server code, it's django and relatively straight forward. The screenshots can also be seen at: http://people.freebsd.org/~alfred/crashreporter/ We could modify our framework for FreeBSD to do so by checking for a sentinel file depending on the host type and only auto-sending if we see that. -Alfred ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
Colin Percival wrote, On 11/04/2013 11:41: After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have now added sysutils/panicmail to the FreeBSD ports tree. The pkesh script is probably still in need of a big review (S00N(TM)...). ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
On 11/04/13 10:49, d...@gmx.com wrote: Colin Percival wrote, On 11/04/2013 11:41: After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have now added sysutils/panicmail to the FreeBSD ports tree. The pkesh script is probably still in need of a big review (S00N(TM)...). Go for it! It's a very simple script. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
On 11/04/13 04:49, Alfred Perlstein wrote: Colin, have you had a few minutes to check out the crash reporting facilities in FreeNAS? Yes. The reason I ask is that: 1) we would like to share code. 2) we have this running for a few months now and have a huge corpus of information. 3) we are building a nice UI (screenshots attached) over it, we have a couple of thousands of lines of code we can share for this. Once I have a useful number of panics collected, I was hoping to take the best pieces from FreeNAS's processing, from the SoC project, and from the processing I've been doing of automatic panic reports from EC2 instances. We send a minimal set of information: kernel stack trace, ddb buffer and hardware. Just enough to get some very, very handy stuff. I'm currently sending the dump header and what I get from kgdb 'bt'. If I find that I'm missing something important, I can always add it to a new version of the panicmail port. ;-) I can share with you offline the crash server code, it's django and relatively straight forward. I'll come back to you about this once I have some data. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
Hi all, After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have now added sysutils/panicmail to the FreeBSD ports tree. If you install this and add panicmail_enable=YES to your /etc/rc.conf, a panic report will be generated and sent to root@ for you to review and submit (via email). You can skip the reviewing step and submit panics automatically by setting panicmail_autosubmit=YES. The panics submitted are encrypted to an RSA key which I hold in order to keep them secure in transit; and I intend to keep the raw panic reports confidential except to the minimum extent necessary for other developers to help me process the incoming reports. If I receive enough panic reports to be useful, I hope to provide developers with aggregate statistics. This may include: * regular email reports listing the top panics, to help guide developers towards the most fertile areas for stability improvements; * email to specific developers alerting them to recurring panics in code they maintain (especially if it becomes clear that the panic has been recently introduced); and * guidance to re@ and secteam@ about how often a particular panic occurs if an errata notice is being considered as well as other yet-to-be-imagined reports of a similarly aggregate and anonymized nature. So please install the sysutils/panicmail port and enable it in rc.conf! This all depends on getting useful data, and I can't do that without your help. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid Question that arises is how does the system know where to send the email, and through what SMTP server, especially if panicmail_autosubmit=YES. In the case of a kernel panic, wouldn't the system crash/freeze, and would it then be able to compose an email message? I use mail/mpop and mail/msmtp rather than messing with sendmail or postfix; have multiple email accounts and inboxes. Now come to think of it, I don't think I ever sent an email from FreeBSD as root, only as nonroot. Something like panicmail ought to be ported to NetBSD pkgsrc, considering that NetBSD seems so much more unstable and crash-prone than FreeBSD on my hardware. Tom ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
On Mon, Nov 4, 2013, at 20:26, Thomas Mueller wrote: Hi all, After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have now added sysutils/panicmail to the FreeBSD ports tree. If you install this and add panicmail_enable=YES to your /etc/rc.conf, a panic report will be generated and sent to root@ for you to review and submit (via email). You can skip the reviewing step and submit panics automatically by setting panicmail_autosubmit=YES. The panics submitted are encrypted to an RSA key which I hold in order to keep them secure in transit; and I intend to keep the raw panic reports confidential except to the minimum extent necessary for other developers to help me process the incoming reports. If I receive enough panic reports to be useful, I hope to provide developers with aggregate statistics. This may include: * regular email reports listing the top panics, to help guide developers towards the most fertile areas for stability improvements; * email to specific developers alerting them to recurring panics in code they maintain (especially if it becomes clear that the panic has been recently introduced); and * guidance to re@ and secteam@ about how often a particular panic occurs if an errata notice is being considered as well as other yet-to-be-imagined reports of a similarly aggregate and anonymized nature. So please install the sysutils/panicmail port and enable it in rc.conf! This all depends on getting useful data, and I can't do that without your help. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid Question that arises is how does the system know where to send the email, and through what SMTP server, especially if panicmail_autosubmit=YES. Every computer on the planet has the capability of being able to send email directly without an SMTP server. The only question is if the receiving end is willing to accept it, or discard it as spam. In the case of a kernel panic, wouldn't the system crash/freeze, and would it then be able to compose an email message? This is all handled on the next boot after the panic. I use mail/mpop and mail/msmtp rather than messing with sendmail or postfix; have multiple email accounts and inboxes. Does it provide a compatible /usr/sbin/sendmail binary? If so, it will just work^TM. Now come to think of it, I don't think I ever sent an email from FreeBSD as root, only as nonroot. Something like panicmail ought to be ported to NetBSD pkgsrc, considering that NetBSD seems so much more unstable and crash-prone than FreeBSD on my hardware. I hope more projects pick this up too. :-) ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Automated submission of kernel panic reports: sysutils/panicmail
On 11/04/13 18:26, Thomas Mueller wrote: Question that arises is how does the system know where to send the email, and through what SMTP server, especially if panicmail_autosubmit=YES. The code assumes that your system knows how to deliver email. An out-of-the-box FreeBSD install has sendmail and can do this. If you don't enable panicmail_autosubmit then it also assumes you're reading or forwarding root's email -- which you should be doing anyway. In the case of a kernel panic, wouldn't the system crash/freeze, and would it then be able to compose an email message? The email is generated from the crashdump when the system next boots. I use mail/mpop and mail/msmtp rather than messing with sendmail or postfix; have multiple email accounts and inboxes. Now come to think of it, I don't think I ever sent an email from FreeBSD as root, only as nonroot. Don't you get daily run output and security run output emails? Something like panicmail ought to be ported to NetBSD pkgsrc, considering that NetBSD seems so much more unstable and crash-prone than FreeBSD on my hardware. Go right ahead. It's a small shell script -- might even work fine without any changes. It's BSD licensed, of course. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
