Re: [PATCH 0/4] coredump: core dump masking support v3
Kawai, Hidehiro wrote: Requirements are: (1) a user can change the core dump settings _anytime_ - sometimes want to dump anonymous shared memory segments and sometimes don't want to dump them I might not have been sufficiently clear about this in my previous e-mail. Currently, the Google coredumper does not have the feature that you asked about, but adding it would be trivial -- it just hadn't been needed, yet, as on-the-fly compression was good enough for most users. Answering your question, I don't see any reason why the new API would not be able to make changes at any time. (2) a user can change the core dump settings of _any processes_ (although permission checks are performed) - in a huge application which forks many processes, a user hopes that some processes dump anonymous shared memory segments and some processes don't dump them The Google coredumper is a library that needs to be linked into the application and needs to be called from appropriate signal handlers. As such, it is the application's responsibility what management API it wants to expose externally, and what tools it wants to provide for managing a group of processes. And reliability of the core dump feature is also important. We have generally had very good reliability with the Google coredumper. In some cases, it even works a little more reliably than the default in-kernel dumper (e.g. because we can control where to write the file, and whether it should be compressed on-the-fly; or because we can get multi-threaded coredumps even in situations where the particular combination of libc and kernel doesn't support this), and in other cases the in-kernel dumper works a little better (e.g. if an application got too corrupted to even run any signal handlers). Realistically, it just works. But we did have to make sure that we set up alternate stacks for signal processing, and that we made sure that these stacks have been dirtied in order to avoid problems with memory overcomitting. And all the software vendors don't necessarily apply google-coredumper. If the vendor doesn't apply it, the user will be bothered by huge core dumps or the buggy application which remains unfixed. So I believe that in kernel solution is still needed. I agree that the Google coredumper is only one possible solution to your problem. Depending on how your production environment looks like, it might help a lot, or it might be completely useless. If it is cheap for you to modify your applications, but expensive to upgrade your kernels, the Google coredumper is the way to go. Also, if you need the extra features, such as the ability to compress core files on-the-fly, or the ability to send corefiles to somewhere other than an on-disk file, you definitely should look at a user-space solution. On the other hand, if you can easily upgrade all your kernels, but you don't even have access to the source of your applications, then an in-kernel solution is much better. Markus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Hi, Markus Gutschke wrote: > Kawai, Hidehiro wrote: > >> This patch series is version 3 of the core dump masking feature, >> which provides a per-process flag not to dump anonymous shared >> memory segments. > > I just wanted to remind you that you need to be careful about dumping > the [vdso] segment no matter whether you omit other segments. I didn't > actually try running your patches, and if the kernel doesn't actually > consider this segment anonymous and shared, things might already work > fine as is. Thank you for your advice and sorry for not replying soon. Fortunately, the latest kernel uses VM_ALWAYSDUMP flag to always dump the vdso segment. My patchset doesn't change this behavior. So we don't need to worry about the vdso segment. > As an alternative to your kernel patch, you could achieve the same goal > in user space, by linking my coredumper > http://code.google.com/p/google-coredumper/ into your binaries and > setting up appropriate signal handlers. An equivalent patch for > selectively omitting memory regions would be trivial to add. As far as I can see, google-coredumper has more flexibility. Can google-coredumper satisfy the following requirements easily? Requirements are: (1) a user can change the core dump settings _anytime_ - sometimes want to dump anonymous shared memory segments and sometimes don't want to dump them (2) a user can change the core dump settings of _any processes_ (although permission checks are performed) - in a huge application which forks many processes, a user hopes that some processes dump anonymous shared memory segments and some processes don't dump them And reliability of the core dump feature is also important. > While this > does give you more flexibility, it of course has the drawback of > requiring you to change your applications, so there still is some > benefit in a kernelspace solution. And all the software vendors don't necessarily apply google-coredumper. If the vendor doesn't apply it, the user will be bothered by huge core dumps or the buggy application which remains unfixed. So I believe that in kernel solution is still needed. Thanks, -- Hidehiro Kawai Hitachi, Ltd., Systems Development Laboratory - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Hi, Markus Gutschke wrote: Kawai, Hidehiro wrote: This patch series is version 3 of the core dump masking feature, which provides a per-process flag not to dump anonymous shared memory segments. I just wanted to remind you that you need to be careful about dumping the [vdso] segment no matter whether you omit other segments. I didn't actually try running your patches, and if the kernel doesn't actually consider this segment anonymous and shared, things might already work fine as is. Thank you for your advice and sorry for not replying soon. Fortunately, the latest kernel uses VM_ALWAYSDUMP flag to always dump the vdso segment. My patchset doesn't change this behavior. So we don't need to worry about the vdso segment. As an alternative to your kernel patch, you could achieve the same goal in user space, by linking my coredumper http://code.google.com/p/google-coredumper/ into your binaries and setting up appropriate signal handlers. An equivalent patch for selectively omitting memory regions would be trivial to add. As far as I can see, google-coredumper has more flexibility. Can google-coredumper satisfy the following requirements easily? Requirements are: (1) a user can change the core dump settings _anytime_ - sometimes want to dump anonymous shared memory segments and sometimes don't want to dump them (2) a user can change the core dump settings of _any processes_ (although permission checks are performed) - in a huge application which forks many processes, a user hopes that some processes dump anonymous shared memory segments and some processes don't dump them And reliability of the core dump feature is also important. While this does give you more flexibility, it of course has the drawback of requiring you to change your applications, so there still is some benefit in a kernelspace solution. And all the software vendors don't necessarily apply google-coredumper. If the vendor doesn't apply it, the user will be bothered by huge core dumps or the buggy application which remains unfixed. So I believe that in kernel solution is still needed. Thanks, -- Hidehiro Kawai Hitachi, Ltd., Systems Development Laboratory - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Kawai, Hidehiro wrote: Requirements are: (1) a user can change the core dump settings _anytime_ - sometimes want to dump anonymous shared memory segments and sometimes don't want to dump them I might not have been sufficiently clear about this in my previous e-mail. Currently, the Google coredumper does not have the feature that you asked about, but adding it would be trivial -- it just hadn't been needed, yet, as on-the-fly compression was good enough for most users. Answering your question, I don't see any reason why the new API would not be able to make changes at any time. (2) a user can change the core dump settings of _any processes_ (although permission checks are performed) - in a huge application which forks many processes, a user hopes that some processes dump anonymous shared memory segments and some processes don't dump them The Google coredumper is a library that needs to be linked into the application and needs to be called from appropriate signal handlers. As such, it is the application's responsibility what management API it wants to expose externally, and what tools it wants to provide for managing a group of processes. And reliability of the core dump feature is also important. We have generally had very good reliability with the Google coredumper. In some cases, it even works a little more reliably than the default in-kernel dumper (e.g. because we can control where to write the file, and whether it should be compressed on-the-fly; or because we can get multi-threaded coredumps even in situations where the particular combination of libc and kernel doesn't support this), and in other cases the in-kernel dumper works a little better (e.g. if an application got too corrupted to even run any signal handlers). Realistically, it just works. But we did have to make sure that we set up alternate stacks for signal processing, and that we made sure that these stacks have been dirtied in order to avoid problems with memory overcomitting. And all the software vendors don't necessarily apply google-coredumper. If the vendor doesn't apply it, the user will be bothered by huge core dumps or the buggy application which remains unfixed. So I believe that in kernel solution is still needed. I agree that the Google coredumper is only one possible solution to your problem. Depending on how your production environment looks like, it might help a lot, or it might be completely useless. If it is cheap for you to modify your applications, but expensive to upgrade your kernels, the Google coredumper is the way to go. Also, if you need the extra features, such as the ability to compress core files on-the-fly, or the ability to send corefiles to somewhere other than an on-disk file, you definitely should look at a user-space solution. On the other hand, if you can easily upgrade all your kernels, but you don't even have access to the source of your applications, then an in-kernel solution is much better. Markus - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Pavel Machek <[EMAIL PROTECTED]> wrote: > By same argument, we should just give up the coredumping in kernel. It > is rarely tested, so someone will just get it wrong. Not so rare... Lots of people still use Evolution after all:-) However, to counter your point, I'll point out that there's just one main code path to do coredumping in the kernel (for ELF; there are other binfmts that do coredumping too), as opposed to lots of applications all trying to set up coredumping themselves. > Remember: we are having people with huge apps, and therefore huge > coredumps. They want to hack a kernel in ugly way to make their dumps > smaller. MMAP_NODUMP or MADV_NODUMP might be better. Let the apps mark out for themselves what they want. > ...but there's better solution for them, create (& debug) userland > coredumping library. No need to hack a kernel. Actually, a better way to do this may be similar the way I assume Windows to work. On fatal fault: start up a debugger and PT_ATTACH corpse to it. The debugger could then be something that'll just save the core. No need to make sure you link in the core dumping library which might not catch the event if it's bad enough as it's working from *inside* the program, and so is subject to being corrupted by beserk code. David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
> > While I am sure you could construct scenarios where this would happen, > > realistically the only one I have run into were stack overflows, and they > > can > > be handled by carefully setting up an alternate stack for signal handlers -- > > just make sure the entire stack is already dirtied before you run out of > > memory (or, turn of overcommitting). > > Duff SIGSEGV or SIGBUS signal handlers are just as realistic. All that takes > is for someone to make a programming error. Remember: error paths are the > least frequently tested. > > And any time you say "by carefully setting up" you can guarantee someone's > going to do it wrong. By same argument, we should just give up the coredumping in kernel. It is rarely tested, so someone will just get it wrong. Remember: we are having people with huge apps, and therefore huge coredumps. They want to hack a kernel in ugly way to make their dumps smaller. ...but there's better solution for them, create (& debug) userland coredumping library. No need to hack a kernel. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Markus Gutschke <[EMAIL PROTECTED]> wrote: > > How does it work when you can't actually get back to userspace to have > > userspace do the coredump? You still have to handle the userspace > > equivalents of double/triple faults. > > My experience shows that there are only very rare occurrences of situations > where you cannot get back into userspace to do the coredump, and the > coredumper tries very hard never to cause additional faults. So what? If they can occur, you have to handle them. > While I am sure you could construct scenarios where this would happen, > realistically the only one I have run into were stack overflows, and they can > be handled by carefully setting up an alternate stack for signal handlers -- > just make sure the entire stack is already dirtied before you run out of > memory (or, turn of overcommitting). Duff SIGSEGV or SIGBUS signal handlers are just as realistic. All that takes is for someone to make a programming error. Remember: error paths are the least frequently tested. And any time you say "by carefully setting up" you can guarantee someone's going to do it wrong. David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Markus Gutschke [EMAIL PROTECTED] wrote: How does it work when you can't actually get back to userspace to have userspace do the coredump? You still have to handle the userspace equivalents of double/triple faults. My experience shows that there are only very rare occurrences of situations where you cannot get back into userspace to do the coredump, and the coredumper tries very hard never to cause additional faults. So what? If they can occur, you have to handle them. While I am sure you could construct scenarios where this would happen, realistically the only one I have run into were stack overflows, and they can be handled by carefully setting up an alternate stack for signal handlers -- just make sure the entire stack is already dirtied before you run out of memory (or, turn of overcommitting). Duff SIGSEGV or SIGBUS signal handlers are just as realistic. All that takes is for someone to make a programming error. Remember: error paths are the least frequently tested. And any time you say by carefully setting up you can guarantee someone's going to do it wrong. David - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
While I am sure you could construct scenarios where this would happen, realistically the only one I have run into were stack overflows, and they can be handled by carefully setting up an alternate stack for signal handlers -- just make sure the entire stack is already dirtied before you run out of memory (or, turn of overcommitting). Duff SIGSEGV or SIGBUS signal handlers are just as realistic. All that takes is for someone to make a programming error. Remember: error paths are the least frequently tested. And any time you say by carefully setting up you can guarantee someone's going to do it wrong. By same argument, we should just give up the coredumping in kernel. It is rarely tested, so someone will just get it wrong. Remember: we are having people with huge apps, and therefore huge coredumps. They want to hack a kernel in ugly way to make their dumps smaller. ...but there's better solution for them, create ( debug) userland coredumping library. No need to hack a kernel. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Pavel Machek [EMAIL PROTECTED] wrote: By same argument, we should just give up the coredumping in kernel. It is rarely tested, so someone will just get it wrong. Not so rare... Lots of people still use Evolution after all:-) However, to counter your point, I'll point out that there's just one main code path to do coredumping in the kernel (for ELF; there are other binfmts that do coredumping too), as opposed to lots of applications all trying to set up coredumping themselves. Remember: we are having people with huge apps, and therefore huge coredumps. They want to hack a kernel in ugly way to make their dumps smaller. MMAP_NODUMP or MADV_NODUMP might be better. Let the apps mark out for themselves what they want. ...but there's better solution for them, create ( debug) userland coredumping library. No need to hack a kernel. Actually, a better way to do this may be similar the way I assume Windows to work. On fatal fault: start up a debugger and PT_ATTACH corpse to it. The debugger could then be something that'll just save the core. No need to make sure you link in the core dumping library which might not catch the event if it's bad enough as it's working from *inside* the program, and so is subject to being corrupted by beserk code. David - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
David Howells wrote: How does it work when you can't actually get back to userspace to have userspace do the coredump? You still have to handle the userspace equivalents of double/triple faults. My experience shows that there are only very rare occurrences of situations where you cannot get back into userspace to do the coredump, and the coredumper tries very hard never to cause additional faults. While I am sure you could construct scenarios where this would happen, realistically the only one I have run into were stack overflows, and they can be handled by carefully setting up an alternate stack for signal handlers -- just make sure the entire stack is already dirtied before you run out of memory (or, turn of overcommitting). Markus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
> Kawai, Hidehiro wrote: > >This patch series is version 3 of the core dump masking feature, > >which provides a per-process flag not to dump anonymous shared > >memory segments. > > I just wanted to remind you that you need to be careful about dumping > the [vdso] segment no matter whether you omit other segments. I didn't > actually try running your patches, and if the kernel doesn't actually > consider this segment anonymous and shared, things might already work > fine as is. > > In any case, you can check with "readelf -a", if the [vdso] segment is > there. And you will find that if you forget to dump it, "gdb" can no > longer give you stack traces on call chains that involve signal handlers. > > As an alternative to your kernel patch, you could achieve the same goal > in user space, by linking my coredumper > http://code.google.com/p/google-coredumper/ into your binaries and > setting up appropriate signal handlers. An equivalent patch for > selectively omitting memory regions would be trivial to add. While this > does give you more flexibility, it of course has the drawback of > requiring you to change your applications, so there still is some > benefit in a kernelspace solution. "We are too lazy to change 0.01% of apps that actually need it" is not good enough reason to push the feature into kernel, I'd say. -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Markus Gutschke <[EMAIL PROTECTED]> wrote: > As an alternative to your kernel patch, you could achieve the same goal in > user space, by linking my coredumper How does it work when you can't actually get back to userspace to have userspace do the coredump? You still have to handle the userspace equivalents of double/triple faults. David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Markus Gutschke [EMAIL PROTECTED] wrote: As an alternative to your kernel patch, you could achieve the same goal in user space, by linking my coredumper How does it work when you can't actually get back to userspace to have userspace do the coredump? You still have to handle the userspace equivalents of double/triple faults. David - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Kawai, Hidehiro wrote: This patch series is version 3 of the core dump masking feature, which provides a per-process flag not to dump anonymous shared memory segments. I just wanted to remind you that you need to be careful about dumping the [vdso] segment no matter whether you omit other segments. I didn't actually try running your patches, and if the kernel doesn't actually consider this segment anonymous and shared, things might already work fine as is. In any case, you can check with readelf -a, if the [vdso] segment is there. And you will find that if you forget to dump it, gdb can no longer give you stack traces on call chains that involve signal handlers. As an alternative to your kernel patch, you could achieve the same goal in user space, by linking my coredumper http://code.google.com/p/google-coredumper/ into your binaries and setting up appropriate signal handlers. An equivalent patch for selectively omitting memory regions would be trivial to add. While this does give you more flexibility, it of course has the drawback of requiring you to change your applications, so there still is some benefit in a kernelspace solution. We are too lazy to change 0.01% of apps that actually need it is not good enough reason to push the feature into kernel, I'd say. -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
David Howells wrote: How does it work when you can't actually get back to userspace to have userspace do the coredump? You still have to handle the userspace equivalents of double/triple faults. My experience shows that there are only very rare occurrences of situations where you cannot get back into userspace to do the coredump, and the coredumper tries very hard never to cause additional faults. While I am sure you could construct scenarios where this would happen, realistically the only one I have run into were stack overflows, and they can be handled by carefully setting up an alternate stack for signal handlers -- just make sure the entire stack is already dirtied before you run out of memory (or, turn of overcommitting). Markus - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Kawai, Hidehiro wrote: This patch series is version 3 of the core dump masking feature, which provides a per-process flag not to dump anonymous shared memory segments. I just wanted to remind you that you need to be careful about dumping the [vdso] segment no matter whether you omit other segments. I didn't actually try running your patches, and if the kernel doesn't actually consider this segment anonymous and shared, things might already work fine as is. In any case, you can check with "readelf -a", if the [vdso] segment is there. And you will find that if you forget to dump it, "gdb" can no longer give you stack traces on call chains that involve signal handlers. As an alternative to your kernel patch, you could achieve the same goal in user space, by linking my coredumper http://code.google.com/p/google-coredumper/ into your binaries and setting up appropriate signal handlers. An equivalent patch for selectively omitting memory regions would be trivial to add. While this does give you more flexibility, it of course has the drawback of requiring you to change your applications, so there still is some benefit in a kernelspace solution. Markus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Kawai, Hidehiro wrote: This patch series is version 3 of the core dump masking feature, which provides a per-process flag not to dump anonymous shared memory segments. I just wanted to remind you that you need to be careful about dumping the [vdso] segment no matter whether you omit other segments. I didn't actually try running your patches, and if the kernel doesn't actually consider this segment anonymous and shared, things might already work fine as is. In any case, you can check with readelf -a, if the [vdso] segment is there. And you will find that if you forget to dump it, gdb can no longer give you stack traces on call chains that involve signal handlers. As an alternative to your kernel patch, you could achieve the same goal in user space, by linking my coredumper http://code.google.com/p/google-coredumper/ into your binaries and setting up appropriate signal handlers. An equivalent patch for selectively omitting memory regions would be trivial to add. While this does give you more flexibility, it of course has the drawback of requiring you to change your applications, so there still is some benefit in a kernelspace solution. Markus - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Hi, David Howells wrote: > Kawai, Hidehiro <[EMAIL PROTECTED]> wrote: > >>To avoid the above situation we can limit the core file size by >>setrlimit(2) or ulimit(1). But this method can lose important data >>such as stack because core dumping is terminated halfway. >>So I suggest keeping shared memory segments from being dumped for >>particular processes. > > A better way might be to place the shared memory segments last if that's > possible (I'm not sure ELF supports out-of-order segments). Placing the shared memory segments last and limiting by ulimit -c is one of the solutions. But there is no guarantee that the memory segments other than anonymous shared memory are always dumped. So your idea cannot alternate my suggesting feature. But if your idea has a merit which my idea doesn't have, I try to consider coexistence of both idea. >>Because the shared memory attached to processes is common in them, we don't >>need to dump the shared memory every time. > > So there's no guarantee that they'll be dumped at all... I'm not sure there's > any way around that, however. Indeed. However, some people don't want to dump anonymous shared memory at all. Taking into account this requirement, I don't guarantee that. But this feature allows per-process setting. So if you want to dump the shared memory at least once, you can manage to do that in userland. Thanks, -- Hidehiro Kawai Hitachi, Ltd., Systems Development Laboratory - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Hi, David Howells wrote: Kawai, Hidehiro [EMAIL PROTECTED] wrote: To avoid the above situation we can limit the core file size by setrlimit(2) or ulimit(1). But this method can lose important data such as stack because core dumping is terminated halfway. So I suggest keeping shared memory segments from being dumped for particular processes. A better way might be to place the shared memory segments last if that's possible (I'm not sure ELF supports out-of-order segments). Placing the shared memory segments last and limiting by ulimit -c is one of the solutions. But there is no guarantee that the memory segments other than anonymous shared memory are always dumped. So your idea cannot alternate my suggesting feature. But if your idea has a merit which my idea doesn't have, I try to consider coexistence of both idea. Because the shared memory attached to processes is common in them, we don't need to dump the shared memory every time. So there's no guarantee that they'll be dumped at all... I'm not sure there's any way around that, however. Indeed. However, some people don't want to dump anonymous shared memory at all. Taking into account this requirement, I don't guarantee that. But this feature allows per-process setting. So if you want to dump the shared memory at least once, you can manage to do that in userland. Thanks, -- Hidehiro Kawai Hitachi, Ltd., Systems Development Laboratory - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Kawai, Hidehiro <[EMAIL PROTECTED]> wrote: > To avoid the above situation we can limit the core file size by > setrlimit(2) or ulimit(1). But this method can lose important data > such as stack because core dumping is terminated halfway. > So I suggest keeping shared memory segments from being dumped for > particular processes. A better way might be to place the shared memory segments last if that's possible (I'm not sure ELF supports out-of-order segments). > Because the shared memory attached to processes is common in them, we don't > need to dump the shared memory every time. So there's no guarantee that they'll be dumped at all... I'm not sure there's any way around that, however. David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH 0/4] coredump: core dump masking support v3
Hi, This patch series is version 3 of the core dump masking feature, which provides a per-process flag not to dump anonymous shared memory segments. In this version, /proc//coredump_omit_anonymous_shared file is provided as an interface instead of the previous /proc//core_flags. If you have written a non-zero value to the file, anonymous shared memory segments of the process not to be dumped. Another change of this version is removal of kernel.core_flags_enables sysctl which enables/disables core dump flags globally. The purpose of this sysctl is for the system administrator to force all anonymous memory to be dumped. But ulimit -c 0 breaks it. So this sysctl is not helpful for the current linux. This patch series can be applied against 2.6.20-mm1. The supported core file formats are ELF and ELF-FDPIC. ELF has been tested, but ELF-FDPIC has not been build and tested because I don't have the test environment. Background: Some software programs share huge memory among hundreds of processes. If a failure occurs on one of these processes, they can be signaled by a monitoring process to generate core files and restart the service. However, it can develop into a system-wide failure such as system slow down for a long time and disk space shortage because the total size of the core files is very huge! To avoid the above situation we can limit the core file size by setrlimit(2) or ulimit(1). But this method can lose important data such as stack because core dumping is terminated halfway. So I suggest keeping shared memory segments from being dumped for particular processes. Because the shared memory attached to processes is common in them, we don't need to dump the shared memory every time. Usage: Get all shared memory segments of pid 1234 not to dump: $ echo 1 > /proc/1234/coredump_omit_anonymous_shared When a new process is created, the process inherits the flag status from its parent. It is useful to set the core dump flags before the program runs. For example: $ echo 1 > /proc/self/coredump_omit_anonymous_shared $ ./some_program ChangeLog: v3: - remove `/proc//core_flags' proc entry - add `/proc//coredump_anonymous_shared' as a named flag - remove kernel.core_flags_enable sysctl parameter v2: http://groups.google.com/group/linux.kernel/browse_frm/thread/cb254465971d4a42/ http://groups.google.com/group/linux.kernel/browse_frm/thread/da78f2702e06fa11/ - rename `coremask' to `core_flags' - change `core_flags' member in mm_struct to a bit field next to `dumpable' - introduce a global spin lock to protect adjacent two bit fields (core_flags and dumpable) from race condition - fix a bug that the generated core file can be corrupted when core dumping and updating core_flags occur concurrently - add kernel.core_flags_enable sysctl parameter to enable/disable flags in /proc//core_flags - support ELF-FDPIC binary format, but not tested v1: http://groups.google.com/group/linux.kernel/browse_frm/thread/1381fc54d716e3e6/ -- Hidehiro Kawai Hitachi, Ltd., Systems Development Laboratory E-mail: [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH 0/4] coredump: core dump masking support v3
Hi, This patch series is version 3 of the core dump masking feature, which provides a per-process flag not to dump anonymous shared memory segments. In this version, /proc/pid/coredump_omit_anonymous_shared file is provided as an interface instead of the previous /proc/pid/core_flags. If you have written a non-zero value to the file, anonymous shared memory segments of the process not to be dumped. Another change of this version is removal of kernel.core_flags_enables sysctl which enables/disables core dump flags globally. The purpose of this sysctl is for the system administrator to force all anonymous memory to be dumped. But ulimit -c 0 breaks it. So this sysctl is not helpful for the current linux. This patch series can be applied against 2.6.20-mm1. The supported core file formats are ELF and ELF-FDPIC. ELF has been tested, but ELF-FDPIC has not been build and tested because I don't have the test environment. Background: Some software programs share huge memory among hundreds of processes. If a failure occurs on one of these processes, they can be signaled by a monitoring process to generate core files and restart the service. However, it can develop into a system-wide failure such as system slow down for a long time and disk space shortage because the total size of the core files is very huge! To avoid the above situation we can limit the core file size by setrlimit(2) or ulimit(1). But this method can lose important data such as stack because core dumping is terminated halfway. So I suggest keeping shared memory segments from being dumped for particular processes. Because the shared memory attached to processes is common in them, we don't need to dump the shared memory every time. Usage: Get all shared memory segments of pid 1234 not to dump: $ echo 1 /proc/1234/coredump_omit_anonymous_shared When a new process is created, the process inherits the flag status from its parent. It is useful to set the core dump flags before the program runs. For example: $ echo 1 /proc/self/coredump_omit_anonymous_shared $ ./some_program ChangeLog: v3: - remove `/proc/pid/core_flags' proc entry - add `/proc/pid/coredump_anonymous_shared' as a named flag - remove kernel.core_flags_enable sysctl parameter v2: http://groups.google.com/group/linux.kernel/browse_frm/thread/cb254465971d4a42/ http://groups.google.com/group/linux.kernel/browse_frm/thread/da78f2702e06fa11/ - rename `coremask' to `core_flags' - change `core_flags' member in mm_struct to a bit field next to `dumpable' - introduce a global spin lock to protect adjacent two bit fields (core_flags and dumpable) from race condition - fix a bug that the generated core file can be corrupted when core dumping and updating core_flags occur concurrently - add kernel.core_flags_enable sysctl parameter to enable/disable flags in /proc/pid/core_flags - support ELF-FDPIC binary format, but not tested v1: http://groups.google.com/group/linux.kernel/browse_frm/thread/1381fc54d716e3e6/ -- Hidehiro Kawai Hitachi, Ltd., Systems Development Laboratory E-mail: [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 0/4] coredump: core dump masking support v3
Kawai, Hidehiro [EMAIL PROTECTED] wrote: To avoid the above situation we can limit the core file size by setrlimit(2) or ulimit(1). But this method can lose important data such as stack because core dumping is terminated halfway. So I suggest keeping shared memory segments from being dumped for particular processes. A better way might be to place the shared memory segments last if that's possible (I'm not sure ELF supports out-of-order segments). Because the shared memory attached to processes is common in them, we don't need to dump the shared memory every time. So there's no guarantee that they'll be dumped at all... I'm not sure there's any way around that, however. David - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
