Re: [Dovecot] segfault - (imap|pop3)-login during nessus scan
On Feb 19, 2010, at 9:23 PM, Timo Sirainen wrote: On Fri, 2010-02-19 at 15:28 -0600, Todd Rinaldo wrote: pop3-login[24451]: segfault at 000c rip 003c7de610a2 rsp 7fff07116968 error 4 I'm having a really hard time getting a core dump Yeah, it's difficult to get login processes to core dump. In v1.2 it's easier though. But there's an alternative way to get the backtrace: First set login_process_per_connection=no. Then: gdb -p `pidof imap-login` cont wait for crash bt full Tim, Thanks for the feedback. In the other email you sent about re-producing with nessus, note that we're using the checkpassword system, however from strace info so far we think the error happens before any fork happens to the custon auth program. Your suggestion for trapping with gdb worked for me! Though I had to do this in gdb first cause I kept getting sigpipe breaks: handle SIGPIPE nostop noprint pass This is my stack trace without debug symbols. How much would symbols help you here? Program received signal SIGSEGV, Segmentation fault. 0x003c7de610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3 (gdb) bt full #0 0x003c7de610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3 No symbol table info available. #1 0x003c7de48ade in krb5_kt_get_entry () from /usr/lib64/libkrb5.so.3 No symbol table info available. #2 0x003c7fe3871e in kssl_keytab_is_available () from /lib64/libssl.so.6 No symbol table info available. #3 0x003c7fe1e345 in ssl3_choose_cipher () from /lib64/libssl.so.6 No symbol table info available. #4 0x003c7fe19aeb in ssl3_get_client_hello () from /lib64/libssl.so.6 No symbol table info available. #5 0x003c7fe1a465 in ssl3_accept () from /lib64/libssl.so.6 No symbol table info available. #6 0x003c7fe22602 in ssl23_get_client_hello () from /lib64/libssl.so.6 No symbol table info available. #7 0x003c7fe22d99 in ssl23_accept () from /lib64/libssl.so.6 No symbol table info available. #8 0x004093f9 in ssl_step () No symbol table info available. #9 0x004095e4 in ssl_proxy_new () No symbol table info available. #10 0x004073b7 in login_accept_ssl () No symbol table info available. #11 0x00411dc8 in io_loop_handler_run () No symbol table info available. #12 0x00410edd in io_loop_run () No symbol table info available. #13 0x0040706e in main () No symbol table info available.
Re: [Dovecot] segfault - (imap|pop3)-login during nessus scan
On 22.2.2010, at 19.49, Todd Rinaldo wrote: gdb -p `pidof imap-login` cont wait for crash bt full Tim, Thanks for the feedback. In the other email you sent about re-producing with nessus, note that we're using the checkpassword system, however from strace info so far we think the error happens before any fork happens to the custon auth program. The crash comes from login process. All authentication is done by dovecot-auth process, so it doesn't matter what kind of auth stuff you're using. Program received signal SIGSEGV, Segmentation fault. 0x003c7de610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3 (gdb) bt full #0 0x003c7de610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3 No symbol table info available. #1 0x003c7de48ade in krb5_kt_get_entry () from /usr/lib64/libkrb5.so.3 No symbol table info available. #2 0x003c7fe3871e in kssl_keytab_is_available () from /lib64/libssl.so.6 No symbol table info available. #3 0x003c7fe1e345 in ssl3_choose_cipher () from /lib64/libssl.so.6 No symbol table info available. Well, that's coming from Kerberos library, which is called by OpenSSL for some reason.. Are you using Kerberos? Anyway it looks to me more like OpenSSL or Kerberos bug.
Re: [Dovecot] segfault - (imap|pop3)-login during nessus scan
On Feb 22, 2010, at 11:57 AM, Timo Sirainen wrote:
Well, that's coming from Kerberos library, which is called by OpenSSL for
some reason.. Are you using Kerberos? Anyway it looks to me more like OpenSSL
or Kerberos bug.
Tim,
Below is the stack trace with symbols. The bug appears to manifest only in
64bit redhat/centos 5 only but happens against multiple versions of openssl
that existed over 5's life. Unfortunately, RedHat decided to compile in
kerberos so I can't control that. We played around but couldn't find a way to
make it stop by manipulating ssl_cipher_list.
I have seen dovecot crash when the following packages are installed:
openssl-0.9.8e-12.el5, openssl-0.9.8e-12.el5_4.1
I've reduced the test case to this:
31705 (SSL Cipher Suites Supported) -
http://www.nessus.org/plugins/index.php?view=singleid=21643
When run manually from command line, I had to replace 443 with 993 or 995
inside the ssl_supported_ciphers.nasl script.
Then I can just run this to make it happen: nasl -t target_host
ssl_supported_ciphers.nasl
While this is clearly an openssl bug, I cannot reproduce this on courier, but I
did find a reference to a similar backtrace with stunnel:
http://tinyurl.com/yeyo7t9
Can you think of any way I could disable kerberos for dovecot so this does not
segfault? Is there any check we could put in the code to prevent the segfault?
Thanks,
Todd
---
Program received signal SIGSEGV, Segmentation fault.
0x003adf4610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3
(gdb) bt full
#0 0x003adf4610a2 in krb5_is_referral_realm () from /usr/lib64/libkrb5.so.3
No symbol table info available.
#1 0x003adf448ade in krb5_kt_get_entry () from /usr/lib64/libkrb5.so.3
No symbol table info available.
#2 0x003ae083876e in kssl_keytab_is_available () from /lib64/libssl.so.6
No symbol table info available.
#3 0x003ae081e385 in ssl3_choose_cipher () from /lib64/libssl.so.6
No symbol table info available.
#4 0x003ae0819b2b in ssl3_get_client_hello () from /lib64/libssl.so.6
No symbol table info available.
#5 0x003ae081a4a5 in ssl3_accept () from /lib64/libssl.so.6
No symbol table info available.
#6 0x003ae0822642 in ssl23_get_client_hello () from /lib64/libssl.so.6
No symbol table info available.
#7 0x003ae0822dd9 in ssl23_accept () from /lib64/libssl.so.6
No symbol table info available.
#8 0x0040a8b2 in ssl_handshake (proxy=0x1a793920) at
ssl-proxy-openssl.c:399
ret = 0
#9 0x0040ab50 in ssl_step (proxy=0x1a793920) at ssl-proxy-openssl.c:456
No locals.
#10 0x00417927 in io_loop_handler_run (ioloop=0x1a789d70) at
ioloop-epoll.c:209
ctx = (struct ioloop_handler_context *) 0x1a78bf00
events = (struct epoll_event *) 0x1a78d670
event = (const struct epoll_event *) 0x1a78d670
list = (struct io_list *) 0x1a7907f0
io = (struct io_file *) 0x1a795e50
tv = {tv_sec = 179, tv_usec = 999415}
events_count = 7
t_id = 2
msecs = 18
ret = 1
i = 0
j = 0
call = true
#11 0x00416b32 in io_loop_run (ioloop=0x1a789d70) at ioloop.c:336
No locals.
#12 0x00408dbd in main (argc=1, argv=0x7fffeae55498,
envp=0x7fffeae554a8) at main.c:482
[Dovecot] segfault - (imap|pop3)-login during nessus scan
We've been struggling with a problem for the past couple of days which to this point I've only gotten to be able to boil down to this: 1. Install nessus home edition (less pluggins I assume) 2. run all scans (sequentially or in parallel, doesn't seem to matter) 3. about 3 minutes in /var/log/messages will show segfaults on imap and/or pop3 imap-login[22185]: segfault at 000c rip 003c7de610a2 rsp 7fffa2342068 error 4 or sometimes... pop3-login[24451]: segfault at 000c rip 003c7de610a2 rsp 7fff07116968 error 4 I'm having a really hard time getting a core dump and I'm having a really hard time narrowing down the list of nessus tests which cause this. So far, I have repeated this failure in 1.1.19 and 1.1.20 Additionally we've seen something similar on 1.2 and reverted back to 1.1 a year ago. At the time we could not re-produce a test case and finally gave up. Has anyone seen something along these lines? Can anyone recommend how I could narrow this down further so we can find the problem? Thanks, Todd
Re: [Dovecot] segfault - (imap|pop3)-login during nessus scan
On Fri, 2010-02-19 at 15:28 -0600, Todd Rinaldo wrote: pop3-login[24451]: segfault at 000c rip 003c7de610a2 rsp 7fff07116968 error 4 I'm having a really hard time getting a core dump Yeah, it's difficult to get login processes to core dump. In v1.2 it's easier though. But there's an alternative way to get the backtrace: First set login_process_per_connection=no. Then: gdb -p `pidof imap-login` cont wait for crash bt full signature.asc Description: This is a digitally signed message part
Re: [Dovecot] segfault - (imap|pop3)-login during nessus scan
On Sat, 2010-02-20 at 05:23 +0200, Timo Sirainen wrote: On Fri, 2010-02-19 at 15:28 -0600, Todd Rinaldo wrote: pop3-login[24451]: segfault at 000c rip 003c7de610a2 rsp 7fff07116968 error 4 BTW. I just tried with Nessus, but couldn't reproduce this. signature.asc Description: This is a digitally signed message part
