[Dovecot] fts_squat hanging on some messages

Henning Rohlfs Fri, 17 Sep 2010 02:32:06 -0700

Hello,

I recently upgraded to dovecot 2.0.2 (on gentoo ~amd64) and nowfts_squat's index building seems to hang for some of my mailboxes. Theimap process gets stuck at 100% cpu usage and dovecot.index.search isnever touched.

I traced the problem to a message (attached) for one of theproblematic mailboxes. Also attached are a full backtrace and theoutput of dovecot -n.


The problem is reproducible:
step 1) /usr/libexec/dovecot/imap
step 2) . select Junk
step 3) . search body sometext

--- Begin Message ---

<?xml version="1.0" encoding="utf-8" ?>
<html><head><title>Sicher ist Sicher! Richten Sie jetzt Ihren Hotmail Junk-Mail 
Filter ein!</title></head><body style="margin:0px;"><table dir="ltr" 
cellpadding="0" cellspacing="0" width="641" border="0"><thead><tr><tr><td><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/de/WLNewsletterBanner_de.jpg"; 
alt=""></td></tr><tr><td style="font:normal normal normal 11px 
Verdana,sans-serif;color:#4D7DDD;vertical-align:bottom;text-align:right;padding-right:20px;padding-bottom:4px;"></td></tr></tr></thead><tbody><tr><td
 colspan="2"><table cellpadding="0" cellspacing="10" width="630" 
border="0"><tbody><tr><td style="border:1px solid 
#D3D8D2;padding:10px;vertical-align:top;" colspan="2"><table cellpadding="0" 
cellspacing="0" border="0" width="376"><tr><td colspan="2" 
style="padding-bottom:4px;"><a name="ID0EED"></a><a 
href=http://microsoft.msn.com/Key=12023.9JTg.M.DL.Lklx4  title="" 
style="font-size:30.56px;font-family:'Segoe 
light',Verdana,sans-serif;color:#4D7DDD;text-decoration:none;"
 ><strong>Junk-E-Mails m&#252;ssen drau&#223;en 
 >bleiben!</strong></a></td></tr><tr><td colspan="2" 
 >style="font-size:18.4px;font-family:'Segoe 
 >light',Verdana,sans-serif;color:#777;padding-bottom:20px;"><strong>Tolle 
 >Sache: der Junk-E-Mail-Filter in Hotmail</strong></td></tr><tr><td 
 >valign="top" style="padding-right:8px;" rowspan="2"><a 
 >href=http://microsoft.msn.com/Key=12023.9JTg.M.DL.Lklx4  title="" 
 >style="text-decoration:none;"><img 
 >src="http://ads1.msn.com/ads/pronws/CIQ/HMML//de/HMML0804_T1_Junk.jpg"; 
 >alt="Junk-E-Mail-Filter" border="0"></a></td><td valign="top" 
 >style="font:normal normal normal 11px 
 >Verdana,sans-serif;color:#333;padding-bottom:8px;"><p><strong>Spam-, Trash- 
 >und Junk-Mails nerven! Gut, dass man da seinen Hotmail-Zugang rundum 
 >absichern kann. W&#228;hlen Sie dazu einfach eine der drei Sicherheitsstufen 
 >im Junk-E-Mail-Filter und schon k&#246;nnen Sie sicher sein, dass die meisten 
 >unerw&#252;nschten Mails fern bleiben!</strong></p><p>Und wenn Sie absolut 
 >sicher
  gehen wollen, dass keine fremde Mail Ihren Posteingang erreicht, k&#246;nnen 
Sie einen weiteren Filter einstellen. Legen Sie dabei einfach fest, wem es 
gestattet ist, Ihnen Mails zu schreiben, und wem nicht.</p></td></tr><tr><td 
colspan="2"><table cellpadding="0" cellspacing="0" border="0"><tr><td 
valign="top" style="width:22px;padding-top:4px;"><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/MemberLetterArrow.gif"; alt="" 
width="18" height="14"></td><td style="font:normal normal normal 11px 
Verdana,sans-serif;color:#333;"><a 
href=http://microsoft.msn.com/Key=12023.9JTg.M.DL.Lklx4  title="" 
style="color:#181399;">Gleich Junk-E-Mail-Filter 
aktivieren!</a></td></tr></table></td></tr></table></td><td style="border:1px 
solid #D3D8D2;padding:10px;vertical-align:top;width:380px;"><div 
style="padding-bottom:8px;"><span style="font-size:22.4px;font-family:'Segoe 
light',Verdana,sans-serif;color:#4D7DDD;text-decoration:none;">In dieser 
Ausgabe</span></div><table cellpadding="0" cell
 spacing="0" width="180" border="0"><tr><td colspan="2" 
style="padding-bottom:8px;font-size:22.4px;font-family:'Segoe 
light',Verdana,sans-serif;color:#4D7DDD;"></td></tr><tr><td valign="top" 
style="width:22px;padding-top:4px;"><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/MemberLetterArrow.gif"; alt="" 
width="18" height="14"></td><td style="font:normal normal normal 11px 
Verdana,sans-serif;color:#333;"><a href="#ID0EED" title="" 
style="color:#181399;">Hotmail: Junk-E-Mail-Filter</a></td></tr><tr><td 
valign="top" style="width:22px;padding-top:4px;"><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/MemberLetterArrow.gif"; alt="" 
width="18" height="14"></td><td style="font:normal normal normal 11px 
Verdana,sans-serif;color:#333;"><a href="#ID0E1E" title="" 
style="color:#181399;">Neu: Windows Live FolderShare Beta</a></td></tr><tr><td 
valign="top" style="width:22px;padding-top:4px;"><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/MemberLetterArrow.gif"; alt="" 
width="18" heigh
 t="14"></td><td style="font:normal normal normal 11px 
Verdana,sans-serif;color:#333;"><a href="#ID0E5F" title="" 
style="color:#181399;">Messenger: Ihr Profil 
aktualisieren</a></td></tr></table></td></tr><tr><td style="border:1px solid 
#D3D8D2;padding:10px;vertical-align:top;width:298px;"><table cellpadding="0" 
cellspacing="0" border="0"><tr><td style="padding-bottom:8px;"><a 
name="ID0E1E"></a><a href=http://microsoft.msn.com/Key=12023.9JTg.N.DL.mHW22  
title="" style="font-size:22.4px;font-family:'Segoe 
light',Verdana,sans-serif;color:#4D7DDD;text-decoration:none;"><strong>Neu: 
Auch unterwegs auf die eigenen Dateien zugreifen! 
</strong></a></td></tr><tr><td style="padding-bottom:8px;font:normal normal 
normal 11px Verdana,sans-serif;color:#333;"><a 
href=http://microsoft.msn.com/Key=12023.9JTg.N.DL.mHW22  title="" 
style="text-decoration:none;float:left;margin-right:10px;margin-bottom:5px;"><img
 src="http://ads1.msn.com/ads/pronws/CIQ/HMML//de/HMML0804_T2_FolderShare.jpg"; 
alt="F
 olderShare Beta kostenlos" border="0"></a><p><strong>Es klingt so einfach, wie 
es tats&#228;chlich ist: Mit Windows Live FolderShare Beta k&#246;nnen Sie ab 
sofort von &#252;berall her auf die eigenen Dateien 
zugreifen!</strong></p><p>&#220;ber eine"pers&#246;nliche Bibliothek" 
k&#246;nnen Sie Dateien auch gemeinsam mit Freunden und Bekannten nutzen - bei 
bis zu 2 GB Dateigr&#246;&#223;e. Und ist FolderShare auf zwei oder mehreren 
Rechnern installiert, lassen sich diese sogar synchronisieren! 
Genial!</p></td></tr><tr><td><table cellpadding="0" cellspacing="0" 
border="0"><tr><td valign="top" style="width:22px;padding-top:4px;"><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/MemberLetterArrow.gif"; alt="" 
width="18" height="14"></td><td style="font:normal normal normal 11px 
Verdana,sans-serif;color:#333;"><a 
href=http://microsoft.msn.com/Key=12023.9JTg.N.DL.mHW22  title="" 
style="color:#181399;">Sofort kostenlos 
downloaden!</a></td></tr></table></td></tr></table></td><td styl
 e="border:1px solid #D3D8D2;padding:10px;vertical-align:top;width:298px;" 
colspan="2"><table cellpadding="0" cellspacing="0" border="0"><tr><td 
style="padding-bottom:8px;"><a name="ID0E5F"></a><a 
href=http://microsoft.msn.com/Key=12023.9JTg.P.DL.KKJHS2  title="" 
style="font-size:22.4px;font-family:'Segoe 
light',Verdana,sans-serif;color:#4D7DDD;text-decoration:none;"><strong>Zeigen 
Sie Profil im Messenger! </strong></a></td></tr><tr><td 
style="padding-bottom:8px;font:normal normal normal 11px 
Verdana,sans-serif;color:#333;"><a 
href=http://microsoft.msn.com/Key=12023.9JTg.P.DL.KKJHS2  title="" 
style="text-decoration:none;float:left;margin-right:10px;margin-bottom:5px;"><img
 src="http://ads1.msn.com/ads/pronws/CIQ/HMML//de/HMML0804_T3_Profil.jpg"; 
alt="" border="0"></a><p><strong>Es kann so einfach sein, im Messenger neue 
Freunde zu finden &ndash; oder sich von alten Freunden finden zu 
lassen!</strong></p><p>Geben Sie unter der Option &bdquo;&#214;ffentliches 
Profil" ganz einfac
 h Ihren Namen ein, Ihren Wohnort, Ihre Hobbys, Ihr Geburtsdatum etc. &ndash; 
schon wird es f&#252;r alte Freunde leichter, Sie zu finden. Und wer neue 
Freunde sucht, der zeigt ebenfalls &bdquo;Profil"!</p></td></tr><tr><td><table 
cellpadding="0" cellspacing="0" border="0"><tr><td valign="top" 
style="width:22px;padding-top:4px;"><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/MemberLetterArrow.gif"; alt="" 
width="18" height="14"></td><td style="font:normal normal normal 11px 
Verdana,sans-serif;color:#333;"><a 
href=http://microsoft.msn.com/Key=12023.9JTg.P.DL.KKJHS2  title="" 
style="color:#181399;">Jetzt Profil 
aktualisieren!</a></td></tr></table></td></tr></table></td></tr><tr><td><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/clear.gif"; alt="" width="299" 
height="1"></td><td><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/clear.gif"; alt="" width="88" 
height="1"></td><td><img 
src="http://ads1.msn.com/ads/pronws/CIQ/HMML/clear.gif"; alt="" width="200" 
height="1"></td></tr
 ></tbody></table></td></tr></tbody><tfoot><tr><td colspan="2"><table 
 >cellpadding="0" cellspacing="0" width="641" border="0" 
 >background="http://ads1.msn.com/ads/pronws/CIQ/HMML/WLMemberLetterFooter.jpg"; 
 >style="background-repeat:no-repeat;background-position:bottom left;"><tr><td 
 >style="padding:0px 15px 0px 10px"><img 
 >src="http://ads1.msn.com/ads/pronws/CIQ/HMML/Microsoft.gif"; width="82" 
 >height="13"></td></tr><tr><td style="font:normal normal normal 10px 
 >Verdana,sans-serif;color:#333;padding:5px 15px 50px 10px;"><img 
 >src="http://ads1.msn.com/ads/pronws/CIQ/HMML/clear.gif"; width="1" height="77" 
 >alt="" style="float:left;"><div class="Footnote"><sup>Weitere Infos und Tipps 
 >finden Sie im </sup><a 
 >href=http://microsoft.msn.com/Key=12023.9JTg.Q.DL.CHYKcp  title=""><sup 
 >xmlns:control="control">MailPower-Blog</sup></a><sup>.<br><br>Sie erhalten 
 >diese Nachricht als gesch&#228;tztes MSN-Mitglied. Microsoft respektiert Ihre 
 >Privatsph&#228;re. Weitere Informationen erhalten Sie online in 
 unseren </sup><a href=http://microsoft.msn.com/Key=12023.9JTg.R.DL.5sVWB  
title=""><sup xmlns:control="control">Datenschutzbestimmungen</sup></a><sup>. 
Wir hoffen, dass diese Information f&#252;r Sie n&#252;tzlich ist. Sollten Sie 
es jedoch trotzdem vorziehen, keine Informationen dieser Art per E-Mail mehr zu 
erhalten, &#228;ndern Sie bitte Ihre </sup><a 
href=http://microsoft.msn.com/Key=12023.9JTg.S.DL.Ktp9QC  title=""><sup 
xmlns:control="control">Marketingeinstellungen</sup></a><sup>.<br><br>Microsoft 
Corporation, One Microsoft Way, Redmond, WA 98052, U.S.A. <br>&copy; 2008 
Microsoft Corporation. Alle Rechte vorbehalten. </sup><a 
href=http://microsoft.msn.com/Key=12023.9JTg.T.DL.DF73MZ  title=""><sup 
xmlns:control="control">Impressum</sup></a></div></td></tr></table></td></tr></tfoot></table></body></html>

..Message-Id: <[email protected]>

--- End Message ---

(gdb) bt full
#0  0x00007ff667019002 in memchr () from /lib/libc.so.6
No symbol table info available.
#1  0x00007ff667332f48 in parse_next_body_to_boundary (ctx=0xcd0d70, 
block_r=0x7fff6dc020a0)
    at message-parser.c:350
        boundary = 0x0
        data = 0xcd64e0 "\n<html><head><title>Sicher ist Sicher! Richten Sie 
jetzt Ihren Hotmail Junk-Mail Filter ein!</title></head><body 
style=\"margin:0px;\"><table dir=\"ltr\" cellpadding=\"0\" cellspacing=\"0\" 
width=\"641\" border"...
        cur = 0xcd64e1 "<html><head><title>Sicher ist Sicher! Richten Sie jetzt 
Ihren Hotmail Junk-Mail Filter ein!</title></head><body 
style=\"margin:0px;\"><table dir=\"ltr\" cellpadding=\"0\" cellspacing=\"0\" 
width=\"641\" border="...
        next = 0xcd64e0 "\n<html><head><title>Sicher ist Sicher! Richten Sie 
jetzt Ihren Hotmail Junk-Mail Filter ein!</title></head><body 
style=\"margin:0px;\"><table dir=\"ltr\" cellpadding=\"0\" cellspacing=\"0\" 
width=\"641\" border"...
        end = 0xcd84e0 ""
        boundary_start = 0
        ret = -1
        full = false
        __FUNCTION__ = "parse_next_body_to_boundary"
#2  0x00007ff667333f55 in message_parser_parse_next_block (ctx=0xcd0d70, 
block_r=0x7fff6dc020a0)
    at message-parser.c:759
        ret = 1
        eof = false
        full = true
        __FUNCTION__ = "message_parser_parse_next_block"
#3  0x00007ff66676ae92 in fts_build_mail (ctx=0xcc7110, uid=6470) at 
fts-storage.c:170
        decoder_flags = MESSAGE_DECODER_FLAG_DTCASE
        input = 0xcd0ba0
---Type <return> to continue, or q <return> to quit---
        parser = 0xcd0d70
        decoder = 0xcd1140
        raw_block = {part = 0xc6e580, hdr = 0x0, 
          data = 0xcd64e0 "\n<html><head><title>Sicher ist Sicher! Richten Sie 
jetzt Ihren Hotmail Junk-Mail Filter ein!</title></head><body 
style=\"margin:0px;\"><table dir=\"ltr\" cellpadding=\"0\" cellspacing=\"0\" 
width=\"641\" border"..., size = 8192}
        block = {part = 0xc6e580, hdr = 0x0, data = 0xcd3250 "<?XML 
VERSION=\"1.0\" ENCODING=\"UTF-8\" ?>", 
          size = 39}
        prev_part = 0xc6e580
        parts = 0x0
        skip_body = false
        body_part = true
        ret = 1
        __FUNCTION__ = "fts_build_mail"
#4  0x00007ff66676c1ed in fts_build_more (ctx=0xcc7110) at fts-storage.c:625
        _data_stack_cur_id = 4
        count = 0
        ret = 0
#5  0x00007ff66676c5c2 in fts_mailbox_search_next_nonblock (ctx=0xcc4f60, 
mail=0xccbec0, 
    tryagain_r=0x7fff6dc0221b) at fts-storage.c:728
        fbox = 0xc9ebd0
        fctx = 0xcc5120
        ret = 0
#6  0x00007ff6675c465a in mailbox_search_next_nonblock (ctx=0xcc4f60, 
mail=0xccbec0, 
    tryagain_r=0x7fff6dc0221b) at mail-storage.c:1141
        box = 0xc9c150
#7  0x000000000041bf79 in cmd_search_more (cmd=0xc95110) at imap-search.c:346
        ctx = 0xc95210
---Type <return> to continue, or q <return> to quit---
        opts = SEARCH_RETURN_ALL
        sync_flags = 32767
        end_time = {tv_sec = 1024, tv_usec = 8}
        range = 0xcd06e0
        count = 0
        id = 128
        id_min = 0
        id_max = 0
        ok_reply = 0x7ff667595040 ""
        time_msecs = 13193816
        tryagain = 103
        minmax = false
        lost_data = false
        __FUNCTION__ = "cmd_search_more"
#8  0x000000000041c6e9 in imap_search_start (ctx=0xc95210, sargs=0xcc3950, 
sort_program=0x0)
    at imap-search.c:550
        cmd = 0xc95110
        wanted_fields = 0
        wanted_headers = 0x0
#9  0x000000000040f77c in cmd_search (cmd=0xc95110) at cmd-search.c:45
        ctx = 0xc95210
        sargs = 0xcc3950
        args = 0xc99978
        charset = 0x421004 "UTF-8"
        ret = 1
#10 0x0000000000413f9f in client_command_input (cmd=0xc95110) at 
imap-client.c:660
        client = 0xc8d090
        command = 0xc95110
---Type <return> to continue, or q <return> to quit---
        __FUNCTION__ = "client_command_input"
#11 0x00000000004141fe in client_command_input (cmd=0xc95110) at 
imap-client.c:710
        client = 0xc8d090
        command = 0xc77860
        __FUNCTION__ = "client_command_input"
#12 0x0000000000414329 in client_handle_next_command (client=0xc8d090, 
remove_io_r=0x7fff6dc0241d)
    at imap-client.c:751
        size = 21
#13 0x00000000004143a9 in client_handle_input (client=0xc8d090) at 
imap-client.c:763
        _data_stack_cur_id = 3
        ret = false
        remove_io = false
        handled_commands = false
        __FUNCTION__ = "client_handle_input"
#14 0x000000000041451f in client_input (client=0xc8d090) at imap-client.c:802
        cmd = 0x6dc0248c
        output = 0xc8d678
        bytes = 21
        __FUNCTION__ = "client_input"
#15 0x00007ff667358502 in io_loop_handler_run (ioloop=0xc768e0) at 
ioloop-epoll.c:212
        ctx = 0xc76af0
        events = 0xc76b60
        event = 0xc76b60
        list = 0xc8d750
        io = 0xc8d710
        tv = {tv_sec = 1786, tv_usec = 785121}
        events_count = 2
        t_id = 2
---Type <return> to continue, or q <return> to quit---
        msecs = 1786786
        ret = 1
        i = 0
        j = 0
        call = true
#16 0x00007ff6673570cd in io_loop_run (ioloop=0xc768e0) at ioloop.c:350
No locals.
#17 0x00007ff66733f826 in master_service_run (service=0xc767b0, 
callback=0x42004c <client_connected>)
    at master-service.c:496
No locals.
#18 0x00000000004202b6 in main (argc=1, argv=0xc76370) at main.c:358
        set_roots = {0x4239a0, 0x0}
        service_flags = 3
        storage_service_flags = 0
        postlogin_socket_path = 0x0
        username = 0x0
        c = -1

# 2.0.2: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.35.4 x86_64 Gentoo Base System release 2.0.1 
first_valid_gid = 100
first_valid_uid = 1000
hostname = xehonk.de
last_valid_gid = 100
last_valid_uid = 1100
mail_location = maildir:~/.maildir
managesieve_notify_capability = mailto
managesieve_sieve_capability = comparator-i;octet comparator-i;ascii-casemap 
fileinto reject envelope encoded-character vacation subaddress 
comparator-i;ascii-numeric relational regex imap4flags copy include variables 
body enotify environment mailbox date
passdb {
  args = *
  driver = pam
}
plugin {
  antispam_backend = mailtrain
  antispam_mail_notspam = --class=innocent
  antispam_mail_sendmail = /usr/bin/dspam
  antispam_mail_sendmail_args = --source=error
  antispam_mail_spam = --class=spam
  antispam_mail_tmpdir = /tmp
  antispam_signature = X-DSPAM-Signature
  antispam_signature_missing = move
  antispam_spam_pattern_ignorecase = Junk;Junk.*
  antispam_trash_pattern_ignorecase = Trash
  fts = squat
  fts_squat = partial=4 full=10
  sieve = ~/.maildir/sieve/main-filter
  sieve_dir = ~/.maildir/sieve/filters
}
postmaster_address = [email protected]
protocols = imap sieve
service imap-login {
  inet_listener imap {
    address = ::1
  }
  inet_listener imaps {
    address = [::], *
  }
}
service managesieve-login {
  inet_listener sieve {
    address = ::1, *
  }
}
ssl_cert = </etc/dovecot/honk.dyndns.info.crt
ssl_key = </etc/dovecot/honk.dyndns.info.key.insecure
userdb {
  driver = passwd
}
protocol lda {
  mail_plugins = sieve
}
protocol imap {
  mail_plugins = fts fts_squat antispam
}

[Dovecot] fts_squat hanging on some messages

Reply via email to