Re: Assertion during dsync receive
Thanks. I've had the user clear out that mailbox, and replication is working
fine for them again.
Is there a better way to catch this than watch for crashes and read the
backtrace to find what mailbox needs to be shrunk?
Where is the threshold for "too big"?
-- Ian
On 2/23/18 11:33 AM, Aki Tuomi wrote:
> The mailbox is too big.
>
>
>
> ---
> Aki Tuomi
> Dovecot oy
>
> ---- Original message
> From: Ian Bobbitt <ibobb...@globalnoc.iu.edu>
> Date: 23/02/2018 17:52 (GMT+02:00)
> To: dovecot@dovecot.org
> Subject: Assertion during dsync receive
>
> Hi,
>
> I'm getting an assertion failed on the receiving side, causing syncs to fail
> for one user. The servers are setup so that
> only one is receiving any traffic other than replication at any time. The one
> that's only receiving replications is the
> one that's failing.
>
> I've tried deleting the user's home on the receiving server, but it still
> crashes during the sync. Oddly, the user's
> home is 7.4G on the sending server, but ends up at 42G on the receiving side,
> even after deleting and trying a fresh sync.
>
> The mailbox implicated in the backtrace ("Spam") does have a very large
> number of messages in it.
> On sender:
> Spam messages=1217764 recent=0 uidnext=1218103 uidvalidity=1379509105
> unseen=16 highestmodseq=744588 vsize=34468460093
> guid=090ed93a7a09abf10200fdf6807a firstsaved=1498744186
> On receiver:
> Spam messages=1217766 recent=352 uidnext=1218105 uidvalidity=1379509105
> unseen=16 highestmodseq=744589 vsize=34468496809
> guid=090ed93a7a09abf10200fdf6807a firstsaved=1519396172
>
> Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking index file
> /gnoc/mail/home/bgeels/mail/storage/dovecot.map.index
> Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox
> /gnoc/mail/home/bgeels/mail/storage: rebuilding indexes
> Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file mail-index-util.c:
> line 10 (mail_index_uint32_to_offset):
> assertion failed: (offset < 0x4000)
> Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw backtrace:
> /usr/lib64/dovecot/libdovecot.so.0(+0x9f3de)
> [0x7feb584143de] -> /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be)
> [0x7feb584144be] ->
> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0)
> [0x7feb587906d0] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854)
> [0x7feb58774f34] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f)
> [0x7feb587884ff] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de)
> [0x7feb5870b3ae] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858)
> [0x7feb5870ccd8] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c) [0x7feb5870ce7c]
> ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b)
> [0x7feb5870cf3b] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
> [0x7feb586f2834] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
> [0x7feb586f28d7] ->
> dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475) [0x445495] ->
> dovecot/doveadm-server() [0x43edc0] ->
> dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653] ->
> dovecot/doveadm-server(dsync_brain_run+0x541)
> [0x43acf1] -> dovecot/doveadm-server() [0x43b070] -> dovecot/doveadm-server()
> [0x44fe5f] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
> [0x7feb5842b3bf] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c]
> ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7feb58429f28] ->
> dovecot/doveadm-server() [0x4209c5] ->
> dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server() [0x4377f4] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
> [0x7feb5842b3bf] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c]
> Feb 23 14:57:33 dsync-local(bgeels): Fatal: master: service(doveadm): child
> 82098 killed with signal 6 (core dumped)
>
> I've attached the output of `doveconf -n` and the full backtrace from a core
> dump.
>
> Dovecot 2.2.33.2 (GhettoForge package)
> CentOS 7 x86_64
> XFS, no NFS.
>
>
> -- Ian
smime.p7s
Description: S/MIME Cryptographic Signature
Assertion during dsync receive
Hi,
I'm getting an assertion failed on the receiving side, causing syncs to fail
for one user. The servers are setup so that
only one is receiving any traffic other than replication at any time. The one
that's only receiving replications is the
one that's failing.
I've tried deleting the user's home on the receiving server, but it still
crashes during the sync. Oddly, the user's
home is 7.4G on the sending server, but ends up at 42G on the receiving side,
even after deleting and trying a fresh sync.
The mailbox implicated in the backtrace ("Spam") does have a very large number
of messages in it.
On sender:
Spam messages=1217764 recent=0 uidnext=1218103 uidvalidity=1379509105 unseen=16
highestmodseq=744588 vsize=34468460093
guid=090ed93a7a09abf10200fdf6807a firstsaved=1498744186
On receiver:
Spam messages=1217766 recent=352 uidnext=1218105 uidvalidity=1379509105
unseen=16 highestmodseq=744589 vsize=34468496809
guid=090ed93a7a09abf10200fdf6807a firstsaved=1519396172
Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking index file
/gnoc/mail/home/bgeels/mail/storage/dovecot.map.index
Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox
/gnoc/mail/home/bgeels/mail/storage: rebuilding indexes
Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file mail-index-util.c:
line 10 (mail_index_uint32_to_offset):
assertion failed: (offset < 0x4000)
Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw backtrace:
/usr/lib64/dovecot/libdovecot.so.0(+0x9f3de)
[0x7feb584143de] -> /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be)
[0x7feb584144be] ->
/usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0)
[0x7feb587906d0] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854)
[0x7feb58774f34] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f)
[0x7feb587884ff] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de)
[0x7feb5870b3ae] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858)
[0x7feb5870ccd8] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c) [0x7feb5870ce7c] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b)
[0x7feb5870cf3b] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
[0x7feb586f2834] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37) [0x7feb586f28d7]
->
dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475) [0x445495] ->
dovecot/doveadm-server() [0x43edc0] ->
dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653] ->
dovecot/doveadm-server(dsync_brain_run+0x541)
[0x43acf1] -> dovecot/doveadm-server() [0x43b070] -> dovecot/doveadm-server()
[0x44fe5f] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
[0x7feb5842b3bf] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7feb58429f28] ->
dovecot/doveadm-server() [0x4209c5] ->
dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server() [0x4377f4] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
[0x7feb5842b3bf] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c]
Feb 23 14:57:33 dsync-local(bgeels): Fatal: master: service(doveadm): child
82098 killed with signal 6 (core dumped)
I've attached the output of `doveconf -n` and the full backtrace from a core
dump.
Dovecot 2.2.33.2 (GhettoForge package)
CentOS 7 x86_64
XFS, no NFS.
-- Ian
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7_4.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
Proper way to enable plugin for some users
What's the best way to enable a plugin (in this case, mail_log) for only some users? I have this as a field returned in my sql user_query that seems to be doing the job: if(debug_log != 0 and '%s' = 'imap', 'listescape acl notify replication mail_log', 'listescape acl notify replication') as mail_plugins But it feels clunky and error prone having to replicate the entire list of plugins in my userdb. Is this the best way to do it, or is there something I should be doing differently? OS: CentOS 7 Dovecot: 2.2.31, GF RPMs smime.p7s Description: S/MIME Cryptographic Signature
dotlock causing crashes
OS: CentOS 7 x86_64
Dovecot version: 2.2.31 (65cde28) (GhettoForge RPM)
Filesystem: GlusterFS, but working on changing that. Only one server is
receiving activity.
Was getting messages about corrupt dovecot.map.index files. Changed to dotlock
from fcntl to try to fix that.
Reading symbols from /usr/libexec/dovecot/imap...(no debugging symbols
found)...done.
[New LWP 74012]
Core was generated by `dovecot/imap'.
Program terminated with signal 6, Aborted.
#0 0x7fa262c741d7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt full
#0 0x7fa262c741d7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 74012
selftid = 74012
#1 0x7fa262c758c8 in __GI_abort () at abort.c:90
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x7ffd7009f401, sa_sigaction
= 0x7ffd7009f401}, sa_mask = {__val =
{0, 0, 140335431377968, 140335423109592, 140335422613219, 4246482,
140335418575669, 12278048, 4192326493288016896,
12278592, 140335423192931, 0, 0, 140335425698848, 12280232, 140726483153732}},
sa_flags = 1657305400, sa_restorer = 0x79a}
sigs = {__val = {32, 0 }}
#2 0x7fa26309eac6 in default_fatal_finish (type=,
status=status@entry=0) at failures.c:201
backtrace = 0xbb5958 "/usr/lib64/dovecot/libdovecot.so.0(+0x9eace)
[0x7fa26309eace] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x9ebae) [0x7fa26309ebae] ->
/usr/lib64/dovecot/libdovecot.so.0(i_fatal+0)
[0x7fa26303012c] -> /usr"...
#3 0x7fa26309ebae in i_internal_fatal_handler (ctx=0x7ffd7009f4d0,
format=, args=) at
failures.c:670
status = 0
#4 0x7fa26303012c in i_panic (format=format@entry=0x7fa2630d11de "file %s:
line %d: unreached") at failures.c:275
ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0,
timestamp_usecs = 0}
args = {{gp_offset = 24, fp_offset = 48, overflow_arg_area =
0x7ffd7009f5d0, reg_save_area = 0x7ffd7009f510}}
#5 0x7fa2630a344f in file_lock_do (fd=fd@entry=20, path=path@entry=0xbb5868
"/gnoc/mail/home/bgeels/mail/mailboxes/Junk/dbox-Mails/.vsize.lock23f657caa43d8796",
lock_type=lock_type@entry=1,
lock_method=lock_method@entry=FILE_LOCK_METHOD_DOTLOCK, timeout_secs=0,
error_r=error_r@entry=0x7ffd7009f768) at
file-lock.c:285
lock_type_str = 0x7fa2630e6948 "write-lock"
started = 1502905468
ret =
__FUNCTION__ = "file_lock_do"
#6 0x7fa2630a3796 in file_wait_lock_error (fd=20, path=0xbb5868
"/gnoc/mail/home/bgeels/mail/mailboxes/Junk/dbox-Mails/.vsize.lock23f657caa43d8796",
lock_type=1,
lock_method=FILE_LOCK_METHOD_DOTLOCK, timeout_secs=,
lock_r=0xc4ec10, error_r=0x7ffd7009f768) at
file-lock.c:314
ret =
#7 0x7fa2630a3813 in file_try_lock_error (fd=,
path=, lock_type=lock_type@entry=1,
lock_method=lock_method@entry=FILE_LOCK_METHOD_DOTLOCK,
lock_r=lock_r@entry=0xc4ec10,
error_r=error_r@entry=0x7ffd7009f768) at file-lock.c:66
No locals.
#8 0x7fa2630a0955 in try_create_new (error_r=0x7ffd7009f768,
lock_r=0xc4ec10, fd_r=0x7ffd7009f700,
set=0x7ffd7009f770, path=0xc2f930
"/gnoc/mail/home/bgeels/mail/mailboxes/Junk/dbox-Mails/.vsize.lock") at
file-create-locked.c:65
fd = 20
orig_errno =
ret = -1
temp_path = 0xbb5830
mode = 0
uid =
gid = 4294967295
#9 file_create_locked (path=0xc2f930
"/gnoc/mail/home/bgeels/mail/mailboxes/Junk/dbox-Mails/.vsize.lock",
set=set@entry=0x7ffd7009f770, lock_r=lock_r@entry=0xc4ec10,
created_r=created_r@entry=0x7ffd7009f767,
error_r=error_r@entry=0x7ffd7009f768) at file-create-locked.c:118
i = 0
fd =
ret =
__FUNCTION__ = "file_create_locked"
#10 0x7fa2633e8f80 in vsize_update_lock_full (update=0xc4ebd0,
lock_secs=lock_secs@entry=0) at index-mailbox-size.c:150
box = 0xc2e268
perm = 0xc2e440
set = {lock_timeout_secs = 0, lock_method = FILE_LOCK_METHOD_DOTLOCK,
mode = 384, uid = 0, gid = 4294967295,
gid_origin = 0xc2ea58 "/gnoc/mail/home/bgeels/mail/mailboxes/Junk"}
error = 0x7fa2633f2062
"1\300[]A\\\303\017\037\200"
created = false
#11 0x7fa2633e9057 in index_mailbox_vsize_update_try_lock
(update=) at index-mailbox-size.c:167
No locals.
#12 0x7fa2633e9755 in index_mailbox_vsize_update_appends (box=) at index-mailbox-size.c:479
update = 0xc4ebd0
status = {messages = 1323, recent = 0, unseen = 0, uidvalidity =
1413091786, uidnext = 6750, first_unseen_seq =
0, first_recent_uid = 5886, last_cached_seq = 0, highest_modseq = 0,
highest_pvt_modseq = 0, keywords = 0x0,
permanent_flags = 0, flags = 0, permanent_keywords = 0, allow_new_keywords = 0,
nonpermanent_modseqs = 0,
no_modseq_tracking = 0, have_guids = 1,
have_save_guids = 1, have_only_guid128 = 0}
#13 0x7fa2633f633c in
