Re: Panic: data stack: Out of memory when allocating bytes

2018-02-22 Thread Josef 'Jeff' Sipek 
On Mon, Jan 29, 2018 at 15:23:28 +0100, Thomas Robers wrote:
> Any idea what the problem could be? Is there anything more i could do
> to encircle the problem? Or perhaps is the information i provided
> uncomplete?

Sorry for the delay... your mail scrolled off the screen in my INBOX and
then I forgot about it.

The good news is that I have a general idea what is happening.  The bad news
is, it's not a trivial fix.  The ACL code is *far* from efficient.  It has
never gotten any real optimizations, so the code is a bit memory hungry.
Your heavy use of acls and shared mailboxes just happens to run that code
enough times that it chews through the 128MB limit.

You could increase the vsz_limit for the imap service [1] to something
higher than 128MB, but that'll eventually break if you create more
mailboxes/share them, and it allows imap processes to use more of the
servers RAM.


If I had the time to work on it, I'd try to implement a replacement acl
format to use instead of vfile.  I'd probably try to base it on top of
dovecot dictionaries [2], so that it could use existing key-value storages.
This should make acl handling faster since the entries could be modified "in
place" (instead of having to reconstruct the whole file)... or at least it
should make it possible to change the acl code to do these improvements.

Then, the acl setting would be as flexible as the acl_shared_dict setting.

Jeff.

[1] https://wiki2.dovecot.org/Services?highlight=(vsz_limit)#Service_limits
[2] https://wiki2.dovecot.org/Dictionary

> Am 25.01.2018 um 16:24 schrieb Thomas Robers:
> > Hi,
> > 
> > Am 24.01.2018 um 23:39 schrieb Josef 'Jeff' Sipek:
> > > It looks like the binaries are stripped.  There should be a "debug"
> > > package
> > > you can install with symbol information.  Then, the backtrace should
> > > be much
> > > more helpful.
> > 
> > I installed the debug package and the backtrace now is:
> > 
> > --- snip ---
> > (gdb) bt full
> > #0  0x7f73f1386495 in raise () from /lib64/libc.so.6
> > No symbol table info available.
> > #1  0x7f73f1387c75 in abort () from /lib64/libc.so.6
> > No symbol table info available.
> > #2  0x7f73f17ab822 in mem_block_alloc (min_size=520) at
> > data-stack.c:356
> >      block = 
> >      prev_size = 
> >      alloc_size = 134217728
> > #3  0x7f73f17abc18 in t_malloc_real (size=,
> > permanent=true) at data-stack.c:415
> >      block = 
> >      ret = 
> >      alloc_size = 520
> > #4  0x7f73f17abdeb in t_malloc0 (size=513) at data-stack.c:468
> >      mem = 
> > #5  0x7f73f17a95dd in p_malloc (buf=0x7f73ebdcef28, size=513) at
> > mempool.h:99
> > No locals.
> > #6  buffer_alloc (buf=0x7f73ebdcef28, size=513) at buffer.c:34
> >      __func__ = "buffer_alloc"
> > #7  0x7f73f17a967b in buffer_create_dynamic (pool= > out>, init_size=512) at buffer.c:143
> >      buf = 0x7f73ebdcef28
> > #8  0x7f73f17a803b in backtrace_get (backtrace_r=0x7ffd0ddd8358) at
> > backtrace-string.c:86
> >      str = 
> > #9  0x7f73f17af1da in default_fatal_finish (type= > out>, status=0) at failures.c:221
> >      backtrace = 
> >      recursed = 1
> > #10 0x7f73f17af766 in i_internal_fatal_handler (ctx=0x7ffd0ddd83a0,
> > format=, args=) at
> > failures.c:718
> >      status = 0
> > #11 0x7f73f1723e11 in i_panic (format=0x1310  > bounds>) at failures.c:306
> >      ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0,
> > timestamp_usecs = 0, log_prefix = 0x0}
> >      args = {{gp_offset = 16, fp_offset = 48, overflow_arg_area =
> > 0x7ffd0ddd84a0, reg_save_area = 0x7ffd0ddd83e0}}
> > #12 0x7f73f17ab83a in mem_block_alloc (min_size=512) at
> > data-stack.c:360
> >      block = 
> >      prev_size = 
> >      alloc_size = 134217728
> > #13 0x7f73f17abc18 in t_malloc_real (size=,
> > permanent=false) at data-stack.c:415
> >      block = 
> >      ret = 
> >      alloc_size = 512
> > #14 0x7f73f17abd3b in t_buffer_get (size=512) at data-stack.c:543
> >      ret = 0x0
> > #15 0x7f73f17e1d60 in vstrconcat (str1=0x7f73ebdceeb0
> > "/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe",
> > args=0x7ffd0ddd8550,
> >      ret_len=0x7ffd0ddd8570) at strfuncs.c:183
> >      str = 0x7f73ebdceeb0
> > "/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe"
> >      temp = 
> >      bufsize = 512
> >      i = 
> >      len = 
> >      __func__ = "vstrconcat"
> > #16 0x7f73f17b7d03 in i_strconcat (str1=0x7f73ebdceeb0
> > "/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe")
> > at imem.c:65
> >      temp = 
> >      _data_stack_cur_id = 5
> >      args = {{gp_offset = 8, fp_offset = 48, overflow_arg_area =
> > 0x7ffd0ddd8650, reg_save_area = 0x7ffd0ddd8580}}
> >      ret = 
> >      len = 
> >      __func__ = "i_strconcat"
> > #17 0x7f73f0b1fa39 in acl_backend_vfile_object

Re: Panic: data stack: Out of memory when allocating bytes

2018-01-29 Thread Thomas Robers 

Any idea what the problem could be? Is there anything more i could do
to encircle the problem? Or perhaps is the information i provided
uncomplete?

Am 25.01.2018 um 16:24 schrieb Thomas Robers:

Hi,

Am 24.01.2018 um 23:39 schrieb Josef 'Jeff' Sipek:
It looks like the binaries are stripped.  There should be a "debug" 
package
you can install with symbol information.  Then, the backtrace should 
be much

more helpful.


I installed the debug package and the backtrace now is:

--- snip ---
(gdb) bt full
#0  0x7f73f1386495 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x7f73f1387c75 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x7f73f17ab822 in mem_block_alloc (min_size=520) at 
data-stack.c:356

     block = 
     prev_size = 
     alloc_size = 134217728
#3  0x7f73f17abc18 in t_malloc_real (size=, 
permanent=true) at data-stack.c:415

     block = 
     ret = 
     alloc_size = 520
#4  0x7f73f17abdeb in t_malloc0 (size=513) at data-stack.c:468
     mem = 
#5  0x7f73f17a95dd in p_malloc (buf=0x7f73ebdcef28, size=513) at 
mempool.h:99

No locals.
#6  buffer_alloc (buf=0x7f73ebdcef28, size=513) at buffer.c:34
     __func__ = "buffer_alloc"
#7  0x7f73f17a967b in buffer_create_dynamic (pool=out>, init_size=512) at buffer.c:143

     buf = 0x7f73ebdcef28
#8  0x7f73f17a803b in backtrace_get (backtrace_r=0x7ffd0ddd8358) at 
backtrace-string.c:86

     str = 
#9  0x7f73f17af1da in default_fatal_finish (type=out>, status=0) at failures.c:221

     backtrace = 
     recursed = 1
#10 0x7f73f17af766 in i_internal_fatal_handler (ctx=0x7ffd0ddd83a0, 
format=, args=) at failures.c:718

     status = 0
#11 0x7f73f1723e11 in i_panic (format=0x1310 bounds>) at failures.c:306
     ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, 
timestamp_usecs = 0, log_prefix = 0x0}
     args = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 
0x7ffd0ddd84a0, reg_save_area = 0x7ffd0ddd83e0}}
#12 0x7f73f17ab83a in mem_block_alloc (min_size=512) at 
data-stack.c:360

     block = 
     prev_size = 
     alloc_size = 134217728
#13 0x7f73f17abc18 in t_malloc_real (size=, 
permanent=false) at data-stack.c:415

     block = 
     ret = 
     alloc_size = 512
#14 0x7f73f17abd3b in t_buffer_get (size=512) at data-stack.c:543
     ret = 0x0
#15 0x7f73f17e1d60 in vstrconcat (str1=0x7f73ebdceeb0 
"/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe", 
args=0x7ffd0ddd8550,

     ret_len=0x7ffd0ddd8570) at strfuncs.c:183
     str = 0x7f73ebdceeb0 
"/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe"

     temp = 
     bufsize = 512
     i = 
     len = 
     __func__ = "vstrconcat"
#16 0x7f73f17b7d03 in i_strconcat (str1=0x7f73ebdceeb0 
"/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe") 
at imem.c:65

     temp = 
     _data_stack_cur_id = 5
     args = {{gp_offset = 8, fp_offset = 48, overflow_arg_area = 
0x7ffd0ddd8650, reg_save_area = 0x7ffd0ddd8580}}

     ret = 
     len = 
     __func__ = "i_strconcat"
#17 0x7f73f0b1fa39 in acl_backend_vfile_object_init 
(_backend=0x7f73f255c498, name=0x7f73ebdceda8 "bla_blub.foo_bar.John 
Doe") at acl-backend-vfile.c:169

     _data_stack_cur_id = 4
     backend = 0x7f73f255c498
     aclobj = 0x7f73f25b5420
     dir = 
     vname = 0x7f73ebdcee68 
"shared/b...@tutech.de/bla_blub/foo_bar/John Doe"

     error = 0x0
#18 0x7f73f0b20740 in acllist_append (backend=0x7f73f255c498) at 
acl-backend-vfile-acllist.c:187

     iter = 0x0
     ret = 
     aclobj = 0x0
     rights = {id_type = ACL_ID_USER, identifier = 0x7f73f25c7930 
"be...@tutech.de", rights = 0x7f73f25c7948, neg_rights = 0x0, global = 
false}
     acllist = {mtime = 1512404820, name = 0x7f73f26369b8 
"bla_blub.foo_bar.Some_Text&AOQ-ge"}

     name = 0x7f73ebdceda8 "bla_blub.foo_bar.John Doe"
#19 acl_backend_vfile_acllist_try_rebuild (backend=0x7f73f255c498) at 
acl-backend-vfile-acllist.c:278

     list = 
     ns = 0x7f73f2547fd0
     iter = 0x7f73f25aad48
     type = MAILBOX_LIST_PATH_TYPE_DIR
     info = 
     rootdir = 0x7f73f255bf58 
"/export/home/imap/b...@tutech.de/Maildir"

     acllist_path = 
     output = 0x7f73f27d9990
     st = {st_dev = 140135963676352, st_ino = 140135948493088, 
st_nlink = 8192, st_mode = 333072, st_uid = 0, st_gid = 4068358896, 
__pad0 = 32627, st_rdev = 8192,
   st_size = 140135948493088, st_blksize = 140135945267666, 
st_blocks = 139, st_atim = {tv_sec = 8192, tv_nsec = 8192}, st_mtim = 
{tv_sec = 0, tv_nsec = 34}, st_ctim = {tv_sec = 34,
     tv_nsec = 140135965564672}, __unused = {140135949466998, 
140135966284032, 140135949496358}}
     perm = {file_uid = 1, file_gid = 1, file_create_m

Re: Panic: data stack: Out of memory when allocating bytes

2018-01-25 Thread Thomas Robers 

Hi,

Am 24.01.2018 um 23:39 schrieb Josef 'Jeff' Sipek:

It looks like the binaries are stripped.  There should be a "debug" package
you can install with symbol information.  Then, the backtrace should be much
more helpful.


I installed the debug package and the backtrace now is:

--- snip ---
(gdb) bt full
#0  0x7f73f1386495 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x7f73f1387c75 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x7f73f17ab822 in mem_block_alloc (min_size=520) at data-stack.c:356
block = 
prev_size = 
alloc_size = 134217728
#3  0x7f73f17abc18 in t_malloc_real (size=, 
permanent=true) at data-stack.c:415

block = 
ret = 
alloc_size = 520
#4  0x7f73f17abdeb in t_malloc0 (size=513) at data-stack.c:468
mem = 
#5  0x7f73f17a95dd in p_malloc (buf=0x7f73ebdcef28, size=513) at 
mempool.h:99

No locals.
#6  buffer_alloc (buf=0x7f73ebdcef28, size=513) at buffer.c:34
__func__ = "buffer_alloc"
#7  0x7f73f17a967b in buffer_create_dynamic (pool=out>, init_size=512) at buffer.c:143

buf = 0x7f73ebdcef28
#8  0x7f73f17a803b in backtrace_get (backtrace_r=0x7ffd0ddd8358) at 
backtrace-string.c:86

str = 
#9  0x7f73f17af1da in default_fatal_finish (type=out>, status=0) at failures.c:221

backtrace = 
recursed = 1
#10 0x7f73f17af766 in i_internal_fatal_handler (ctx=0x7ffd0ddd83a0, 
format=, args=) at failures.c:718

status = 0
#11 0x7f73f1723e11 in i_panic (format=0x1310 bounds>) at failures.c:306
ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, 
timestamp_usecs = 0, log_prefix = 0x0}
args = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 
0x7ffd0ddd84a0, reg_save_area = 0x7ffd0ddd83e0}}

#12 0x7f73f17ab83a in mem_block_alloc (min_size=512) at data-stack.c:360
block = 
prev_size = 
alloc_size = 134217728
#13 0x7f73f17abc18 in t_malloc_real (size=, 
permanent=false) at data-stack.c:415

block = 
ret = 
alloc_size = 512
#14 0x7f73f17abd3b in t_buffer_get (size=512) at data-stack.c:543
ret = 0x0
#15 0x7f73f17e1d60 in vstrconcat (str1=0x7f73ebdceeb0 
"/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe", 
args=0x7ffd0ddd8550,

ret_len=0x7ffd0ddd8570) at strfuncs.c:183
str = 0x7f73ebdceeb0 
"/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe"

temp = 
bufsize = 512
i = 
len = 
__func__ = "vstrconcat"
#16 0x7f73f17b7d03 in i_strconcat (str1=0x7f73ebdceeb0 
"/export/home/imap/b...@tutech.de/Maildir/.bla_blub.foo_bar.John Doe") 
at imem.c:65

temp = 
_data_stack_cur_id = 5
args = {{gp_offset = 8, fp_offset = 48, overflow_arg_area = 
0x7ffd0ddd8650, reg_save_area = 0x7ffd0ddd8580}}

ret = 
len = 
__func__ = "i_strconcat"
#17 0x7f73f0b1fa39 in acl_backend_vfile_object_init 
(_backend=0x7f73f255c498, name=0x7f73ebdceda8 "bla_blub.foo_bar.John 
Doe") at acl-backend-vfile.c:169

_data_stack_cur_id = 4
backend = 0x7f73f255c498
aclobj = 0x7f73f25b5420
dir = 
vname = 0x7f73ebdcee68 
"shared/b...@tutech.de/bla_blub/foo_bar/John Doe"

error = 0x0
#18 0x7f73f0b20740 in acllist_append (backend=0x7f73f255c498) at 
acl-backend-vfile-acllist.c:187

iter = 0x0
ret = 
aclobj = 0x0
rights = {id_type = ACL_ID_USER, identifier = 0x7f73f25c7930 
"be...@tutech.de", rights = 0x7f73f25c7948, neg_rights = 0x0, global = 
false}
acllist = {mtime = 1512404820, name = 0x7f73f26369b8 
"bla_blub.foo_bar.Some_Text&AOQ-ge"}

name = 0x7f73ebdceda8 "bla_blub.foo_bar.John Doe"
#19 acl_backend_vfile_acllist_try_rebuild (backend=0x7f73f255c498) at 
acl-backend-vfile-acllist.c:278

list = 
ns = 0x7f73f2547fd0
iter = 0x7f73f25aad48
type = MAILBOX_LIST_PATH_TYPE_DIR
info = 
rootdir = 0x7f73f255bf58 "/export/home/imap/b...@tutech.de/Maildir"
acllist_path = 
output = 0x7f73f27d9990
st = {st_dev = 140135963676352, st_ino = 140135948493088, 
st_nlink = 8192, st_mode = 333072, st_uid = 0, st_gid = 4068358896, 
__pad0 = 32627, st_rdev = 8192,
  st_size = 140135948493088, st_blksize = 140135945267666, 
st_blocks = 139, st_atim = {tv_sec = 8192, tv_nsec = 8192}, st_mtim = 
{tv_sec = 0, tv_nsec = 34}, st_ctim = {tv_sec = 34,
tv_nsec = 140135965564672}, __unused = {140135949466998, 
140135966284032, 140135949496358}}
perm = {file_uid = 1, file_gid = 1, file_create_mode = 
384, dir_create_mode = 448, file_create_gid = 4294967295,
  file_create_gid_origin = 0x7f73f255c3a8 
"/export/home/imap/b...@tutech.de/Maildir", gid_origin_is_mailbox_path = 
false, mail_index_permissions_set = false}

path = 0x7f73eb9a52a

Re: Panic: data stack: Out of memory when allocating bytes

2018-01-24 Thread Josef 'Jeff' Sipek 
On Wed, Jan 24, 2018 at 18:55:47 +0100, Thomas Robers wrote:
> Am 23.01.2018 um 20:07 schrieb Josef 'Jeff' Sipek:
> > On Tue, Jan 23, 2018 at 14:03:27 -0500, Josef 'Jeff' Sipek wrote:
> > > On Tue, Jan 23, 2018 at 18:21:38 +0100, Thomas Robers wrote:
...
> > > 1. Do you have any idea what the imap process was doing at the time of the
> > > allocation failure?
> 
> Yes perhaps. We use shared mailboxes and at the time of failure the
> imap process is reading acl files in a shared mailbox (and subfolder).
> This shared mailbox has about 2800 subfolder and the acl files are read
> in about 10sec and and during this reading the imap process dies with
> the already mentioned error. It seems it has something to do with the
> shared mailbox, because since yesterday morning 5 core dumps have been
> created, 4 of them by one user accessing the shared mailbox and 1
> by the user who is the owner of the shared mailbox. No other mailboxes
> are affected until now.

Based on the stack trace, the client was creating a new mailbox, which
caused an acl list rebuild, which ended up triggering this oddly sized
allocation.

> > > 2. You snipped all the important parts of the back trace. :)  It *starts* 
> > > on
> > > the line:
> > >   #0  0x7f73f1386495 in raise () from /lib64/libc.so.6
> > 
> > In case you haven't used gdb before...  after starting up gdb, run "bt full"
> > at the gdb prompt.  That'll print out a very detailed backtrace.  (You might
> > want to sanity check it to make sure there aren't any user passwords in it
> > before posting it here...)
> 
> Yes, sorry I'm not very familiar in using gdb, but here is the full
> backtrace:

It looks like the binaries are stripped.  There should be a "debug" package
you can install with symbol information.  Then, the backtrace should be much
more helpful.

> > > Having the backtrace should help answer question number 1.
> > > 3. How big is this process when it dies?
> 
> I don't know which imap process dies beforehand so how do i get this
> information?

The size of the core file will give you an general idea.  gdb can also print
out lots of info via "maintenance info sections", but I don't think that'll
help figuring out why things blow up.

Jeff.

-- 
Keyboard not found!
Press F1 to enter Setup

Re: Panic: data stack: Out of memory when allocating bytes

2018-01-24 Thread Thomas Robers 

Hi,

Am 23.01.2018 um 20:07 schrieb Josef 'Jeff' Sipek:

On Tue, Jan 23, 2018 at 14:03:27 -0500, Josef 'Jeff' Sipek wrote:

On Tue, Jan 23, 2018 at 18:21:38 +0100, Thomas Robers wrote:

Hello,

I'm using Dovecot 2.3 and sometimes i get this:

--- snip ---
Jan 23 14:23:13 mail dovecot: imap(b...@tutech.de)<4880>:
Panic: data stack: Out of memory when allocating 134217768 bytes


Interesting... imap is trying to allocate 128MB and failing.  A couple of
questions:

0. Does this user have any unusually large emails?


No, not usually but there are some mails which are larger than 15mb.
But that's not the normal size. Most e-mail are between some kb up to
5mb.



1. Do you have any idea what the imap process was doing at the time of the
allocation failure?


Yes perhaps. We use shared mailboxes and at the time of failure the
imap process is reading acl files in a shared mailbox (and subfolder).
This shared mailbox has about 2800 subfolder and the acl files are read
in about 10sec and and during this reading the imap process dies with
the already mentioned error. It seems it has something to do with the
shared mailbox, because since yesterday morning 5 core dumps have been
created, 4 of them by one user accessing the shared mailbox and 1
by the user who is the owner of the shared mailbox. No other mailboxes
are affected until now.


2. You snipped all the important parts of the back trace. :)  It *starts* on
the line:
#0  0x7f73f1386495 in raise () from /lib64/libc.so.6


In case you haven't used gdb before...  after starting up gdb, run "bt full"
at the gdb prompt.  That'll print out a very detailed backtrace.  (You might
want to sanity check it to make sure there aren't any user passwords in it
before posting it here...)


Yes, sorry I'm not very familiar in using gdb, but here is the full
backtrace:

--- snip ---
(gdb) bt full
#0  0x7f73f1386495 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x7f73f1387c75 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x7f73f17ab822 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#3  0x7f73f17abc18 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#4  0x7f73f17abdeb in t_malloc0 () from 
/usr/lib64/dovecot/libdovecot.so.0

No symbol table info available.
#5  0x7f73f17a95dd in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#6  0x7f73f17a967b in buffer_create_dynamic () from 
/usr/lib64/dovecot/libdovecot.so.0

No symbol table info available.
#7  0x7f73f17a803b in backtrace_get () from 
/usr/lib64/dovecot/libdovecot.so.0

No symbol table info available.
#8  0x7f73f17af1da in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#9  0x7f73f17af766 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#10 0x7f73f1723e11 in i_panic () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#11 0x7f73f17ab83a in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#12 0x7f73f17abc18 in ?? () from /usr/lib64/dovecot/libdovecot.so.0
No symbol table info available.
#13 0x7f73f17abd3b in t_buffer_get () from 
/usr/lib64/dovecot/libdovecot.so.0

No symbol table info available.
#14 0x7f73f17e1d60 in vstrconcat () from 
/usr/lib64/dovecot/libdovecot.so.0

No symbol table info available.
#15 0x7f73f17b7d03 in i_strconcat () from 
/usr/lib64/dovecot/libdovecot.so.0

No symbol table info available.
#16 0x7f73f0b1fa39 in ?? () from /usr/lib64/dovecot/lib01_acl_plugin.so
No symbol table info available.
#17 0x7f73f0b20740 in ?? () from /usr/lib64/dovecot/lib01_acl_plugin.so
No symbol table info available.
#18 0x7f73f0b20b7d in acl_backend_vfile_acllist_rebuild () from 
/usr/lib64/dovecot/lib01_acl_plugin.so

No symbol table info available.
#19 0x7f73f0b21569 in acl_backend_vfile_object_update () from 
/usr/lib64/dovecot/lib01_acl_plugin.so

No symbol table info available.
#20 0x7f73f0b24bd8 in ?? () from /usr/lib64/dovecot/lib01_acl_plugin.so
No symbol table info available.
#21 0x7f73f1aa1973 in mailbox_create () from 
/usr/lib64/dovecot/libdovecot-storage.so.0

No symbol table info available.
#22 0x7f73f1fd1654 in cmd_create ()
No symbol table info available.
#23 0x7f73f1fde585 in command_exec ()
No symbol table info available.
#24 0x7f73f1fdb7b0 in ?? ()
No symbol table info available.
#25 0x7f73f1fdb848 in ?? ()
No symbol table info available.
#26 0x7f73f1fdbc35 in client_handle_input ()
No symbol table info available.
#27 0x7f73f1fdc17e in client_input ()
No symbol table info available.
#28 0x7f73f17c5ec5 in io_loop_call_io () from 
/usr/lib64/dovecot/libdovecot.so.0

No symbol table info available.
#29 0x7f73f17c7dcf in io_loop_handler_run_internal () from 
/usr/lib64/dovecot/libdovecot.so.0

No symbol table info availabl

Re: Panic: data stack: Out of memory when allocating bytes

2018-01-23 Thread Josef 'Jeff' Sipek 
On Tue, Jan 23, 2018 at 14:03:27 -0500, Josef 'Jeff' Sipek wrote:
> On Tue, Jan 23, 2018 at 18:21:38 +0100, Thomas Robers wrote:
> > Hello,
> > 
> > I'm using Dovecot 2.3 and sometimes i get this:
> > 
> > --- snip ---
> > Jan 23 14:23:13 mail dovecot: imap(b...@tutech.de)<4880>:
> > Panic: data stack: Out of memory when allocating 134217768 bytes
> 
> Interesting... imap is trying to allocate 128MB and failing.  A couple of
> questions:
> 
> 0. Does this user have any unusually large emails?
> 1. Do you have any idea what the imap process was doing at the time of the
>allocation failure?
> 2. You snipped all the important parts of the back trace. :)  It *starts* on
>the line:
>   #0  0x7f73f1386495 in raise () from /lib64/libc.so.6

In case you haven't used gdb before...  after starting up gdb, run "bt full"
at the gdb prompt.  That'll print out a very detailed backtrace.  (You might
want to sanity check it to make sure there aren't any user passwords in it
before posting it here...)

Jeff.

>Having the backtrace should help answer question number 1.
> 3. How big is this process when it dies?
> 4. Do you have any sort of ulimit/rlimit set on dovecot in whatever startup
>script you use?
> 5. Do you use cgroups to limit memory usage?
> 6. Did you disable memory overcommit on the system?
> 
> Jeff.
> 
> > Jan 23 14:23:13 mail dovecot: imap(b...@tutech.de)<4880>:
> > Fatal: master: service(imap): child 4880 killed with signal 6 (core dumped)
> > --- snip ---
> > 
> > The gdb backtrace is:
> > 
> > --- snip ---
> > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
> > Copyright (C) 2010 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > 
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-redhat-linux-gnu".
> > For bug reporting instructions, please see:
> > ...
> > Reading symbols from /usr/libexec/dovecot/imap...(no debugging symbols
> > found)...done.
> > Attaching to program: /usr/libexec/dovecot/imap, process 4880
> > ptrace: Kein passender Prozess gefunden.
> > [New Thread 4880]
> > Reading symbols from /usr/lib64/dovecot/libdovecot-storage.so.0...(no
> > debugging symbols found)...done.
> > Loaded symbols for /usr/lib64/dovecot/libdovecot-storage.so.0
> > Reading symbols from /usr/lib64/dovecot/libdovecot.so.0...(no debugging
> > symbols found)...done.
> > Loaded symbols for /usr/lib64/dovecot/libdovecot.so.0
> > Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
> > Loaded symbols for /lib64/libc.so.6
> > Reading symbols from /lib64/librt.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/librt.so.1
> > Reading symbols from /lib64/libdl.so.2...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libdl.so.2
> > Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/ld-linux-x86-64.so.2
> > Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> > found)...done.
> > [Thread debugging using libthread_db enabled]
> > Loaded symbols for /lib64/libpthread.so.0
> > Reading symbols from /usr/lib64/dovecot/lib01_acl_plugin.so...(no debugging
> > symbols found)...done.
> > Loaded symbols for /usr/lib64/dovecot/lib01_acl_plugin.so
> > Reading symbols from /usr/lib64/dovecot/lib02_imap_acl_plugin.so...(no
> > debugging symbols found)...done.
> > Loaded symbols for /usr/lib64/dovecot/lib02_imap_acl_plugin.so
> > Reading symbols from /usr/lib64/dovecot/lib15_notify_plugin.so...(no
> > debugging symbols found)...done.
> > Loaded symbols for /usr/lib64/dovecot/lib15_notify_plugin.so
> > Reading symbols from /usr/lib64/dovecot/lib20_mail_log_plugin.so...(no
> > debugging symbols found)...done.
> > Loaded symbols for /usr/lib64/dovecot/lib20_mail_log_plugin.so
> > Reading symbols from /usr/lib64/dovecot/lib20_zlib_plugin.so...(no debugging
> > symbols found)...done.
> > Loaded symbols for /usr/lib64/dovecot/lib20_zlib_plugin.so
> > Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
> > Loaded symbols for /lib64/libz.so.1
> > Reading symbols from /lib64/libbz2.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib64/libbz2.so.1
> > Reading symbols from /usr/lib64/dovecot/lib30_imap_zlib_plugin.so...(no
> > debugging symbols found)...done.
> > Loaded symbols for /usr/lib64/dovecot/lib30_imap_zlib_plugin.so
> > Core was generated by `dovecot/imap'.
> > Program terminated with signal 6, Aborted.
> > #0  0x7f73f1386495 in raise () from /lib64/libc.so.6
> > --- snip ---
> > 
> > I searched for that error message but only found some entries regarding
> > older dovecot versions and setting "mail_process_size" but i couldn't
> > fi

Re: Panic: data stack: Out of memory when allocating bytes

2018-01-23 Thread Josef 'Jeff' Sipek 
On Tue, Jan 23, 2018 at 18:21:38 +0100, Thomas Robers wrote:
> Hello,
> 
> I'm using Dovecot 2.3 and sometimes i get this:
> 
> --- snip ---
> Jan 23 14:23:13 mail dovecot: imap(b...@tutech.de)<4880>:
> Panic: data stack: Out of memory when allocating 134217768 bytes

Interesting... imap is trying to allocate 128MB and failing.  A couple of
questions:

0. Does this user have any unusually large emails?
1. Do you have any idea what the imap process was doing at the time of the
   allocation failure?
2. You snipped all the important parts of the back trace. :)  It *starts* on
   the line:
#0  0x7f73f1386495 in raise () from /lib64/libc.so.6
   Having the backtrace should help answer question number 1.
3. How big is this process when it dies?
4. Do you have any sort of ulimit/rlimit set on dovecot in whatever startup
   script you use?
5. Do you use cgroups to limit memory usage?
6. Did you disable memory overcommit on the system?

Jeff.

> Jan 23 14:23:13 mail dovecot: imap(b...@tutech.de)<4880>:
> Fatal: master: service(imap): child 4880 killed with signal 6 (core dumped)
> --- snip ---
> 
> The gdb backtrace is:
> 
> --- snip ---
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> ...
> Reading symbols from /usr/libexec/dovecot/imap...(no debugging symbols
> found)...done.
> Attaching to program: /usr/libexec/dovecot/imap, process 4880
> ptrace: Kein passender Prozess gefunden.
> [New Thread 4880]
> Reading symbols from /usr/lib64/dovecot/libdovecot-storage.so.0...(no
> debugging symbols found)...done.
> Loaded symbols for /usr/lib64/dovecot/libdovecot-storage.so.0
> Reading symbols from /usr/lib64/dovecot/libdovecot.so.0...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/dovecot/libdovecot.so.0
> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/librt.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libdl.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> found)...done.
> [Thread debugging using libthread_db enabled]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /usr/lib64/dovecot/lib01_acl_plugin.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/dovecot/lib01_acl_plugin.so
> Reading symbols from /usr/lib64/dovecot/lib02_imap_acl_plugin.so...(no
> debugging symbols found)...done.
> Loaded symbols for /usr/lib64/dovecot/lib02_imap_acl_plugin.so
> Reading symbols from /usr/lib64/dovecot/lib15_notify_plugin.so...(no
> debugging symbols found)...done.
> Loaded symbols for /usr/lib64/dovecot/lib15_notify_plugin.so
> Reading symbols from /usr/lib64/dovecot/lib20_mail_log_plugin.so...(no
> debugging symbols found)...done.
> Loaded symbols for /usr/lib64/dovecot/lib20_mail_log_plugin.so
> Reading symbols from /usr/lib64/dovecot/lib20_zlib_plugin.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/dovecot/lib20_zlib_plugin.so
> Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libz.so.1
> Reading symbols from /lib64/libbz2.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libbz2.so.1
> Reading symbols from /usr/lib64/dovecot/lib30_imap_zlib_plugin.so...(no
> debugging symbols found)...done.
> Loaded symbols for /usr/lib64/dovecot/lib30_imap_zlib_plugin.so
> Core was generated by `dovecot/imap'.
> Program terminated with signal 6, Aborted.
> #0  0x7f73f1386495 in raise () from /lib64/libc.so.6
> --- snip ---
> 
> I searched for that error message but only found some entries regarding
> older dovecot versions and setting "mail_process_size" but i couldn't
> find anything regarding Dovecot Version 2.3. Maybe it's something that is
> fixed already but if so i can't find it. Or is this a configuration issue or
> a bug?
> 
> Here my Dovecot configuration:
> 
> --- snip ---
> # 2.3.0 (c8b89eb): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.0.1 (d33dca20)
> # OS: Linux 2.6.32-696.3.1.el6.x86_64 x86_64 CentOS release 6.9 (Final) ext4
> auth_debug = yes
> auth_debug_passwords = yes
> auth_master_user_separator = *
> auth_mechanisms = plain login
> auth_verbose = yes
> disable_plain