A NOTE has been added to this issue. ====================================================================== http://www.dbmail.org/mantis/view.php?id=433 ====================================================================== Reported By: rbutler Assigned To: paul ====================================================================== Project: DBMail Issue ID: 433 Category: Command-Line programs (dbmail-users, dbmail-util) Reproducibility: always Severity: major Priority: normal Status: feedback target: ====================================================================== Date Submitted: 03-Nov-06 21:20 CET Last Modified: 08-Nov-06 15:12 CET ====================================================================== Summary: converting from 2.0->Trunk causes glib allocation error Description: When running dbmail-util -by, during the envelope cache creation:
GLib-ERROR **: gmem.c:172: failed to allocate 536870912 bytes top doesn't show the system especially low on swap at the time this happens... Repairing DBMAIL for cached envelopes... Ok. Found [120614] missing envelope values. GLib-ERROR **: gmem.c:172: failed to allocate 536870912 bytes aborting... Aborted System has 512 meg of RAM and 1 gig of swap. ====================================================================== ---------------------------------------------------------------------- aaron - 03-Nov-06 21:36 ---------------------------------------------------------------------- Even if you had a terabyte of RAM to allocate, you shouldn't be getting messages with 536 megabyte envelopes! This is probably an uninitialized value bug. As for narrowing down the revision you're seeing this in, please don't use trunk, use dbmail_2_2_branch. Please post the SVN rev you're at, too. ---------------------------------------------------------------------- paul - 04-Nov-06 09:08 ---------------------------------------------------------------------- Ryan, I can't reproduce this on the 2.2 branch. I've just reinitialized my envelopes table on one of my production servers with 500MB ram and 170k messages. ---------------------------------------------------------------------- rbutler - 06-Nov-06 21:38 ---------------------------------------------------------------------- Path: . URL: https://svn.ic-s.nl/svn/dbmail/branches/dbmail_2_2_branch Repository Root: https://svn.ic-s.nl/svn/dbmail Repository UUID: 7b491191-dbf0-0310-aff6-d879d4d69008 Revision: 2357 Node Kind: directory Schedule: normal Last Changed Author: aaron Last Changed Rev: 2356 Last Changed Date: 2006-11-05 10:43:48 -0600 (Sun, 05 Nov 2006) Problem still occurs with the same message. Below is the versions of packages involved as far as glib goes. ii dbus-glib-1 0.23.4-8 simple interprocess messaging system (GLib-b ii libdbus-glib-1-2 0.71-2 simple interprocess messaging system (GLib-b ii libglib1.2 1.2.10-14 The GLib library of C routines ii libglib2.0-0 2.12.4-1 The GLib library of C routines ii libglib2.0-data 2.12.4-1 Common files for GLib library ii libglib2.0-dev 2.12.4-1 Development files for the GLib library I have approximately 240k messages I believe and when I rerun dbmail-util -by after a failure I have around 114,000 left. It may be a specific message that it doesn't like as it errors out without printing any dots on subsequent tries but does print a lot of dots on the original database first. However, it shouldn't choke on any messages (and flushing my message store is not an option, if there's some debug information to tell me which message in the db it is, I might be able to toss that message). ---------------------------------------------------------------------- paul - 06-Nov-06 22:42 ---------------------------------------------------------------------- Could you please test with the patch I'm uploading? It should bail-out with a more descriptive message in the log. ---------------------------------------------------------------------- rbutler - 06-Nov-06 23:24 ---------------------------------------------------------------------- I'll give it a try tomorrow, I reverted to the previous database for the night. ---------------------------------------------------------------------- rbutler - 07-Nov-06 20:48 ---------------------------------------------------------------------- Gave the glib error again, did not print the additional trace detail anywhere. I was running it in gdb and here is the backtrace: (gdb) bt http://www.dbmail.org/mantis/view.php?id=0 0xb7c5f947 in raise () from /lib/tls/libc.so.6 http://www.dbmail.org/mantis/view.php?id=1 0xb7c610c9 in abort () from /lib/tls/libc.so.6 http://www.dbmail.org/mantis/view.php?id=2 0xb7de3074 in g_logv () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libglib-2.0.so.0 http://www.dbmail.org/mantis/view.php?id=3 0xb7de30a9 in g_log () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libglib-2.0.so.0 http://www.dbmail.org/mantis/view.php?id=4 0xb7de1bcf in g_realloc () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libglib-2.0.so.0 http://www.dbmail.org/mantis/view.php?id=5 0xb7dbcd14 in g_ptr_array_new () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libglib-2.0.so.0 http://www.dbmail.org/mantis/view.php?id=6 0xb7dbd0a4 in g_array_set_size () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libglib-2.0.so.0 http://www.dbmail.org/mantis/view.php?id=7 0xb7dbd149 in g_byte_array_set_size () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libglib-2.0.so.0 http://www.dbmail.org/mantis/view.php?id=8 0xb7ed28c9 in g_mime_stream_mem_new () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libgmime-2.0.so.2 http://www.dbmail.org/mantis/view.php?id=9 0xb7ece5b3 in g_mime_stream_write () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libgmime-2.0.so.2 http://www.dbmail.org/mantis/view.php?id=10 0xb7ed1444 in g_mime_stream_filter_new_with_stream () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libgmime-2.0.so.2 http://www.dbmail.org/mantis/view.php?id=11 0xb7ece5b3 in g_mime_stream_write () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libgmime-2.0.so.2 http://www.dbmail.org/mantis/view.php?id=12 0xb7ece7ef in g_mime_stream_write_string () from /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../libgmime-2.0.so.2 http://www.dbmail.org/mantis/view.php?id=13 0xb7d76f0a in _set_content_from_stream (self=0x805a7b0, stream=<value optimized out>, type=DBMAIL_STREAM_PIPE) at dbmail-message.c:446 http://www.dbmail.org/mantis/view.php?id=14 0xb7d776b0 in _set_content (self=0x805a7b0, content=<value optimized out>) at dbmail-message.c:395 http://www.dbmail.org/mantis/view.php?id=15 0xb7d776f7 in dbmail_message_init_with_string (self=0x805a7b0, content=0x8195fa0) at dbmail-message.c:254 http://www.dbmail.org/mantis/view.php?id=16 0xb7d7785b in _retrieve (self=0x805a7b0, query_template=<value optimized out>) at dbmail-message.c:738 http://www.dbmail.org/mantis/view.php?id=17 0xb7d779bd in dbmail_message_retrieve (self=0x805a7b0, physid=791288, filter=1) at dbmail-message.c:773 http://www.dbmail.org/mantis/view.php?id=18 0xb7d827fb in db_set_envelope (lost=0x8ad4b70) at db.c:1701 http://www.dbmail.org/mantis/view.php?id=19 0x0804999b in do_header_cache () at maintenance.c:700 http://www.dbmail.org/mantis/view.php?id=20 0x0804adfe in main (argc=Cannot access memory at address 0x4da9 ) at maintenance.c:250 Frame http://www.dbmail.org/mantis/view.php?id=17 gives a physid of 791288, that message is a 294 meg message sent from Outlook containing many many high resolution photos. ---------------------------------------------------------------------- rbutler - 08-Nov-06 04:47 ---------------------------------------------------------------------- After I deleted that large message, it did eventually finish processing. It did also have a couple of random segfaults. I now have 2.2.x running and the speedup is impressive. I kept my backup of the original database with the large message and would be willing to test any fix for the large message issue. ---------------------------------------------------------------------- aaron - 08-Nov-06 10:23 ---------------------------------------------------------------------- Wait, you actually had a 500MB message? And we barfed on it? Shoot. Good news is that this isn't a general showstopper for 2.2.0. Bad news is that there should be a better way to handle this situation and we haven't found it. ---------------------------------------------------------------------- paul - 08-Nov-06 10:57 ---------------------------------------------------------------------- We can handle this: glib2.8 offers g_try_malloc and friends which will allow us to catch allocation errors. We only need to load the whole message into memory to deal with the envelope cache. We can always provide a dummy envelope for messages that fail (just like uw-imapd does). ---------------------------------------------------------------------- rbutler - 08-Nov-06 15:12 ---------------------------------------------------------------------- Aaron, The message was actually (according to the rfcsize in the table) closer to 294 meg. I'm not so sure I agree that it's not a blocker for 2.2.0 as there's bound to be a lot of people that go from 2.0->2.2 when the "stable" version of it comes out. I can't believe it is just me that has large messages in the store. Issue History Date Modified Username Field Change ====================================================================== 03-Nov-06 21:20 rbutler New Issue 03-Nov-06 21:36 aaron Note Added: 0001521 04-Nov-06 09:08 paul Note Added: 0001527 04-Nov-06 09:08 paul Assigned To => paul 04-Nov-06 09:08 paul Status new => assigned 04-Nov-06 09:08 paul Fixed in Version => 2.2 branch 04-Nov-06 09:08 paul Status assigned => closed 04-Nov-06 09:08 paul Resolution open => unable to reproduce 06-Nov-06 21:38 rbutler Status closed => feedback 06-Nov-06 21:38 rbutler Resolution unable to reproduce => reopened 06-Nov-06 21:38 rbutler Note Added: 0001541 06-Nov-06 22:42 paul File Added: try.diff 06-Nov-06 22:42 paul Note Added: 0001542 06-Nov-06 23:24 rbutler Note Added: 0001546 07-Nov-06 20:48 rbutler Note Added: 0001558 08-Nov-06 04:47 rbutler Note Added: 0001559 08-Nov-06 10:23 aaron Note Added: 0001560 08-Nov-06 10:57 paul Note Added: 0001562 08-Nov-06 15:12 rbutler Note Added: 0001563 ======================================================================
