Re: [Dbmail] OT: DBMail Administrator (DBMA) Performance Fix
All,
Been a long time since I posted anything but I wanted to give some number
for the group as well.
Just an FYI it also maybe hardware and version dependent, that is to say
that the performance measurements are not cpu bound therefor it may be IO
that is saturated which will happen on either mysql or postgresql but as
show below on MySQL 5.0.67-community on my customer's production dbmail
cluster sql server
Dell 6850 quad 3.2Ghz cpus
2.6.18-92.1.18.el5.centos.plus #1 SMP Wed Nov 26 07:28:20 EST 2008 x86_64
x86_64 x86_64 GNU/Linux
CentOS 5.2
16G ram
Dell MD3000 sas raid5 array 200G storage over 13 drives at 15k rpm using
LVM2 and ext3
InnoDB, binary logging turned on
mysql SELECT COUNT(1) FROM dbmail_messageblks;
+--+
| COUNT(1) |
+--+
| 997728 |
+--+
1 row in set (0.58 sec)
mysql SELECT COUNT(1) FROM dbmail_messageblks;
+--+
| COUNT(1) |
+--+
| 997728 |
+--+
1 row in set (0.00 sec)
mysql SELECT COUNT(*) FROM dbmail_messageblks;
+--+
| COUNT(*) |
+--+
| 997728 |
+--+
1 row in set (0.57 sec)
mysql SELECT COUNT(*) FROM dbmail_messageblks;
+--+
| COUNT(*) |
+--+
| 997728 |
+--+
1 row in set (0.00 sec)
However it should be said that this box is very large and the customer never
fully utilized it so it is unloaded being Friday night at 9pm and he has
business customers but still it shows the time to load a similar dataset's
indexs into memory where mysql innodb has 8G of ram dedicated to it.
I would say that untill dbmail sql processes are CPU bound these kinds of
discussions on performance are not the most critical.
The slow query log on this box shows stuff like (taken with the fact that
this schema has been thru upgrades and maybe missing something, but I am
fairly certian it is up to date with indexs):
# Time: 090402 4:02:33
# u...@host: mail[mail] @ dbmail00 [10.1.1.204]
# Query_time: 3 Lock_time: 0 Rows_sent: 0 Rows_examined: 493594
SELECT p.id FROM dbmail_physmessage p LEFT JOIN dbmail_headervalue h
ON p.id= h.physmessage_id WHERE h.physmessage_id IS NULL;
# Time: 090403 4:02:24
# u...@host: mail[mail] @ dbmail00 [10.1.1.204]
# Query_time: 4 Lock_time: 0 Rows_sent: 0 Rows_examined: 994156
SELECT MIN(messageblk_idnr),MAX(is_header) FROM dbmail_messageblks GROUP BY
physmessage_id HAVING MAX(is_header)=0;
# Time: 090403 4:02:33
# u...@host: mail[mail] @ dbmail00 [10.1.1.204]
# Query_time: 5 Lock_time: 0 Rows_sent: 1 Rows_examined: 1022067
SELECT usr.user_idnr, sum(pm.messagesize), usr.curmail_size FROM
dbmail_users usr LEFT JOIN dbmail_mailboxes mbx ON mbx.owner_idnr =
usr.user_idnr LEFT JOIN dbmail_messages msg ON msg.mailbox_idnr =
mbx.mailbox_idnr LEFT JOIN dbmail_physmessage pm ON pm.id =
msg.physmessage_id AND msg.status 2 GROUP BY usr.user_idnr,
usr.curmail_size HAVING ((SUM(pm.messagesize) usr.curmail_size) OR (NOT
(SUM(pm.messagesize) IS NOT NULL) AND usr.curmail_size 0));
# Time: 090403 4:02:37
# u...@host: mail[mail] @ dbmail00 [10.1.1.204]
# Query_time: 4 Lock_time: 0 Rows_sent: 0 Rows_examined: 1679722
SET timestamp=1238745757;
SELECT message_idnr FROM dbmail_messages m JOIN dbmail_physmessage p ON
m.physmessage_id=p.id JOIN dbmail_headervalue v ON
v.physmessage_id=p.idJOIN dbmail_headername n ON v.headername_id=
n.id WHERE m.mailbox_idnr=1816 AND n.headername IN
('resent-message-id','message-id') AND
v.headervalue='20090403080233.c12443b...@dbmail00.xxx' AND p.internal_date
NOW() - INTERVAL 3 DAY;
the last one seems to be the common but these three are consistant this may
be a better place to look in opimizations, I was under the impression that
any time you use a HAVING clause it will cause a table scan?
Also for interest of comparison to the explain details below:
mysql EXPLAIN EXTENDED SELECT COUNT(1) FROM dbmail_messageblks\G
*** 1. row ***
id: 1
select_type: SIMPLE
table: dbmail_messageblks
type: index
possible_keys: NULL
key: PRIMARY
key_len: 8
ref: NULL
rows: 5428069
Extra: Using index
1 row in set, 1 warning (0.00 sec)
mysql EXPLAIN EXTENDED SELECT COUNT(*) FROM dbmail_messageblks\G
*** 1. row ***
id: 1
select_type: SIMPLE
table: dbmail_messageblks
type: index
possible_keys: NULL
key: PRIMARY
key_len: 8
ref: NULL
rows: 5428069
Extra: Using index
1 row in set, 1 warning (0.00 sec)
On Fri, Apr 3, 2009 at 2:04 PM, Wallace Tan wa...@pacific.net.sg wrote:
Michael Monnerie wrote:
On Freitag 03 April 2009 Wallace Tan wrote:
select count(1) from t1;
That would have been my next question. I've spoken once to Paul, because
dbmail uses lots of count(*), but PostgreSQL optimizes this out. Now it
seems MySQL would have a performance boost using count(1).
Could you please try:
1) first, SELECT
Re: [Dbmail-dev] dbmail-imapd memory leak
I belive I have found the memleak for Peters issue:
Index: dbmail-mailbox.c
===
--- dbmail-mailbox.c(revision 2574)
+++ dbmail-mailbox.c(working copy)
@@ -1312,7 +1312,7 @@
GTree * dbmail_mailbox_get_set(struct DbmailMailbox *self, const char
*set, gboolean uid)
{
- GList *ids = NULL, *sets = NULL;
+ GList *topids, *ids = NULL, *sets = NULL;
GString *t;
char *rest;
u64_t i, l, r, lo = 0, hi = 0;
@@ -1329,12 +1329,12 @@
TRACE(TRACE_DEBUG,[%s] uid [%d], set, uid);
if (uid) {
- ids = g_tree_keys(self-ids);
- ids = g_list_last(ids);
+ topids = g_tree_keys(self-ids);
+ ids = g_list_last(topids);
hi = *((u64_t *)ids-data);
- ids = g_list_first(ids);
+ ids = g_list_first(topids);
lo = *((u64_t *)ids-data);
- g_list_free(ids);
+ g_list_free(topids);
} else {
lo = 1;
hi = g_tree_nnodes(self-ids);
It is so small a leak it was not notices but peter's imapsync is doing a
uid ic_fetch on hundreds of ids per fetch command over and over so the
glist of ids leak would build and build on our servers but on his it
goes nuts! :)
Thanks,
Leif
p.s. Peter I am having trouble getting your test script to run on my dev
box please try the change above and let me/ the group know.
Peter Rabbitson wrote:
Paul J Stevens wrote:
==1631== 71,603,456 bytes in 288,752 blocks are still reachable in loss
record 26 of 26
This could be a side effect of the slab allocator. Use
G_DEBUG=always_malloc valgrind ... to circumvent the glib default
allocator.
Tested with 'export G_DEBUG=always_malloc' yielding nearly identical
results (71,604,456 bytes this time). New logs at
http://rabbit.us/pool/misc/dbmail_bug584_memcheck_svn2574_always_malloc.tar.bz2
This URL has some interesting points
http://www.tinymail.org/trac/tinymail/wiki/MemoryStats
Erm, being a n00b and all, wouldn't this affect both imap daemons
involved in the test? In my case only 'host2' leaks, 'host1' stabilizing
at about 40M VSZ.
Peter
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail-imapd memory leak
cool! But rev 2579 doesn't seem to include this particular issue as well
as what may be one with dbmail_imap_session_bodyfetch_rewind which I
don't have a quick answer as to how much of an issue it is?
-leif
Paul J Stevens wrote:
Leif Jackson wrote:
I belive I have found the memleak for Peters issue:
It's one instance of a whole class of leakage we uncovered here over the
last few days.
Before I did:
g_list_foreach(somelist, (GFunc)g_free, NULL);
g_list_free(somelist);
wheras we should always feed the head of the list to such calls:
somelist = g_list_first(somelist);
g_list_foreach(somelist, (GFunc)g_free, NULL);
somelist = g_list_first(somelist);
g_list_free(somelist);
which can be optimized by storing a reference to the start of the list
like you do below.
I've just committed a patch that adds the rewind to all instances of
freeing lists.
Index: dbmail-mailbox.c
===
--- dbmail-mailbox.c(revision 2574)
+++ dbmail-mailbox.c(working copy)
@@ -1312,7 +1312,7 @@
GTree * dbmail_mailbox_get_set(struct DbmailMailbox *self, const char
*set, gboolean uid)
{
- GList *ids = NULL, *sets = NULL;
+ GList *topids, *ids = NULL, *sets = NULL;
GString *t;
char *rest;
u64_t i, l, r, lo = 0, hi = 0;
@@ -1329,12 +1329,12 @@
TRACE(TRACE_DEBUG,[%s] uid [%d], set, uid);
if (uid) {
- ids = g_tree_keys(self-ids);
- ids = g_list_last(ids);
+ topids = g_tree_keys(self-ids);
+ ids = g_list_last(topids);
hi = *((u64_t *)ids-data);
- ids = g_list_first(ids);
+ ids = g_list_first(topids);
lo = *((u64_t *)ids-data);
- g_list_free(ids);
+ g_list_free(topids);
} else {
lo = 1;
hi = g_tree_nnodes(self-ids);
It is so small a leak it was not notices but peter's imapsync is doing a
uid ic_fetch on hundreds of ids per fetch command over and over so the
glist of ids leak would build and build on our servers but on his it
goes nuts! :)
Thanks,
Leif
p.s. Peter I am having trouble getting your test script to run on my dev
box please try the change above and let me/ the group know.
Peter Rabbitson wrote:
Paul J Stevens wrote:
==1631== 71,603,456 bytes in 288,752 blocks are still reachable in loss
record 26 of 26
This could be a side effect of the slab allocator. Use
G_DEBUG=always_malloc valgrind ... to circumvent the glib default
allocator.
Tested with 'export G_DEBUG=always_malloc' yielding nearly identical
results (71,604,456 bytes this time). New logs at
http://rabbit.us/pool/misc/dbmail_bug584_memcheck_svn2574_always_malloc.tar.bz2
This URL has some interesting points
http://www.tinymail.org/trac/tinymail/wiki/MemoryStats
Erm, being a n00b and all, wouldn't this affect both imap daemons
involved in the test? In my case only 'host2' leaks, 'host1' stabilizing
at about 40M VSZ.
Peter
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail-imapd memory leak re: g_mime_init rev 2579
Paul,
I am not sure that the g_mime_init and shutdown moving to the session
is a good idea? Once/ If dbmail ever goes threaded this won't work as it
would only need it in the main function and since g_mime uses a global
object can we assume that each session will be only on one forked imapd
instance? If not I supsect g_mime_init and shutdown in the session will
cause untold problems.
Just my .02
Thanks,
Leif
Paul J Stevens wrote:
Leif Jackson wrote:
I belive I have found the memleak for Peters issue:
It's one instance of a whole class of leakage we uncovered here over the
last few days.
Before I did:
g_list_foreach(somelist, (GFunc)g_free, NULL);
g_list_free(somelist);
wheras we should always feed the head of the list to such calls:
somelist = g_list_first(somelist);
g_list_foreach(somelist, (GFunc)g_free, NULL);
somelist = g_list_first(somelist);
g_list_free(somelist);
which can be optimized by storing a reference to the start of the list
like you do below.
I've just committed a patch that adds the rewind to all instances of
freeing lists.
Index: dbmail-mailbox.c
===
--- dbmail-mailbox.c(revision 2574)
+++ dbmail-mailbox.c(working copy)
@@ -1312,7 +1312,7 @@
GTree * dbmail_mailbox_get_set(struct DbmailMailbox *self, const char
*set, gboolean uid)
{
- GList *ids = NULL, *sets = NULL;
+ GList *topids, *ids = NULL, *sets = NULL;
GString *t;
char *rest;
u64_t i, l, r, lo = 0, hi = 0;
@@ -1329,12 +1329,12 @@
TRACE(TRACE_DEBUG,[%s] uid [%d], set, uid);
if (uid) {
- ids = g_tree_keys(self-ids);
- ids = g_list_last(ids);
+ topids = g_tree_keys(self-ids);
+ ids = g_list_last(topids);
hi = *((u64_t *)ids-data);
- ids = g_list_first(ids);
+ ids = g_list_first(topids);
lo = *((u64_t *)ids-data);
- g_list_free(ids);
+ g_list_free(topids);
} else {
lo = 1;
hi = g_tree_nnodes(self-ids);
It is so small a leak it was not notices but peter's imapsync is doing a
uid ic_fetch on hundreds of ids per fetch command over and over so the
glist of ids leak would build and build on our servers but on his it
goes nuts! :)
Thanks,
Leif
p.s. Peter I am having trouble getting your test script to run on my dev
box please try the change above and let me/ the group know.
Peter Rabbitson wrote:
Paul J Stevens wrote:
==1631== 71,603,456 bytes in 288,752 blocks are still reachable in loss
record 26 of 26
This could be a side effect of the slab allocator. Use
G_DEBUG=always_malloc valgrind ... to circumvent the glib default
allocator.
Tested with 'export G_DEBUG=always_malloc' yielding nearly identical
results (71,604,456 bytes this time). New logs at
http://rabbit.us/pool/misc/dbmail_bug584_memcheck_svn2574_always_malloc.tar.bz2
This URL has some interesting points
http://www.tinymail.org/trac/tinymail/wiki/MemoryStats
Erm, being a n00b and all, wouldn't this affect both imap daemons
involved in the test? In my case only 'host2' leaks, 'host1' stabilizing
at about 40M VSZ.
Peter
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail-imapd memory leak
Humm, well from your valgrind log it still must be in fetch: ==28403== ==28403== 97,544 bytes in 383 blocks are possibly lost in loss record 25 of 26 ==28403==at 0x40226DB: memalign (vg_replace_malloc.c:332) ==28403==by 0x4022735: posix_memalign (vg_replace_malloc.c:425) ==28403==by 0x415E0A3: (within /usr/lib/libglib-2.0.so.0.1200.12) ==28403==by 0x415EAC7: g_slice_alloc (in /usr/lib/libglib-2.0.so.0.1200.12) ==28403==by 0x4145725: g_list_prepend (in /usr/lib/libglib-2.0.so.0.1200.12) ==28403==by 0x41D6F5A: traverse_tree_keys (misc.c:1117) ==28403==by 0x4168E44: g_tree_foreach (in /usr/lib/libglib-2.0.so.0.1200.12) ==28403==by 0x41D6AD4: g_tree_keys (misc.c:1130) ==28403==by 0x41C41FA: dbmail_mailbox_get_set (dbmail-mailbox.c:1333) ==28403==by 0x804EC61: _ic_fetch (imapcommands.c:1411) ==28403==by 0x804F1B3: _ic_uid (imapcommands.c:1790) ==28403==by 0x804D0DF: IMAPClientHandler (imap4.c:290) ==28403== I am out of time for today but maybe Paul or Aaron will see something and we can put this one to rest. Thanks for you help peter. -leif Peter Rabbitson wrote: Leif Jackson wrote: Peter try 2579 with my changes from the previous message. Same. Logs at http://rabbit.us/pool/misc/bug584_memcheck_svn2579_LJpatch.tar.bz2 Peter ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail-imapd memory leak - one more time...
Peter,
On second look please test this patch against rev 2579 vanila:
Index: dbmail-mailbox.c
===
--- dbmail-mailbox.c(revision 2579)
+++ dbmail-mailbox.c(working copy)
@@ -1313,7 +1313,7 @@
GTree * dbmail_mailbox_get_set(struct DbmailMailbox *self, const char
*set, gboolean uid)
{
- GList *ids = NULL, *sets = NULL;
+ GList *topids, *ids = NULL, *sets = NULL;
GString *t;
char *rest;
u64_t i, l, r, lo = 0, hi = 0;
@@ -1330,12 +1330,13 @@
TRACE(TRACE_DEBUG,[%s] uid [%d], set, uid);
if (uid) {
- ids = g_tree_keys(self-ids);
- ids = g_list_last(ids);
+ topids = g_tree_keys(self-ids);
+ ids = g_list_last(topids);
hi = *((u64_t *)ids-data);
- ids = g_list_first(ids);
+ ids = g_list_first(topids);
lo = *((u64_t *)ids-data);
- g_list_free(g_list_first(ids));
+ //g_list_free(g_list_first(ids));
+ g_list_free(topids);
} else {
lo = 1;
hi = g_tree_nnodes(self-ids);
I think it was cause you tried to use my patch from the old code and
Paul's changes might have not helped. let me know
Thanks,
Leif
Peter Rabbitson wrote:
Leif Jackson wrote:
Peter try 2579 with my changes from the previous message.
Same. Logs at
http://rabbit.us/pool/misc/bug584_memcheck_svn2579_LJpatch.tar.bz2
Peter
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail] dbmail-imapd memory leak
Peter, You need to install libtool, I got the same thing on one of my devel boxes, once libtool is installed re-run autoreconf. This svn version has now been running for a while on a semi-production box and it looks like imapd doesn't leak memory anymore. -leif Peter Rabbitson wrote: Aaron Stone wrote: Ah, sorry, forgot to mention that it requires an autoreconf. How do I do that? I installed the autoconf and automake debian packages (sid) and I get this: [EMAIL PROTECTED]:~/devel/dbmail$ autoreconf /usr/share/aclocal/smpeg.m4:13: warning: underquoted definition of AM_PATH_SMPEG /usr/share/aclocal/smpeg.m4:13: run info '(automake)Extending aclocal' /usr/share/aclocal/smpeg.m4:13: or see http://sources.redhat.com/automake/automake.html#Extending-aclocal configure.in:40: warning: macro `AM_ENABLE_SHARED' not found in library configure.in:41: warning: macro `AM_ENABLE_STATIC' not found in library configure.in:40: error: possibly undefined macro: AM_ENABLE_SHARED If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. configure.in:41: error: possibly undefined macro: AM_ENABLE_STATIC configure.in:116: error: possibly undefined macro: AC_PROG_LIBTOOL autoreconf: /usr/bin/autoconf failed with exit status: 1 [EMAIL PROTECTED]:~/devel/dbmail$ If I just try to make I get the same error as Jorge (about DBMAIL_VERSION). Please help :) ___ DBmail mailing list DBmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail ___ DBmail mailing list DBmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail] dbmail-imapd memory leak
Peter Rabbitson wrote: Aaron Stone wrote: First let's look at some debug logs and see what's going on. Maybe the program is caught in a tight loop somewhere? If there's a particular IMAP command string that immediately triggers the problem, that'd be sort of the ideal situation, but not necessarily likely. Since we use Glib, these instructions for profiling GNOME applications apply to us as well: http://developer.gnome.org/doc/guides/optimisation/Massif.html Please ask any and all questions along the way, we're more than happy to bring you up to speed on how to debug! Aaron Hi Aaron I played around with the latest rc2 build and my problem is still there (bug 584). I spent more time on it and did a fully logged run of valgrind as described in the url you gave me. Problem is I do not know what to do about the output I get. You can find a tar of all the information I could scramble at: http://rabbit.us/pool/misc/dbmail_bug584.tar.bz2 The file includes: dbmail_test - the script that made this happen dbmail.conf - my config file netstat.log - a netstat snapshot showing active connections/pids mem.log - a `ps aux` showing the fat process dbmail.err - a level 5 trace of the entire process the valgrind/massif output for all dbmail processes I understand that this is way too much information, but hopefully you can just glance at it and tell me right away what I should do next. Thanks for the help! Peter, If you run your tests with valgrind in memcheck mode using --suppressions=dbmail.supp in the contrib directory I created a while back. Use options valgrind --suppressions=contrib/dbmail.supp --tool=memcheck --leak-check=full --show- reachable=yes --trace-children=yes --log-file=/tmp/valtrace the logfile directory can be any you choose it will make a file per pid and log the leaks that arn't suppressed by the valgrind suppession file. That should help us track this down futher. As arron said the options that you specified may also be needed but the tool should be memcheck. Thanks for helping track this down. Thanks, Leif Jackson ___ DBmail mailing list DBmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail] dbmail-imapd memory leak
Humm looks like it might be in the ic_fetch: ==26918== 37,800 bytes in 54 blocks are still reachable in loss record 23 of 27 ==26918==at 0x40235AB: realloc (vg_replace_malloc.c:306) ==26918==by 0x428CF4A: vasprintf (in /lib/i686/cmov/libc-2.5.so) ==26918==by 0x41708C6: g_vasprintf (in /usr/lib/libglib-2.0.so.0.1200.12) ==26918==by 0x41629E5: g_strdup_vprintf (in /usr/lib/libglib-2.0.so.0.1200.12) ==26918==by 0x41D73AE: g_list_append_printf (in /usr/lib/dbmail/libdbmail.so.0.0.0) ==26918==by 0x41C57B7: g_list_slices (in /usr/lib/dbmail/libdbmail.so.0.0.0) ==26918==by 0x8052534: dbmail_imap_session_mailbox_update_recent (in /usr/sbin/dbmail-imapd) ==26918==by 0x8055D74: dbmail_imap_session_fetch_get_items (in /usr/sbin/dbmail-imapd) ==26918==by 0x804E707: _ic_fetch (in /usr/sbin/dbmail-imapd) ==26918==by 0x804EBCA: _ic_uid (in /usr/sbin/dbmail-imapd) ==26918==by 0x804CD50: IMAPClientHandler (in /usr/sbin/dbmail-imapd) ==26918==by 0x41DD7F2: PerformChildTask (in /usr/lib/dbmail/libdbmail.so.0.0.0) but I don't have enough time untill maybe this weekend to dig into it maybe Aaron or Paul have a sec to look into it (In my other debuging I noticed a lot of the valgrind complaints for g_list_append_printf, but if there is a memleak there it must be in glib? humm...). But since your script does a lot of fetches with large arguments I suspect it is somewhere in there. Thanks for the debug info! -Leif Here it is: http://rabbit.us/pool/misc/dbmail_bug584_memcheck.tar.bz2 The tar contains the same information as the previous one. None of the options I used before work - I guess they are specific to massif. Let me know what to do next. Peter ___ DBmail mailing list DBmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail ___ DBmail mailing list DBmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail-dev] default_iconv stuff with convert_8bit_field_to_utf8
Anton Paul, humm sorry I missed the fact it was a static variable, but there is some wierdness with this code then and gmime/glib as far as valgrind is concerned. I will move on to other things for now. Thanks, Leif Paul J Stevens wrote: Anton Zakatov wrote: Paul and Leif, in functions convert_8bit_field_to_utf8 and convert_8bit_db_to_mime all staic variable init only once on first call. If you release memory of iconv struct and not set it value to (iconv_t)-1 we get segfault on next call function. THIS IS NOT MEMORY LEAK! Anton, That's why I've remove all g_mime_iconv_close calls for now. I was getting segfaults all over the place on g_mime_shutdown that all led to the g_mime_iconv_close calls. I guess I was a bit tired, and running out of patience with debugging and fixing half-tested patches and BOFH mode kicked in :-( I'm always happy to accept patches and I definitely don't want to discourage anyone from helping out on the code. It's just that I sometimes stop being so happy after running 'make check' or 'python test-scripts/testimap.py'... ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] default_iconv stuff with convert_8bit_field_to_utf8
Paul J Stevens wrote:
Leif,
could you plz resent this patch as attachment? I can't get it to apply
like this.
You can simply do 'svn diff misc.c' to produce valid patches. No need to
keep an original tree around.
Thats really cool, still learning svn. Anyway the svn diff for misc.c is
attached. I looked through the code and a few more iconv's needed
closing before returns. So all that is included in this patch.
Thanks,
Leif
Index: misc.c
===
--- misc.c (revision 2482)
+++ misc.c (working copy)
@@ -2036,12 +2036,19 @@
default_iconv=g_mime_iconv_open(UTF-8,default_charset);
if (default_iconv == (iconv_t)-1) {
TRACE(TRACE_DEBUG,incorrect default encoding [%s], default_charset);
- }
+ }else
+ allocated_default_iconv = 1;
}
- if (str_in==NULL)
+ if (str_in==NULL) {
+ if (allocated_default_iconv)
+ g_mime_iconv_close(default_iconv);
return NULL;
+ }
+
if (g_utf8_validate((const gchar *)str_in, -1, NULL) || !g_mime_utils_text_is_8bit((unsigned char *)str_in, strlen(str_in))) {
+ if (allocated_default_iconv)
+ g_mime_iconv_close(default_iconv);
// Conversion not needed
return g_strdup(str_in);
}
@@ -2057,7 +2064,6 @@
if ((conv_iconv=g_mime_iconv_open(UTF-8,charset))==(iconv_t)-1) {
TRACE(TRACE_DEBUG,incorrect encoding [%s] base [UTF-8], charset);
-allocated_default_iconv = 1;
} else {
subj=g_mime_iconv_strdup(conv_iconv,str_in);
g_mime_iconv_close(conv_iconv);
@@ -2066,7 +2072,6 @@
}
if (subj==NULL) {
subj=g_mime_iconv_strdup(default_iconv,str_in);
- if (allocated_default_iconv) g_mime_iconv_close(default_iconv);
}
if (subj==NULL) {
@@ -2077,6 +2082,9 @@
if(*p 0x80) *p='?';
}
+ if (allocated_default_iconv)
+ g_mime_iconv_close(default_iconv);
+
return subj;
}
@@ -2127,29 +2135,44 @@
default_iconv=g_mime_iconv_open(base_charset,default_charset);
if (default_iconv == (iconv_t)-1) {
TRACE(TRACE_DEBUG,incorrect default encoding [%s], default_charset);
- }
- allocated_default_iconv = 1;
+ }else
+ allocated_default_iconv = 1;
}
if (str_in==NULL) {
- if(allocated_default_iconv) g_mime_iconv_close(default_iconv);
+ if(allocated_default_iconv)
+ g_mime_iconv_close(default_iconv);
+ if (base_iconv != (iconv_t)-1)
+ g_mime_iconv_close(base_iconv);
return NULL;
}
if (!g_mime_utils_text_is_8bit((unsigned char *)str_in, strlen(str_in))) {
+ if(allocated_default_iconv)
+ g_mime_iconv_close(default_iconv);
+ if (base_iconv != (iconv_t)-1)
+ g_mime_iconv_close(base_iconv);
// Conversion not needed
- if(allocated_default_iconv) g_mime_iconv_close(default_iconv);
return g_strdup(str_in);
}
if ((subj=g_mime_iconv_strdup(base_iconv,str_in))!=NULL) {
+ if(allocated_default_iconv)
+ g_mime_iconv_close(default_iconv);
+ if (base_iconv != (iconv_t)-1)
+ g_mime_iconv_close(base_iconv);
// Conversion already done by header decode ? May insert to database
- if(allocated_default_iconv) g_mime_iconv_close(default_iconv);
return subj;
}
-
+
+ //base_iconv not needed after here.
+ if (base_iconv != (iconv_t)-1)
+ g_mime_iconv_close(base_iconv);
+
+
// Get message encode codepage
if (message)
mess_obj=g_mime_message_get_mime_part(message);
+
if (mess_obj) {
if ((charset=g_mime_object_get_content_type_parameter(mess_obj,charset))) {
// codepage not set in message header use default
@@ -2163,6 +2186,7 @@
}
}
}
+
if (subj==NULL) {
subj=g_mime_iconv_strdup(default_iconv,str_in);
}
@@ -2211,9 +2235,13 @@
}
if (str_in==NULL)
+ if (base_iconv != (iconv_t)-1)
+ g_mime_iconv_close(base_iconv);
return NULL;
if (!g_mime_utils_text_is_8bit((unsigned char *)str_in, strlen(str_in))) {
+ if (base_iconv != (iconv_t)-1)
+ g_mime_iconv_close(base_iconv);
// Conversion not needed
return g_strdup(str_in);
}
@@ -2221,8 +2249,13 @@
if ((subj=g_mime_iconv_strdup(base_iconv,str_in))!=NULL) {
subj2 = g_mime_utils_header_encode_text((unsigned char *)subj);
g_free(subj);
+ if (base_iconv != (iconv_t)-1)
+ g_mime_iconv_close(base_iconv);
return subj2;
}
+
+ if (base_iconv != (iconv_t)-1)
+ g_mime_iconv_close(base_iconv);
return g_mime_utils_header_encode_text((unsigned char *)str_in);
}
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] imapcommands.c again?
Paul,
This explains my confusion one devel machine I have was downloading
from trunk and the other was use dbmail_2_2_branch, I didn't relize they
were diffrent. I figured since dbmail 2.3 or other devel version hasn't
really been started that trunk would have been pointing to 2_2 by
default. I get it now will work with dbmail_2_2_branch only now. Is
there any way with svn to verifiy that all files are in sync with the
tree? (I don't use svn enough) .
Thanks,
Leif
Paul J Stevens wrote:
Leif,
I've merged all recent changes from dbmail_2_2_branch into the trunk again.
But apart from the ingres driver (unfinished) there's not much to see in
the trunk atm.
Paul J Stevens wrote:
Leif,
You should stay away from trunk for now. I merge changes into the trunk
from time to time but it's the dbmail_2_2_branch where current
development is taking place.
Leif Jackson wrote:
I just pulled a fresh copy to make sure that it wasn't me this time paul
but as I see it in trunk _ic_select in imapcommands.c
looks like this:
/* show idx of first unseen msg (if present) */
if (ud-mailbox.exists) {
if (! (key = db_first_unseen(ud-mailbox.uid))) {
dbmail_imap_session_printf(self, * BYE internal
dbase error\r\n);
return -1;
}
if ((msn = g_tree_lookup(self-mailbox-ids, key))) {
dbmail_imap_session_printf(self,
* OK [UNSEEN %llu] first unseen
message\r\n, *msn);
}
}
this re-introduces the issue with select all failing..etc as of 3/19
paul you showed they looked like:
The lines in svn look like this:
/* show idx of first unseen msg (if present) */
if (ud-mailbox.exists) { key = db_first_unseen(ud-mailbox.uid); if (
(key 0) (msn =
g_tree_lookup(self-mailbox-ids, key))) {
dbmail_imap_session_printf(self, * OK [UNSEEN %llu] first unseen
message\r\n, *msn); }
}
any ideas?
Thanks,
leif
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] imapcommands.c again?
Cool I just got the 2_2 branch from svn is this suposed to have the configure script and buildtools? usualy i use autoreconf -f -i to get the configure script. It seems the buildtools and configure script are not allowing me to build on my x86_64 I have a -fPIC error, Aaron don't worrie about this just was wondering if there is a quick way to get to a clean distro make distclean dosn't seem to get clear enough. Thanks, Leif Aaron Stone wrote: Just a simple 'svn update' with a working directory pointed to the correct branch will get all the relevant files updated. 'svn status' will then tell you which files are different. Use 'svn switch' to switch from trunk to dbmail_2_2_branch. The command should be: svn switch https://svn.ic-s.nl/svn/dbmail/branches/dbmail_2_2_branch Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] explicit initalize of the config variable and 1 minor memleak for branch 2_2 rev 2480
Aaron it seem the path I sent with the config initializers got taken back out other than the change to dsn.c that is all this patch is. Otherwise the 2_2 branch is looking good as far as memleaks :) I was looking at the dbmail-message.c code and saw a new function convert_8bit_field_to_utf8 is used that looks like the function I patched for the iconv default leak... misc.c anyway I was wondering does the the new utf8 version also need the check for if default_iconv was allocated rather than just the g_mime_iconv_close if subj==NULL? just my .02 but I think it is more explicit for all the possible cases. I would assube that the base_charset is always null and so default_iconv will always be allocated but then there are if statments that return before closing default_iconv that was my idea behind the if(allocated_default_iconv) looking at the code now tho it seems like it should always try and close it? What do you guys think? Side note to this if the dbmail code is calling this function a lot would it make sense to open the default_iconv once for each session at the least? Thanks, Leif diff -urN dbmail-svn-2.2.4-2480.orig/dsn.c dbmail-svn-2.2.4-2480.leif/dsn.c --- dbmail-svn-2.2.4-2480.orig/dsn.c2007-03-24 18:40:07.0 -0400 +++ dbmail-svn-2.2.4-2480.leif/dsn.c2007-03-24 19:35:06.0 -0400 @@ -234,6 +234,8 @@ g_free(dsnuser-forwards); dsnuser-address = NULL; + if (dsnuser-mailbox) + g_free(dsnuser-mailbox); //allocated by dbmail-smtp main.c line ~ 319 g_strdup dsnuser-mailbox = NULL; dsnuser-userids = NULL; dsnuser-forwards = NULL; diff -urN dbmail-svn-2.2.4-2480.orig/imapd.c dbmail-svn-2.2.4-2480.leif/imapd.c --- dbmail-svn-2.2.4-2480.orig/imapd.c 2007-03-24 18:40:07.0 -0400 +++ dbmail-svn-2.2.4-2480.leif/imapd.c 2007-03-24 19:21:15.0 -0400 @@ -40,6 +40,9 @@ g_mime_init(0); openlog(PNAME, LOG_PID, LOG_MAIL); + //fixes valgrind Conditional jump or move depends on uninitialised value(s) + memset(config, 0, sizeof(serverConfig_t)); + result = serverparent_getopt(config, IMAP, argc, argv); if (result == -1) goto shutdown; diff -urN dbmail-svn-2.2.4-2480.orig/lmtpd.c dbmail-svn-2.2.4-2480.leif/lmtpd.c --- dbmail-svn-2.2.4-2480.orig/lmtpd.c 2007-03-24 18:40:07.0 -0400 +++ dbmail-svn-2.2.4-2480.leif/lmtpd.c 2007-03-24 19:22:07.0 -0400 @@ -40,6 +40,9 @@ g_mime_init(0); openlog(PNAME, LOG_PID, LOG_MAIL); +//fixes valgrind Conditional jump or move depends on uninitialised value(s) +memset(config, 0, sizeof(serverConfig_t)); + result = serverparent_getopt(config, LMTP, argc, argv); if (result == -1) goto shutdown; diff -urN dbmail-svn-2.2.4-2480.orig/pop3d.c dbmail-svn-2.2.4-2480.leif/pop3d.c --- dbmail-svn-2.2.4-2480.orig/pop3d.c 2007-03-24 18:40:07.0 -0400 +++ dbmail-svn-2.2.4-2480.leif/pop3d.c 2007-03-24 19:21:25.0 -0400 @@ -44,6 +44,9 @@ g_mime_init(0); openlog(PNAME, LOG_PID, LOG_MAIL); +//fixes valgrind Conditional jump or move depends on uninitialised value(s) +memset(config, 0, sizeof(serverConfig_t)); + result = serverparent_getopt(config, POP, argc, argv); if (result == -1) goto shutdown; diff -urN dbmail-svn-2.2.4-2480.orig/timsieved.c dbmail-svn-2.2.4-2480.leif/timsieved.c --- dbmail-svn-2.2.4-2480.orig/timsieved.c 2007-03-24 18:40:07.0 -0400 +++ dbmail-svn-2.2.4-2480.leif/timsieved.c 2007-03-24 19:21:35.0 -0400 @@ -42,6 +42,9 @@ g_mime_init(0); openlog(PNAME, LOG_PID, LOG_MAIL); +//fixes valgrind Conditional jump or move depends on uninitialised value(s) +memset(config, 0, sizeof(serverConfig_t)); + result = serverparent_getopt(config, SIEVE, argc, argv); if (result == -1) goto shutdown; ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] [DBMail 0000546]: Memory corrupted in IMAP daemon
[EMAIL PROTECTED] wrote: -- aaron - 24-Mar-07 22:19 -- Oh crap, I've confirmed that I SIGHUP handling very badly. Probably will rearrange the code more significantly to make it very obvious what is happening. Too many functions have side-effects on the config struct. -- aaron - 24-Mar-07 23:32 -- Ok, I see what this bug might actually be about, and SIGHUP was me being stupid and has nothing to do with the issue here. In dbmail-imapsession.c, 'the_args' is an array of parsed out imap command arguments. It is static memory that is passed around a lot. Someplace this must not be going according to plan. Would it be possible to trace the imap commands leading up to this error? I was just looking at this. Since the_args is a global var, it doesn't make sence to return it does it? Maybe there is some namespace corruption... ? -leif ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] default_iconv stuff with convert_8bit_field_to_utf8
Just a suggestion :) I belive without closing it everywhere in that
function there are still leaks from iconv.
-leif
diff -urN dbmail-svn-2.2.4-2480.leif2/misc.c
dbmail-svn-2.2.4-2480.leif3/misc.c
--- dbmail-svn-2.2.4-2480.leif2/misc.c 2007-03-24 18:40:07.0 -0400
+++ dbmail-svn-2.2.4-2480.leif3/misc.c 2007-03-25 00:38:19.0 -0400
@@ -2019,6 +2019,7 @@
char * convert_8bit_field_to_utf8(GMimeMessage *message,const char* str_in)
{
char * subj=NULL;
+ int allocated_default_iconv = 0;
static iconv_t default_iconv=(iconv_t)-1;
static const char *default_charset=NULL;
const char *charset;
@@ -2035,13 +2036,18 @@
default_iconv=g_mime_iconv_open(UTF-8,default_charset);
if (default_iconv == (iconv_t)-1) {
TRACE(TRACE_DEBUG,incorrect default encoding
[%s], default_charset);
- }
+ }else
+ allocated_default_iconv = 1;
}
- if (str_in==NULL)
+ if (str_in==NULL) {
+ if(allocated_default_iconv)
g_mime_iconv_close(default_iconv);
return NULL;
+ }
+
if (g_utf8_validate((const gchar *)str_in, -1, NULL) ||
!g_mime_utils_text_is_8bit((unsigned char *)str_in, strlen(str_in))) {
// Conversion not needed
+ if(allocated_default_iconv)
g_mime_iconv_close(default_iconv);
return g_strdup(str_in);
}
@@ -2058,13 +2064,11 @@
TRACE(TRACE_DEBUG,incorrect encoding
[%s] base [UTF-8], charset);
} else {
subj=g_mime_iconv_strdup(conv_iconv,str_in);
- g_mime_iconv_close(conv_iconv);
}
}
}
if (subj==NULL) {
subj=g_mime_iconv_strdup(default_iconv,str_in);
- g_mime_iconv_close(default_iconv);
}
if (subj==NULL) {
@@ -2075,6 +2079,9 @@
if(*p 0x80) *p='?';
}
+ if(allocated_default_iconv)
+ g_mime_iconv_close(default_iconv);
+
return subj;
}
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] Possible minor memleaks in svn trunk 2474
Paul Aaron,
I was reviewing the current trunk 2474 from today, to see if I can help
make 2.2.5 bug free :) Anyway I came across some possible memleak issues
patch is attached. This is from valgrind I deduced the issues. Please
verify and apply to the trunk if you see fit.
Thanks,
Leif Jackson
diff -urN dbmail-svn-2.2.4-2474.orig/dbmail-message.c dbmail-svn-2.2.4-2474.new/dbmail-message.c
--- dbmail-svn-2.2.4-2474.orig/dbmail-message.c 2007-03-23 14:09:34.0 -0400
+++ dbmail-svn-2.2.4-2474.new/dbmail-message.c 2007-03-23 14:49:59.0 -0400
@@ -1109,6 +1109,7 @@
g_string_printf(q, INSERT INTO %s%sfield (id, physmessage_id, %sfield)
VALUES (%s, %llu,'%s'), DBPFX, field, field, id, physid, clean_value);
g_free(id);
+ g_free(sid);
g_free(clean_value);
if (db_query(q-str)) {
diff -urN dbmail-svn-2.2.4-2474.orig/imapd.c dbmail-svn-2.2.4-2474.new/imapd.c
--- dbmail-svn-2.2.4-2474.orig/imapd.c 2007-03-23 14:09:34.0 -0400
+++ dbmail-svn-2.2.4-2474.new/imapd.c 2007-03-23 16:59:46.0 -0400
@@ -40,6 +40,10 @@
g_mime_init(0);
openlog(PNAME, LOG_PID, LOG_MAIL);
+ //fixes valgrind Conditional jump or move depends on uninitialised value(s)
+ config.iplist = NULL;
+ config.listenSockets = NULL;
+
result = serverparent_getopt(config, IMAP, argc, argv);
if (result == -1)
goto shutdown;
diff -urN dbmail-svn-2.2.4-2474.orig/lmtpd.c dbmail-svn-2.2.4-2474.new/lmtpd.c
--- dbmail-svn-2.2.4-2474.orig/lmtpd.c 2007-03-23 14:09:33.0 -0400
+++ dbmail-svn-2.2.4-2474.new/lmtpd.c 2007-03-23 16:59:41.0 -0400
@@ -40,6 +40,10 @@
g_mime_init(0);
openlog(PNAME, LOG_PID, LOG_MAIL);
+ //fixes valgrind Conditional jump or move depends on uninitialised value(s)
+config.iplist = NULL;
+config.listenSockets = NULL;
+
result = serverparent_getopt(config, LMTP, argc, argv);
if (result == -1)
goto shutdown;
diff -urN dbmail-svn-2.2.4-2474.orig/misc.c dbmail-svn-2.2.4-2474.new/misc.c
--- dbmail-svn-2.2.4-2474.orig/misc.c 2007-03-23 14:09:34.0 -0400
+++ dbmail-svn-2.2.4-2474.new/misc.c 2007-03-23 16:58:04.0 -0400
@@ -2015,6 +2015,7 @@
}
if (subj==NULL) {
subj=g_mime_iconv_strdup(default_iconv,str_in);
+ g_mime_iconv_close(default_iconv);
}
if (subj==NULL) {
@@ -2036,6 +2037,7 @@
//char * str_out=NULL;
char * subj=NULL;
//int err_flg=1;
+ int allocated_default_iconv = 0;
static const char * base_charset=NULL;
static iconv_t base_iconv=(iconv_t)-1;
static iconv_t default_iconv=(iconv_t)-1;
@@ -2075,17 +2077,22 @@
if (default_iconv == (iconv_t)-1) {
TRACE(TRACE_DEBUG,incorrect default encoding [%s], default_charset);
}
+ allocated_default_iconv = 1;
}
- if (str_in==NULL)
+ if (str_in==NULL) {
+ if(allocated_default_iconv) g_mime_iconv_close(default_iconv);
return NULL;
+ }
if (!g_mime_utils_text_is_8bit((unsigned char *)str_in, strlen(str_in))) {
// Conversion not needed
+ if(allocated_default_iconv) g_mime_iconv_close(default_iconv);
return g_strdup(str_in);
}
if ((subj=g_mime_iconv_strdup(base_iconv,str_in))!=NULL) {
// Conversion already done by header decode ? May insert to database
+ if(allocated_default_iconv) g_mime_iconv_close(default_iconv);
return subj;
}
@@ -2117,6 +2124,9 @@
if(*p 0x80) *p='?';
}
+ if(allocated_default_iconv)
+ g_mime_iconv_close(default_iconv);
+
return subj;
}
diff -urN dbmail-svn-2.2.4-2474.orig/pop3d.c dbmail-svn-2.2.4-2474.new/pop3d.c
--- dbmail-svn-2.2.4-2474.orig/pop3d.c 2007-03-23 14:09:34.0 -0400
+++ dbmail-svn-2.2.4-2474.new/pop3d.c 2007-03-23 17:00:07.0 -0400
@@ -44,6 +44,10 @@
g_mime_init(0);
openlog(PNAME, LOG_PID, LOG_MAIL);
+//fixes valgrind Conditional jump or move depends on uninitialised value(s)
+config.iplist = NULL;
+config.listenSockets = NULL;
+
result = serverparent_getopt(config, POP, argc, argv);
if (result == -1)
goto shutdown;
diff -urN dbmail-svn-2.2.4-2474.orig/serverparent.c dbmail-svn-2.2.4-2474.new/serverparent.c
--- dbmail-svn-2.2.4-2474.orig/serverparent.c 2007-03-23 14:09:34.0 -0400
+++ dbmail-svn-2.2.4-2474.new/serverparent.c 2007-03-23 16:04:22.0 -0400
@@ -167,6 +167,7 @@
* e.g. on SIGHUP or other graceful restart condition. */
DoConfig(config, service); }
+ ClearConfig(config);
TRACE(TRACE_INFO, leaving main loop);
return 0;
}
diff -urN dbmail-svn-2.2.4-2474.orig/timsieved.c dbmail-svn-2.2.4-2474.new/timsieved.c
--- dbmail-svn-2.2.4-2474.orig/timsieved.c 2007-03-23 14:09:33.0 -0400
+++ dbmail-svn-2.2.4-2474.new/timsieved.c 2007-03-23 17:00:26.0 -0400
@@ -42,6 +42,10 @@
g_mime_init(0);
openlog(PNAME, LOG_PID, LOG_MAIL);
+//fixes valgrind Conditional jump or move depends on uninitialised value(s)
+config.iplist = NULL;
+config.listenSockets = NULL;
+
result = serverparent_getopt(config, SIEVE, argc, argv);
if (result == -1)
goto shutdown
[Dbmail-dev] imapcommands.c again?
I just pulled a fresh copy to make sure that it wasn't me this time paul
but as I see it in trunk _ic_select in imapcommands.c
looks like this:
/* show idx of first unseen msg (if present) */
if (ud-mailbox.exists) {
if (! (key = db_first_unseen(ud-mailbox.uid))) {
dbmail_imap_session_printf(self, * BYE internal
dbase error\r\n);
return -1;
}
if ((msn = g_tree_lookup(self-mailbox-ids, key))) {
dbmail_imap_session_printf(self,
* OK [UNSEEN %llu] first unseen
message\r\n, *msn);
}
}
this re-introduces the issue with select all failing..etc as of 3/19
paul you showed they looked like:
The lines in svn look like this:
/* show idx of first unseen msg (if present) */
if (ud-mailbox.exists) { key = db_first_unseen(ud-mailbox.uid); if ( (key 0)
(msn =
g_tree_lookup(self-mailbox-ids, key))) {
dbmail_imap_session_printf(self, * OK [UNSEEN %llu] first unseen
message\r\n, *msn); }
}
any ideas?
Thanks,
leif
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] dbmail 2.2.4 svn 2472 issue with server dropping connections on non-error
This was a show stopper when I did an upgrade used svn to get the other
minor fixes, this was first talked about for dbmail 2.2.3 did this somehow
get back into svn? I found it at
http://www.mail-archive.com/dbmail-dev@dbmail.org/msg08488.html anyway
when a user was selecting their Sent mailbox is was doing a SELECT ALL for
non-seen messages and the DB sent back NULL which was correct but the code
below was disconnecting the client rather than showing 0 msgs as
not-seen..etc.
diff -urN dbmail-svn-2.2.4-2472.orig/imapcommands.c
dbmail-svn-2.2.4-2472.new/imapcommands.c
--- dbmail-svn-2.2.4-2472.orig/imapcommands.c 2007-03-17
23:21:45.0 -0400
+++ dbmail-svn-2.2.4-2472.new/imapcommands.c2007-03-19
13:37:38.0 -0400
@@ -240,7 +240,7 @@
/* show idx of first unseen msg (if present) */
if (ud-mailbox.exists) {
- if (! (key = db_first_unseen(ud-mailbox.uid))) {
+ if ((key = db_first_unseen(ud-mailbox.uid)) == (u64_t)
(-1)) {
dbmail_imap_session_printf(self, * BYE internal
dbase error\r\n);
return -1;
}
Thanks,
Leif Jackson
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail 2.2.4 svn 2472 issue with server dropping connections on non-error
Thanks Paul,
I have no idea how this got corrupted the current 2472 svn tree seems to
work fine with regards to this issue.
-leif
On Mon, March 19, 2007 2:29 pm, Paul J Stevens wrote:
Leif,
You're tree is not in sync. This patch doesn't apply to 2472 at all.
The lines in svn look like this:
/* show idx of first unseen msg (if present) */
if (ud-mailbox.exists) { key = db_first_unseen(ud-mailbox.uid); if ( (key
0) (msn =
g_tree_lookup(self-mailbox-ids, key))) {
dbmail_imap_session_printf(self, * OK [UNSEEN %llu] first unseen
message\r\n, *msn); }
}
Leif Jackson wrote:
This was a show stopper when I did an upgrade used svn to get the other
minor fixes, this was first talked about for dbmail 2.2.3 did this
somehow get back into svn? I found it at
http://www.mail-archive.com/dbmail-dev@dbmail.org/msg08488.html anyway
when a user was selecting their Sent mailbox is was doing a SELECT ALL
for non-seen messages and the DB sent back NULL which was correct but
the code below was disconnecting the client rather than showing 0 msgs
as not-seen..etc.
diff -urN dbmail-svn-2.2.4-2472.orig/imapcommands.c
dbmail-svn-2.2.4-2472.new/imapcommands.c ---
dbmail-svn-2.2.4-2472.orig/imapcommands.c 2007-03-17
23:21:45.0 -0400
+++ dbmail-svn-2.2.4-2472.new/imapcommands.c2007-03-19
13:37:38.0 -0400
@@ -240,7 +240,7 @@
/* show idx of first unseen msg (if present) */
if (ud-mailbox.exists) { - if (! (key =
db_first_unseen(ud-mailbox.uid))) { + if ((key =
db_first_unseen(ud-mailbox.uid)) == (u64_t) (-1)) {
dbmail_imap_session_printf(self, * BYE internal dbase error\r\n); return
-1;
}
Thanks,
Leif Jackson
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
--
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlandshttp://www.nfg.nl
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Multifoo architecture update
Aaron, From your wiki page: On the issue of a database connection pool, I think that issue is mooted by this architecture. There are relatively few commands that do not require any database interaction, and the ones that dont are so fast to process as to not warrant the additional abstraction so that they wont tie up a database handle for their 0.05ms of processing time. Sorry not familar with editing the wiki so I will respond here. The need for thread pools is limited but the database pool is a good thing to have due to the fact that you will end up making a new connection when you get a new pop3/imap connect, sql connections take some time vs requesting the data in the response and so if you have a pool of open database connections waiting there you remove that connect lag, which gets large when MySQL gets busy. The thread of execution then just returns the connection back to the pool when the request is completed and thus you elemenate the churn of opening and closing DB connections. just my .02 -Leif On Tue, February 6, 2007 3:28 pm, Aaron Stone wrote: Hey folks, I've updated the multifoo architecture page on the wiki to reflect some of my latest work on the subject. At this point I'm starting to build the framework for the threaded server core. I think I've eliminated the need for a thread pool library and for any super complicated back and forth message passing between threads. http://www.dbmail.org/dokuwiki/doku.php?id=multifoo_architecture Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] Re dbmail bug report 405
Paul,
even though these warning are harmless they should be fixed, this is
minor yes but still will save you and aaron and the list time not
answering this question again so I belive this is the best fix:
diff -urNb dbmail-svn-2.1.8-2270.orig/modules/dbmysql.c
dbmail-svn-2.1.8-2270/modules/dbmysql.c
--- dbmail-svn-2.1.8-2270.orig/modules/dbmysql.c2006-09-18
14:27:31.0 -0400
+++ dbmail-svn-2.1.8-2270/modules/dbmysql.c 2006-09-19
07:17:57.0 -0400
@@ -236,7 +236,7 @@
return NULL;
}
- if ((row = db_num_rows()) || (field = db_num_fields())) {
+ if ((row db_num_rows()) || (field db_num_fields())) {
trace(TRACE_WARNING, %s, %s:
row = %u, field = %u, bigger than size of result set,
__FILE__, __func__, row, field);
@@ -266,7 +266,7 @@
result = last_row[field];
if (result == NULL)
- trace(TRACE_WARNING, %s,%s: result is null\n,
+ trace(TRACE_DEBUG, %s,%s: result is null\n,
__FILE__, __func__);
return result;
}
the first diff is because you get a erounous bigger than size of result
set for row == 0 and field == 0 which is not what I belive that warning
was ment for. and the second addresses report 405 and just makes it a
debuging message only if you and aaron don't want to remove that
completly.
as always just my 0.02 :)
Thanks,
Leif
Re: [Dbmail-dev] Re dbmail bug report 405
can you please also get the row is NULL above the result is NULL I missed
that in my patch.
Thanks,
Leif
On Tue, September 19, 2006 9:23 am, Paul J Stevens wrote:
fix applied (to dbpgsql.c as well).
thanks,
Leif Jackson wrote:
Paul,
even though these warning are harmless they should be fixed, this is
minor yes but still will save you and aaron and the list time not
answering this question again so I belive this is the best fix:
diff -urNb dbmail-svn-2.1.8-2270.orig/modules/dbmysql.c
dbmail-svn-2.1.8-2270/modules/dbmysql.c ---
dbmail-svn-2.1.8-2270.orig/modules/dbmysql.c2006-09-18
14:27:31.0 -0400
+++ dbmail-svn-2.1.8-2270/modules/dbmysql.c 2006-09-19
07:17:57.0 -0400
@@ -236,7 +236,7 @@
return NULL; }
- if ((row = db_num_rows()) || (field = db_num_fields())) {
+ if ((row db_num_rows()) || (field db_num_fields())) {
trace(TRACE_WARNING, %s, %s: row = %u, field = %u, bigger than size
of result set, __FILE__, __func__, row, field);
@@ -266,7 +266,7 @@
result = last_row[field]; if (result == NULL) -
trace(TRACE_WARNING, %s,%s: result is null\n, +
trace(TRACE_DEBUG, %s,%s: result is null\n, __FILE__, __func__);
return result; }
the first diff is because you get a erounous bigger than size of result
set for row == 0 and field == 0 which is not what I belive that
warning was ment for. and the second addresses report 405 and just makes
it a debuging message only if you and aaron don't want to remove that
completly.
as always just my 0.02 :)
Thanks,
Leif
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
--
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlandshttp://www.nfg.nl
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] [DBMail 0000390]: DEF_QUERYSIZE too small
We could make the query string dynamicly size it self with a realloc or somthing to that effect, then the question just would be would that open up a DDoS of some sort... But just for the utils and such that might come accross a query larger than the default it might solve and issue such as this one? -leif On Fri, August 4, 2006 2:11 pm, [EMAIL PROTECTED] wrote: A NOTE has been added to this issue. == http://www.dbmail.org/mantis/view.php?id=390 == Reported By:jnorell Assigned To: == Project:DBMail Issue ID: 390 Category: Command-Line programs (dbmail-users, dbmail-util) Reproducibility:always Severity: block Priority: normal Status: new target: == Date Submitted: 03-Aug-06 20:19 CEST Last Modified: 04-Aug-06 20:11 CEST == Summary:DEF_QUERYSIZE too small Description: DEF_QUERYSIZE is currently 1024, causing db_set_isheader() to fail during dbmail-util -ay. Repairing DBMAIL for incorrect is_header flags... Ok. Found [4291] incorrect is_header flags. Error dbpgsql.c, db_query: Error executing query [UPDATE dbmail_messageblks SET is_header = '1' WHERE messageblk_idnr IN (93793903,98060014,98077374,99083639,99084409,99198119,99198261,99219437, 99546559,99911465,101252056,101268114,101707614,102670265,102927551,10366 8187,103715340,104823570,105023569,105178339,106255768,107202646,10727114 7,109169596,109198078,109670036,109863443,109868049,109884055,109903270,1 09911632,109912467,109914168,109928479,109938422,109945972,109953386,1099 62557,109966403,109986404,109993587,110017640,110018292,110028237,1100337 08,110036495,110054553,110056874,110060590,110066643,110067484,110077596, 110077753,110078354,110086154,110130349,110130776,110143373,110144375,110 158164,110160247,110167293,110169681,110187387,110210148,110211965,110212 061,110223102,110223960,110225223,110235523,110243032,110254844,110260252 ,110268562,110330175,110429441,110551666,110667559,110775322,110859792,11 0883271,110896930,110921282,110930608,110946869,110956154,110980251,11099 0557,111030305,111033592,111040643,111057696,111093491,19017,4388 0,1] : [ERROR: syntax error at end of input at character 1024 ] Error db.c,db_set_isheader: could not access messageblks table Error setting the is_header flags == -- aaron - 04-Aug-06 12:41 -- I count 96 commas in that query. Tell me if changing the value 100 in the call to g_list_slices down to, say, 90 fixes this. You'll probably have to zap all of the is_header values before testing it again. -- jnorell - 04-Aug-06 20:11 -- Yeah, cutting down to 90 gets that to run. To size the correctly, what's the largest (in # digits) numbers that would ever be there? We've got 9 digits, but if you had 10, that would again go over the 1024 chars. Issue History Date Modified Username FieldChange == 03-Aug-06 20:19 jnorellNew Issue 04-Aug-06 12:41 aaron Note Added: 0001330 04-Aug-06 20:11 jnorellNote Added: 0001331 == ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] Current Trunk broken Compile issue with modules/sortsieve.c
Guys,
It seems current trunk is broken for compiling with sieve, the define
IMAP_NFLASGS is missing where should that be? also typo on the g_free.
Thanks,
Leif
diff -urNab dbmail-svn-2.1.7-2218.orig/modules/sortsieve.c
dbmail-svn-2.1.7-2218/modules/sortsieve.c
--- dbmail-svn-2.1.7-2218.orig/modules/sortsieve.c 2006-07-30
12:53:34.21449 -0400
+++ dbmail-svn-2.1.7-2218/modules/sortsieve.c 2006-07-30
13:11:44.207982927 -0400
@@ -181,6 +181,8 @@
return SIEVE2_OK;
}
+#define IMAP_NFLASGS 256
+
int sort_fileinto(sieve2_context_t *s, void *my)
{
struct sort_context *m = (struct sort_context *)my;
@@ -223,7 +225,7 @@
}
if (msgflags)
- g_free(msgflasgs);
+ g_free(msgflags);
return SIEVE2_OK;
}
Re: [Dbmail] init.d script not starting dbmail
DK, I have them at 95 with a chkconfig version of the dbmail init scripts on CentOS 4.3 64bit But as long as they are after your network and if sql is on the same machine your mysql startup it should be fine at any init number. -leif On Wed, July 26, 2006 7:37 pm, DK wrote: One more thing while I was trying to fix all this I remove all the rc links init.d for dbmail. Then I recreated them with default values but I am not so sure about those default values. I think 25 is wrong, I remember something in the 90s. Could someone check for me please? /etc/rc0.d/K25dbmail - ../init.d/dbmail /etc/rc1.d/K25dbmail - ../init.d/dbmail /etc/rc2.d/S25dbmail - ../init.d/dbmail /etc/rc3.d/S25dbmail - ../init.d/dbmail /etc/rc4.d/S25dbmail - ../init.d/dbmail /etc/rc5.d/S25dbmail - ../init.d/dbmail /etc/rc6.d/K25dbmail - ../init.d/dbmail - Demi ___ Dbmail mailing list Dbmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail] dbmail-smtp 2.1.7 killed by sig9 with specific message
Martin, Based on the link with the logs and your message in question I tried to reproduce this issue under valgrind and was unable to with 2.1.7 dbmail-smtp as well as the retrival of the message after storage matches the original. So I would guess that this sig9 is somewhere in your exim chain or a library diffrence I don't have, what version of glib gmime and glibc do you have dbmail compiled against as well as mysql versions. -Leif On Thu, July 27, 2006 7:59 am, Martin Hierling wrote: Hi, dbmail-smtp gets killed by signal 9 with a specific mail. All other mails get in there boxes, only this on not, bug? 006-07-27 12:37:08 1G63Eq-0004gv-Ji ** [EMAIL PROTECTED] F=[EMAIL PROTECTED] R=localuser T=transport_dbmail_p: Child process of transport_dbmail_p transport (running command /usr/sbin/dbmail-smtp -d [EMAIL PROTECTED]) was terminated by signal 9 (Killed) I dont want to pollute the list with tonns of logs so they are here: http://www.fh-luh.de/~mad/dbmail-smtp-4325234.txt Part1: Exim Log with some examples that dbmail-smtp delivers other mails correct and the sig9 delivery attempt. Part2: level 5 debug of dbmail-smtp Part3: the message, but i cannot detect something special this is dbmail-2.1.7 with libsieve 2.1.12, mysql 4.1.20 running on gentoo. regards Martin -- DOS ERROR: Please remove cat from drive A: ___ Dbmail mailing list Dbmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
[Dbmail-dev] dbmail 2.1.7 minor memory leak in db.c and other questions
Paul Aaron,
It seems that there is a minor memory leak in the 2.1.7 release. Sorry.
see minor patch below.
diff -urNab dbmail-2.1.7.orig/db.c dbmail-2.1.7/db.c
--- dbmail-2.1.7.orig/db.c 2006-07-24 11:35:41.0 -0400
+++ dbmail-2.1.7/db.c 2006-07-26 02:04:47.596208079 -0400
@@ -2885,6 +2885,9 @@
*mailboxes = g_list_reverse(*mailboxes);
*errmsg = Everything is peachy keen;
+ g_strfreev(chunks);
+ dm_free(cpy);
+
return DM_SUCCESS;
equery:
I am also wondering about the dm_strnesc in misc.c (line 452) any guesses
if the valgrind error:
==10854== Invalid read of size 1
==10854==at 0x40392E9: dm_strnesc (misc.c:452)
==10854==by 0x4029FF4: _header_cache (dbmail-message.c:942)
==10854==by 0xC9FEE8: (within /usr/lib/libglib-2.0.so.0.902.0)
==10854==by 0xC9FECB: (within /usr/lib/libglib-2.0.so.0.902.0)
==10854==by 0xC9FF22: (within /usr/lib/libglib-2.0.so.0.902.0)
==10854==by 0xC9FF22: (within /usr/lib/libglib-2.0.so.0.902.0)
==10854==by 0x4029D44: dbmail_message_headers_cache
(dbmail-message.c:857)
==10854==by 0x4029A22: dbmail_message_store (dbmail-message.c:778)
==10854==by 0x404075C: insert_messages (pipe.c:581)
==10854==by 0x80496C1: main (main.c:325)
...
is dangerous, I was thinking that since this is dbmail-smtp I was checking
that it might have somthing to do with from address parsing? To recreate
this you should be able to do a valgrind (recomend new version 3.2.0 btw)
on dbmail-smtp and feed it the test-scripts/testbox: e.g.
valgrind --suppressions=contrib/dbmail.supp --tool=memcheck
--leak-check=full --show-reachable=yes .libs/lt-dbmail-smtp -u testuser1
test-scripts/testbox
Also Paul, I seems to be having an error with the current testimap.py
script with the past few releases and svn versions:
..E
==
ERROR: testLsub (__main__.testImapServer)
--
Traceback (most recent call last):
File ./test-scripts/testimap.py, line 343, in testLsub
self.assertEquals('(\\hasnochildren) / %s' % mailboxes[6],
self.o.lsub()[1][7])
IndexError: list index out of range
--
Ran 31 tests in 155.825s
FAILED (errors=1)
any ideas?
Thanks,
Leif
Re: [Dbmail-dev] dbmail 2.1.7 minor memory leak in db.c and other questions
On Wed, July 26, 2006 3:38 am, UEMURA (fka. MAENAKA) Tetsuya wrote: Posted on Wed, 26 Jul 2006 02:55:20 -0400 (EDT) by author Leif Jackson [EMAIL PROTECTED] It seems that there is a minor memory leak in the 2.1.7 release. Sorry. see minor patch below. Leif, your patch isn't a minor. It fixes a major leak which keeps me annoying for a few months now. Without the patch, I can make imapd consuming hundreds MB within a minute. Now I can't do. Thanks a lot. Glad to hear that, did you file a bug report on this? I would recomend next time you see somthing like this let the guys know, that usualy can find it quickly with the steps to replicate. But your welcome glad I count help out. Thanks, Leif -- UEMURA (fka. MAENAKA) Tetsuya [EMAIL PROTECTED] ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] svn HEAD (2117) minor patch
Cool! :) -leif On Sat, May 13, 2006 9:43 pm, Aaron Stone wrote: Thanks! There was even one more in dbmail-imapsession, if !ln failed then re was not freed, about three lines above your fix. Now in SVN. Aaron On Sat, 2006-05-13 at 15:28 -0400, Leif Jackson wrote: Guys, Hey looking real good on the current svn HEAD (rev 2117). I belive I found a few minor memleaks in the imap daemon. Attached is a patch. Thanks, Leif ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Troubles: rev-1996 and 1997 - one failure on testimap
Aaron Paul,
I belive I have come up with a fix for this and I have also found a few
extra possible memory leaks, I will submit a patch for your review here
in a bit but I have a question for Paul and the testimap.py with the
current head compiled shared with mysql and no sieve i got one failed
test:
==
FAIL: testSetacl (__main__.testImapServer)
--
Traceback (most recent call last):
File ../testimap.py, line 439, in testSetacl
self.assertEquals(self.o.setacl('testaclbox','testuser2','slrw')[0],'OK')
File /usr/src/build/394694-i386/install/usr/lib/python2.3/unittest.py,
line 302, in failUnlessEqual
raise self.failureException, \
AssertionError: 'NO' != 'OK'
--
Ran 29 tests in 140.009s
FAILED (failures=1)
Is this truly a failure? or can you update testimap.py to use the correct
check? ..etc.
Thanks,
leif
On Tue, February 28, 2006 5:29 pm, Leif Jackson wrote:
On Tue, February 28, 2006 4:59 pm, Aaron Stone wrote:
On Tue, Feb 28, 2006, Leif Jackson [EMAIL PROTECTED] said:
Guys before 2.1.4 rel, what about that memory leak issue in imapd?,
did you find a workaround? This is the malloc for () keys in the
command parser I am talking about.
Oh. Hrmm... I'll try to take a look at it tonight. I think the
lazy/stupid approach will be to change that global the_args array to be
a static variable inside of the build_args function. Every time the
function is called, it will reap the memory from the previous call.
Exactly the
same fix you made for db_query ;-)
That was the issue, you can't reap/free the memory for each allocation
because you loose the pointer refernece, it leaks because when the the_args
array is free'ed after use you only free the ref to the main allocation. I
belive the correct solution would be to change it to a resizeing realloc
or somthing of that nature so that when freeing the allocated args it
free's all the memory in the buffer because it was realloced each time to
point to the whole thing...etc.
-leif
Aaron
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Troubles: rev-1996 and 1997 - one failure on testimap
On Tue, February 28, 2006 5:54 pm, Leif Jackson wrote: Aaron Paul, I belive I have come up with a fix for this and I have also found a few extra possible memory leaks, I will submit a patch for your review here in a bit what i tried didn't quite workout right so Aaron if you have some ideas go for it I will keep hacking as well, but I have to get back to another project. Thanks, Leif
Re: [Dbmail-dev] Troubles: rev-1996 and 1997 - updated dbmail.supp file
See attached valgrind suppression file. I have added the libmodule leaks to the suppression file. So it make debugging the other stuff a bit better. Didn't know if you guys had already done this Should the suppressions file be included in contrib ? Aaron ..etc. I have also attached the valgrind output for the tests in order from testimap.py if they will help. -leif dbmail.supp Description: Binary data testimap-svn-1999-logs.tar.gz Description: GNU Zip compressed data
Re: [Dbmail-dev] Troubles: rev-1996 and 1997 - one failure on testimap
On Wed, March 1, 2006 1:03 am, Aaron Stone wrote: On Tue, 2006-02-28 at 19:02 -0500, Leif Jackson wrote: On Tue, February 28, 2006 5:54 pm, Leif Jackson wrote: I belive I have come up with a fix for this and I have also found a few extra possible memory leaks, I will submit a patch for your review here in a bit what i tried didn't quite workout right so Aaron if you have some ideas go for it I will keep hacking as well, but I have to get back to another project. The use of g_strfreev here (ironically, for the out-of-memory cases) is unsafe because strfreev tries to free the array itself -- but the_args is static! So I replaced that with a little function to do the same, and do it at the top of every call to build_args_array_ext. The only leak left appears to be this: ==23026== 843 bytes in 12 blocks are definitely lost in loss record 16 of 23 at 0x1B905E6A: calloc (vg_replace_malloc.c:176) by 0x1BA5FE1E: g_malloc0 (in /usr/lib/libglib-2.0.so.0.800.5) by 0x804D8A9: IMAPClientHandler (imap4.c:269) by 0x1B9475EB: manage_start_cli_server (serverchild.c:373) by 0x1B94613C: StartCliServer (server.c:102) by 0x8055270: main (imapd.c:119) The best I can figure is that it's leaking a list, because the only line of code in all of DBMail that calls g_malloc0 is dm_list_nodeadd. Cool!, I doub't this is an accual leak now, as the calloc calls from glib don't agree with valgrind, that is why they are supressed in my dbmail.supp file. Is the size always the same? I suspect it is and we are now in my opinion ready for 2.1.4 :)! Thanks, Leif Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] humm... rev 2000 memleaks?
Aaron, Maybe I spoke too soon just checked out svn rev 2000 and took at look I think you got the main issues with build_args, but I am curious are you running valgrind via the testimap.py? As it seems there might be a few more issues it is finding that you didn't see, I will check and send a patch if I can resolve them or a updated dbmail.supp if they are glib related. Thanks! -leif On Wed, March 1, 2006 1:32 am, Leif Jackson wrote: On Wed, March 1, 2006 1:03 am, Aaron Stone wrote: On Tue, 2006-02-28 at 19:02 -0500, Leif Jackson wrote: On Tue, February 28, 2006 5:54 pm, Leif Jackson wrote: I belive I have come up with a fix for this and I have also found a few extra possible memory leaks, I will submit a patch for your review here in a bit what i tried didn't quite workout right so Aaron if you have some ideas go for it I will keep hacking as well, but I have to get back to another project. The use of g_strfreev here (ironically, for the out-of-memory cases) is unsafe because strfreev tries to free the array itself -- but the_args is static! So I replaced that with a little function to do the same, and do it at the top of every call to build_args_array_ext. The only leak left appears to be this: ==23026== 843 bytes in 12 blocks are definitely lost in loss record 16 of 23 at 0x1B905E6A: calloc (vg_replace_malloc.c:176) by 0x1BA5FE1E: g_malloc0 (in /usr/lib/libglib-2.0.so.0.800.5) by 0x804D8A9: IMAPClientHandler (imap4.c:269) by 0x1B9475EB: manage_start_cli_server (serverchild.c:373) by 0x1B94613C: StartCliServer (server.c:102) by 0x8055270: main (imapd.c:119) The best I can figure is that it's leaking a list, because the only line of code in all of DBMail that calls g_malloc0 is dm_list_nodeadd. Cool!, I doub't this is an accual leak now, as the calloc calls from glib don't agree with valgrind, that is why they are supressed in my dbmail.supp file. Is the size always the same? I suspect it is and we are now in my opinion ready for 2.1.4 :)! Thanks, Leif Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] humm... rev 2000 memleaks?
Here i go again checked dbmail-lmtpd and it has 2 minor problems:
first:
--- dbmail-svn-2.1.3-2000.orig/lmtp.c 2006-03-01 01:46:40.939279000 -0500
+++ dbmail-svn-2.1.3-2000.new/lmtp.c2006-03-01 03:45:56.648141000 -0500
@@ -476,6 +476,8 @@
if (goodtogo) {
/* Sure fine go ahead. */
dm_list_nodeadd(from, tmpaddr, strlen(tmpaddr)+1);
+ /* free tmpaddr as dm_list_nodeadd makes a copy */
+ dm_free(tmpaddr);
ci_write((FILE *) stream,
250 Sender %s OK\r\n,
(char *)(dm_list_getstart(from)-data));
Simple fix but an issue none the less. Then there is the 2nd one
in lmtp.c around line 516:
/* find_bounded() allocated tmpaddr for us, and that's ok
* since dsnuser_free() will free it for us later on. */
dsnuser.address = tmpaddr;
but the comment isn't true thus this leaks ? What do you guys want to do
with this?
void dsnuser_free(deliver_to_user_t * dsnuser)
{
dsnuser-useridnr = 0;
dsnuser-dsn.class = 0;
dsnuser-dsn.subject = 0;
dsnuser-dsn.detail = 0;
/* These are nominally const, but
* we really do want to free them. */
/*
dm_free((char *) dsnuser-address);
dm_free((char *) dsnuser-mailbox);
*/
dsnuser-address = NULL;
Thanks,
Leif
p.s. fix these 2 minor issues and then rel 2.1.4 :) blah..blah just my .02
as always.
On Wed, March 1, 2006 3:09 am, Leif Jackson wrote:
Disregard this. It appears it is all calloc glib stuff and no more memory
leaks exist that I can find in dbmail-pop3d or dbmail-imapd, one issue
for paul to see if it is just me tho is the testimap.py script the
testSetacl doesn't seem to complete when running in network mode, it does
work however in stream mode, I found what I did wrong before that made it
fail.
Thanks,
Leif
p.s. Maybe we will see 2.1.4 tomarrow? :)
On Wed, March 1, 2006 1:50 am, Leif Jackson wrote:
Aaron,
Maybe I spoke too soon just checked out svn rev 2000 and took at look I
think you got the main issues with build_args, but I am curious are
you running valgrind via the testimap.py? As it seems there might be a
few more issues it is finding that you didn't see, I will check and send
a patch if I can resolve them or a updated dbmail.supp if they are glib
related.
Thanks!
-leif
On Wed, March 1, 2006 1:32 am, Leif Jackson wrote:
On Wed, March 1, 2006 1:03 am, Aaron Stone wrote:
On Tue, 2006-02-28 at 19:02 -0500, Leif Jackson wrote:
On Tue, February 28, 2006 5:54 pm, Leif Jackson wrote:
I belive I have come up with a fix for this and I have also
found a few extra possible memory leaks, I will submit a patch
for your review here in a bit
what i tried didn't quite workout right so Aaron if you have some
ideas go for it I will keep hacking as well, but I have to get
back to another project.
The use of g_strfreev here (ironically, for the out-of-memory
cases) is unsafe because strfreev tries to free the array itself --
but the_args is static! So I replaced that with a little function to
do the same, and do it at the top of every call to
build_args_array_ext. The only leak
left appears to be this:
==23026==
843 bytes in 12 blocks are definitely lost in loss record 16 of 23
at 0x1B905E6A: calloc (vg_replace_malloc.c:176) by 0x1BA5FE1E:
g_malloc0 (in /usr/lib/libglib-2.0.so.0.800.5) by 0x804D8A9:
IMAPClientHandler (imap4.c:269) by 0x1B9475EB:
manage_start_cli_server (serverchild.c:373) by 0x1B94613C:
StartCliServer
(server.c:102)
by 0x8055270: main (imapd.c:119)
The best I can figure is that it's leaking a list, because the only
line of code in all of DBMail that calls g_malloc0 is
dm_list_nodeadd.
Cool!, I doub't this is an accual leak now, as the calloc calls from
glib don't agree with valgrind, that is why they are supressed in my
dbmail.supp file. Is the size always the same? I suspect it is and we
are now in my opinion ready for 2.1.4 :)!
Thanks,
Leif
Aaron
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Troubles: rev-1996 and 1997
On Tue, February 28, 2006 3:42 pm, Aaron Stone wrote: On Tue, Feb 28, 2006, Paul J Stevens [EMAIL PROTECTED] said: Aaron Stone wrote: This fixes it for me: --- ../dbmail/imapd.c 2006-02-28 04:15:22.556837664 -0800 +++ imapd.c 2006-02-28 10:16:50.398784536 -0800 case 0: /* child process */ drop_privileges(config.serverUser, config.serverGroup); result = StartServer(config); trace(TRACE_INFO, %s,%s: server done, restart = [%d], __FILE__, __func__, result); - + exit(result); Looks like on-target to me :-). I'm away from my dev box until 6pm -- would you mind checking in this fix? I also think that we should review the other daemons to apply the same fix-pattern. The stress test that every daemon should pass: for i in `seq 1 10`; do kill -HUP `cat /var/run/dbmail-*.pid`; done I had a little fun with the imap daemon this morning, and did this: while [ 1 -eq 1 ]; do kill -HUP... and eventually smashed the signal stack ;-) The daemon code held up until the stack was smashed, though! The server code offers many opportunities for redesign (understatement). That's very much a set target for post-2.2 development, as far as I am concerned. Agreed. No rush replacing something what works for now. But bugs in the current, simple setup should and will be fixed during 2.2's lifetime. I'm still a bit unclear though as to why these problems didn't surface earlier. Yeah, weird. Let's get 2.1.4 out the door! :-D Guys before 2.1.4 rel, what about that memory leak issue in imapd?, did you find a workaround? This is the malloc for () keys in the command parser I am talking about. -leif Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Troubles: rev-1996 and 1997
On Tue, February 28, 2006 4:59 pm, Aaron Stone wrote: On Tue, Feb 28, 2006, Leif Jackson [EMAIL PROTECTED] said: Guys before 2.1.4 rel, what about that memory leak issue in imapd?, did you find a workaround? This is the malloc for () keys in the command parser I am talking about. Oh. Hrmm... I'll try to take a look at it tonight. I think the lazy/stupid approach will be to change that global the_args array to be a static variable inside of the build_args function. Every time the function is called, it will reap the memory from the previous call. Exactly the same fix you made for db_query ;-) That was the issue, you can't reap/free the memory for each allocation because you loose the pointer refernece, it leaks because when the the_args array is free'ed after use you only free the ref to the main allocation. I belive the correct solution would be to change it to a resizeing realloc or somthing of that nature so that when freeing the allocated args it free's all the memory in the buffer because it was realloced each time to point to the whole thing...etc. -leif Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] timsieved
On Tue, February 21, 2006 1:42 am, Oleg Lapshin wrote:
But there is error: dbmysql.c,db_query: res was not freed after the
previous query!
This isn't really an error just a warning message to developers about the
fact that on the previous call to db_query they didn't free the result
before calling db_query again. Aaron added my patch for memory leaks and
in it was this sort of catch all check as I was seeing a lot of query's
that wern't being freeded as the default db_query stores all result's even
if it isn't really needed..etc. I will let Aaron comment on the sieve
stuff.
-leif
If script has error, temporarily record don't deleted from table
dbmail_sievescripts
Log:
pool.c,child_reg_connected: [27272]
serverchild.c,PerformChildTask: incoming connection from [127.0.0.1]
serverchild.c,PerformChildTask: client info init complete, calling client
handler tims(): incoming buffer: [AUTHENTICATE PLAIN {16+}]
tims(): command issued: cmd [AUTHENTICATE], val [PLAIN {16+}]
tims(): command looked up as commandtype 4
misc.c, find_bounded: Found nothing between '' and '' misc.c,
find_bounded: Found [16] of length [2] between '{' and '+' so next
skip [4] db.c,db_usermap_resolve: checking userid 'loe' in usermap
db.c,db_usermap_resolve: client on inet socket [inet:127.0.0.1:2]
dbmysql.c,db_query: executing query [SELECT login, sock_allow, sock_deny,
userid FROM dbmail_usermap WHERE login in ('loe','ANY') ORDER BY
sock_allow, sock_deny] db.c,db_usermap_resolve: login 'loe' not found in
usermap dbmysql.c,db_query: executing query [SELECT user_idnr FROM
dbmail_users WHERE lower(userid) = lower('loe')] dbmysql.c,db_query:
executing query [SELECT user_idnr, passwd, encryption_type FROM
dbmail_users WHERE user_idnr = '2'] authsql.c,auth_validate: validating
using plaintext passwords dbmysql.c,db_query: executing query [UPDATE
dbmail_users SET last_login = '2006-02-21 09:35:58' WHERE user_idnr = '2']
dbmysql.c,db_query: res was not freed after the previous query!
tims(): incoming buffer: [LOGOUT]
tims(): command looked up as commandtype 0
serverchild.c,PerformChildTask: client handling complete, closing streams
serverchild.c,client_close: closing write stream
serverchild.c,client_close: closing read stream
serverchild.c,PerformChildTask: connection closed
serverchild.c,PerformChildTask: waiting for connection
pool.c,child_reg_disconnected: [27272]
pool.c,child_reg_connected: [27270]
serverchild.c,PerformChildTask: incoming connection from [127.0.0.1]
serverchild.c,PerformChildTask: client info init complete, calling client
handler tims(): incoming buffer: [AUTHENTICATE PLAIN {16+}]
tims(): command issued: cmd [AUTHENTICATE], val [PLAIN {16+}]
tims(): command looked up as commandtype 4
misc.c, find_bounded: Found nothing between '' and '' misc.c,
find_bounded: Found [16] of length [2] between '{' and '+' so next
skip [4] db.c,db_usermap_resolve: checking userid 'loe' in usermap
db.c,db_usermap_resolve: client on inet socket [inet:127.0.0.1:2]
dbmysql.c,db_query: executing query [SELECT login, sock_allow, sock_deny,
userid FROM dbmail_usermap WHERE login in ('loe','ANY') ORDER BY
sock_allow, sock_deny] db.c,db_usermap_resolve: login 'loe' not found in
usermap dbmysql.c,db_query: executing query [SELECT user_idnr FROM
dbmail_users WHERE lower(userid) = lower('loe')] dbmysql.c,db_query:
executing query [SELECT user_idnr, passwd, encryption_type FROM
dbmail_users WHERE user_idnr = '2'] authsql.c,auth_validate: validating
using plaintext passwords dbmysql.c,db_query: executing query [UPDATE
dbmail_users SET last_login = '2006-02-21 09:35:58' WHERE user_idnr = '2']
dbmysql.c,db_query: res was not freed after the previous query!
tims(): incoming buffer: [LISTSCRIPTS]
tims(): command looked up as commandtype 3
dbmysql.c,db_query: executing query [SELECT name,active from
dbmail_sievescripts where owner_idnr = 2] tims(): incoming buffer:
[GETSCRIPT spam]
tims(): command issued: cmd [GETSCRIPT], val [spam]
tims(): command looked up as commandtype 6
misc.c, find_bounded: Found [spam] of length [4] between '' and '' so
next skip [5] dbmysql.c,db_query: executing query [SELECT script from
dbmail_sievescripts where owner_idnr = '2' and name = 'spam'] tims():
incoming buffer: [PUTSCRIPT spam {305+}] tims(): command issued: cmd
[PUTSCRIPT], val [spam {305+}]
tims(): command looked up as commandtype 9
misc.c, find_bounded: Found [spam] of length [4] between '' and '' so
next skip [5] misc.c, find_bounded: Found [305] of length [3] between '{'
and '+' so next skip [6] timsieve.c, tims: Client sending script of length
[305]
db.c,db_check_sievescript_quota: updating 2 sievescript quota with 305
dbmysql.c,db_query: executing query [BEGIN]
dbmysql.c,db_query: executing query [SELECT COUNT(*) FROM
dbmail_sievescripts WHERE owner_idnr = 2 AND name = '@!temp-script!@']
dbmysql.c,db_query: executing query [DELETE FROM dbmail_sievescripts WHERE
owner_idnr = 2
Re: [Dbmail-dev] Current svn build problems
On Fri, February 17, 2006 10:29 pm, Aaron Stone wrote: On Fri, 2006-02-17 at 17:35 -0500, Leif Jackson wrote: - dbmail_message_set_header(msg, Return-Path, from-data); + dbmail_message_set_header(msg, Return-Path, (char *)(dm_list_getstart(from)-data)); Committed to SVN. +#ifdef SIEVE Well, no, because the whole point of the module loader is that you can build DBMail without a module, decide you want that module later on, build it, and load it into the same set of binaries you originally had. This is hugely valuable for packagers, they can do this: dbmail.rpm/deb/pkg (whatever) dbmail-pgsql.whatever dbmail-mysql.whatever dbmail-sieve.whatever I'm trying to figure out why sortmodule.c doesn't seem to be building unless --with-sieve is specified. The Makefile.am doesn't separate sortmodule.c from the other two module loaders... Right ok, well I belive it has to with other ifdef's then on the function defs and having at least stub functions if the module is not ld'ed. But I didn't really understand that was the goal. :) -leif Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Current svn build problems
On Sat, February 18, 2006 2:57 am, Aaron Stone wrote: On Sat, 2006-02-18 at 01:59 -0500, Leif Jackson wrote: On Fri, February 17, 2006 10:29 pm, Aaron Stone wrote: I'm trying to figure out why sortmodule.c doesn't seem to be building unless --with-sieve is specified. The Makefile.am doesn't separate sortmodule.c from the other two module loaders... Right ok, well I belive it has to with other ifdef's then on the function defs and having at least stub functions if the module is not ld'ed. But I didn't really understand that was the goal. :) So the broken combination is static build without sieve. I suppose a null driver, sortnull.c, could stub the functions and give a warning: Sieve enabled in config file but was not enabled at compile time. Either that or #ifdef (SIEVE || SHARED). The jury is out. You know as painfull as it can be this may be a good time to work out a loadable at runtime module system using function pointers. Incidentally, the default build appears to be static? acinclude.m4 seems to indicate that builds should be shared by default... From what I get from svn it seams shared is my default. Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] Current svn build problems
On Sat, February 18, 2006 3:33 pm, Aaron Stone wrote: On Sat, 2006-02-18 at 04:47 -0500, Leif Jackson wrote: Either that or #ifdef (SIEVE || SHARED). The jury is out. You know as painfull as it can be this may be a good time to work out a loadable at runtime module system using function pointers. I did that two months ago ;-) dbmodule.c, authmodule.c, sortmodule.c Sorry, shows how long it has been since I have really dug into dbmail's source. I belive the problem is that sortmodule.c is not included in libdbmail.so when --with-sieve is not on the configure line. And thus the function not found issue. However from the looks of my autoconf'ed compile tree the dbmodule authmodule are not the only things being linked in at compile time: gcc -g -O2 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/gmime-2.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -W -Wall -Wpointer-arith -Wstrict-prototypes -o .libs/dbmail-smtp main.o -L/usr/lib ./.libs/libdbmail.so /root/projects/dbmail-svn-2.1.3-1987/modules/.libs/libmysql.so -lmysqlclient -lm /root/projects/dbmail-svn-2.1.3-1987/modules/.libs/libsort_sieve.so /root/projects/dbmail-svn-2.1.3-1987/modules/.libs/libauth_sql.so /usr/lib/libgmime-2.0.so -lgmodule-2.0 -ldl -lgthread-2.0 -lz -lnsl -lgobject-2.0 -lglib-2.0 -lcrypt -Wl,--rpath -Wl,/usr/local/lib -Wl,--rpath -Wl,/usr/local/lib/dbmail -Wl,--rpath -Wl,/usr/lib Correct me if I am wrong but shouldn't optional modules shuch as libsort_sieve.so be a config option for the library as well? and then loaded when the config item is found only? Leif
Re: [Dbmail-dev] Current svn build problems - found it
I belive I have found it. Little case of autoconf/automake nuttiness. I still don't understand how the y do half of what they do but anyway if you change in Makefile.in the SHARED_TRUE to a SHARED_FALSE and vice versa for the other line it compiles fine without the #ifdef SIEVE, the reason is that when defined enabled then on the true autoconf adds the define as # thus commenting it out when we want it uncommented. Also anyone know how to make only the shared the default and leave static no? e.g. how to make STATIC in configure.in have a default ? AM_CONDITIONAL(SHARED, [ test $enable_shared = yes ]) AM_CONDITIONAL(STATIC, [ test $enable_static = yes ]) Thanks, Leif p.s. Aaron you were right a day ago when you said something about the compile having both static and shared and seeming to default to static even though both were enabled. :) diff -urN dbmail-svn-2.1.3-1987.orig/configure dbmail-svn-2.1.3-1987/configure --- dbmail-svn-2.1.3-1987.orig/configure2006-02-18 15:43:45.0 -0500 +++ dbmail-svn-2.1.3-1987/configure 2006-02-18 16:16:58.0 -0500 @@ -6754,7 +6754,7 @@ ;; esac else - enable_static=yes + enable_static=no fi; # Check whether --enable-fast-install or --disable-fast-install was given. diff -urN dbmail-svn-2.1.3-1987.orig/Makefile.in dbmail-svn-2.1.3-1987/Makefile.in --- dbmail-svn-2.1.3-1987.orig/Makefile.in 2006-02-18 15:43:49.0 -0500 +++ dbmail-svn-2.1.3-1987/Makefile.in 2006-02-18 16:19:53.0 -0500 @@ -105,11 +105,11 @@ libdbmail_la-dsn.lo libdbmail_la-sort.lo am__objects_4 = libdbmail_la-dbmodule.lo libdbmail_la-authmodule.lo \ libdbmail_la-sortmodule.lo [EMAIL PROTECTED]@am_libdbmail_la_OBJECTS = $(am__objects_1) \ [EMAIL PROTECTED]@ $(am__objects_2) $(am__objects_3) @[EMAIL PROTECTED] = $(am__objects_1) \ [EMAIL PROTECTED]@ $(am__objects_2) $(am__objects_3) \ [EMAIL PROTECTED]@ $(am__objects_4) [EMAIL PROTECTED]@ $(am__objects_2) $(am__objects_3) [EMAIL PROTECTED]@am_libdbmail_la_OBJECTS = $(am__objects_1) \ [EMAIL PROTECTED]@ $(am__objects_2) $(am__objects_3) \ [EMAIL PROTECTED]@ $(am__objects_4) libdbmail_la_OBJECTS = $(am_libdbmail_la_OBJECTS) @[EMAIL PROTECTED] = check_dbmail_common$(EXEEXT) \ @WITHCHECK_TRUE@ check_dbmail_server$(EXEEXT) \ On Sat, February 18, 2006 3:58 pm, Leif Jackson wrote: On Sat, February 18, 2006 3:33 pm, Aaron Stone wrote: On Sat, 2006-02-18 at 04:47 -0500, Leif Jackson wrote: Either that or #ifdef (SIEVE || SHARED). The jury is out. You know as painfull as it can be this may be a good time to work out a loadable at runtime module system using function pointers. I did that two months ago ;-) dbmodule.c, authmodule.c, sortmodule.c Sorry, shows how long it has been since I have really dug into dbmail's source. I belive the problem is that sortmodule.c is not included in libdbmail.so when --with-sieve is not on the configure line. And thus the function not found issue. However from the looks of my autoconf'ed compile tree the dbmodule authmodule are not the only things being linked in at compile time: gcc -g -O2 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/gmime-2.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -W -Wall -Wpointer-arith -Wstrict-prototypes -o .libs/dbmail-smtp main.o -L/usr/lib ./.libs/libdbmail.so /root/projects/dbmail-svn-2.1.3-1987/modules/.libs/libmysql.so -lmysqlclient -lm /root/projects/dbmail-svn-2.1.3-1987/modules/.libs/libsort_sieve.so /root/projects/dbmail-svn-2.1.3-1987/modules/.libs/libauth_sql.so /usr/lib/libgmime-2.0.so -lgmodule-2.0 -ldl -lgthread-2.0 -lz -lnsl -lgobject-2.0 -lglib-2.0 -lcrypt -Wl,--rpath -Wl,/usr/local/lib -Wl,--rpath -Wl,/usr/local/lib/dbmail -Wl,--rpath -Wl,/usr/lib Correct me if I am wrong but shouldn't optional modules shuch as libsort_sieve.so be a config option for the library as well? and then loaded when the config item is found only? Leif ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] (no subject)
Current svn build problems... minor fixes.. Without sieve enabled there is
a missing ifdef and the Return-Path argument is of the wrong type...
diff -urN dbmail-svn-2.1.3-1987.orig/lmtp.c dbmail-svn-2.1.3-1987/lmtp.c
--- dbmail-svn-2.1.3-1987.orig/lmtp.c 2006-02-17 17:23:43.0 -0500
+++ dbmail-svn-2.1.3-1987/lmtp.c2006-02-17 17:44:23.244784376 -0500
@@ -616,7 +616,7 @@
return 1;
}
- dbmail_message_set_header(msg,
Return-Path, from-data);
+ dbmail_message_set_header(msg,
Return-Path, (char *)(dm_list_getstart(from)-data));
if (insert_messages(msg,
headerfields, rcpt) == -1) {
ci_write((FILE *) stream,
503 Message not
received\r\n);
} else {
diff -urN dbmail-svn-2.1.3-1987.orig/sort.c dbmail-svn-2.1.3-1987/sort.c
--- dbmail-svn-2.1.3-1987.orig/sort.c 2006-02-17 17:41:10.323157000 -0500
+++ dbmail-svn-2.1.3-1987/sort.c2006-02-17 17:41:45.705486863 -0500
@@ -42,6 +42,7 @@
}
}
+#ifdef SIEVE
/* Sieve. */
config_get_value(SIEVE, DELIVERY, val);
if (strcasecmp(val, yes) == 0) {
@@ -52,6 +53,7 @@
sort_free_result(sort_result);
}
}
+#endif
/* Sieve actions:
* (m = must implement, s = should implement, e = extension)
Thanks,
Leif
p.s. Aaron Paul I am excited to see all the great changes going into
svn! Thanks for all your time and efforts!
Sorry Subject Current svn build problems, minor patch Re: [Dbmail-dev] (no subject)
On Fri, February 17, 2006 5:35 pm, Leif Jackson wrote:
Current svn build problems... minor fixes.. Without sieve enabled there
is a missing ifdef and the Return-Path argument is of the wrong type...
diff -urN dbmail-svn-2.1.3-1987.orig/lmtp.c dbmail-svn-2.1.3-1987/lmtp.c
--- dbmail-svn-2.1.3-1987.orig/lmtp.c 2006-02-17 17:23:43.0
-0500
+++ dbmail-svn-2.1.3-1987/lmtp.c2006-02-17 17:44:23.244784376
-0500
@@ -616,7 +616,7 @@
return 1; }
- dbmail_message_set_header(msg,
Return-Path, from-data);
+ dbmail_message_set_header(msg,
Return-Path, (char *)(dm_list_getstart(from)-data));
if (insert_messages(msg, headerfields, rcpt) == -1) {
ci_write((FILE *) stream, 503 Message not
received\r\n); } else {
diff -urN dbmail-svn-2.1.3-1987.orig/sort.c dbmail-svn-2.1.3-1987/sort.c
--- dbmail-svn-2.1.3-1987.orig/sort.c 2006-02-17 17:41:10.323157000
-0500
+++ dbmail-svn-2.1.3-1987/sort.c2006-02-17 17:41:45.705486863
-0500
@@ -42,6 +42,7 @@
}
}
+#ifdef SIEVE
/* Sieve. */
config_get_value(SIEVE, DELIVERY, val); if (strcasecmp(val, yes) ==
0) {
@@ -52,6 +53,7 @@
sort_free_result(sort_result); }
}
+#endif
/* Sieve actions:
* (m = must implement, s = should implement, e = extension)
Thanks,
Leif
p.s. Aaron Paul I am excited to see all the great changes going into
svn! Thanks for all your time and efforts!
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] [DBMail 0000297]: memory leaks patch
Aaron, Thanks for getting this into CVS, odd tho the md5 issue didn't seem to leak anywhere else. But I am sure it would have. Any ideas for the currently supressed build_args_array_ext leak, this is kinda a big issue in my mind as it leaks anywhere from 16bytes to 1600bytes on each connection and depending on the daemon...etc. Thoughts? -leif p.s. maybe dbmail.supp and the valgrind info should be added to SVN with some hacking info? On Wed, February 8, 2006 1:36 am, [EMAIL PROTECTED] wrote: The following issue has been RESOLVED. == http://www.dbmail.org/mantis/view.php?id=297 == Reported By:ljackson Assigned To:aaron == Project:DBMail Issue ID: 297 Category: General Reproducibility:always Severity: major Priority: normal Status: resolved Resolution: fixed Fixed in Version: SVN Trunk == Date Submitted: 01-Feb-06 22:58 CET Last Modified: 08-Feb-06 07:36 CET == Summary:memory leaks patch Description: I did some digging in the current svn trunk and discoverd a few memory leaks. I was able to create a patch for them, However there is one I have found that doesn't have an easy fix. That is build_args_array_ext function in imap session code. I wille attache to this report and valgrind suppressions file to help in narrowing down the memory leaks or any other issues in dbmail, this supresses a whole bunch of mess from glib and gmime. This is still includes the fix for authsql memleak as it doesn't appear that it got into svn trunk. I added a check before db_query that looks for unfreeed result sets and frees them automaticaly to avoid this in the future. == -- ljackson - 07-Feb-06 19:18 -- Aarron or Paul have you guys had a chance to look at this? And maybe the dbmail.supp file will help you with your sieve code..etc. Oh there is a proccess in which you can debug/memcheck dbmail in your working directory without re-compiling to static. when you finish building your working directory, run dbmail-daemon -n once and then exit it. you will find in .libs/lt-dbmail-daemon which then can be passed to gdb and or valgrind memcheck: e.g. valgrind --suppressions=dbmail.supp --tool=memcheck --leak-check=full --show-reachable=yes .libs/lt-dbmail-pop3d -n or if you want to accualy watch it in deamon mode set your dbmail.conf down to 2 children and run: valgrind --suppressions=dbmail.supp --tool=memcheck --leak-check=full --show-reachable=yes --trace-children=yes --log-file=/tmp/valtrace libs/lt-dbmail-pop3d this will make after a exit a file /tmp/valtrace.masterpid that will have the memory debugging in it. Thanks, Leif p.s. Paul any ideas on the x86_64 issue? -- aaron - 07-Feb-06 19:32 -- I've been using the dbmail.supp file this week, and it's really useful. Thanks! I admit that I haven't looked at the patch at all until just now; it's fairly short, so I'll go over it this evening. Thanks again! -- aaron - 08-Feb-06 07:36 -- Nice patch! Now in SVN. The makemd5-as-argument bug is really a class of memory leaks. I fixed makemd5 in two more places in authsql.c, and grepped to ensure it wasn't broken anywhere else (fortunately it wasn't). Issue History Date Modified Username FieldChange == 01-Feb-06 22:58 ljackson New Issue 01-Feb-06 22:58 ljackson File Added: dbmail-svn-2.1.3-1970-memleaks.patch 01-Feb-06 22:58 ljackson File Added: dbmail.supp 07-Feb-06 19:18 ljackson Note Added: 985 07-Feb-06 19:32 aaron Note Added: 986 08-Feb-06 07:36 aaron Status new = resolved 08-Feb-06 07:36 aaron Fixed in Version = SVN Trunk 08-Feb-06 07:36 aaron Resolution open = fixed 08-Feb-06 07:36 aaron Assigned To = aaron 08-Feb-06 07:36 aaron Note Added: 987
[Dbmail-dev] current trunk v 1967 authsql memory leak on connection
Hey guys Just taking a new look at the code you guys have been working
so hard on. Looks really good! I have a minor patch that doesn't even
warrent a bug report so I have attached it here.
On another note I wanted to know if either Arron or Paul have tried the
current svn trunk on x86_64 bit it seems that with the trace function in
debug.c that there is a 64bit issue, I can't seem to track it down, if I
turn off syslog tracing and just use stderr all works fine. Os is CentOS
4.2 with current updates, gmime 2.1.17 and glib2 2.9.3, BTW the 2.9.2
doesn't work on x86_64 and nither does 2.9.4 :/ Thread issues in the
verification.
Thanks,
Leif
diff -urN dbmail-svn-2.1.3-1967.orig/modules/authsql.c dbmail-svn-2.1.3-1967/modules/authsql.c
--- dbmail-svn-2.1.3-1967.orig/modules/authsql.c2006-01-25 12:09:08.0 -0500
+++ dbmail-svn-2.1.3-1967/modules/authsql.c 2006-01-25 16:33:50.0 -0500
@@ -479,6 +479,7 @@
}
if (is_validated) {
+ db_free_result();
/* log login in the dbase */
snprintf(__auth_query_data, AUTH_QUERY_SIZE,
UPDATE %susers SET last_login = '%s'
@@ -488,6 +489,7 @@
if (__auth_query(__auth_query_data) == -1)
trace(TRACE_ERROR, %s,%s: could not update user login time,
__FILE__, __func__);
+ db_free_result();
} else {
*user_idnr = 0;
db_free_result();
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
All, Just as a note I never intended for this code to get back ported to 2.0, so I guess if you guys want to work it out that is cool, however my vote would be to remove it rather than fix it and have a patch on the side if someone wants to fool with it. Otherwise I think it is a waste of time to work on the code for 2.0 as it is limted by the parsing system and other issues people are improving on in 2.1. just my .02 -leif I'm putting the two strands of this thread back together ;-) ... On Tue, Jan 25, 2005, Leif Jackson [EMAIL PROTECTED] said: That US-ASCII isn't recognized at all means that a fallback is always used. I would be interested in seeing the command that SquirrelMail uses. Perhaps we'd be much better off replacing the tries-to-be-general _ic_sort with an actual hack -- one that recognizes specific SORT commands that we know we can implement and that we've seen used by clients in the wild. I thought about making it a specific hack but then I knew it would be a wast of time when we got to x.x version of dbmail with header cahcing where it can be done right. Anyway I agree with you guys, and say leave 2.0.x alone with the _ic_sort and just make a push to make it right for 2.1 ..etc. Tho latly I have been way to busy to help so you can ignore me if you want :) We're not getting to 2.1 anytime soon, and I don't want to leave this broken in 2.0 if we can make a sane set of workarounds. Which is to say, all of the strings that are recognized and then skipped because the code says TODO are essentially silent failures to perform the requested sort command. That's not OK. If we get a string that we don't have the SQL to handle, then we should give a suitable error message and allow the client to work around us. I tried that yesterday. Won't work even for ARRIVAL, because of the way the result set is constructed in db_search. But perhaps I was just being dense. Accualy paul you could do this with a search of the cmd string to set a key for reverse and then just tell my btree code not to re-order the mid's as it takes them in reverse to being with if I rember correctly. I'll start reading through this part of the code, as I've never done much work on the IMAP side of things. What I'd like to do is take a survey of the SORT command options, and then see what sort of SQL we can reasonably do, and then work backwards saying, We have SQL than can do X, Y and Z so we'll only accept SORT strings involving X, Y and Z. If the SORT string asks for Q, then we give an error. Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
On Tue, Jan 25, 2005, Paul J Stevens [EMAIL PROTECTED] said:
I was pleasantly surprised by your command trace yesterday. Apparently
c-client is smart. Smart enough to hide server-errors from the client.
The fact that
A01 SORT (REVERSE ARRIVAL) US-ASCII ALL
failed is very important in this respect. The build_imap_search function
is full of TODO statements for search keys that are silently ignored
whereas US-ASCII and UTF-8 which are both explicitely required...
[snip]
Looks like SORT (ARRIVAL) US-ASCII ALL kinda works in 2.0.3, because for
ARRIVAL, there's:
if(sorted (strcasecmp(search_keys[*idx], arrival) == 0)) {
key.type = IST_SORT;
strncpy(key.search, order by pms.internal_date, MAX_SEARCH_LEN);
(*idx)++;
Whereas (REVERSE ARRIVAL) fails because REVERSE doesn't work. In 2.0.2,
US-ASCII itself threw an error, and so in both cases fallbacks were used.
So REVERSE ARRIVAL can be done just with a desc in the order by
clause. However REVERSE with any of the other sorts, working or not, won't
work.
That US-ASCII isn't recognized at all means that a fallback is always
used. I would be interested in seeing the command that SquirrelMail uses.
Perhaps we'd be much better off replacing the tries-to-be-general _ic_sort
with an actual hack -- one that recognizes specific SORT commands that we
know we can implement and that we've seen used by clients in the wild.
I thought about making it a specific hack but then I knew it would be a
wast of time when we got to x.x version of dbmail with header cahcing
where it can be done right. Anyway I agree with you guys, and say leave
2.0.x alone with the _ic_sort and just make a push to make it right for
2.1 ..etc. Tho latly I have been way to busy to help so you can ignore me
if you want :)
-leif
Aaron
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron Stone wrote:
On Tue, Jan 25, 2005, Paul J Stevens [EMAIL PROTECTED] said:
I was pleasantly surprised by your command trace yesterday. Apparently
c-client is smart. Smart enough to hide server-errors from the client.
The fact that
A01 SORT (REVERSE ARRIVAL) US-ASCII ALL
failed is very important in this respect. The build_imap_search function
is full of TODO statements for search keys that are silently ignored
whereas US-ASCII and UTF-8 which are both explicitely required...
[snip]
Looks like SORT (ARRIVAL) US-ASCII ALL kinda works in 2.0.3, because for
ARRIVAL, there's:
if(sorted (strcasecmp(search_keys[*idx], arrival) == 0)) {
key.type = IST_SORT;
strncpy(key.search, order by pms.internal_date, MAX_SEARCH_LEN);
(*idx)++;
Whereas (REVERSE ARRIVAL) fails because REVERSE doesn't work. In 2.0.2,
US-ASCII itself threw an error, and so in both cases fallbacks were
used.
So REVERSE ARRIVAL can be done just with a desc in the order by
clause. However REVERSE with any of the other sorts, working or not,
won't
work.
I tried that yesterday. Won't work even for ARRIVAL, because of the way
the
result set is constructed in db_search. But perhaps I was just being
dense.
Accualy paul you could do this with a search of the cmd string to set a
key for reverse and then just tell my btree code not to re-order the mid's
as it takes them in reverse to being with if I rember correctly.
Thanks,
Leif
p.s. Until there is header caching anything but Date sort is painfully
slow becuase the code has to reparse all the messages headers with the
mime code. If we could somehow make the extensible table for header values
and keys and then a table of key refs we could make the mime parsers just
check for the value for that uniq key in the cache if it isn't there add
it and then just have a ref to the messages uniq id, then anytime the mime
parse header is called it could check for an already parsed header. This
is my .02 of how I would quickly whip it out, I wanted to get a little
crazy also and add memcached interface to the header caching code for
large sites like my customers where many many people will be grinding
throught the cached headers..etc.
Re: [Dbmail-dev] [DBMail 0000166]: IMAP sort breakage in 2.0.3
Aaron, I've reverted _ic_sort behaviour to pre-2.0.3. Can you confirm this wrt twig? This has given me something to think about: - Leif interleaved the sort with the search code. There are some valid reason's for doing so, but we need to treat the search arguments fundamentally differently from the sort keys, where they are currently part of the same struct. Messy. - Given that my understanding of the current sort code is less than perfect, I suggest that the sort keys *not* be used for search commands, as is now (partly) the case. Instead, the search keys should be used for determining the result set, and next the sort keys should be used for reordering the set. This, it seems, would require the search run to use the sort keys for retrieving all the message attributes that will later on be used for sorting. I would say that as the _ic_sort was mostly just a stop gap untill we had header caching available, that it is obvious the whole sort part of dbmail should be reworked, however the current implementation is near optimal for at least the sort needed for squirrelmail, as you mentioned before I never intended this to be anything but for SM. If anyone has time to work on code I would urge we focus on header caching so as to be able to make the sort and other items work well for 2.1 and leave the sort code that was merged into 2.0 as is and make a not that it is intened for SM only, or add a option to enable the sort code and runtime...etc. Thanks, Leif
Re: [Dbmail-dev] pthreads support?
On Wed, Sep 29, 2004 at 04:24:38PM +0200, Paul J Stevens wrote: Agreed. And that's what I'm working on already. However, doing so would be a lot easier if we implement some kind of test-frame. Its not a solution, tho it may help. A real solution as I said earlier is to simplify processes, enhance function calls, and probably remove hand crafted parsers for ones which are created by bison. This would really help because then there is always a maintainable interface to the parser and bison generally tries to optimize to efficiency. I belive i have the starting of the bison grammer worked out using another GPL'ed project so I was going to focus on that when I had the next block of time I will be send a patch in a few days for the GList replacement for the current list code this will make things a lot cleaner and more elegant in my mind or rather how it works is more elegant. On another note paul you have a patch you can send me directly on that Gmime code so I can check that out and maybe help? On another note Aaron have you had a moment to look at the sieve patch and see if you can put it into cvs head? If not wait I guess as it needs some work to be thread safe *shrug* like I said it was just a framework when I did it. Thanks, Leif
Re: [Dbmail-dev] pthreads support?
Just check out the nfg-0-1 branch. Its in a unfinished state at the moment, so you won't be able to compile it completely yet. Like I said, I'm removing the mime, cache, and list code, so you can imagine there's a lot of changes going into imapcommands.c and imaputils.c. OK will do going to bed now, but I will send you for your sake on the liswt code my diff that I have been working on mind you just use the list.c and related functions and I belive it will help you to understand a little bit more about glib which has it's own quirks. But at least you will have a tested and working list.c implementation and you have good example code through most of imap and all of pop3. Thanks, leif
Re: [Dbmail-dev] pthreads support?
Leif Jackson wrote: Could you please explain how Oracle's connection overhead directly translates to we need threads? A new thread still has to open up a new connection to the DB. It sounds to me that Oracle's connection overhead (or Postgresql or whatever) is more an argument for persistent connections and pre forking more than threads. But if I'm missing something please let me/us know. I know a good bit about oracle (and maybe re-writing a new driver for it) but that wasn't my driving force behind the code work... as for my implementation it uses thread pool's and so it only opens a connection the first time a thread is used then it keeps that connection open in memory for the next client and then closes all of them (up to MAXCHILDREN) at the end of the life of the daemon. Ok, so you are using thread pools to implement persistent connections. I would still like to hear more details about the 60% speedup of the threaded pop3 daemon vs the non-threaded. Exactly what was 60% faster? I still think that the Multi-Process Model can be just as effective as a threaded model with the benefit of simplified code. Yeah, the overhead of fork and not having to have shared memory segments is where i see the speedup. I belive the rest of the speedup comes from chaning out Glist and some other things. But I have some neat things up my sleave..etc. I accualy think I just found a faster mime parser that is GPL'ed than GMime. it is stable and well tested as well as thread-safe which GMime may or maynot be acording to thier docs...etc. I will see what I feel like when I get to that point.. :) The other library is called EPS: http://www.inter7.com/index.php?page=eps I guess the mime stuff is a RFC...etc. BTW, I'm not trying to be negative about your work, I'm just trying to discuss the options and get down to the real details. I don't want us making radicle changes without good well thought out, discussed and agreed upon reasoning. Yeah I know not thinking that at all, I am doing it for someone else weather or not the dbmail developers decied to use any or all of it. So it works out for all of us :) You can thank my customer more they are the ones paying me to fiddle with it :P Thanks, Leif Thanks for the hard work! Matthew ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] pthreads support?
Leif Jackson wrote: Ok, so you are using thread pools to implement persistent connections. I would still like to hear more details about the 60% speedup of the threaded pop3 daemon vs the non-threaded. Exactly what was 60% faster? I still think that the Multi-Process Model can be just as effective as a threaded model with the benefit of simplified code. Yeah, the overhead of fork and not having to have shared memory segments is where i see the speedup. I belive the rest of the speedup comes from chaning out Glist and some other things. But I have some neat things up my sleave..etc. I accualy think I just found a faster mime parser that is GPL'ed than GMime. it is stable and well tested as well as thread-safe which GMime may or maynot be acording to thier docs...etc. I will see what I feel like when I get to that point.. :) The other library is called EPS: http://www.inter7.com/index.php?page=eps I guess the mime stuff is a RFC...etc. Interesting, I look forward to hearing about this new parser. I'm sure there is room for improvement in our parsing routines. Going over it with dmalloc now to see if it is even worth the effort so far not bad :) Very fast on it's test mails going to feed it junk in a bit, Anyone on the list have the e-mails that dbmail used to barf on in a tgz they could send me for testing? While the overhead of creating a new thread is less than the overhead of calling fork, I still don't think this will buy us much since the fork call typically doesn't account for a significant percentage of the work done after connection, and even if it did, preforking can handle a lot of this. Again there is a distinction between the workload a light http server that doesn't to much thinking vs. an IMAP server that does a lot of work during the average connection so as to make the fork overhead negligible. Maybe as everyone has said the proof will be when others try my patches as I have seen a drastic improment in the pop3 code, and as I have said i am not totaly done with imap. As for the overhead of shared memory segments, I claim no expertiece here, but I'm surprised to hear that this is a performance bottleneck. Are you sure? Well not to take anything away for paul's great work! (thanks paul for the pre-fork stuff btw! ) I have seen on my test bed that the attach and detach way that it is currently done in dbmail is a bottleneck for number of simultanious connections but not throughput. Yeah I know not thinking that at all, I am doing it for someone else weather or not the dbmail developers decied to use any or all of it. So it works out for all of us :) You can thank my customer more they are the ones paying me to fiddle with it :P Well I would think I speak for the whole community when I say that we would like to benefit from you work :-). One general comment which I think was alluded to earlier in this thread. If you want to get some of this applied, I think it best to separate out different parts of the patch. It may not make sense for you right now if you are trying to throw together a proof of concept but in the long run all the GMime changes etc, should probably be made separate from the threading changes. Yeah understood, definaly not somthing I can do right now. Maybe once I am to a testing point for the public (dbmail-dev) someone can take on that task as I am not sure I will have the time. Thanks, Leif
Re: [Dbmail-dev] pthreads support?
Gerrit P. Haase wrote: Leif Jackson wrote: Hello, please consider using one single process with threads, for each client one thread, instead the slower 'fork a child for every client' or Apache like pre-fork technique. Our current 2.1 cvs head has pre-fork and I am currently working on a high performance version with pthreads which is coming along well but still is a while out. Hey, this is great news, I just started using lighttpd which uses pthreads too and it is a lot faster than Apache therefore. So I thought there are chances that it will speed up dbmail too. Threads may speed up dbmail a little, but I don't think the forking model currently is much of a bottleneck. Combined with a redesign of the imap codebase however, dbmail on pthreads will kick ass. Well going to bed now but I thought I would respond, the threading patch has been completed for the pop server and halfway through making the imap daemon thread safe... It is a major bunch of changes as well as I changed out the list.c for the GList stuff because we have glib in the requirments might as well start using it :) As for the threading code it makes about a 60% speedup on the pop3 server alone and I can't state much yet for the imap server but between the pthreads and the new list code I have finaly eradicated the last few minor memory leaks I used to see from the list_nodeadd stuff that bugged me! The diff so far is almost 400k so it is a major re-think of some areas...etc.. Anyway a little punchy 4:36am here, been coding almost non-stop all weekend just got a bug up my $#% if you know what I mean... Later, Leif
Re: [Dbmail-dev] refactoring plans for 2.1
Along with this I would like to start a discussion on the possiblity for doing away with the fork code and going to a pthread model with a thread pool, I have already starting some proof of concept code on my own, but with the performance on threads in linux 2.6.8 and the performance of threads on my Sun boxes this would give DBmail a leap ahead. Let me know the thoughts ...etc from everyone on if you guys think it would make any since to switch to a threading pooled model. Thanks, Leif Hi all, As promised I've been busy reworking imapcommands.c lately. Since what I'm doing will be pretty invasive wrt the imap code I want to share with you some of the progress made and what I plan to do... 1) change the signature for all _ic_XXX functions 2) change the way results are gathered and flushed to to client 3) use glib and gmime extensively ad 1) Currently all _ic_XXX functions look like: int _ic_XXX(char *tag, char **args, ClientInfo * ci) I plan to change this to int _ic_XXX(struct ImapSession *self) where a struct ImapSession contains much of the currently globalized variables in imapcommands.c and imaputil.c This will enable splitting up of _ic_fetch and friends, and generally make life easier for people who want to refactor the imap code. ad 2) Currently all results are written to the client by calling fprintf all over the place (or ci_write which is a wrapper). I'm almost finished with rewriting the imapcode and replace all those fprintf's with a GList setup. This helps building up clean lists of items to be returned to the client, and sanatize their formatting. No more trailing spaces in parentisized lists, etc... ad3) While working at (2) I began using GLib extensively, but now I've also started working on the actual message-retrieval code. Using gmime there makes a lot of sense, and I've already rewritten some the rfcsize code. Using GMime will help clean up the message parsing *a lot*. All this is being worked on on the nfg-0-1 branch. Code is currently working well, in sync with head as of sept 16, and tested with both the testframe and thunderbird. Of course, doing this started out with a plan to phase-in the is_header field, and start working toward a header-caching setup. Those plans will largely determine what I'll be working on in this branch. -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
RE: [Dbmail-dev] refactoring plans for 2.1
If we're using Glib, then let's also use Glib's threading interface. It gives us transparency over a variety of thread implementations and platforms, and provides for graceful failure in case there is no thread implementation at all. The issue is that glib's thead pool won't work for an application server like dbmail, it is a pool or worker functions I wrote another program using glib's thread interface and it is a nightmare. I have a thread pool implementaion that is mostly from the pthreads book by o'reilly with real world updates and improments to it. It should be portable and if we make the threads configurable that would take care of the systems that don't allow for threads. In the event of no thread implementation, or threads turned off, there would basically be one thread per process and we could act as we do now. In such a case, I think it would be ideal if the main parent process could keep track of how many threads are running, and spawn processes until the proper thread count is reached. On another note, dbmail shouldn't be run on systems without threads as off the top of my head I can't think of one that it would even compile on right now. Sun Solaris, linux and the bsd's have pthreads without any issues it is more of an issue to support glib on other system other than linux than it is to support pthreads. Just my thoughts as always -leif
Re: [Dbmail-dev] CVS Patch for dynmaic table prefixes...
Paul,
On the default where if the prefix is not specified it needs to be
changed to look like this. Didn't feel like doing a diff. This is because
we need pople to be able to specify a prefix of your fix doesn't allow
for this I have changed my config.c function to look like such:
void GetDBParams(db_param_t * db_params)
{
field_t port_string;
field_t sock_string;
if (GetConfigValue(host, DBMAIL, db_params-host) 0)
trace(TRACE_FATAL, %s,%s: error getting config!,
__FILE__, __func__);
if (GetConfigValue(db, DBMAIL, db_params-db) 0)
trace(TRACE_FATAL, %s,%s: error getting config!,
__FILE__, __func__);
if (GetConfigValue(user, DBMAIL, db_params-user) 0)
trace(TRACE_FATAL, %s,%s: error getting config!,
__FILE__, __func__);
if (GetConfigValue(pass, DBMAIL, db_params-pass) 0)
trace(TRACE_FATAL, %s,%s: error getting config!,
__FILE__, __func__);
if (GetConfigValue(sqlport, DBMAIL, port_string) 0)
trace(TRACE_FATAL, %s,%s: error getting config!,
__FILE__, __func__);
if (GetConfigValue(sqlsocket, DBMAIL, sock_string) 0)
trace(TRACE_FATAL, %s,%s: error getting config!,
__FILE__, __func__);
if (GetConfigValue(table_prefix, DBMAIL, db_params-pfx) 0){
trace(TRACE_FATAL, %s,%s: error getting config!,
__FILE__, __func__);
if (strlen(db_params-pfx) == 0)
strncpy(db_params-pfx, DEFAULT_DBPFX, FIELDSIZE);
}
/* check if port_string holds a value */
if (strlen(port_string) != 0) {
db_params-port =
(unsigned int) strtoul(port_string, NULL, 10);
if (errno == EINVAL || errno == ERANGE)
trace(TRACE_FATAL,
%s,%s: wrong value for sqlport in
config file, __FILE__, __func__);
} else
db_params-port = 0;
/* same for sock_string */
if (strlen(sock_string) != 0)
strncpy(db_params-sock, sock_string, FIELDSIZE);
else
db_params-sock[0] = '\0';
}
notice the DEFAULT_DBPFX is withing the case check for the item in the
config file.
Thanks,
leif
Re: [Dbmail-dev] sort patch now in CVS
Paul, Glad it will work in the current CVS head, thanks for putting it in there... As you said and I had planed it is just a stop gap till we have header caching. But the btree works well it needs some work for other clients than sqmail as other clients use diffrent sort parameters and I didn't finish all the options to the SORT command per the RFC. Thanks, Leif Hi all, Following up on the performance thread I decided to checkout Leif's patch. It works great. I had to merge his code manually mostly due to Leif's indent and braces styles and the table changes since rc3, but that was no big deal. I've committed the patch to cvs-head. Please let me know how it works for you. If this hold up ok, I'll include this patch in the 2.0.0 debian packages as well. Thank you Leif. The sqmail crowd will love this. -- Paul Stevens [EMAIL PROTECTED] NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31 The Netherlands___www.nfg.nl ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
[Dbmail-dev] Patch for dynmaic table prefixes... please consider ASAP
Hi All, Paul or whomever is the one to review and add patches these days, I have made a patch for current CVS head (2.1) that allows for dynamic table prefixes as I am tired of patching the source each time I want to use the latest CVS on my test platforms... :) Please consider and apply to the codebase for 2.1 asap. This requires the addition of table_prefix to the dbmail.conf Thanks, Leif Jackson dbmail-2.1cvs091404-dynamicprefix.diff Description: Binary data
Re: [Dbmail-dev] Patch for dynmaic table prefixes... please consider ASAP
On Tue, 14 Sep 2004 15:59:51 -0400 (EDT), Leif Jackson [EMAIL PROTECTED] wrote: Hi All, Paul or whomever is the one to review and add patches these days, I have made a patch for current CVS head (2.1) that allows for dynamic table prefixes as I am tired of patching the source each time I want to use the latest CVS on my test platforms... :) Please consider and apply to the codebase for 2.1 asap. This requires the addition of table_prefix to the dbmail.conf I'll have a look at this tomorrow (it's 10:15PM here now.. Why am I reading my e-mail?) I take it that this patch has been working for you for some time already? nope just wrote it 15 min before sending the patch but it has been tested and it doesn't add anything but one option to the db_params struct. -leif Ilja ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] LTDL Code
Guys, I disagree with the statement that LTDL makes it impossible to debug. I fact it is quite easy to debug dynamic libraries with gdb, as well as valgrind. What about this code can you not figure out. I haven't taken the time to look at it yet since I only use MySQL but I understand the need for it. -Leif Hi Dan, Dan Weber wrote: Ilja Booij wrote: 2 things in pgsql/dbgsql.c #include string.h is missing there's a typo in the db_api struct: it reads db_num_fielqs where it should read db_num_fields. After this I get some compiler warnings and it won't run because of some crash in the configuration code (config.c) Ilja Yeah I know. ltdl makes it impossible to debug the library code so all I know is it fails at db_connect. I have been the mysql code compiles and is what should be tested at the moment. ltdl makes it impossible to debug? Would this be resolved by having an option to do normal linking? I wouldn't like to have a situation where it's almost impossible to debug our code. Ilja ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] messageblks logic
Yes, it makes the code more complicated, but I think it's only because we still haven't fully adapted our thinking to our model. It occurred to me today that if we were MIME parsing at delivery time, we could also store the mime structure of a message in the database. An IMAP BODY.PEEK for mime structures would be nearly instantaneous. This is probably the single most common request from an email client, especially web-based ones that need to refresh their message list on almost every page hit. This can be very interesting to do. Of course, we still need to store the message in its original format, but to store some information on mime-parts would be very benificial for performance. This would also lend to the header caching, or at least let me finish up a sort and thread implementation that would be very fast. just my 0.02 -leif
Re: [Dbmail] DBMail and IMAP ... dropping connections
I am betting that my mysql auto-reconnect change that went into 2.x devel tree didn't get into 1.2.x Ilja? maybe we can get out a 1.2.7b anyway in the db_connect area you have to add a line like... /* auto re-connect */ conn.reconnect = 1; this is right after the mysql_init in the dbmysql.c (around line 68) and then re-compile it shouldn't ever have this issue again. -Leif I got the 1.2.6 dbmail installed and working perfect...except... I'm running dbmail-pop3 and dbmail-imapd. They both work fine for about a day. Almost exactly 24 hours (don't think that's important) later the IMAP server starts dropping connections and it won't let me check my mail. The pop3 server is still functioning fine...and postfix is working fine too...this seems to be only related to the dbmail-imapd server. This has occurred for the last 4 days in a row. MySQL Server is running fine and max connections was around 121 (configured to allow 500 connections). The maillog file is as follows: Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: PerformChildTask(): incoming connection from [127.0.0.1 (MAIL-SRV)] Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: PerformChildTask(): client info init complete, calling client handler Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: IMAPClientHandler(): line read for PID 26393 Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: COMMAND: [A001 LOGIN [EMAIL PROTECTED] admin99366] Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: arg[0]: 'hidden-email' Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: arg[1]: 'hidden-pass' Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: IMAPClientHandler(): Executing command login... Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: _ic_login(): trying to validate user Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: __auth_query(): connection failure Apr 30 07:15:24 MAIL-SRV dbmail/imap4d[26393]: auth_validate(): could not select user information Apr 30 07:15:25 MAIL-SRV dbmail/imap4d[26393]: _ic_login(): user (id:18446744073709551615, name [EMAIL PROTECTED]) tries login^M Apr 30 07:15:25 MAIL-SRV dbmail/imap4d[26393]: _ic_login(): db-validate error while validating user hidden-email (pass hidden-pass). Apr 30 07:15:25 MAIL-SRV dbmail/imap4d[26393]: IMAPClientHandler(): Finished command login Apr 30 07:15:25 MAIL-SRV dbmail/imap4d[26393]: IMAPClientHandler(): Closing connection for client from IP [127.0.0.1] Apr 30 07:15:25 MAIL-SRV dbmail/imap4d[26393]: PerformChildTask(): client handling complete, closing streams Apr 30 07:15:25 MAIL-SRV dbmail/imap4d[26393]: PerformChildTask(): connection closed Apr 30 07:15:25 MAIL-SRV dbmail/imap4d[26393]: PerformChildTask(): waiting for connection Any suggestions on what to look at? It looks like it's the _auth_query() connection problem...but I don't really know. ___ Dbmail mailing list Dbmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail] __auth_query(): connection failure
I just posed a message to the list for John's issue you are having the same one. if Ilja is around maybe we can get out a 1.2.7b with this minor fix in it. See the other message I sent to the list. -leif Hello, I use dbmail for a couple of days and have been monitoring the maillog very closely. No errors, except for the following which occured once. Apr 30 22:25:06 mailer dbmail/pop3d[211]: PerformChildTask(): incoming connection from [192.168.1.3 (sharon)] Apr 30 22:25:06 mailer dbmail/pop3d[211]: __auth_query(): connection failure Apr 30 22:25:06 mailer dbmail/pop3d[211]: auth_validate(): could not select user information Apr 30 22:25:06 mailer dbmail/pop3d[211]: pop3_handle_connection(): authorization layer failure Apr 30 22:25:48 mailer dbmail/pop3d[209]: PerformChildTask(): incoming connection from [192.168.1.3 (sharon)] Apr 30 22:25:48 mailer dbmail/pop3d[209]: __auth_query(): connection failure Apr 30 22:25:48 mailer dbmail/pop3d[209]: auth_validate(): could not select user information Apr 30 22:25:48 mailer dbmail/pop3d[209]: pop3_handle_connection(): authorization layer failure When I closed my mailclient and restarted it, the problem was gone (I was able to login). Why this behavior? I installed dbmail 1.2.7a on freebsd 4.9 with postfix 2.0.19 and mysql 4.0.18. On the client (windows xp) I have outlook 2003. Pop trace_level logging is set to 2, so not much detail I am afraid. Jan-Kees ___ Dbmail mailing list Dbmail@dbmail.org https://mailman.fastxs.nl/mailman/listinfo/dbmail
[Dbmail-dev] SORT patch for current CVS, just in case anyone wants the clean diff
I have attached the same version 2 patch for the sort command I sent the list a few days ago, but this is against current cvs so it should apply cleanly. Might be nice to sneek it into 2.0 for the people who use dbmail for a webmail backend, but that is up to you guys :) It is usable I just don't know if it is good enough to sneek in. At least it is obviously a needed feature Thanks, Leif dbmail-2.0cvs031804-ic_sort-v2.diff Description: Binary data
Re: [Dbmail] Squirrelmail 1.4.2 integration problem w/ dbmail-imapd 1.2.2
Augusto Micah,
This issues has to do with the fact that until dbmail 2.1 (where they
will add my sort patch), dbmail lacks support for the server side sort.
I have a patch for the functinality required for squirrel mail for
2.0rc4 but it is only functional, not fully optimized due to some onther
internals we will be chaning for 2.1 devlopment cycle. Every time
squirrelmail connects to the imap server it must download all message
headers and then sort them in php and then return to your browser the 25
you want to see. This currently cannot be overcome with dbmail 1.x, and
only with my beta patch to 2.0rc4.
Thanks,
Leif Jackson.
Micah,
I'm running the very same configuration here - SM 1.4.2 and dbmail 1.2.5.
Have you experienced perfomance problems?
For benchmarking purposes, i created 30 folders and filled each one of
them with 100 to 1500 messages (a total of 16000 messages) - it's
extremeley slow - taking almost 40 seconds to show the folder list. I
the other hand i took the messages from an uw-imap (again, 30 folders,
16000 messages) - it took only 3 secs to show me the folder bar.
Any tips on perfomance would be greatly appreciated.
Augusto Bott
Micah wrote:
I'm running SM 1.4.2 with the dbmail 1.2.5 distro, and it's working
great. I
didn't have to do anything special to get it to work either, I did set
the
IMAP server to 'other' is all, everything else was standard.
-Micah
On Monday 29 March 2004 04:20 pm, James XMS wrote:
I started seeing this issue with the release of 1.4.
When i was using 1.2.x it was working fine. Maybe give 1.2.x a go to see
if
the same happenes to you?
Cheers
James
On Mon, 29 Mar 2004 16:43 , Micah [EMAIL PROTECTED] sent:
I'd say it's likely a php session issue.. IMAP errors will not cause
this
error. It's caused when you access a protected page with no session
information. Check the normal session stuff, cookies, passed SID's
stuff
like that.
hope that helps.
-Micah
On Monday 29 March 2004 03:14 pm, Will Berry wrote:
I have posted this problem to the SquirrelMail list already, but I am
starting to question my initial assumption that it is a SM
configuration
problem. I am starting to wonder whether I am doing something wrong
on
the DBMail side. I could not find anything like this in the archives.
We are preparing to deploy SM for our customers and I've got an
install
on our test box. All services are running on the same box (MySQL,
Apache, Postfix/DBMail, etc.), and the browser is on the same box as
well.
The basic problem is that when I correctly login to Squirrelmail I get
an error message: You must be logged in to access this page. This
is
not an HTTP authentication window; this is the resultant PHP document
from logging in. (When I login incorrectly, I get the expected bad
password massage instead.)
I have turned the trace level up to 5 on the dbmail-imapd server.
Unfortunately, it does not log the responses to client commands, only
the commands themselves. (Feature request?) But here are the four
commands I see from the client. Does this look strange to anyone?
A001 LOGIN tld_domain__user password
A002 CAPABILITY
. LIST INBOX
A003 LOGOUT
The IMAP server logs do not show any errors. It can talk to MySQL
just
fine, etc. (The POP3 daemon works fine, and I can use this mailbox
via
IMAP in Mozilla 1.6 without problems.) I created the account with
this
command line:
$ dbmail-adduser a tld_domain__user {crypt:}password 0 0
[EMAIL PROTECTED]
I have made no changes to the user, mailboxes, or any other tables
by hand. All flags in the INBOX row for this account are set to 0
(the
default for all), and 'permission' is 2 (the default).
Does anybody see any configuration error on my part with DBMail? Has
anyone seen this exact problem before? Thanks for your attention; I
promise to post the resolution when I reach it.
___
Dbmail mailing list
Dbmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail
--- Msg sent via @Mail - http://atmail.nl/
___
Dbmail mailing list
Dbmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail
___
Dbmail mailing list
Dbmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail
___
Dbmail mailing list
Dbmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail] Squirrelmail 1.4.2 integration problem w/ dbmail-imapd 1.2.2
Augusto,
It is not in the 2.0rc4 source, there are a few show stoppers in the rc4
source so you would need to use cvs version. I will send you the patch
and all details offline.
Thanks,
Leif
Leif,
I *really* need this speedup right now - I'm using dbmail with
squirrelmail. I'm willing to migrate my dbmail 1.2.5 to the 2.0 RC4 if I
can apply your speedup patch (server side search) - where can i get it?
Is it bundled in the 2.0 RC4 source?
Thanks in advance,
Augusto Bott
Leif Jackson wrote:
Augusto Micah,
This issues has to do with the fact that until dbmail 2.1 (where they
will add my sort patch), dbmail lacks support for the server side sort.
I have a patch for the functinality required for squirrel mail for
2.0rc4 but it is only functional, not fully optimized due to some onther
internals we will be chaning for 2.1 devlopment cycle. Every time
squirrelmail connects to the imap server it must download all message
headers and then sort them in php and then return to your browser the 25
you want to see. This currently cannot be overcome with dbmail 1.x, and
only with my beta patch to 2.0rc4.
Thanks,
Leif Jackson.
Micah,
I'm running the very same configuration here - SM 1.4.2 and dbmail
1.2.5.
Have you experienced perfomance problems?
For benchmarking purposes, i created 30 folders and filled each one of
them with 100 to 1500 messages (a total of 16000 messages) - it's
extremeley slow - taking almost 40 seconds to show the folder list. I
the other hand i took the messages from an uw-imap (again, 30 folders,
16000 messages) - it took only 3 secs to show me the folder bar.
Any tips on perfomance would be greatly appreciated.
Augusto Bott
Micah wrote:
I'm running SM 1.4.2 with the dbmail 1.2.5 distro, and it's working
great. I
didn't have to do anything special to get it to work either, I did set
the
IMAP server to 'other' is all, everything else was standard.
-Micah
On Monday 29 March 2004 04:20 pm, James XMS wrote:
I started seeing this issue with the release of 1.4.
When i was using 1.2.x it was working fine. Maybe give 1.2.x a go to
see
if
the same happenes to you?
Cheers
James
On Mon, 29 Mar 2004 16:43 , Micah [EMAIL PROTECTED] sent:
I'd say it's likely a php session issue.. IMAP errors will not cause
this
error. It's caused when you access a protected page with no session
information. Check the normal session stuff, cookies, passed SID's
stuff
like that.
hope that helps.
-Micah
On Monday 29 March 2004 03:14 pm, Will Berry wrote:
I have posted this problem to the SquirrelMail list already, but I
am
starting to question my initial assumption that it is a SM
configuration
problem. I am starting to wonder whether I am doing something wrong
on
the DBMail side. I could not find anything like this in the
archives.
We are preparing to deploy SM for our customers and I've got an
install
on our test box. All services are running on the same box (MySQL,
Apache, Postfix/DBMail, etc.), and the browser is on the same box as
well.
The basic problem is that when I correctly login to Squirrelmail I
get
an error message: You must be logged in to access this page. This
is
not an HTTP authentication window; this is the resultant PHP
document
from logging in. (When I login incorrectly, I get the expected bad
password massage instead.)
I have turned the trace level up to 5 on the dbmail-imapd server.
Unfortunately, it does not log the responses to client commands,
only
the commands themselves. (Feature request?) But here are the four
commands I see from the client. Does this look strange to anyone?
A001 LOGIN tld_domain__user password
A002 CAPABILITY
. LIST INBOX
A003 LOGOUT
The IMAP server logs do not show any errors. It can talk to MySQL
just
fine, etc. (The POP3 daemon works fine, and I can use this mailbox
via
IMAP in Mozilla 1.6 without problems.) I created the account with
this
command line:
$ dbmail-adduser a tld_domain__user {crypt:}password 0 0
[EMAIL PROTECTED]
I have made no changes to the user, mailboxes, or any other
tables
by hand. All flags in the INBOX row for this account are set to 0
(the
default for all), and 'permission' is 2 (the default).
Does anybody see any configuration error on my part with DBMail?
Has
anyone seen this exact problem before? Thanks for your attention; I
promise to post the resolution when I reach it.
___
Dbmail mailing list
Dbmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail
--- Msg sent via @Mail - http://atmail.nl/
___
Dbmail mailing list
Dbmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail
___
Dbmail mailing list
Dbmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail
___
Dbmail mailing list
Dbmail@dbmail.org
https://mailman.fastxs.nl/mailman/listinfo/dbmail
Re: [Dbmail-dev] Sort Command - second version - still alpha.
All, Here is version two of the imap SORT command i got to thinking about that first one using msort on the list, and that was just stupid. So I made up a quick binary tree and all is much happier now. This still needs a lot of work and still needs the header caching but at least now it does swamp the cpu. Anything over about 1000 messages tring to sort by subject, to or from just takes to much time for a webmail browser to keep connected. If there wasn't the overhead of parsing the headers I belive this would be a valid implementation, also you could just do the sorting in the SQL order by command if the headers where chached :). The only thing I add that is nessary is the addition of the mid appended to the string to make it uniq, the imap SORT rfc accualy states that if two messages are the same string for the sort then you need to compare order of arrival, this is done by default with the mid's. Comments? Thanks, leif dbmail-2.0cvs031804-ic_sort-v2.diff Description: Binary data
Re: [Dbmail-dev] SORT Problem - first patch for sort command ***VERY ALPHA***
ALl, Well people, attached is the first attempt, this code works for the SORT options required for my customers webmail system it will need some enhancment to fully follow the rfc draft for the SORT extention, It does subject,to,cc,from and arrival. Arrival was the easiest so it works perfectly, the others involved a new merge sort I added to list.c and obviously pulling all the headers in a mailbox which blows as far as performance. I would like to warn everyone while this code works it is no where near ready to go into any environment without a lot more help. But I needed to get it going so here it is. Please feel free to help/give comments and sugestions. For this first version I used a high perfomance merge sort I found with a relativly compatible grant on the licence. If anyone has a singly-link list merge sort or other better performing sort that is compatiable with the list.c code I would like to see options. A requirment for the sorting of header fields is the header caching I don't see this working with any kind of speed without it. Thanks, Leif p.s. a sample of the sort commands I used for testing would be A001 LOGIN username password A002 select INBOX A003 SORT (ARRIVAL) ISO-8859-1 ALL p.p.s. see http://www.ietf.org/internet-drafts/draft-ietf-imapext-sort-14.txt Yay!!! You are a life saver Leif. SB Leif Jackson [EMAIL PROTECTED] wrote: This observation was made before :) SORT and THREAD will have to be implemented for version 2.1 Or as a patch to 2.0 if I can get it done shortly, I really need it for our application which is a web based mail desktop system. So my client is asking for this almost every day. I hope to have something by next week:) -leif Ilja Sanjiv Bawa wrote: I tried a larger mailbox in OpenGroupware. It is way to slow It takes 8-10 seconds for a mailbox 200 messages. Cyrus does it instantaneously. It makes sense. OpenGroupware does not have a local repository (just as SquirrelMail does not - or any other web mail program). Folks, the SORT problem seems to be a show stopper for web mail implementations. I cant imagine this would be difficult to implement. Ultimately it is an ORDER BY on the SQL statement. I dont know what is involved on the IMAP side. This does not appear to be a big deal for Outlook/Express since they have local repositories and sorts can be done locally efficiently. Do you Yahoo!? Yahoo! Search - Find what youre looking for faster. ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev Do you Yahoo!? Yahoo! Mail - More reliable, more storage, less spam___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev dbmail-2.0cvs031804-ic_sort.diff Description: Binary data
Re: [Dbmail-dev] BackPlane Database ?
All, I am with arron on this one, I have been looking to implement a sqlrelay interface for a while now, the benifit of it is that it gives us an easy migration to other RDBM's such as oracle, which is relatively hard to code natively for. As Aaron pointed out it also solves some if not all of the load balancing and redundancy issues we are facing in a large scale implementation of dbmail. With the right options in my tests against the dbmail schema in mysql it shows little to no performance hit. Except for the way it caches the result sets. -leif There's another project out there that, although not specifically targetted towards fault tolerance, adresses some of those same issues: http://sqlrelay.sourceforge.net/ Incidentally, only implementing the varchar datatype isn't as much of a problem as you might think; right now, all results come back as char arrays and any numerical types need to be converted using atoi, strtoull, etc. Aaron Patrick [EMAIL PROTECTED] said: Hi, In a recent interview with Matt Dillon of the DragonFlyBsd project on OsNews on of his other projects was mentioned as well : BackPlane Fault-Tolerant Distributed RDMS The idea from what i can gather with this project is to create a scalable (for lack of a better word) unkillable fast database server that can be easily plugged into other programs. This project would help alot with a whole lot of the problems that people have with DBMail ( DB Redundancy, Issues with DB Replication Engines ) if of course it can be plugged into DBMail and its free... There are docs on its C API at : http://www.backplane.com/docs.shtml?doc=1 The only problems i have see so far is : Only one datatype is implemented, 'varchar'. In otherwords, everything is a string. We do not support UNIQUE ( more info at : http://www.backplane.com/docs.shtml?doc=2 ) Comments ? P ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev -- ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
RE: [Dbmail-dev] NNTP
Ed, I read your spec, seems like this will work fine but make sure you add and implement XOVER section 2.8 from rfc2980 this is the _most_ important command in an nntp server as far as current clients today. Be carefully to take into account that article numbers are not uniq on the servers out in the wild, so you will need to implement a renumbering system for any postings. For ease of use I would also suggest you implement XPAT as a feature as well. -leif jn, Since dbmail is primarily a mail server, it may require many changes to dbmail into a full fledged nntp server. That is not my intent. I just want to export the mailbox as a newsgroup. You could always subscribe the mailbox to the mailing-list and satisfy one of the features you mention. But, allowing the user to inject a news posting into the mailbox would be designed, but not implemented. Also, dbmail-nntpd could act as a client, with the help of suck, and fetch news from an upstream nntp server. But this is designed, and not planed for implementation. You are correct, that is the IHAVE command. Is there any progress on the cached_header_types? I see you email at: http://mailman.fastxs.net/pipermail/dbmail/2003-December/003886.html Right, the daemon should be called dbmail-nntpd ed On Wed, 17 Mar 2004, Jesse Norell wrote: Hello, I've been thinking a little about a mailing-list software that would work with dbmail (basically just an sql-backed list of subscribers and preferences, and probably a dbmail aliases table lookup to see if the recipient is handled locally or not, in which case direct-inject the message to their mailbox). This is a little different objective than what you're trying to handle, but could well be related/complimentary. Anyone want to take that on? :) You throw in a web-based client that can handle the mail, newsgroups, etc., and you could have a very nice little user portal. :) As far as nntp is concerned, for performance so you don't have to parse every message in a mailbox (for thread lookups, etc.), you'll need some table for some of the header data. It might be a very opportune time to add in the generic header caching that's planned for dbmail, and you can simply use that to also cache the news headers you need. Do you plan on making an nntp client component? Perhaps client is the wrong term - I'm thinking the ability for your nntp server to connect to an upstream and be able to pull specific newsgroups off the net to archive locally. Perhaps that's handled in the commands you list (I'm not that familiar with nntp), but it didn't appear to be (unless it's that IHAVE command). You might consider calling your daemon component dbmail-nntpd. Jn Original Message From: Ed K. dbmail-dev@dbmail.org To: dbmail-dev@dbmail.org Subject: [Dbmail-dev] NNTP Sent: Tue, 16 Mar 2004 22:10:51 -0500 (EST) As I said before, I'm serious about writing the nntp connector for dbmail. Have a look at the current project specifications at: http://www.hp.uab.edu/~ed/dbmail-nntp/ Before I start, is there anyone that has tried to tackle this before? Hints are welcome. Comments are welcome. ed -- End Original Message -- -- Jesse Norell [EMAIL PROTECTED] is not my email address; change administrator to my first name. -- ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev Security on the internet is impossible without strong, open, and unhindered encryption. ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail2/mozilla woes
Ed, For good reason, I used to work for one of the worlds largest NNTP providers. And believe me from experience we do not want to get into NNTP stuff. The reason none of the currently available NNTP server code uses an RDBM is that the file IO is so heavy and requires so much IO that it takes all of a fiber channel disk system and Sun 420R's with 4G of ram to keep up with a full NNTP feed. Much less serve any end users. Roughly about 65Mbps (to 90Mbps) for a full feed now, scary eh? my suggestion is to leave nntp alone :) -leif Let me know when the NNTP service starts development. No other NNTP servers that I've found use a RDBMS backend. ed On Mon, 15 Mar 2004, Aaron Stone wrote: This is really, really good... I wonder if it can be combined with a macro or built into a macro so that the if (ci_write) return -1 could be ci_write and since there's only ever one stream to write, ci-tx could be assumed. The LMTP code doesn't use the ClientInfo structure, but in the process of refactoring the server code these sorts of micro-differences should be brought together so that the ClientInfo structure is either generic enough or beefy enough (just toss all of the different elements into it) to handle IMAP, POP, LMTP, and whatever else comes next (NNTP anyone? ;-) Aaron Paul J Stevens [EMAIL PROTECTED] said: Hi all, Since there were no takers, I did a follow-up on my earlier conclusion that the networking code could do with some improvements. For those of you who haven't followed the list; dbmail2 won't work with mozilla and derivatives. I've wrapped the code talking to clients in simple wrappers with error checking, and lo; mozilla suddenly works just fine with dbmail2 :-) As a proof-of-concept I'm attaching my patch against the HEAD branch as of today. This is not for inclusion in CVS just yet. It's still kind of rough, but I wanted to share my relief that this really does appear to fix the bug. Also, pop3.c and lmtpd.c would probably also benefit from this approach as well. IMO imap4.c and friends could do with some re-indenting. Looks like there are several different indent-styles intertwined in the code. Any consensus on that regard ? -- Paul Stevens mailto:[EMAIL PROTECTED] NET FACILITIES GROUP PGP: finger [EMAIL PROTECTED] The Netherlandshttp://www.nfg.nl -- ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev Security on the internet is impossible without strong, open, and unhindered encryption. ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] SORT Problem - OpenGroupware
This observation was made before :) SORT and THREAD will have to be implemented for version 2.1 Or as a patch to 2.0 if I can get it done shortly, I really need it for our application which is a web based mail desktop system. So my client is asking for this almost every day. I hope to have something by next week:) -leif Ilja Sanjiv Bawa wrote: I tried a larger mailbox in OpenGroupware. It is way to slow It takes 8-10 seconds for a mailbox 200 messages. Cyrus does it instantaneously. It makes sense. OpenGroupware does not have a local repository (just as SquirrelMail does not - or any other web mail program). Folks, the SORT problem seems to be a show stopper for web mail implementations. I cant imagine this would be difficult to implement. Ultimately it is an ORDER BY on the SQL statement. I dont know what is involved on the IMAP side. This does not appear to be a big deal for Outlook/Express since they have local repositories and sorts can be done locally efficiently. Do you Yahoo!? Yahoo! Search - Find what youre looking for faster. http://search.yahoo.com/?fr=ad-mailsig-home ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] MD5 bug reproduced on Opteron
well that makes sence. I belive solaris figures out the uint32 as it is really emulating 32bits on the 64bit kernel. I did some reading and it seems that dbmail as a whole may not benifit much from accualy being compiled 64bit, in fact it may make it slower. What is your test showing? I am glad at least we figured it out. I guess we could agree on some test data and compile it into testmd5.c instead of using stdin and then we can make sure the md5 is correct on the diff arch and 64bit vs 32bit? -leif Fixed. Turned out to be quite simple: In md5.h, find the #ifdef __alpha and change it to this: #if (defined(__alpha) || defined(__x86_64__)) typedef unsigned int uint32; #else typedef unsigned long uint32; #endif For whatever reason, the Sparc is still using a 32-bit int? Or perhaps it is not crashing but still giving wrong results? Being a big endian machine, there is an additional byte order reversing routine that kicks in on the Sparc; perhaps the side effect of which is not crashing? Also, clearly, we now know why my Alpha produced proper results... it was already hacked to work ;-) The correct way to handle this is with a sizeof test from configure, and an appropriate entry in config.h to give hints to word-size sensitive code. I'll make the appropriate changes this weekend :-) Aaron Aaron Stone [EMAIL PROTECTED] said: Works in 32 bit mode. Also, the resulting md5 is different... There are a couple of compiler warnings in header.c regarding size_t arguments to printf; basically, there's a different size_t size in effect, and of course a different pointer size, too. So either GCC is generating faulty 64-bit code on the Opteron, or there's just a simple 64-bit-unclean portion of the md5 code that isn't generating any warnings but sure is mucking up the output. Aaron [EMAIL PROTECTED]:~/dbmail-2.0rc3 gcc -g -O2 -I . -m32 -o testmd5 testmd5.c *o [EMAIL PROTECTED]:~/dbmail-2.0rc3 file *o testmd5 dbmd5.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped debug.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped header.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped md5.o:ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped testmd5: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), not stripped [EMAIL PROTECTED]:~/dbmail-2.0rc3 ./testmd5 ;lkajsdf;kljasdf asdf 0a8c8e4fd15849ec600d9aac8e53bf93 [EMAIL PROTECTED]:~/dbmail-2.0rc3 Leif Jackson [EMAIL PROTECTED] said: Humm well it isn't our code that's not 64bit clean, as I emailed in the previous message, try compiling just thouse objects with -m32 or the equiv if you can and see what happens. -leif Looks like 64 bit by default on the Opteron... [EMAIL PROTECTED]:~/dbmail-2.0rc3 file *o testmd5 dbmd5.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped debug.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped header.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped md5.o:ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped testmd5: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped Leif Jackson [EMAIL PROTECTED] said: Valgrind emulates a Pentium instruction set, so it's not useful on an x86-64 processor. They have gdb 5.3 installed, and I just compiled gdb 6.0 for myself, both of which give me this when I try to read the backtrace: Make sence on valgrind... hey does the gcc on an operton default to a 64bit linking or do you have to do something like -64 for all the objects? I am thinking about sun and gcc for example where you cannot compile 64 bit without explicity telling the linker to link the 64 bit libs. Just a thought I know sun solaris is totaly diffrent than linux on operton. If it is accessing a 64bit NULL value, but only after completing the output from md5. very odd... I will check this on a sun after compiling 64bit. will let you know in a sec. on sun 32bit it works just find. -leif [EMAIL PROTECTED]:~/gdb-6.0 ./gdb/gdb GNU gdb 6.0 Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as x86_64-unknown-linux-gnu. (gdb) file ../dbmail-2.0rc3/testmd5 Reading symbols from ../dbmail-2.0rc3/testmd5...done. (gdb) run Starting program: /home/users/s/so/sodabrew/dbmail-2.0rc3/testmd5 ;lkajsdf;kljasdf
Re: [Dbmail-dev] MD5 bug reproduced on Opteron
odd, just checked with your same chars pasted into the test app: jsdf;kljasdf asdf sun sparc 64 (no change to header) dc0778f4e49bd14622a016e705cbdb15 changed header: 0988915eac85ce5fed09f29252b0cc7f sun sparc 32 bit (no header change) 0988915eac85ce5fed09f29252b0cc7f linux 32 bit adb3013a42bdec7987a807bc750b47d5 linux 32 bit md5sum command on file with only the 2 lines plus the blank line: adb3013a42bdec7987a807bc750b47d5 I would say this askes the question is any md5 value we are getting from other archs valid? what did you get from your alpha for the sum? I guess this needs to be a str array to make sure.. I will write it up. thx. Thanks, Leif well that makes sence. I belive solaris figures out the uint32 as it is really emulating 32bits on the 64bit kernel. I did some reading and it seems that dbmail as a whole may not benifit much from accualy being compiled 64bit, in fact it may make it slower. What is your test showing? I am glad at least we figured it out. I guess we could agree on some test data and compile it into testmd5.c instead of using stdin and then we can make sure the md5 is correct on the diff arch and 64bit vs 32bit? -leif Fixed. Turned out to be quite simple: In md5.h, find the #ifdef __alpha and change it to this: #if (defined(__alpha) || defined(__x86_64__)) typedef unsigned int uint32; #else typedef unsigned long uint32; #endif For whatever reason, the Sparc is still using a 32-bit int? Or perhaps it is not crashing but still giving wrong results? Being a big endian machine, there is an additional byte order reversing routine that kicks in on the Sparc; perhaps the side effect of which is not crashing? Also, clearly, we now know why my Alpha produced proper results... it was already hacked to work ;-) The correct way to handle this is with a sizeof test from configure, and an appropriate entry in config.h to give hints to word-size sensitive code. I'll make the appropriate changes this weekend :-) Aaron Aaron Stone [EMAIL PROTECTED] said: Works in 32 bit mode. Also, the resulting md5 is different... There are a couple of compiler warnings in header.c regarding size_t arguments to printf; basically, there's a different size_t size in effect, and of course a different pointer size, too. So either GCC is generating faulty 64-bit code on the Opteron, or there's just a simple 64-bit-unclean portion of the md5 code that isn't generating any warnings but sure is mucking up the output. Aaron [EMAIL PROTECTED]:~/dbmail-2.0rc3 gcc -g -O2 -I . -m32 -o testmd5 testmd5.c *o [EMAIL PROTECTED]:~/dbmail-2.0rc3 file *o testmd5 dbmd5.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped debug.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped header.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped md5.o:ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped testmd5: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), not stripped [EMAIL PROTECTED]:~/dbmail-2.0rc3 ./testmd5 ;lkajsdf;kljasdf asdf 0a8c8e4fd15849ec600d9aac8e53bf93 [EMAIL PROTECTED]:~/dbmail-2.0rc3 Leif Jackson [EMAIL PROTECTED] said: Humm well it isn't our code that's not 64bit clean, as I emailed in the previous message, try compiling just thouse objects with -m32 or the equiv if you can and see what happens. -leif Looks like 64 bit by default on the Opteron... [EMAIL PROTECTED]:~/dbmail-2.0rc3 file *o testmd5 dbmd5.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped debug.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped header.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped md5.o:ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped testmd5: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped Leif Jackson [EMAIL PROTECTED] said: Valgrind emulates a Pentium instruction set, so it's not useful on an x86-64 processor. They have gdb 5.3 installed, and I just compiled gdb 6.0 for myself, both of which give me this when I try to read the backtrace: Make sence on valgrind... hey does the gcc on an operton default to a 64bit linking or do you have to do something like -64 for all the objects? I am thinking about sun and gcc for example where you cannot compile 64 bit without explicity telling the linker to link the 64 bit libs. Just a thought I know sun solaris is totaly diffrent than linux on operton. If it is accessing a 64bit NULL value, but only after completing the output from md5. very odd... I will check this on a sun after compiling 64bit. will let you know in a sec. on sun 32bit
Re: [Dbmail-dev] MD5 bug reproduced on Opteron
The answer is simple... the tags BYTE_ORDER are not specified on sparc because sparc doesn't have endian.h, simple fix add to md5.h #ifdef __sparc #define LITTLE_ENDIAN 1234 #define BIG_ENDIAN4321 #define BYTE_ORDER BIG_ENDIAN #endif and them it works perfectly I get adaXXX on both 64bit and 32bit binaries! you were right we had an endian problem, it just wasn't doing the code from md5.c that was there to fix the endian problem. :) Thanks, Leif I'm getting the adb... on both the Opteron and my Pentium, both with typedef unsigned int uint32 So I wonder if we have an endianness issue, too... I've got a Sparc at school, I'll give it a whirl on that machine, too. And my Alpha is out of reach again, so I can't double check the results on it :-( Aaron Leif Jackson [EMAIL PROTECTED] said: odd, just checked with your same chars pasted into the test app: jsdf;kljasdf asdf sun sparc 64 (no change to header) dc0778f4e49bd14622a016e705cbdb15 changed header: 0988915eac85ce5fed09f29252b0cc7f sun sparc 32 bit (no header change) 0988915eac85ce5fed09f29252b0cc7f linux 32 bit adb3013a42bdec7987a807bc750b47d5 linux 32 bit md5sum command on file with only the 2 lines plus the blank line: adb3013a42bdec7987a807bc750b47d5 I would say this askes the question is any md5 value we are getting from other archs valid? what did you get from your alpha for the sum? I guess this needs to be a str array to make sure.. I will write it up. thx. Thanks, Leif well that makes sence. I belive solaris figures out the uint32 as it is really emulating 32bits on the 64bit kernel. I did some reading and it seems that dbmail as a whole may not benifit much from accualy being compiled 64bit, in fact it may make it slower. What is your test showing? I am glad at least we figured it out. I guess we could agree on some test data and compile it into testmd5.c instead of using stdin and then we can make sure the md5 is correct on the diff arch and 64bit vs 32bit? -leif Fixed. Turned out to be quite simple: In md5.h, find the #ifdef __alpha and change it to this: #if (defined(__alpha) || defined(__x86_64__)) typedef unsigned int uint32; #else typedef unsigned long uint32; #endif For whatever reason, the Sparc is still using a 32-bit int? Or perhaps it is not crashing but still giving wrong results? Being a big endian machine, there is an additional byte order reversing routine that kicks in on the Sparc; perhaps the side effect of which is not crashing? Also, clearly, we now know why my Alpha produced proper results... it was already hacked to work ;-) The correct way to handle this is with a sizeof test from configure, and an appropriate entry in config.h to give hints to word-size sensitive code. I'll make the appropriate changes this weekend :-) Aaron Aaron Stone [EMAIL PROTECTED] said: Works in 32 bit mode. Also, the resulting md5 is different... There are a couple of compiler warnings in header.c regarding size_t arguments to printf; basically, there's a different size_t size in effect, and of course a different pointer size, too. So either GCC is generating faulty 64-bit code on the Opteron, or there's just a simple 64-bit-unclean portion of the md5 code that isn't generating any warnings but sure is mucking up the output. Aaron [EMAIL PROTECTED]:~/dbmail-2.0rc3 gcc -g -O2 -I . -m32 -o testmd5 testmd5.c *o [EMAIL PROTECTED]:~/dbmail-2.0rc3 file *o testmd5 dbmd5.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped debug.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped header.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped md5.o:ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped testmd5: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), not stripped [EMAIL PROTECTED]:~/dbmail-2.0rc3 ./testmd5 ;lkajsdf;kljasdf asdf 0a8c8e4fd15849ec600d9aac8e53bf93 [EMAIL PROTECTED]:~/dbmail-2.0rc3 Leif Jackson [EMAIL PROTECTED] said: Humm well it isn't our code that's not 64bit clean, as I emailed in the previous message, try compiling just thouse objects with -m32 or the equiv if you can and see what happens. -leif Looks like 64 bit by default on the Opteron... [EMAIL PROTECTED]:~/dbmail-2.0rc3 file *o testmd5 dbmd5.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped debug.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped header.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped md5.o:ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped testmd5: ELF 64-bit LSB executable, AMD
Re: [Dbmail-dev] MD5 bug reproduced on Opteron
Aaron, I would try valgrind, they should have it installed. It does well on all kinds of bounds checking, as well as memory and cache checks. -Leif Hey, So I whipped up a little wrapper program around read_header() and makemd5() that crashes on the Opteron server at SourceForge, but works properly on my Pentium. Just one problem: what tools can I use to debug this thing on Opteron!? I've attached my test program. It compiles in the dbmail build tree, like so: gcc -g -O -I. -o testmd5 testmd5.c header.o dbmd5.o md5.o debug.o Aaron
Re: [Dbmail-dev] MD5 bug reproduced on Opteron
Aaron, no problems being 64bit clean on sun solaris9: bash-2.05$ gcc -DHAVE_CONFIG_H -I. -I. -I.-fomit-frame-pointer -g -m64 -O2 -W -Wall -Wpointer-arith -Wstrict-prototypes -o testmd5 testmd5.c header.o dbmd5.o md5.o debug.o testmd5.c: In function `main': testmd5.c:58: warning: unused parameter `argc' testmd5.c:58: warning: unused parameter `argv' bash-2.05$ bash-2.05$ file testmd5 testmd5:ELF 64-bit MSB executable SPARCV9 Version 1, dynamically linked, not stripped bash-2.05$ ./testmd5 asdlkjaskdfj asdkfj bd800d96270606bca89656c9df095f2f bash-2.05$ I wonder if this is an optomising thing try it without opt, also is there the -m32 options for gcc on that platform to force 32bit? -leif Valgrind emulates a Pentium instruction set, so it's not useful on an x86-64 processor. They have gdb 5.3 installed, and I just compiled gdb 6.0 for myself, both of which give me this when I try to read the backtrace: Make sence on valgrind... hey does the gcc on an operton default to a 64bit linking or do you have to do something like -64 for all the objects? I am thinking about sun and gcc for example where you cannot compile 64 bit without explicity telling the linker to link the 64 bit libs. Just a thought I know sun solaris is totaly diffrent than linux on operton. If it is accessing a 64bit NULL value, but only after completing the output from md5. very odd... I will check this on a sun after compiling 64bit. will let you know in a sec. on sun 32bit it works just find. -leif [EMAIL PROTECTED]:~/gdb-6.0 ./gdb/gdb GNU gdb 6.0 Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as x86_64-unknown-linux-gnu. (gdb) file ../dbmail-2.0rc3/testmd5 Reading symbols from ../dbmail-2.0rc3/testmd5...done. (gdb) run Starting program: /home/users/s/so/sodabrew/dbmail-2.0rc3/testmd5 ;lkajsdf;kljasdf asdf b3dd95bad20e039aa898a75cdab51a4d Program received signal SIGSEGV, Segmentation fault. 0x in ?? () Leif Jackson [EMAIL PROTECTED] said: Aaron, I would try valgrind, they should have it installed. It does well on all kinds of bounds checking, as well as memory and cache checks. -Leif Hey, So I whipped up a little wrapper program around read_header() and makemd5() that crashes on the Opteron server at SourceForge, but works properly on my Pentium. Just one problem: what tools can I use to debug this thing on Opteron!? I've attached my test program. It compiles in the dbmail build tree, like so: gcc -g -O -I. -o testmd5 testmd5.c header.o dbmd5.o md5.o debug.o Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev -- ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] MD5 bug reproduced on Opteron
Humm well it isn't our code that's not 64bit clean, as I emailed in the previous message, try compiling just thouse objects with -m32 or the equiv if you can and see what happens. -leif Looks like 64 bit by default on the Opteron... [EMAIL PROTECTED]:~/dbmail-2.0rc3 file *o testmd5 dbmd5.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped debug.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped header.o: ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped md5.o:ELF 64-bit LSB relocatable, AMD x86-64, version 1 (SYSV), not stripped testmd5: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped Leif Jackson [EMAIL PROTECTED] said: Valgrind emulates a Pentium instruction set, so it's not useful on an x86-64 processor. They have gdb 5.3 installed, and I just compiled gdb 6.0 for myself, both of which give me this when I try to read the backtrace: Make sence on valgrind... hey does the gcc on an operton default to a 64bit linking or do you have to do something like -64 for all the objects? I am thinking about sun and gcc for example where you cannot compile 64 bit without explicity telling the linker to link the 64 bit libs. Just a thought I know sun solaris is totaly diffrent than linux on operton. If it is accessing a 64bit NULL value, but only after completing the output from md5. very odd... I will check this on a sun after compiling 64bit. will let you know in a sec. on sun 32bit it works just find. -leif [EMAIL PROTECTED]:~/gdb-6.0 ./gdb/gdb GNU gdb 6.0 Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as x86_64-unknown-linux-gnu. (gdb) file ../dbmail-2.0rc3/testmd5 Reading symbols from ../dbmail-2.0rc3/testmd5...done. (gdb) run Starting program: /home/users/s/so/sodabrew/dbmail-2.0rc3/testmd5 ;lkajsdf;kljasdf asdf b3dd95bad20e039aa898a75cdab51a4d Program received signal SIGSEGV, Segmentation fault. 0x in ?? () Leif Jackson [EMAIL PROTECTED] said: Aaron, I would try valgrind, they should have it installed. It does well on all kinds of bounds checking, as well as memory and cache checks. -Leif Hey, So I whipped up a little wrapper program around read_header() and makemd5() that crashes on the Opteron server at SourceForge, but works properly on my Pentium. Just one problem: what tools can I use to debug this thing on Opteron!? I've attached my test program. It compiles in the dbmail build tree, like so: gcc -g -O -I. -o testmd5 testmd5.c header.o dbmd5.o md5.o debug.o Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev -- ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev -- ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] adding FreeConfig while tracking down memory leaks .. was format errors in calls to trace()
All, I found one ting this breaks I will send a patch shortly.. Bascialy bounce.c uses the config as a global. I don't see that this is a great idea? I will be moving: field_t dbmail_from_address, sendmail, postmaster; to be global instead, so I can still FreeConfig in the main function. Thanks, Leif Cool, Ilja, I have attached a patch that adds a FreeConfig function, this doesn't solve my memory leaks from list.c addnode, but makes it a lot easier to debug as it free's the config list's right after their used instead of at the end of the main (). Please commit it to cvs if you feel it warrants it. Thanks, leif I've put the patch in CVS, because we'll need it anyway. This way, it's also easier to debug it, because a simple cvs update will get everybody the new code :) The delivery chain is still buggy: When delivering messages through LMTP, all messages get delivered twice. Ilja Leif Jackson wrote: Ilja Aaron, I am looking for this patch. I cannot access it from sourceforge? Or do I have to checkout from cvs differently than the default dbmail root? I would be glad to look at this Also Aaron, i was working on your sieve code but there are some issues between the current libsieve you have in cvs and the last one posed on sourceforge, the api and the lib doesn't match quite right or at least not to the code you have in the dbmail cvs tree, i am really exited about this feature and would like to help. Thanks, Leif Hi, don't know why, but I just cannot find the reason why the delivery user_idnr is added to the userids-list in the dsn twice. It does not happen when using dbmail-smtp, only when using dbmail-lmtp. Aaron (or anybody else), can you take a look at this? I'm completely lost at the moment.. :( Also, there is the problem with the read_header() function. Some testing has revealed that it always functions the first time an instance of dbmail-lmtp gets a message. The second time that the same instance of the daemon gets a message, it reads the output from the MTA (postfix in my case) very slowly. Are we forgetting to cleanup something? Ilja Ilja Booij wrote: I haven't found the cause of this bug yet. There is also still a problem with the read_header() function. It's constantly hanging on an fgets there.. Ilja Ilja Booij wrote: There is no problem when sending messages through dbmail-smtp, only when using lmtp. Strange.. looking further Ilja Ilja Booij wrote: Now there's another problem with deliveries. I get every mail twice! I'm firing up gdb as I type.. Ilja Aaron Stone wrote: Posted to SourceForge. A little patch to pipe.c and header.c, which fixes a buffer boundary issue in the newline/rfc counting, the forgotten delivery useridnr loop and a missing rfcsize argument to sort_and_deliver. It's also a proper forwards patch now :-P Aaron Aaron Stone [EMAIL PROTECTED] said: Excuses, excuses. See SourceForge for an updated version of the previously posted patch; I neglected to add the new rfcsize arguments to the sort call, and something else gone wrong read_header(). Valgrinding as we speak! Aaron Aaron Stone [EMAIL PROTECTED] said: Now I remember! I continued fixing a bug or two in the 2.0rc2-fixes-snap1 tree after I'd made the patches from it. But to start clean, I took a fresh 2.0rc2, applied the patches, and then started working towards the next set of patches... apparently without bringing this bugfix into the new tree :-\ I read over the rest of the diff to make sure that I didn't leave anything else out, and this does seem to be the only thing I missed. Apply the attached patch, *reversed* (because I really need sleep :-P) Aaron Aaron Stone [EMAIL PROTECTED] said: Oh, weird. I really did fix that already; I'll see if maybe I fixed it in an older tree by accident. Will post a patch this afternoon. Aaron Ilja Booij [EMAIL PROTECTED] said: I've applied the patch (have not updated CVS yet). I ran into the following problem: When delivering a message, all message go into the mailbox of user_idnr 0 (that is: zero). The problem seems to be, that the user_idnrs to deliver the messages to are kept in delivery-userids (in a list), but that this list is never used when attempting delivery. The delivery-userids field is not used when calling sort_and_deliver(). In that call, only delivery-useridnr is used, which defaults to zero. Ilja Aaron Stone wrote: Here it goes... I'll also post to SourceForge. Aaron Ilja Booij [EMAIL PROTECTED] said: HEAD is completely updated. I'm having some trouble updating dbmail_2_0_branch, due to conflicts when applying patches. I guess I'll wait with updating that branch. Or, like somebody suggested a while ago, do the branching on release of 2.0 final (and abandon the current branch for now). Good luck finishing your project :) Ilja Aaron Stone wrote: If you have CVS updated to your latest working tree I'll patch against it in a few
Re: [Dbmail-dev] Tons of format errors in calls to trace()
Ilja, Correction I wiped my cvs out and re-checked it out and the changes are there. I was expecting the update command to merge the changes is this not the case with the dbmail cvs tree? On the list.c stuff I see this with the dmalloc lib we talked about a while back, do you knotice a state where the imapd and pop3d daemons will not exit when given a sig term? I am having an issue with them while debugging and I don't in the cvs from 1/12/04. Any ideas? If you want this patch for dmalloc let me know I will be glad to send it on so you can check with it yourself. Thanks, leif p.s. Aaron I still want to help on the libsieve and sieve code, and to anyone else if you have the framework for server side sorting and threading I would like to help with that otherwise I may have to write it myself :) Thanks. Ilja Aaron, I am looking for this patch. I cannot access it from sourceforge? Or do I have to checkout from cvs differently than the default dbmail root? I would be glad to look at this Also Aaron, i was working on your sieve code but there are some issues between the current libsieve you have in cvs and the last one posed on sourceforge, the api and the lib doesn't match quite right or at least not to the code you have in the dbmail cvs tree, i am really exited about this feature and would like to help. Thanks, Leif Hi, don't know why, but I just cannot find the reason why the delivery user_idnr is added to the userids-list in the dsn twice. It does not happen when using dbmail-smtp, only when using dbmail-lmtp. Aaron (or anybody else), can you take a look at this? I'm completely lost at the moment.. :( Also, there is the problem with the read_header() function. Some testing has revealed that it always functions the first time an instance of dbmail-lmtp gets a message. The second time that the same instance of the daemon gets a message, it reads the output from the MTA (postfix in my case) very slowly. Are we forgetting to cleanup something? Ilja Ilja Booij wrote: I haven't found the cause of this bug yet. There is also still a problem with the read_header() function. It's constantly hanging on an fgets there.. Ilja Ilja Booij wrote: There is no problem when sending messages through dbmail-smtp, only when using lmtp. Strange.. looking further Ilja Ilja Booij wrote: Now there's another problem with deliveries. I get every mail twice! I'm firing up gdb as I type.. Ilja Aaron Stone wrote: Posted to SourceForge. A little patch to pipe.c and header.c, which fixes a buffer boundary issue in the newline/rfc counting, the forgotten delivery useridnr loop and a missing rfcsize argument to sort_and_deliver. It's also a proper forwards patch now :-P Aaron Aaron Stone [EMAIL PROTECTED] said: Excuses, excuses. See SourceForge for an updated version of the previously posted patch; I neglected to add the new rfcsize arguments to the sort call, and something else gone wrong read_header(). Valgrinding as we speak! Aaron Aaron Stone [EMAIL PROTECTED] said: Now I remember! I continued fixing a bug or two in the 2.0rc2-fixes-snap1 tree after I'd made the patches from it. But to start clean, I took a fresh 2.0rc2, applied the patches, and then started working towards the next set of patches... apparently without bringing this bugfix into the new tree :-\ I read over the rest of the diff to make sure that I didn't leave anything else out, and this does seem to be the only thing I missed. Apply the attached patch, *reversed* (because I really need sleep :-P) Aaron Aaron Stone [EMAIL PROTECTED] said: Oh, weird. I really did fix that already; I'll see if maybe I fixed it in an older tree by accident. Will post a patch this afternoon. Aaron Ilja Booij [EMAIL PROTECTED] said: I've applied the patch (have not updated CVS yet). I ran into the following problem: When delivering a message, all message go into the mailbox of user_idnr 0 (that is: zero). The problem seems to be, that the user_idnrs to deliver the messages to are kept in delivery-userids (in a list), but that this list is never used when attempting delivery. The delivery-userids field is not used when calling sort_and_deliver(). In that call, only delivery-useridnr is used, which defaults to zero. Ilja Aaron Stone wrote: Here it goes... I'll also post to SourceForge. Aaron Ilja Booij [EMAIL PROTECTED] said: HEAD is completely updated. I'm having some trouble updating dbmail_2_0_branch, due to conflicts when applying patches. I guess I'll wait with updating that branch. Or, like somebody suggested a while ago, do the branching on release of 2.0 final (and abandon the current branch for now). Good luck finishing your project :) Ilja Aaron Stone wrote: If you have CVS updated to your latest working tree I'll patch against it in a few hours. This moment I have to finish up a project
Re: [Dbmail-dev] Tons of format errors in calls to trace()
Nevermind on the not exiting that was a issue I was causing with the debuging options. thx. -leif Ilja, Correction I wiped my cvs out and re-checked it out and the changes are there. I was expecting the update command to merge the changes is this not the case with the dbmail cvs tree? On the list.c stuff I see this with the dmalloc lib we talked about a while back, do you knotice a state where the imapd and pop3d daemons will not exit when given a sig term? I am having an issue with them while debugging and I don't in the cvs from 1/12/04. Any ideas? If you want this patch for dmalloc let me know I will be glad to send it on so you can check with it yourself. Thanks, leif p.s. Aaron I still want to help on the libsieve and sieve code, and to anyone else if you have the framework for server side sorting and threading I would like to help with that otherwise I may have to write it myself :) Thanks. Ilja Aaron, I am looking for this patch. I cannot access it from sourceforge? Or do I have to checkout from cvs differently than the default dbmail root? I would be glad to look at this Also Aaron, i was working on your sieve code but there are some issues between the current libsieve you have in cvs and the last one posed on sourceforge, the api and the lib doesn't match quite right or at least not to the code you have in the dbmail cvs tree, i am really exited about this feature and would like to help. Thanks, Leif Hi, don't know why, but I just cannot find the reason why the delivery user_idnr is added to the userids-list in the dsn twice. It does not happen when using dbmail-smtp, only when using dbmail-lmtp. Aaron (or anybody else), can you take a look at this? I'm completely lost at the moment.. :( Also, there is the problem with the read_header() function. Some testing has revealed that it always functions the first time an instance of dbmail-lmtp gets a message. The second time that the same instance of the daemon gets a message, it reads the output from the MTA (postfix in my case) very slowly. Are we forgetting to cleanup something? Ilja Ilja Booij wrote: I haven't found the cause of this bug yet. There is also still a problem with the read_header() function. It's constantly hanging on an fgets there.. Ilja Ilja Booij wrote: There is no problem when sending messages through dbmail-smtp, only when using lmtp. Strange.. looking further Ilja Ilja Booij wrote: Now there's another problem with deliveries. I get every mail twice! I'm firing up gdb as I type.. Ilja Aaron Stone wrote: Posted to SourceForge. A little patch to pipe.c and header.c, which fixes a buffer boundary issue in the newline/rfc counting, the forgotten delivery useridnr loop and a missing rfcsize argument to sort_and_deliver. It's also a proper forwards patch now :-P Aaron Aaron Stone [EMAIL PROTECTED] said: Excuses, excuses. See SourceForge for an updated version of the previously posted patch; I neglected to add the new rfcsize arguments to the sort call, and something else gone wrong read_header(). Valgrinding as we speak! Aaron Aaron Stone [EMAIL PROTECTED] said: Now I remember! I continued fixing a bug or two in the 2.0rc2-fixes-snap1 tree after I'd made the patches from it. But to start clean, I took a fresh 2.0rc2, applied the patches, and then started working towards the next set of patches... apparently without bringing this bugfix into the new tree :-\ I read over the rest of the diff to make sure that I didn't leave anything else out, and this does seem to be the only thing I missed. Apply the attached patch, *reversed* (because I really need sleep :-P) Aaron Aaron Stone [EMAIL PROTECTED] said: Oh, weird. I really did fix that already; I'll see if maybe I fixed it in an older tree by accident. Will post a patch this afternoon. Aaron Ilja Booij [EMAIL PROTECTED] said: I've applied the patch (have not updated CVS yet). I ran into the following problem: When delivering a message, all message go into the mailbox of user_idnr 0 (that is: zero). The problem seems to be, that the user_idnrs to deliver the messages to are kept in delivery-userids (in a list), but that this list is never used when attempting delivery. The delivery-userids field is not used when calling sort_and_deliver(). In that call, only delivery-useridnr is used, which defaults to zero. Ilja Aaron Stone wrote: Here it goes... I'll also post to SourceForge. Aaron Ilja Booij [EMAIL PROTECTED] said: HEAD is completely updated. I'm having some trouble updating dbmail_2_0_branch, due to conflicts when applying patches. I guess I'll wait with updating that branch. Or, like somebody suggested a while ago, do the branching on release of 2.0 final (and abandon the current branch for now). Good luck finishing your project :) Ilja Aaron Stone wrote: If you have CVS
Re: [Dbmail-dev] Tons of format errors in calls to trace()
I have a series of patches under the thread [ 895107 ] Delivery status in pipe and getopt support in main in the patch tracker... Sometimes I've uploaded files that SourceForge pukes on when other people try to download them. Very odd; methinks SourceForge is a little finicky in places. Thanks will try and look at that thread. The libSieve in CVS is not current with my working tree. I've was cleaning up the API and the data structures and writing documentation for all of it a few months ago, but got really busy with school and DBMail and so I only got around to uploading the documentation :-\ You mind sendming me a tarball offline? Or sending me a place I can get your current working tree? Thx. I understand about getting busy Aaron Leif Jackson [EMAIL PROTECTED] said: Ilja Aaron, I am looking for this patch. I cannot access it from sourceforge? Or do I have to checkout from cvs differently than the default dbmail root? I would be glad to look at this Also Aaron, i was working on your sieve code but there are some issues between the current libsieve you have in cvs and the last one posed on sourceforge, the api and the lib doesn't match quite right or at least not to the code you have in the dbmail cvs tree, i am really exited about this feature and would like to help. Thanks, Leif Hi, don't know why, but I just cannot find the reason why the delivery user_idnr is added to the userids-list in the dsn twice. It does not happen when using dbmail-smtp, only when using dbmail-lmtp. Aaron (or anybody else), can you take a look at this? I'm completely lost at the moment.. :( Also, there is the problem with the read_header() function. Some testing has revealed that it always functions the first time an instance of dbmail-lmtp gets a message. The second time that the same instance of the daemon gets a message, it reads the output from the MTA (postfix in my case) very slowly. Are we forgetting to cleanup something? Ilja Ilja Booij wrote: I haven't found the cause of this bug yet. There is also still a problem with the read_header() function. It's constantly hanging on an fgets there.. Ilja Ilja Booij wrote: There is no problem when sending messages through dbmail-smtp, only when using lmtp. Strange.. looking further Ilja Ilja Booij wrote: Now there's another problem with deliveries. I get every mail twice! I'm firing up gdb as I type.. Ilja Aaron Stone wrote: Posted to SourceForge. A little patch to pipe.c and header.c, which fixes a buffer boundary issue in the newline/rfc counting, the forgotten delivery useridnr loop and a missing rfcsize argument to sort_and_deliver. It's also a proper forwards patch now :-P Aaron Aaron Stone [EMAIL PROTECTED] said: Excuses, excuses. See SourceForge for an updated version of the previously posted patch; I neglected to add the new rfcsize arguments to the sort call, and something else gone wrong read_header(). Valgrinding as we speak! Aaron Aaron Stone [EMAIL PROTECTED] said: Now I remember! I continued fixing a bug or two in the 2.0rc2-fixes-snap1 tree after I'd made the patches from it. But to start clean, I took a fresh 2.0rc2, applied the patches, and then started working towards the next set of patches... apparently without bringing this bugfix into the new tree :-\ I read over the rest of the diff to make sure that I didn't leave anything else out, and this does seem to be the only thing I missed. Apply the attached patch, *reversed* (because I really need sleep :-P) Aaron Aaron Stone [EMAIL PROTECTED] said: Oh, weird. I really did fix that already; I'll see if maybe I fixed it in an older tree by accident. Will post a patch this afternoon. Aaron Ilja Booij [EMAIL PROTECTED] said: I've applied the patch (have not updated CVS yet). I ran into the following problem: When delivering a message, all message go into the mailbox of user_idnr 0 (that is: zero). The problem seems to be, that the user_idnrs to deliver the messages to are kept in delivery-userids (in a list), but that this list is never used when attempting delivery. The delivery-userids field is not used when calling sort_and_deliver(). In that call, only delivery-useridnr is used, which defaults to zero. Ilja Aaron Stone wrote: Here it goes... I'll also post to SourceForge. Aaron Ilja Booij [EMAIL PROTECTED] said: HEAD is completely updated. I'm having some trouble updating dbmail_2_0_branch, due to conflicts when applying patches. I guess I'll wait with updating that branch. Or, like somebody suggested a while ago, do the branching on release of 2.0 final (and abandon the current branch for now). Good luck finishing your project :) Ilja Aaron Stone
[Dbmail] Missing free statements in cvs as of this morning... db.c
Roel, I noticed that some of the original patch I sent you a while ago for memory leaks missed these few db_free_result's I have attached the patch for current cvs. Thanks. I am still trying to track down the issue with the list.c node_add it still seems that there is an instance in the codebase where the list is not fully freed, I am seeing a significant leak in pop3d still. Will let you know when I find it. Any pointers would be great. Also I am really exited about Arron's sieve stuff, will this implement the SORT command for imap and not just filtering? I believe a major performance enhancement for web based clients like squirrellmail which I use full time with dbmail would be the addition of the server side sorting code. Thanks, Leif dbmail-2.0cvs030104-db.c.diff Description: Binary data
