Hi,
I've added this patch to the DBMail 1.1 branch in CVS. I'll also put
it in the 2.0 branch.
greetings,
Ilja
On Monday, Oct 6, 2003, at 23:18 Europe/Amsterdam, Jesse Norell wrote:
Hello,
Please try the attached patch and see if it helps. I think it has
some, but maybe not as much as I thought originally.
Anyone really familiar with network programming? I'm wondering if
this just needs to put in a select() loop for the writes. Does a
constant fwrite() loop end up blocking with high cpu usage when the
tcp send buffer is full, vs. using a select() loop to only write when
it wouldn't block? I can try to test that, but it'd be nice if
someone with experience there would share their thoughts/experience.
Thanks,
Jesse
---- Original Message ----
From: Jesse Norell <[email protected]>
To: [email protected]
Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d
Sent: Mon, 6 Oct 2003 14:14:19 -0600 (MDT)
Hello,
In very little testing, that seems to have fixed or greatly helped
the problem. I changed it to a 4096 byte buffer, flushing every
2048 bytes ... I'll try to get a simple patch made that you can test
too. (Will have to pull out a few of other patches first...)
---- Original Message ----
From: Jesse Norell <[email protected]>
To: [email protected]
Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d
Sent: Mon, 6 Oct 2003 12:44:48 -0600 (MDT)
Ok, taking a quick look at it again, this is what I gather:
The time is spent in db_send_message_lines when sending a RETR reply
from pop3.c calling:
fprintf ((FILE *)stream,"+OK %llu octets\r\n",((struct message
*)tmpelement->data)->msize);
return db_send_message_lines ((void *)stream, ((struct message
*)tmpelement->data)->realmessageid,-2, 0);
Looking in dbpgsql.c, in db_send_message_lines, we allocate a
1MB buffer (ie. READ_BLOCK_SIZE from db.h * 2) and read the db
return data into it one char at a time, handling \n -> \r\n
conversion and escaping periods, and then flush that buffer at every
3000 characters, then memset the whole 1MB buffer back to null,
in this block:
if (rowlength%3000==0) /* purge buffer at every 3000
bytes */
{
/* fprintf ((FILE *)fstream,"%s",buffer); */
/* fflush ((FILE *)fstream); */
fwrite (buffer, sizeof(char), strlen(buffer),
(FILE *)fstream);
/* cleanup the buffer */
memset (buffer, '\0', (READ_BLOCK_SIZE*2));
}
I don't know how long it takes to write to 1MB of memory, but
that seems a bit inefficient. I might see if I can use a smaller
working buffer and see if that fixes it. Any avid C coders see
that as an immediate alarm condition?
jn
---- Original Message ----
From: Jesse Norell <[email protected]>
To: [email protected]
Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d
Sent: Mon, 6 Oct 2003 12:11:37 -0600 (MDT)
Hello,
Yes, that's exactly what it acts like. I looked over the code a
bit once and nothing stood out as definitely wrong, but I really
don't do a lot of C network coding, so it could easily be.
---- Original Message ----
From: Bret Baptist <[email protected]>
To: [email protected]
Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d
Sent: Mon, 6 Oct 2003 12:48:16 -0500
Sarge is testing.
Well I think we can rule out the postgres libs, since I am running
MySQL. I
am thinking it must be the SQL code they are using, instead of
idling while
waiting for the client to download, it is just taking all the cpu.
Just silly.
Bret.
On Monday 06 October 2003 12:36 pm, Jesse Norell wrote:
Hello,
Take comfort in the fact that you're not alone ... but I don't
have a solution for you. This is exactly what we're seeing after
having upgraded to debian unstable (is unstable now sarge?). I
alluded to this briefly in a reply the other day - I think the
problem has to either be in the linux kernel or the postgres libs,
not dbmail itsself, but maybe would be possible to work around
it if it could be nailed down. We've not had much time to look
into it, just have enough pop3 front ends that we can put it off
for a while. It has persisted through a few kernel updates (and
tried smp kernel, same effects), and at least one postgres lib
update also...
---- Original Message ----
From: Bret Baptist <[email protected]>
To: [email protected]
Subject: [Dbmail] 100% cpu usage with dbmail-pop3d
Sent: Mon, 6 Oct 2003 12:24:21 -0500
I recently switched over to a separate dbmail front end server
and
database server. When I set the new front end server up I used
debian
sarge. Since that time I have been having huge problems with
load being
caused by dbmail-pop3d. It appears to attempt to use all the
available
cpu while transferring messages to the client. I had never
seen this
before when running the front end and database on the same
server.
Does anyone have any clues? This is where the logs are at when
it is
taking 100% cpu:
Oct 6 12:02:52 destiny dbmail/pop3d[20773]: pop3():RETR command,
retrieving message
Oct 6 12:02:52 destiny dbmail/pop3d[20773]: pop3(): RETR
command,
selecting message
Oct 6 12:02:52 destiny dbmail/pop3d[20773]:
db_send_message_lines():
request for [-2] lines
Oct 6 12:02:52 destiny dbmail/pop3d[20773]:
db_send_message_lines():
executing query [SELECT * FROM messageblks WHERE
message_idnr=2458388
ORDER BY messageblk_idnr ASC]
Oct 6 12:02:52 destiny dbmail/pop3d[20773]: db_query():
executing
[SELECT * FROM messageblks WHERE message_idnr=2458388 ORDER BY
messageblk_idnr ASC] Oct 6 12:02:52 destiny dbmail/pop3d[20773]:
db_send_message_lines(): sending [-2] lines from message
[2458388]
Oct 6 12:02:52 destiny dbmail/pop3d[20773]:
db_send_message_lines():
getting nextblock [1]
I am running the latest CVS version available, and still seeing
this
issue.
--
Bret Baptist
Systems and Technical Support Specialist
[EMAIL PROTECTED]
Internet Exposure, Inc.
http://www.iexposure.com
(612)676-1946 x17
Web Development-Web Marketing-ISP Services
------------------------------------------
Today is the tomorrow you worried about yesterday.
_______________________________________________
Dbmail mailing list
[email protected]
https://mailman.fastxs.nl/mailman/listinfo/dbmail
-- End Original Message --
--
Jesse Norell
jesse (at) kci.net
_______________________________________________
Dbmail mailing list
[email protected]
https://mailman.fastxs.nl/mailman/listinfo/dbmail
--
Bret Baptist
Systems and Technical Support Specialist
[EMAIL PROTECTED]
Internet Exposure, Inc.
http://www.iexposure.com
(612)676-1946 x17
Web Development-Web Marketing-ISP Services
------------------------------------------
Today is the tomorrow you worried about yesterday.
_______________________________________________
Dbmail mailing list
[email protected]
https://mailman.fastxs.nl/mailman/listinfo/dbmail
-- End Original Message --
--
Jesse Norell
jesse (at) kci.net
_______________________________________________
Dbmail mailing list
[email protected]
https://mailman.fastxs.nl/mailman/listinfo/dbmail
-- End Original Message --
--
Jesse Norell
jesse (at) kci.net
_______________________________________________
Dbmail mailing list
[email protected]
https://mailman.fastxs.nl/mailman/listinfo/dbmail
-- End Original Message --
--
Jesse Norell
jesse (at) kci.net
_______________________________________________
Dbmail mailing list
[email protected]
https://mailman.fastxs.nl/mailman/listinfo/dbmail
-- End Original Message --
--
Jesse Norell
jesse (at) kci.net
<dbmail-20031006-write_buffer_size.patch>
