Hello, Please try the attached patch and see if it helps. I think it has some, but maybe not as much as I thought originally.
Anyone really familiar with network programming? I'm wondering if this just needs to put in a select() loop for the writes. Does a constant fwrite() loop end up blocking with high cpu usage when the tcp send buffer is full, vs. using a select() loop to only write when it wouldn't block? I can try to test that, but it'd be nice if someone with experience there would share their thoughts/experience. Thanks, Jesse ---- Original Message ---- From: Jesse Norell <[email protected]> To: [email protected] Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d Sent: Mon, 6 Oct 2003 14:14:19 -0600 (MDT) > > Hello, > > In very little testing, that seems to have fixed or greatly helped > the problem. I changed it to a 4096 byte buffer, flushing every > 2048 bytes ... I'll try to get a simple patch made that you can test > too. (Will have to pull out a few of other patches first...) > > > > ---- Original Message ---- > From: Jesse Norell <[email protected]> > To: [email protected] > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > Sent: Mon, 6 Oct 2003 12:44:48 -0600 (MDT) > > > > > Ok, taking a quick look at it again, this is what I gather: > > > > The time is spent in db_send_message_lines when sending a RETR reply > > from pop3.c calling: > > > > fprintf ((FILE *)stream,"+OK %llu octets\r\n",((struct message > > *)tmpelement->data)->msize); > > return db_send_message_lines ((void *)stream, ((struct message > > *)tmpelement->data)->realmessageid,-2, 0); > > > > > > Looking in dbpgsql.c, in db_send_message_lines, we allocate a > > 1MB buffer (ie. READ_BLOCK_SIZE from db.h * 2) and read the db > > return data into it one char at a time, handling \n -> \r\n > > conversion and escaping periods, and then flush that buffer at every > > 3000 characters, then memset the whole 1MB buffer back to null, > > in this block: > > > > if (rowlength%3000==0) /* purge buffer at every 3000 bytes */ > > { > > /* fprintf ((FILE *)fstream,"%s",buffer); */ > > /* fflush ((FILE *)fstream); */ > > > > fwrite (buffer, sizeof(char), strlen(buffer), (FILE > > *)fstream); > > > > /* cleanup the buffer */ > > memset (buffer, '\0', (READ_BLOCK_SIZE*2)); > > } > > > > > > I don't know how long it takes to write to 1MB of memory, but > > that seems a bit inefficient. I might see if I can use a smaller > > working buffer and see if that fixes it. Any avid C coders see > > that as an immediate alarm condition? > > > > jn > > > > > > ---- Original Message ---- > > From: Jesse Norell <[email protected]> > > To: [email protected] > > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > > Sent: Mon, 6 Oct 2003 12:11:37 -0600 (MDT) > > > > > > > > Hello, > > > > > > Yes, that's exactly what it acts like. I looked over the code a > > > bit once and nothing stood out as definitely wrong, but I really > > > don't do a lot of C network coding, so it could easily be. > > > > > > > > > ---- Original Message ---- > > > From: Bret Baptist <[email protected]> > > > To: [email protected] > > > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > > > Sent: Mon, 6 Oct 2003 12:48:16 -0500 > > > > > > > Sarge is testing. > > > > > > > > Well I think we can rule out the postgres libs, since I am running > > > > MySQL. I > > > > am thinking it must be the SQL code they are using, instead of idling > > > > while > > > > waiting for the client to download, it is just taking all the cpu. > > > > > > > > Just silly. > > > > > > > > > > > > Bret. > > > > > > > > > > > > > > > > On Monday 06 October 2003 12:36 pm, Jesse Norell wrote: > > > > > Hello, > > > > > > > > > > Take comfort in the fact that you're not alone ... but I don't > > > > > have a solution for you. This is exactly what we're seeing after > > > > > having upgraded to debian unstable (is unstable now sarge?). I > > > > > alluded to this briefly in a reply the other day - I think the > > > > > problem has to either be in the linux kernel or the postgres libs, > > > > > not dbmail itsself, but maybe would be possible to work around > > > > > it if it could be nailed down. We've not had much time to look > > > > > into it, just have enough pop3 front ends that we can put it off > > > > > for a while. It has persisted through a few kernel updates (and > > > > > tried smp kernel, same effects), and at least one postgres lib > > > > > update also... > > > > > > > > > > > > > > > > > > > > ---- Original Message ---- > > > > > From: Bret Baptist <[email protected]> > > > > > To: [email protected] > > > > > Subject: [Dbmail] 100% cpu usage with dbmail-pop3d > > > > > Sent: Mon, 6 Oct 2003 12:24:21 -0500 > > > > > > > > > > > I recently switched over to a separate dbmail front end server and > > > > > > database server. When I set the new front end server up I used > > > > > > debian > > > > > > sarge. Since that time I have been having huge problems with load > > > > > > being > > > > > > caused by dbmail-pop3d. It appears to attempt to use all the > > > > > > available > > > > > > cpu while transferring messages to the client. I had never seen > > > > > > this > > > > > > before when running the front end and database on the same server. > > > > > > > > > > > > Does anyone have any clues? This is where the logs are at when it > > > > > > is > > > > > > taking 100% cpu: > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: pop3():RETR command, > > > > > > retrieving message > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: pop3(): RETR command, > > > > > > selecting message > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > > > > > > db_send_message_lines(): > > > > > > request for [-2] lines > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > > > > > > db_send_message_lines(): > > > > > > executing query [SELECT * FROM messageblks WHERE > > > > > > message_idnr=2458388 > > > > > > ORDER BY messageblk_idnr ASC] > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: db_query(): executing > > > > > > [SELECT * FROM messageblks WHERE message_idnr=2458388 ORDER BY > > > > > > messageblk_idnr ASC] Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > > > > > > db_send_message_lines(): sending [-2] lines from message [2458388] > > > > > > Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > > > > > > db_send_message_lines(): > > > > > > getting nextblock [1] > > > > > > > > > > > > I am running the latest CVS version available, and still seeing this > > > > > > issue. > > > > > > > > > > > > > > > > > > -- > > > > > > Bret Baptist > > > > > > Systems and Technical Support Specialist > > > > > > [EMAIL PROTECTED] > > > > > > Internet Exposure, Inc. > > > > > > http://www.iexposure.com > > > > > > > > > > > > (612)676-1946 x17 > > > > > > Web Development-Web Marketing-ISP Services > > > > > > ------------------------------------------ > > > > > > > > > > > > > > > > > > Today is the tomorrow you worried about yesterday. > > > > > > > > > > > > _______________________________________________ > > > > > > Dbmail mailing list > > > > > > [email protected] > > > > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > > > > > > -- End Original Message -- > > > > > > > > > > > > > > > -- > > > > > Jesse Norell > > > > > jesse (at) kci.net > > > > > > > > > > > > > > > _______________________________________________ > > > > > Dbmail mailing list > > > > > [email protected] > > > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > > > > -- > > > > Bret Baptist > > > > Systems and Technical Support Specialist > > > > [EMAIL PROTECTED] > > > > Internet Exposure, Inc. > > > > http://www.iexposure.com > > > > > > > > (612)676-1946 x17 > > > > Web Development-Web Marketing-ISP Services > > > > ------------------------------------------ > > > > > > > > > > > > Today is the tomorrow you worried about yesterday. > > > > > > > > _______________________________________________ > > > > Dbmail mailing list > > > > [email protected] > > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > > > -- End Original Message -- > > > > > > > > > -- > > > Jesse Norell > > > jesse (at) kci.net > > > > > > > > > _______________________________________________ > > > Dbmail mailing list > > > [email protected] > > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > > > -- End Original Message -- > > > > > > -- > > Jesse Norell > > jesse (at) kci.net > > > > > > _______________________________________________ > > Dbmail mailing list > > [email protected] > > https://mailman.fastxs.nl/mailman/listinfo/dbmail > > > -- End Original Message -- > > > -- > Jesse Norell > jesse (at) kci.net > > > _______________________________________________ > Dbmail mailing list > [email protected] > https://mailman.fastxs.nl/mailman/listinfo/dbmail > -- End Original Message -- -- Jesse Norell jesse (at) kci.net
dbmail-20031006-write_buffer_size.patch
Description: Binary data
