Hello Ilja, Offhand, have you taken a look at this patch: http://sourceforge.net/tracker/index.php?func=detail&aid=773182&group_id=85894&atid=577644 (brief description on that same page)? We've not had any problems with it, and it'd be nice if it could be applied to CVS. Trying to troubleshoot this last issue with various combinations of this patch and stock cvs code and our usermap patch and then breaking discrete pieces out to post patches has been quite a pain as of late. :)
---- Original Message ---- From: Ilja Booij <[email protected]> To: [email protected] Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d Sent: Tue, 7 Oct 2003 12:19:25 +0200 > Hi, > > I've added this patch to the DBMail 1.1 branch in CVS. I'll also put > it in the 2.0 branch. > > greetings, > Ilja > > On Monday, Oct 6, 2003, at 23:18 Europe/Amsterdam, Jesse Norell wrote: > > > Hello, > > > > Please try the attached patch and see if it helps. I think it has > > some, but maybe not as much as I thought originally. > > > > Anyone really familiar with network programming? I'm wondering if > > this just needs to put in a select() loop for the writes. Does a > > constant fwrite() loop end up blocking with high cpu usage when the > > tcp send buffer is full, vs. using a select() loop to only write when > > it wouldn't block? I can try to test that, but it'd be nice if > > someone with experience there would share their thoughts/experience. > > > > Thanks, > > Jesse > > > > > > > > ---- Original Message ---- > > From: Jesse Norell <[email protected]> > > To: [email protected] > > Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > > Sent: Mon, 6 Oct 2003 14:14:19 -0600 (MDT) > > > >> > >> Hello, > >> > >> In very little testing, that seems to have fixed or greatly helped > >> the problem. I changed it to a 4096 byte buffer, flushing every > >> 2048 bytes ... I'll try to get a simple patch made that you can test > >> too. (Will have to pull out a few of other patches first...) > >> > >> > >> > >> ---- Original Message ---- > >> From: Jesse Norell <[email protected]> > >> To: [email protected] > >> Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > >> Sent: Mon, 6 Oct 2003 12:44:48 -0600 (MDT) > >> > >>> > >>> Ok, taking a quick look at it again, this is what I gather: > >>> > >>> The time is spent in db_send_message_lines when sending a RETR reply > >>> from pop3.c calling: > >>> > >>> fprintf ((FILE *)stream,"+OK %llu octets\r\n",((struct message > >>> *)tmpelement->data)->msize); > >>> return db_send_message_lines ((void *)stream, ((struct message > >>> *)tmpelement->data)->realmessageid,-2, 0); > >>> > >>> > >>> Looking in dbpgsql.c, in db_send_message_lines, we allocate a > >>> 1MB buffer (ie. READ_BLOCK_SIZE from db.h * 2) and read the db > >>> return data into it one char at a time, handling \n -> \r\n > >>> conversion and escaping periods, and then flush that buffer at every > >>> 3000 characters, then memset the whole 1MB buffer back to null, > >>> in this block: > >>> > >>> if (rowlength%3000==0) /* purge buffer at every 3000 > >>> bytes */ > >>> { > >>> /* fprintf ((FILE *)fstream,"%s",buffer); */ > >>> /* fflush ((FILE *)fstream); */ > >>> > >>> fwrite (buffer, sizeof(char), strlen(buffer), > >>> (FILE *)fstream); > >>> > >>> /* cleanup the buffer */ > >>> memset (buffer, '\0', (READ_BLOCK_SIZE*2)); > >>> } > >>> > >>> > >>> I don't know how long it takes to write to 1MB of memory, but > >>> that seems a bit inefficient. I might see if I can use a smaller > >>> working buffer and see if that fixes it. Any avid C coders see > >>> that as an immediate alarm condition? > >>> > >>> jn > >>> > >>> > >>> ---- Original Message ---- > >>> From: Jesse Norell <[email protected]> > >>> To: [email protected] > >>> Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > >>> Sent: Mon, 6 Oct 2003 12:11:37 -0600 (MDT) > >>> > >>>> > >>>> Hello, > >>>> > >>>> Yes, that's exactly what it acts like. I looked over the code a > >>>> bit once and nothing stood out as definitely wrong, but I really > >>>> don't do a lot of C network coding, so it could easily be. > >>>> > >>>> > >>>> ---- Original Message ---- > >>>> From: Bret Baptist <[email protected]> > >>>> To: [email protected] > >>>> Subject: Re: [Dbmail] 100% cpu usage with dbmail-pop3d > >>>> Sent: Mon, 6 Oct 2003 12:48:16 -0500 > >>>> > >>>>> Sarge is testing. > >>>>> > >>>>> Well I think we can rule out the postgres libs, since I am running > >>>>> MySQL. I > >>>>> am thinking it must be the SQL code they are using, instead of > >>>>> idling while > >>>>> waiting for the client to download, it is just taking all the cpu. > >>>>> > >>>>> Just silly. > >>>>> > >>>>> > >>>>> Bret. > >>>>> > >>>>> > >>>>> > >>>>> On Monday 06 October 2003 12:36 pm, Jesse Norell wrote: > >>>>>> Hello, > >>>>>> > >>>>>> Take comfort in the fact that you're not alone ... but I don't > >>>>>> have a solution for you. This is exactly what we're seeing after > >>>>>> having upgraded to debian unstable (is unstable now sarge?). I > >>>>>> alluded to this briefly in a reply the other day - I think the > >>>>>> problem has to either be in the linux kernel or the postgres libs, > >>>>>> not dbmail itsself, but maybe would be possible to work around > >>>>>> it if it could be nailed down. We've not had much time to look > >>>>>> into it, just have enough pop3 front ends that we can put it off > >>>>>> for a while. It has persisted through a few kernel updates (and > >>>>>> tried smp kernel, same effects), and at least one postgres lib > >>>>>> update also... > >>>>>> > >>>>>> > >>>>>> > >>>>>> ---- Original Message ---- > >>>>>> From: Bret Baptist <[email protected]> > >>>>>> To: [email protected] > >>>>>> Subject: [Dbmail] 100% cpu usage with dbmail-pop3d > >>>>>> Sent: Mon, 6 Oct 2003 12:24:21 -0500 > >>>>>> > >>>>>>> I recently switched over to a separate dbmail front end server > >>>>>>> and > >>>>>>> database server. When I set the new front end server up I used > >>>>>>> debian > >>>>>>> sarge. Since that time I have been having huge problems with > >>>>>>> load being > >>>>>>> caused by dbmail-pop3d. It appears to attempt to use all the > >>>>>>> available > >>>>>>> cpu while transferring messages to the client. I had never > >>>>>>> seen this > >>>>>>> before when running the front end and database on the same > >>>>>>> server. > >>>>>>> > >>>>>>> Does anyone have any clues? This is where the logs are at when > >>>>>>> it is > >>>>>>> taking 100% cpu: > >>>>>>> Oct 6 12:02:52 destiny dbmail/pop3d[20773]: pop3():RETR command, > >>>>>>> retrieving message > >>>>>>> Oct 6 12:02:52 destiny dbmail/pop3d[20773]: pop3(): RETR > >>>>>>> command, > >>>>>>> selecting message > >>>>>>> Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > >>>>>>> db_send_message_lines(): > >>>>>>> request for [-2] lines > >>>>>>> Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > >>>>>>> db_send_message_lines(): > >>>>>>> executing query [SELECT * FROM messageblks WHERE > >>>>>>> message_idnr=2458388 > >>>>>>> ORDER BY messageblk_idnr ASC] > >>>>>>> Oct 6 12:02:52 destiny dbmail/pop3d[20773]: db_query(): > >>>>>>> executing > >>>>>>> [SELECT * FROM messageblks WHERE message_idnr=2458388 ORDER BY > >>>>>>> messageblk_idnr ASC] Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > >>>>>>> db_send_message_lines(): sending [-2] lines from message > >>>>>>> [2458388] > >>>>>>> Oct 6 12:02:52 destiny dbmail/pop3d[20773]: > >>>>>>> db_send_message_lines(): > >>>>>>> getting nextblock [1] > >>>>>>> > >>>>>>> I am running the latest CVS version available, and still seeing > >>>>>>> this > >>>>>>> issue. > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Bret Baptist > >>>>>>> Systems and Technical Support Specialist > >>>>>>> [EMAIL PROTECTED] > >>>>>>> Internet Exposure, Inc. > >>>>>>> http://www.iexposure.com > >>>>>>> > >>>>>>> (612)676-1946 x17 > >>>>>>> Web Development-Web Marketing-ISP Services > >>>>>>> ------------------------------------------ > >>>>>>> > >>>>>>> > >>>>>>> Today is the tomorrow you worried about yesterday. > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> Dbmail mailing list > >>>>>>> [email protected] > >>>>>>> https://mailman.fastxs.nl/mailman/listinfo/dbmail > >>>>>> > >>>>>> -- End Original Message -- > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Jesse Norell > >>>>>> jesse (at) kci.net > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> Dbmail mailing list > >>>>>> [email protected] > >>>>>> https://mailman.fastxs.nl/mailman/listinfo/dbmail > >>>>> > >>>>> -- > >>>>> Bret Baptist > >>>>> Systems and Technical Support Specialist > >>>>> [EMAIL PROTECTED] > >>>>> Internet Exposure, Inc. > >>>>> http://www.iexposure.com > >>>>> > >>>>> (612)676-1946 x17 > >>>>> Web Development-Web Marketing-ISP Services > >>>>> ------------------------------------------ > >>>>> > >>>>> > >>>>> Today is the tomorrow you worried about yesterday. > >>>>> > >>>>> _______________________________________________ > >>>>> Dbmail mailing list > >>>>> [email protected] > >>>>> https://mailman.fastxs.nl/mailman/listinfo/dbmail > >>>>> > >>>> -- End Original Message -- > >>>> > >>>> > >>>> -- > >>>> Jesse Norell > >>>> jesse (at) kci.net > >>>> > >>>> > >>>> _______________________________________________ > >>>> Dbmail mailing list > >>>> [email protected] > >>>> https://mailman.fastxs.nl/mailman/listinfo/dbmail > >>>> > >>> -- End Original Message -- > >>> > >>> > >>> -- > >>> Jesse Norell > >>> jesse (at) kci.net > >>> > >>> > >>> _______________________________________________ > >>> Dbmail mailing list > >>> [email protected] > >>> https://mailman.fastxs.nl/mailman/listinfo/dbmail > >>> > >> -- End Original Message -- > >> > >> > >> -- > >> Jesse Norell > >> jesse (at) kci.net > >> > >> > >> _______________________________________________ > >> Dbmail mailing list > >> [email protected] > >> https://mailman.fastxs.nl/mailman/listinfo/dbmail > >> > > -- End Original Message -- > > > > > > -- > > Jesse Norell > > jesse (at) kci.net > > > > <dbmail-20031006-write_buffer_size.patch> > > _______________________________________________ > Dbmail mailing list > [email protected] > https://mailman.fastxs.nl/mailman/listinfo/dbmail > -- End Original Message -- -- Jesse Norell jesse (at) kci.net
