You might try the standard "array appending" Perl idiom, which is:
push(@$msgref, $mid); This will perform the array appending operation without accessing every element of the entire array, which your method was doing. It's the difference between the act of sliding a card into the bottom of a card deck, and the act of placing a card on the table and then dealing every card in the deck on top of it. The result is the same, but the dealing takes much longer. Fortunately, perl seems to "consistently stack the deck in our favor!" Most languages don't seem to have array manipulation features like this. Hope this helps, jpt > -----Original Message----- > From: Khosla, Deepak [mailto:[EMAIL PROTECTED]] > Sent: Tuesday, February 11, 2003 9:01 AM > To: Khosla, Deepak; [EMAIL PROTECTED]; Tillman, > James; [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED] > Subject: RE: memory usage issue > > > To clarify: > REMOVING that line made all the difference! I have t figure > out how to do this a different way.. > > -----Original Message----- > From: Khosla, Deepak > Sent: Tuesday, February 11, 2003 7:57 AM > To: [EMAIL PROTECTED]; > [EMAIL PROTECTED]; [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED] > Subject: RE: memory usage issue > > > Found it!! > One little line in the loop: > @$msgref = (@$msgref, $mid); > made all the difference. Now I can process 4500 lines per sec > throughout. > > So the question, is there a way to speed up the above operation? > $mid is a 8 byte string. > At array size 1000, the speed is down to 1000 lines/sec > At 1500 it is 500 lines/sec > At 2000 it is 333 lines/sec > At 3000 it is 166 lines/sec > It does not take it very far into the file get down to 10 > lines/sec! So the reduction is on a bad curve.... > > I tried pre-allocating but it went to a very slow speed from > the very beginning. > Any thoughts! > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]] > Sent: Monday, February 10, 2003 11:28 AM > To: Khosla, Deepak; [EMAIL PROTECTED]; [EMAIL PROTECTED] > Cc: [EMAIL PROTECTED] > Subject: RE: memory usage issue > > > All, > > I wonder if James is referring to a problem I had a couple of > years back > http://aspn.activestate.com/ASPN/Mail/Message/915295 > > I (rather clumsily) posted a bug at Activestate > http://bugs.activestate.com//ActivePerl/show_bug.cgi?id=18559 > > But, I'm pleased to report that this problem is fixed in AS > Perl 5.8.0 build > 804 > > What version of Perl are you using? > > Alistair > > > -----Original Message----- > > From: Tillman, James [mailto:[EMAIL PROTECTED]] > > Sent: Monday, February 10, 2003 5:47 AM > > To: Khosla, Deepak; Michael Krygowski > > Cc: [EMAIL PROTECTED] > > Subject: RE: memory usage issue > > > > > > You might also try searching through the archives for this > > mailing list on this topic. I remember a discussion about a > > year ago involving large variable allocations in a loop and > > memory usage. > > > > jpt > > > > > -----Original Message----- > > > From: Khosla, Deepak [mailto:[EMAIL PROTECTED]] > > > Sent: Friday, February 07, 2003 2:41 PM > > > To: Michael Krygowski > > > Cc: [EMAIL PROTECTED] > > > Subject: RE: memory usage issue > > > > > > > > > Thanks Michael. > > > I'll try it. I think the issue is internally with the hashes > > > - either they are not getting released or there is something > > > I am overlooking. I can see the perl process contiue chewing > > > memory while doing slower and slower I/O.. > > > > > > -----Original Message----- > > > From: Michael Krygowski [mailto:[EMAIL PROTECTED]] > > > Sent: Friday, February 07, 2003 12:19 PM > > > To: Khosla, Deepak > > > Cc: [EMAIL PROTECTED] > > > Subject: RE: memory usage issue > > > > > > > > > Deepak, > > > > > > An application that I use quite a bit for tracking memory > > > issues and such, is called Process Explorer. You can find > > > the application at www.sysinternals.com. They have quite a > > > few other applications that have proved helpful in the past. > > > > > > HTH > > > > > > - Michael > > > > > > > > > -----Original Message----- > > > From: Khosla, Deepak [mailto:[EMAIL PROTECTED]] > > > Sent: Friday, February 07, 2003 6:58 AM > > > To: [EMAIL PROTECTED] > > > Subject: memory usage issue > > > > > > > > > Hi, > > > Is there any way to find what is using up large amounts of > > > memory in a program. > > > I am reading large files in a loop (30-50 MB each). Before > > > each I initialize a few hashes to () but it appears that as I > > > read multiple files, things get slower. Is there a wat to > > > tell where the leak may be. I have looked through the code > > > severla times but possibly could be missing something. > > > > > > > > > ============================================ > > > Deepak Khosla > > > > > > HP Direct: 281-514-9234 > > > Self Direct: 713-339-7201 > > > ============================================ > > > > > > _______________________________________________ > > > Perl-Win32-Users mailing list > > > [EMAIL PROTECTED] > > > To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs > > > _______________________________________________ > > > Perl-Win32-Users mailing list > > > [EMAIL PROTECTED] > > > To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs > > > > > _______________________________________________ > > Perl-Win32-Users mailing list > > [EMAIL PROTECTED] > > To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs > > > > > -------------------------------------------------------------- > --------- > > > Registered Office: > Marks & Spencer p.l.c > Michael House, Baker Street, > London, W1U 8EP > Registered No. 214436 in England and Wales. > > Telephone (020) 7935 4422 > Facsimile (020) 7487 2670 > > www.marksandspencer.com > > Please note that electronic mail may be monitored. > > This e-mail is confidential. If you received it by mistake, > please let us know and then delete it from your system; you > should not copy, disclose, or distribute its contents to > anyone nor act in reliance on this e-mail, as this is > prohibited and may be unlawful. > > The registered office of Marks and Spencer Financial Services > PLC, Marks and Spencer Unit Trust Management Limited, Marks > and Spencer Life Assurance Limited and Marks and Spencer > Savings and Investments Limited is Kings Meadow, Chester, CH99 9FB. > > _______________________________________________ > Perl-Win32-Users mailing list > [EMAIL PROTECTED] > To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs > _______________________________________________ Perl-Win32-Users mailing list [EMAIL PROTECTED] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
