Found it!!
One little line in the loop:
@$msgref = (@$msgref, $mid);
made all the difference. Now I can process 4500 lines per sec throughout.
So the question, is there a way to speed up the above operation?
$mid is a 8 byte string.
At array size 1000, the speed is down to 1000 lines/sec
At 1500 it is 500 lines/sec
At 2000 it is 333 lines/sec
At 3000 it is 166 lines/sec
It does not take it very far into the file get down to 10 lines/sec! So the reduction
is on a bad curve....
I tried pre-allocating but it went to a very slow speed from the very beginning.
Any thoughts!
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
Sent: Monday, February 10, 2003 11:28 AM
To: Khosla, Deepak; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: RE: memory usage issue
All,
I wonder if James is referring to a problem I had a couple of years back
http://aspn.activestate.com/ASPN/Mail/Message/915295
I (rather clumsily) posted a bug at Activestate
http://bugs.activestate.com//ActivePerl/show_bug.cgi?id=18559
But, I'm pleased to report that this problem is fixed in AS Perl 5.8.0 build
804
What version of Perl are you using?
Alistair
> -----Original Message-----
> From: Tillman, James [mailto:[EMAIL PROTECTED]]
> Sent: Monday, February 10, 2003 5:47 AM
> To: Khosla, Deepak; Michael Krygowski
> Cc: [EMAIL PROTECTED]
> Subject: RE: memory usage issue
>
>
> You might also try searching through the archives for this
> mailing list on this topic. I remember a discussion about a
> year ago involving large variable allocations in a loop and
> memory usage.
>
> jpt
>
> > -----Original Message-----
> > From: Khosla, Deepak [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, February 07, 2003 2:41 PM
> > To: Michael Krygowski
> > Cc: [EMAIL PROTECTED]
> > Subject: RE: memory usage issue
> >
> >
> > Thanks Michael.
> > I'll try it. I think the issue is internally with the hashes
> > - either they are not getting released or there is something
> > I am overlooking. I can see the perl process contiue chewing
> > memory while doing slower and slower I/O..
> >
> > -----Original Message-----
> > From: Michael Krygowski [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, February 07, 2003 12:19 PM
> > To: Khosla, Deepak
> > Cc: [EMAIL PROTECTED]
> > Subject: RE: memory usage issue
> >
> >
> > Deepak,
> >
> > An application that I use quite a bit for tracking memory
> > issues and such, is called Process Explorer. You can find
> > the application at www.sysinternals.com. They have quite a
> > few other applications that have proved helpful in the past.
> >
> > HTH
> >
> > - Michael
> >
> >
> > -----Original Message-----
> > From: Khosla, Deepak [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, February 07, 2003 6:58 AM
> > To: [EMAIL PROTECTED]
> > Subject: memory usage issue
> >
> >
> > Hi,
> > Is there any way to find what is using up large amounts of
> > memory in a program.
> > I am reading large files in a loop (30-50 MB each). Before
> > each I initialize a few hashes to () but it appears that as I
> > read multiple files, things get slower. Is there a wat to
> > tell where the leak may be. I have looked through the code
> > severla times but possibly could be missing something.
> >
> >
> > ============================================
> > Deepak Khosla
> >
> > HP Direct: 281-514-9234
> > Self Direct: 713-339-7201
> > ============================================
> >
> > _______________________________________________
> > Perl-Win32-Users mailing list
> > [EMAIL PROTECTED]
> > To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
> > _______________________________________________
> > Perl-Win32-Users mailing list
> > [EMAIL PROTECTED]
> > To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
> >
> _______________________________________________
> Perl-Win32-Users mailing list
> [EMAIL PROTECTED]
> To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
>
-----------------------------------------------------------------------
Registered Office:
Marks & Spencer p.l.c
Michael House, Baker Street,
London, W1U 8EP
Registered No. 214436 in England and Wales.
Telephone (020) 7935 4422
Facsimile (020) 7487 2670
www.marksandspencer.com
Please note that electronic mail may be monitored.
This e-mail is confidential. If you received it by mistake, please let us know and
then delete it from your system; you should not copy, disclose, or distribute its
contents to anyone nor act in reliance on this e-mail, as this is prohibited and may
be unlawful.
The registered office of Marks and Spencer Financial Services PLC, Marks and Spencer
Unit Trust Management Limited, Marks and Spencer Life Assurance Limited and Marks and
Spencer Savings and Investments Limited is Kings Meadow, Chester, CH99 9FB.
_______________________________________________
Perl-Win32-Users mailing list
[EMAIL PROTECTED]
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
