Hi Thanks for response.
I'm pretty sure for one reason and another that hardware is not at fault. I use the windows monitor to track memory usage. I must say I hadn't realised that Python doesn't release memory back to the OS - someone else advised me to try the alpha 2.5 version which cures this problem - I have but regretably the same problem keeps occurring. The programme is about 600 lines and I'd be loth to let anyone have it in its current form until I understand this problem. Interestingly I have programmed this in 3 different ways - as a class, as a modular programme and (currently) as a single file application (using global variables instead of parameter passing purely because in some cases I was passing and returning up to 20 parameters which seemed clumsy). I have profiled the programme to death and there is no doubt that using globals is faster (this is a time-critical application). However I may translate the current version back into modular or class form and see if the problem vanishes. Regards Phil ----- Original Message ----- From: "Danny Yoo" <[EMAIL PROTECTED]> To: "Philip Smith" <[EMAIL PROTECTED]> Cc: <tutor@python.org> Sent: Thursday, May 04, 2006 6:31 PM Subject: Re: [Tutor] Memory Management etc >> I have a problem with a large programme that uses a lot of memory >> (numerous large arrays which are dynamically extended). I keep getting >> unpredictable crashes (on a machine with 1/2 GB memory) whereas on my >> laptop (1 GB memory) it seems OK. > > Hi Philip, > > Can you be more specific about what you mean by "crash"? Does the whole > operating system freeze, or do you get an error message from Python, or > ...? > > The reason I ask is because what you see may not necessarily have to do > with Python --- there may be lower-level issues such as physical hardware, > for example. So more information on symptoms will be very helpful. > > >> There are no debugger messages it just crashes (and reboots the machine >> more often than not). > > Ok, if I understand what you're saying: are you saying that the machine > physically reboots without user prompting? If so, that's almost certainly > NOT Python then. System reboot means that even your operating system is > having difficulty keeping the machine usable. The most likely explanation > in this circumstance is that the physical hardware is defective. > > I'd recommend running diagnostics like a RAM checker. But try running > your program on another machine as another data point to support the > possibility that perhaps the hardware, and not the software is the issue. > > >> I have to say I have noticed (the programme is basically a >> batch-factoring programme for integers) that no matter how I tune gc I >> can't get it to reliably free memory in between factoring each integer. > > How are you measuring this? > > Note that Python does not necessarily give allocated memory back to the > operating system: it keeps a memory pool that it reuses for performance > reasons. > > Is the amount of memory you're using at least bounded? > > >> Because this is a development programme (and for performance reasons) I >> use global variables some of which refer to the large arrays. > > Global variables don't necessarily make programs fast. I would strongly > discourage this kind of ad-hoc performance optimization. Don't guess: let > the machine tell you where the program is slow. If you really want to make > your program fast, use a profiler. > > Also note that parameter passing from one function to another is a > constant-time operation: no object values are being copied. So the cost > assumptions you're making about passing large arrays around functions may > not be correct. > > > >> 1) Does the mere fact that a function cites a variable as global create >> a reference which prevents it being collected by gc? > > This isn't a contributing factor. But global variable values at the > toplevel don't die: that's the point about global variables, because they > always have at least one reference to them and they're always accessible > to the outside. > > >> 3) Is there any way to allocate a pool of available memory to my >> programme at the outset from which I can allocate my large arrays? > > Practially everything in Python is done at runtime, not compile time. At > program startup, I suppose you can preallocate some arrays and keep a pool > of them for your usage. > > But are you finding this to be a significant factor in your program, > though? Again, before you go ahead with optimization, I'd strongly > recommend using a profiler to do a principled analysis of the hotspots of > your program. Have you looked at the Python Profiler yet? > > > >> I'm keen to solve this because I would like to make my programme >> generally available - in every other respect its near complete and >> massively outperforms the only other comparable pure python module >> (nzmath) which does the same job. > > If you would like a code review, and if the program is short enough, I'm > sure people here would be happy to give some pointers. > > Good luck to you! > _______________________________________________ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
