Re: [fpc-pascal] heap size growing
En/na Jonas Maebe ha escrit: [sorry to revive an old thread, but I'm having a similar problem and I prefer to follow-up than to start a new thread. Since it's old, I'm quoting it without trimming] On 14 Jan 2009, at 13:02, Burkhard Carstens wrote: Am Mittwoch, 14. Januar 2009 04:50 schrieb Seth Grover: I had the same problem. You could try to enable BESTMATCH in the heap manager by either compiling the rtl with -dBESTMATCH or changing { define BESTMATCH} to {$define BESTMATCH} in rtl/inc/heap.inc and see if the situation improves. For me, it improved but didn't solve the problem completely. I had to create my own mem pool for some frequently allocated/freed structures. In the general/average case, best fit produces slightly worse results than first fit (which is the default). The reason is that's you're more likely to end up with a bunch of unusably small blocks over time than with first fit (there are of course usage patterns in which this does not hold). IIRC this was caused by usage pattern like this: * free a huge chunk (a) of mem - chunk is returned to heap manager * allocate small chunk - this results in heap manager splitting chunk (a) to return the small piece (b) * now allocating again a huge chunk (same size as (a)) results in heap manager requesting a new chunk of mem from OS because the remainder of (a)-(b) is not sufficient. .. well, in short: memory fragmentation With BESTMATCH enabled, the heap manager tries harder to find a small free block for (b) before splitting (a) .. However, I am not completely sure if this is the same problem .. It probably is. it could also be caused by the reworked heap manager, which now handles mem allocation per thread (if that's allready in 2.2.2 ??) .. No, it is not. It will only be released in 2.4.0. And it will not solve the problem, unless the differently-sized memory blocks are only allocated in different threads. You can also try using your platform's libc memory manager to see whether it deals better with this usage pattern (add uses cmem to the uses clause of your main program). Or, as mentioned above, use your own memory pool (like the default memory manager already does for small allocations). I'm having a similar problem with the heap: I'm capturing an mjpeg stream from an ip camera (well, actually I'm capturing 50 streams from 50 cameras) using TJpegImage (from the LCL). The problem is that memory usage grows, slowly but steadily, and after a while (1 day, 1 week, it depends) the program starts failing (with No buffer space available). Seeing this thread, I used GetFPCHeapStatus and, effectively, CurrHeapSize is growing but CurrHeapUsed isn't, so apparently TJpegImage.LoadFromStream (that's what I'm using) causes heap fragmentation. FYI, using cmem (this program is running under windows xp) overall memory usage stays constant (though I haven't tried it yet in the field). Is there any undesirable side effect due to the use of cmem? Bye -- Luca ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] heap size growing
On 11 May 2009, at 12:00, Luca Olivetti wrote: Is there any undesirable side effect due to the use of cmem? It's usually slower if you perform many allocations and deallocations of small blocks. Other than that, no. Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] heap size growing
Luca Olivetti wrote: Seeing this thread, I used GetFPCHeapStatus and, effectively, CurrHeapSize is growing but CurrHeapUsed isn't, so apparently TJpegImage.LoadFromStream (that's what I'm using) causes heap fragmentation. Can you reproduce this in a small test program? Perhaps there is a simple fix for TJpegImage's allocation pattern. Micha ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] heap size growing
En/na Micha Nelissen ha escrit: Luca Olivetti wrote: Seeing this thread, I used GetFPCHeapStatus and, effectively, CurrHeapSize is growing but CurrHeapUsed isn't, so apparently TJpegImage.LoadFromStream (that's what I'm using) causes heap fragmentation. Can you reproduce this in a small test program? Perhaps there is a simple fix for TJpegImage's allocation pattern. No, and I tried very hard (mainly to isolate the problem). It must be the interaction between everything that's causing the problem, not TJPegImage alone (though just removing the LoadFromStream made the memory usage stay constant). It also depends on the image (not all images caused the heap to grow). Bye -- Luca ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] heap size growing
Am Mittwoch, 14. Januar 2009 04:50 schrieb Seth Grover: I've got a fairly large system (originally written in Delphi, then moved to Kylix, and now finally in FPC 2.2.2) which is doing a lot of different things. Various threads are running which execute reports, communiate with other devices, query data, process notifications, send and receive data on sockets, etc. For analysis of how its performing, I've got a thread which prints out the information from GetFPCHeapStatus every five minutes. What I notice is that CurrHeapUsed is not growing at all, it stays pretty much the same over hours of running. However, CurrHeapSize is pretty much continuously growing. For example (cu is current heap used, cs is current heap size, cf is current heap free, mu is max heap used, ms is max heap size): heap status: cu=1,407,664, cs=34,930,688, cf=33,523,024, mu=33,621,024, ms=35,749,888 heap status: cu=1,411,264, cs=66,715,648, cf=65,304,384, mu=33,635,232, ms=67,239,936 heap status: cu=1,409,296, cs=98,435,072, cf=97,025,776, mu=33,635,232, ms=98,467,840 heap status: cu=1,411,952, cs=161,873,920, cf=160,461,968, mu=33,674,176, ms=161,873,920 heap status: cu=1,409,056, cs=193,593,344, cf=192,184,288, mu=33,674,176, ms=193,593,344 heap status: cu=1,409,360, cs=193,593,344, cf=192,183,984, mu=33,674,176, ms=193,626,112 heap status: cu=1,409,552, cs=225,312,768, cf=223,903,216, mu=33,674,176, ms=225,345,536 heap status: cu=1,408,704, cs=257,032,192, cf=255,623,488, mu=33,674,176, ms=257,064,960 etc. top shows: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ P COMMAND 15196 root 19 0 2323m 1.8g 2780 S0 24.6 0:00.68 4 processname free -m shows: total used free sharedbuffers cached Mem: 7611 7356254 0 66 5023 -/+ buffers/cache: 2266 5344 Swap: 4094 0 4094 If I were allocating memory and not deallocating it, ie., if it were a memory leak, it seems CurrHeapUsed would be growing, right? It seems I should be concerned... what's the best direction to go from here? HEAPTRC? -SG I had the same problem. You could try to enable BESTMATCH in the heap manager by either compiling the rtl with -dBESTMATCH or changing { define BESTMATCH} to {$define BESTMATCH} in rtl/inc/heap.inc and see if the situation improves. For me, it improved but didn't solve the problem completely. I had to create my own mem pool for some frequently allocated/freed structures. IIRC this was caused by usage pattern like this: * free a huge chunk (a) of mem - chunk is returned to heap manager * allocate small chunk - this results in heap manager splitting chunk (a) to return the small piece (b) * now allocating again a huge chunk (same size as (a)) results in heap manager requesting a new chunk of mem from OS because the remainder of (a)-(b) is not sufficient. .. well, in short: memory fragmentation With BESTMATCH enabled, the heap manager tries harder to find a small free block for (b) before splitting (a) .. However, I am not completely sure if this is the same problem .. it could also be caused by the reworked heap manager, which now handles mem allocation per thread (if that's allready in 2.2.2 ??) .. regards Burkhard ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
Re: [fpc-pascal] heap size growing
On 14 Jan 2009, at 13:02, Burkhard Carstens wrote: Am Mittwoch, 14. Januar 2009 04:50 schrieb Seth Grover: I had the same problem. You could try to enable BESTMATCH in the heap manager by either compiling the rtl with -dBESTMATCH or changing { define BESTMATCH} to {$define BESTMATCH} in rtl/inc/heap.inc and see if the situation improves. For me, it improved but didn't solve the problem completely. I had to create my own mem pool for some frequently allocated/freed structures. In the general/average case, best fit produces slightly worse results than first fit (which is the default). The reason is that's you're more likely to end up with a bunch of unusably small blocks over time than with first fit (there are of course usage patterns in which this does not hold). IIRC this was caused by usage pattern like this: * free a huge chunk (a) of mem - chunk is returned to heap manager * allocate small chunk - this results in heap manager splitting chunk (a) to return the small piece (b) * now allocating again a huge chunk (same size as (a)) results in heap manager requesting a new chunk of mem from OS because the remainder of (a)-(b) is not sufficient. .. well, in short: memory fragmentation With BESTMATCH enabled, the heap manager tries harder to find a small free block for (b) before splitting (a) .. However, I am not completely sure if this is the same problem .. It probably is. it could also be caused by the reworked heap manager, which now handles mem allocation per thread (if that's allready in 2.2.2 ??) .. No, it is not. It will only be released in 2.4.0. And it will not solve the problem, unless the differently-sized memory blocks are only allocated in different threads. You can also try using your platform's libc memory manager to see whether it deals better with this usage pattern (add uses cmem to the uses clause of your main program). Or, as mentioned above, use your own memory pool (like the default memory manager already does for small allocations). Jonas ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal
[fpc-pascal] heap size growing
I've got a fairly large system (originally written in Delphi, then moved to Kylix, and now finally in FPC 2.2.2) which is doing a lot of different things. Various threads are running which execute reports, communiate with other devices, query data, process notifications, send and receive data on sockets, etc. For analysis of how its performing, I've got a thread which prints out the information from GetFPCHeapStatus every five minutes. What I notice is that CurrHeapUsed is not growing at all, it stays pretty much the same over hours of running. However, CurrHeapSize is pretty much continuously growing. For example (cu is current heap used, cs is current heap size, cf is current heap free, mu is max heap used, ms is max heap size): heap status: cu=1,407,664, cs=34,930,688, cf=33,523,024, mu=33,621,024, ms=35,749,888 heap status: cu=1,411,264, cs=66,715,648, cf=65,304,384, mu=33,635,232, ms=67,239,936 heap status: cu=1,409,296, cs=98,435,072, cf=97,025,776, mu=33,635,232, ms=98,467,840 heap status: cu=1,411,952, cs=161,873,920, cf=160,461,968, mu=33,674,176, ms=161,873,920 heap status: cu=1,409,056, cs=193,593,344, cf=192,184,288, mu=33,674,176, ms=193,593,344 heap status: cu=1,409,360, cs=193,593,344, cf=192,183,984, mu=33,674,176, ms=193,626,112 heap status: cu=1,409,552, cs=225,312,768, cf=223,903,216, mu=33,674,176, ms=225,345,536 heap status: cu=1,408,704, cs=257,032,192, cf=255,623,488, mu=33,674,176, ms=257,064,960 etc. top shows: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ P COMMAND 15196 root 19 0 2323m 1.8g 2780 S0 24.6 0:00.68 4 processname free -m shows: total used free sharedbuffers cached Mem: 7611 7356254 0 66 5023 -/+ buffers/cache: 2266 5344 Swap: 4094 0 4094 If I were allocating memory and not deallocating it, ie., if it were a memory leak, it seems CurrHeapUsed would be growing, right? It seems I should be concerned... what's the best direction to go from here? HEAPTRC? -SG Computer over. Virus = very yes. Seth Grover sethdgrover[at]gmail[dot]com ___ fpc-pascal maillist - fpc-pascal@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-pascal