[issue478005] possible memory leak in posixmodule.c

Change by admin : -- github: None -> 35464 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22534] Possible Memory Leak with 'shelve'

Ned Deily added the comment: Sorry, I'm not able to reproduce any major memory leak, using a couple of different dbm implementations, and there is not enough information to go on. To pursue further, you should identify: - what platform (operating system distribution and version) - exactly what

[issue22534] Possible Memory Leak with 'shelve'

STINNER Victor added the comment: I modified the example a little bit to display the RSS memory 10 times. The RSS increases by +176 kB at the beginning and then it is stable. I tested on Fedora 20 (Linux): anydbm.open('bla', 'c') creates a 'bsddb._DBWithCursor' object. Can you please give us

[issue22534] Possible Memory Leak with 'shelve'

TJ added the comment: Thanks for the help. For my linux box, I have no issue. This is Ubuntu 13.10: Linux localhost 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux 2.7.5+ (default, Feb 27 2014, 19:37:08) [GCC 4.8.1] However, I'm still

[issue22534] Possible Memory Leak with 'shelve'

severity: normal status: open title: Possible Memory Leak with 'shelve' versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22534 ___ ___ Python

[issue22534] Possible Memory Leak with 'shelve'

Ned Deily added the comment: What happens if you use xrange instead of range? range(100).__sizeof__() 856 xrange(100).__sizeof__() 56 -- nosy: +ned.deily ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22534

[issue22534] Possible Memory Leak with 'shelve'

TJ added the comment: Put it in the while loop. Same result. Memory usage grows about 1 GiB per minute. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22534 ___

Re: Possible memory leak?

The reason it's done in width and heigth is that there is some other non-related processing functions that were going on in the mean time with the problem. I found the source of the slow-down, when 2 non-important lines of code were commented out, the program worked perfectly.

Re: Possible memory leak?

Steven D'Aprano wrote: Of course. I was just trying to make a point about string accumulation being O(n) and not O(n^2). But according to Fredrik, string accumulation is still quadratic, even with the optimizations added to Python 2.4. Quoting: it only means that if the interpreter can

Re: Possible memory leak?

The times that I posted was the time that it took to perform ONE row iteration. As you can see, the time was going up, fairly dramatically. Why on earth could it be doing this? I understand the the time will fluctuate somewhat depending upon what else the CPU is doing, but, why is the base time

Re: Possible memory leak?

I have made one confirmation. The only identifiable difference that I have seen is that one runs on python 2.4.2, and the other 2.4.1. Oddly enough, it's the first one that's the one that is having more problems than the second... Why that is, I don't know. It still could be something else, but...

Re: Possible memory leak?

In [EMAIL PROTECTED], Tuvas wrote: The modified version of my code is now as follows: (Note, a few small changes have been made to simplify things, however, these things don't apply to a full-scale picture, so the shouldn't slow anything down in the slightest.) def

Re: Possible memory leak?

Steven D'Aprano wrote: But the real killer is this one line: row=row+chr(num/64) Bad, bad BAD idea. Every time you add two strings together, Python has to copy BOTH strings. As row gets huge, this takes longer and longer to do. This is no longer true as of CPython 2.4 though. I'm not sure

Re: Possible memory leak?

Giovanni Bajo wrote: Steven D'Aprano wrote: But the real killer is this one line: row=row+chr(num/64) Bad, bad BAD idea. Every time you add two strings together, Python has to copy BOTH strings. As row gets huge, this takes longer and longer to do. This is no longer true as of CPython

Re: Possible memory leak?

Replying to myself... the first step towards perdition... Steven D'Aprano wrote: We really don't know what the optimization recognises, how it works, or how fast a difference it makes. Of course, by we I mean those of us who haven't seen and understood the CPython source code, or run

Re: Possible memory leak?

Steven D'Aprano wrote: Steven D'Aprano wrote: But the real killer is this one line: row=row+chr(num/64) Bad, bad BAD idea. Every time you add two strings together, Python has to copy BOTH strings. As row gets huge, this takes longer and longer to do. This is no longer true as

Re: Possible memory leak?

FYI, to all who asked, I was indeed just simply monitering the system memory. I changed my approach to one that uses arrays and simply joins the statements together at the end, it seems to have improved the speed. However, for some reason it still takes a small eternity to process on one computer,

Re: Possible memory leak?

Steven D'Aprano wrote: No, that's not correct. You are making a false assumption. Did you ever try to measure the code yourself? This is from the What's New for Python 2.4: [quote] String concatenations in statements of the form s = s + abc and s += abc are now performed more efficiently

Re: Possible memory leak?

Very interesting results with the last test. I guess I go back to my other code then, even if it is only a hair faster, it's still faster... It's odd, I just ran another test. There's 2 ways I can call my load_pic function, first of all, through taking a picture, secondly by loading a picture.

Re: Possible memory leak?

Giovanni Bajo wrote: --- foo.py - def iters(n): s = '' for i in xrange(n): s += chr(i%64) return s def iters2(n): L = [] for i in xrange(n): L.append(chr(i%64)) return .join(L) --- foo.py - So, look, it's even faster than the

Re: Possible memory leak?

Fredrik Lundh wrote: Giovanni Bajo wrote: --- foo.py - def iters(n): s = '' for i in xrange(n): s += chr(i%64) return s def iters2(n): L = [] for i in xrange(n): L.append(chr(i%64)) return .join(L) --- foo.py -

Re: Possible memory leak?

Fredrik Lundh wrote: since you know the length, you can preallocate the list def iters3(n): L = [None]*n for i in xrange(n): L[i] = chr(i%64) return .join(L) or use a preallocated array def iters4(n): L = array.array(B, [0])*n

Re: Possible memory leak?

On Wed, 25 Jan 2006 13:17:25 -0800, Tuvas wrote: Very interesting results with the last test. I guess I go back to my other code then, even if it is only a hair faster, it's still faster... Can you say premature optimization? Have you timed your code with realistic data? Not somebody else's

Re: Possible memory leak?

[Fredrik Lundh] ... for the OP's problem, a PIL-based solution would probably be ~100 times faster than the array solution, but that's another story. [Tuvas] What do you mean by a PIL based solution? The reason I need to get the data into the string list is so I can pump it into PIL to give

Re: Possible memory leak?

On Wed, 25 Jan 2006 23:20:08 +, Giovanni Bajo wrote: Of course. I was just trying to make a point about string accumulation being O(n) and not O(n^2). But according to Fredrik, string accumulation is still quadratic, even with the optimizations added to Python 2.4. Quoting: it only means

Re: Possible memory leak?

On Wed, 25 Jan 2006 20:08:56 +0100, Giovanni Bajo wrote: Steven D'Aprano wrote: No, that's not correct. You are making a false assumption. Did you ever try to measure the code yourself? I'm still running Python 2.3, it would give misleading results. This is from the What's New for

Possible memory leak?

I have a function in a program that works something like this. def load_pic_data(width,heigth,inpdat, filt=TRUE): data='' total=0 tnum=0 size=100 for y in range(0,heigth): row='' for x in range(0,width):

Re: Possible memory leak?

Tuvas wrote: I have a function in a program that works something like this. def load_pic_data(width,heigth,inpdat, filt=TRUE): data='' total=0 tnum=0 size=100 for y in range(0,heigth): row='' for x in

Re: Possible memory leak?

On Tue, 24 Jan 2006 13:51:36 -0800, Tuvas wrote: The purpose of this part of a program is to take a 14 bit numerical representation and convert it to an 8 bit representation. This will later be displayed as an image. However, I've noticed the following about this code. I was noticing when I

Re: Possible memory leak?

Steven D'Aprano [EMAIL PROTECTED] writes: row = [] # processing in a loop... row.append(chr(num/64) # finished processing row = .join(row) # convert to a string in one hit print row Probably better to use StringIO or the array module. -- http://mail.python.org/mailman/listinfo/python-list

Re: Possible memory leak?

Paul Rubin http://[EMAIL PROTECTED] writes: Probably better to use StringIO or the array module. That's cStringIO of course. -- http://mail.python.org/mailman/listinfo/python-list

Re: Possible memory leak?

In message [EMAIL PROTECTED], Steven D'Aprano [EMAIL PROTECTED] writes But the real killer is this one line: row=row+chr(num/64) Bad, bad BAD idea. Every time you add two strings together, Python has to copy BOTH strings. As row gets huge, this takes longer and longer to do. A rule of thumb I

Re: Possible memory leak?

Tuvas wrote: I have a function in a program that works something like this. def load_pic_data(width,heigth,inpdat, filt=TRUE): data='' total=0 tnum=0 size=100 for y in range(0,heigth): row='' for x in

Re: Possible memory leak?

Hmm. The problem is that the required form for the image is as a string of characters to convert with the tkimage interface, at least, as I understood it. Perhaps an array would work, I'll try that when I get ahold of the computer in question (One thing required is a linux only library, and I

Re: Possible memory leak?

Oh, I should also mention, I used a memory monitor and saw the amount of memory being used go up with time, even when the function ended, meaning I did the 10 128x128 pictures, never was any memory dealocated until I exited the program. -- http://mail.python.org/mailman/listinfo/python-list

Re: Possible memory leak?

Tuvas wrote: Oh, I should also mention, I used a memory monitor and saw the amount of memory being used go up with time, even when the function ended, meaning I did the 10 128x128 pictures, never was any memory dealocated until I exited the program. Are you really trying to say that the

Re: Possible memory leak?

Tuvas wrote: However, I would like to make a few more statements. The max image size is 1024x1024. The length of time it takes to do the row statement increased fairly substationally every time it was ran, in the order of ~.5%. That seems small, but when you run it 1M times, the last