Re: API/C memory mananegemnt problem
[EMAIL PROTECTED] wrote: Sorry for responding to my own post. I think I understand the original statement now. What you are really saying is that there is a pool of Python float objects (which can, at different times, wrap different values) which can grow but never decrease in size. So the memory held by this pool is dictated by the maximum number of floats that have ever been simultaneously active (accessible). The same goes for integers. All the more reason to avoid range(.) and use xrange(.). Uh? The integers produced as you loop over xrange will be just immortal as those that get into the list built by range, so why is int immortality any reason to use one over the other? If you know you're probably not going to loop over ALL the range or xrange, sure, but that's relatively rare -- and, when it does occur, xrange is seriously preferable: helen:~ alex$ python -mtimeit 'for x in range(100): pass' 10 loops, best of 3: 15.9 usec per loop helen:~ alex$ python -mtimeit 'for x in xrange(100): pass' 10 loops, best of 3: 12.2 usec per loop helen:~ alex$ python -mtimeit 'for x in range(100): break' 10 loops, best of 3: 7.57 usec per loop helen:~ alex$ python -mtimeit 'for x in xrange(100): break' 100 loops, best of 3: 1.5 usec per loop The immediate break only halves the time requirements of the range-based loop, but it slashes by 8 times those of the xrange-based one -- now THAT is big enough to matter, as opposed to the 20% or so difference in overhead when you're always looping all the way, which is unlikely to make an important difference in overall application speed. Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: API/C memory mananegemnt problem
In article [EMAIL PROTECTED], Alex Martelli [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: I think I understand the original statement now. What you are really saying is that there is a pool of Python float objects (which can, at different times, wrap different values) which can grow but never decrease in size. So the memory held by this pool is dictated by the maximum number of floats that have ever been simultaneously active (accessible). The same goes for integers. All the more reason to avoid range(.) and use xrange(.). Uh? The integers produced as you loop over xrange will be just immortal as those that get into the list built by range, so why is int immortality any reason to use one over the other? Because unless you save the ints produced by xrange(), you're reusing slots in the free list as you go through the loop, whereas range() chews up a gob of memory that will never get released even if you never otherwise use all the ints it produces. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ 19. A language that doesn't affect the way you think about programming, is not worth knowing. --Alan Perlis -- http://mail.python.org/mailman/listinfo/python-list
Re: API/C memory mananegemnt problem
This exactly what I was thinking. Are we wrong Alex? -- http://mail.python.org/mailman/listinfo/python-list
Re: API/C memory mananegemnt problem
[EMAIL PROTECTED] wrote: This exactly what I was thinking. Are we wrong Alex? Nope, you're not -- since ints aren't immortal (but rather limited to being reused as other ints), then, if at no _other_ place in your program do you ever need to be storing N ints at the same time, looping on an xrange of N items does reduce maximum memory needed by your program compared to looping on an equivalent range. Alex -- http://mail.python.org/mailman/listinfo/python-list
API/C memory mananegemnt problem
Hi everybody,
I have a problem with Python/C API and memory management.
I'm using
Python 2.3.5 (#1, Jan 4 2006, 16:44:27)
[GCC 4.0.2 20050901 (prerelease) (SUSE Linux)] on linux2
In my C-module I have a loop like this:
***
int size=1000;
output=(double *) calloc(size, sizeof(double));
py_output=PyList_New(0);
for(i=0; isize; i++){
tmp=PyFloat_FromDouble(output[i]);
PyList_Append(py_output, tmp);
}
free(outout);
return py_output;
***
It returns to python module a (very large) list.
Problem: when I delete the list in python module (with python del statement)
not all memory is relased.
It look like all 1000 tmp PyFloat allocated in C code
remain stored in memory.
Somebody can help me?
Thanks.
marco
--
http://mail.python.org/mailman/listinfo/python-list
Re: API/C memory mananegemnt problem
[EMAIL PROTECTED] wrote:
In my C-module I have a loop like this:
***
int size=1000;
output=(double *) calloc(size, sizeof(double));
py_output=PyList_New(0);
for(i=0; isize; i++){
tmp=PyFloat_FromDouble(output[i]);
PyList_Append(py_output, tmp);
Py_DECREF(tmp); // append adds a reference
}
free(outout);
return py_output;
or with proper error handling:
int size=1000;
output=(double *) calloc(size, sizeof(double));
if (!output) {
PyErr_NoMemory();
return NULL;
}
py_output=PyList_New(0);
if (!py_output)
goto error;
for(i=0; isize; i++){
tmp=PyFloat_FromDouble(output[i]);
if (!tmp)
goto error;
if (PyList_Append(py_output, tmp) 0)
goto error;
Py_DECREF(tmp);
}
free(outout);
return py_output;
error:
Py_XDECREF(py_output);
free(output);
return NULL;
or optimized for speed:
...
py_output = PyList_New(size);
if (!py_output)
goto error;
for (i = 0; i size; i++) {
tmp = PyFloat_FromDouble(output[i]);
if (!tmp)
goto error;
PyList_SET_ITEM(py_output, tmp);
// don't decref here
}
...
hope this helps!
/F
--
http://mail.python.org/mailman/listinfo/python-list
API/C memory mananegemnt problem
Thank for your help.
I have try to follow your suggestion but I seem to fail.
Now my C-module (call it C_Core) code is:
***
/* create_list function */
int size=1000;
output=(double *) calloc(size, sizeof(double));
py_output=PyList_New(0);
for(i=0; isize; i++){
tmp=PyFloat_FromDouble(output[i]);
PyList_Append(py_output, tmp);
Py_DECREF(tmp); // append adds a reference
}
free(outout);
return py_output;
**
with del statement all memory is relased, but I have a malformed list.
In python shell:
# Call C function an create a list
alfa=C_Core.create_list()
# check the list
len(alfa)
1000
# OK
alfa[1]
Segmentation fault
On the other size your last option with
PyList_SET_ITEM(py_output, tmp) statement
is quick, but I have still memory problem.
Any idea?
Thank a lot
marco
--
http://mail.python.org/mailman/listinfo/python-list
Re: API/C memory mananegemnt problem
[EMAIL PROTECTED]
Hi everybody,
I have a problem with Python/C API and memory management.
I'm using
Python 2.3.5 (#1, Jan 4 2006, 16:44:27)
[GCC 4.0.2 20050901 (prerelease) (SUSE Linux)] on linux2
In my C-module I have a loop like this:
***
int size=1000;
output=(double *) calloc(size, sizeof(double));
py_output=PyList_New(0);
for(i=0; isize; i++){
tmp=PyFloat_FromDouble(output[i]);
PyList_Append(py_output, tmp);
}
free(outout);
return py_output;
***
It returns to python module a (very large) list.
Problem: when I delete the list in python module (with python del statement)
not all memory is relased.
It look like all 1000 tmp PyFloat allocated in C code
remain stored in memory.
Somebody can help me?
As Fredrik noted, your refcounting is off and you need a great deal of
error-checking code.
However, none of that will help your underlying problem: CPython
floats are allocated in a special free list dedicated to floats, and
that free list is both immortal and unbounded in size. Once Python
has allocated memory to hold a Python float, that memory is never
released. BTW, Python has another, distinct immortal and unbounded
free list dedicated to holding storage for int objects.
There's nothing you can do about that. It may be possible for you to
use an array.array to hold unboxed floats directly; that depends on
details of your app.
--
http://mail.python.org/mailman/listinfo/python-list
Re: API/C memory mananegemnt problem
I have not looked into C programming and Python so I probably don't understand the issues here but the above statement sounds very troubling. Does this mean that any integer or float created anywhere at anytime in a (pure) python program will be stored away for all time (as long as the program is running)? If the above is true my initial reaction is to recoil in horror... Say it ain't so Joe! -- http://mail.python.org/mailman/listinfo/python-list
Re: API/C memory mananegemnt problem
Sorry for responding to my own post. I think I understand the original statement now. What you are really saying is that there is a pool of Python float objects (which can, at different times, wrap different values) which can grow but never decrease in size. So the memory held by this pool is dictated by the maximum number of floats that have ever been simultaneously active (accessible). The same goes for integers. All the more reason to avoid range(.) and use xrange(.). So is this correct? Thanks! -- http://mail.python.org/mailman/listinfo/python-list
