Hi Jim,
You wrote:
(2) Is he allocating new _types_, which I think don't get properly
collected.
(Off-topic) For reference, as far as I know new types are properly
freed. There has been a number of bugs and lots of corner cases to fix,
but I know of no remaining one. This assumes that the
Travis Oliphant wrote:
So, I now believe that his code (plus the array scalar extension type)
was actually exposing a real bug in the memory manager itself. In
theory, the Python memory manager should have been able to re-use the
memory for the array-scalar instances because they are
Martin v. Löwis wrote:
One way (I think the only way) this could happen if:
- the objects being allocated are all smaller than 256 bytes
- when allocating new objects, the requested size was different
from any other size previously deallocated.
So if you first allocate 1,000,000 objects
Hi,
On Thu, Nov 24, 2005 at 01:59:57AM -0800, Robert Kern wrote:
You can get the version of scipy_core just before the fix that Travis
applied:
Now we can start debugging :-)
http://projects.scipy.org/scipy/scipy_core/changeset/1490
This changeset alone fixes the small example you
Hi,
Ok, here is the reason for the leak...
There is in scipy a type called 'int32_arrtype' which inherits from both
another scipy type called 'signedinteger_arrtype', and from 'int'.
Obscure! This is not 100% officially allowed: you are inheriting from
two C types. You're living dangerously!
Armin Rigo wrote:
Hi,
Ok, here is the reason for the leak...
There is in scipy a type called 'int32_arrtype' which inherits from both
another scipy type called 'signedinteger_arrtype', and from 'int'.
Obscure! This is not 100% officially allowed: you are inheriting from
two C types.
Martin v. Löwis wrote:
Travis Oliphant wrote:
So, I now believe that his code (plus the array scalar extension
type) was actually exposing a real bug in the memory manager itself.
In theory, the Python memory manager should have been able to re-use
the memory for the array-scalar
Martin v. Löwis wrote:
Travis Oliphant wrote:
As verified by removing usage of the Python PyObject_MALLOC function,
it was the Python memory manager that was performing poorly. Even
though the array-scalar objects were deleted, the memory manager
would not re-use their memory for later
Travis Oliphant wrote:
Bingo. Yes, definitely allocating new _types_ (an awful lot of them...)
--- that's what the array scalars are: new types created in C.
are you allocating PyTypeObject structures dynamically?
why are you creating an awful lot of new type objects to represent the
Travis Oliphant wrote:
The fact that it did happen is what I'm reporting on. If nothing will
be done about it (which I can understand), at least this thread might
help somebody else in a similar situation track down why their Python
process consumes all of their memory even though their
Travis Oliphant [EMAIL PROTECTED] writes:
Bingo. Yes, definitely allocating new _types_ (an awful lot of them...)
--- that's what the array scalars are: new types created in C.
Ah! And, er, why?
If they don't get properly collected then that would definitely have
created the problem.
Bingo. Yes, definitely allocating new _types_ (an awful lot of
them...)
--- that's what the array scalars are: new types created in C.
Do you really mean that someArray[1] will create a new type to represent
the second element of someArray? I would guess that you create an
instance of
[EMAIL PROTECTED] wrote:
Travis More to the point, however, these scalar objects were allocated
Travis using the standard PyObject_New and PyObject_Del functions which
Travis of course use the Python memory manager. One user ported his
Travis (long-running) code to the new scipy
Travis Oliphant [EMAIL PROTECTED] wrote:
[EMAIL PROTECTED] wrote:
Travis More to the point, however, these scalar objects were allocated
Travis using the standard PyObject_New and PyObject_Del functions which
Travis of course use the Python memory manager. One user ported his
Josiah Carlson wrote:
Travis Oliphant [EMAIL PROTECTED] wrote:
I think definitely, his usage pattern represented a bad corner case.
An unusable corner case in fact. At any rate, moving to use the
system free and malloc fixed the immediate problem. I mainly wanted to
report the problem
Josiah Carlson wrote:
Robert Kern [EMAIL PROTECTED] wrote:
[1] There *is* an array type for general PyObjects in scipy_core, but
that's not being used in the code that blows up and has nothing to do
with the problem Travis is talking about.
I seemed to have misunderstood the
Jim Jewett wrote:
Do you have the code that caused problems?
Yes. I was able to reproduce his trouble and was trying to debug it.
The things I would check first are
(1) Is he allocating (peak usage) a type (such as integers) that
never gets returned to the free pool, in case you need more
On 2005-11-16, Travis Oliphant [EMAIL PROTECTED] wrote:
Josiah Carlson wrote:
I seemed to have misunderstood the discussion. Was the original user
accessing and saving copies of many millions of these doubles?
He *was* accessing them (therefore generating a call to an array-scalar
object
On 17-nov-2005, at 3:15, Travis Oliphant wrote:
Jim Jewett wrote:
(2) Is he allocating new _types_, which I think don't get properly
collected.
Bingo. Yes, definitely allocating new _types_ (an awful lot of
them...)
--- that's what the array scalars are: new types created in C.
19 matches
Mail list logo
