Hi Jim,
You wrote:
> >(2) Is he allocating new _types_, which I think don't get properly
> > collected.
(Off-topic) For reference, as far as I know new types are properly
freed. There has been a number of bugs and lots of corner cases to fix,
but I know of no remaining one. This assumes that t
Martin v. Löwis wrote:
> Travis Oliphant wrote:
>
>> As verified by removing usage of the Python PyObject_MALLOC function,
>> it was the Python memory manager that was performing poorly. Even
>> though the array-scalar objects were deleted, the memory manager
>> would not re-use their memory
Martin v. Löwis wrote:
> Travis Oliphant wrote:
>
>> So, I now believe that his code (plus the array scalar extension
>> type) was actually exposing a real bug in the memory manager itself.
>> In theory, the Python memory manager should have been able to re-use
>> the memory for the array-scal
Armin Rigo wrote:
> Hi,
>
> Ok, here is the reason for the leak...
>
> There is in scipy a type called 'int32_arrtype' which inherits from both
> another scipy type called 'signedinteger_arrtype', and from 'int'.
> Obscure! This is not 100% officially allowed: you are inheriting from
> two C typ
Armin Rigo wrote:
> Hi,
>
> Ok, here is the reason for the leak...
>
> There is in scipy a type called 'int32_arrtype' which inherits from both
> another scipy type called 'signedinteger_arrtype', and from 'int'.
> Obscure! This is not 100% officially allowed: you are inheriting from
> two C typ
[Martin v. Löwis]
> One way (I think the only way) this could happen if:
> - the objects being allocated are all smaller than 256 bytes
> - when allocating new objects, the requested size was different
> from any other size previously deallocated.
>
> So if you first allocate 1,000,000 objects of
Hi,
Ok, here is the reason for the leak...
There is in scipy a type called 'int32_arrtype' which inherits from both
another scipy type called 'signedinteger_arrtype', and from 'int'.
Obscure! This is not 100% officially allowed: you are inheriting from
two C types. You're living dangerously!
N
Hi,
On Thu, Nov 24, 2005 at 01:59:57AM -0800, Robert Kern wrote:
> You can get the version of scipy_core just before the fix that Travis
> applied:
Now we can start debugging :-)
> http://projects.scipy.org/scipy/scipy_core/changeset/1490
This changeset alone fixes the small example you provi
Martin v. Löwis wrote:
> That the code is complex would not so much be a problem: we often
> analyse complex code here. It is a problem that the code is not
> available, and it would be a problem if the problem was not
> reproducable even if you had the code (i.e. if the problem would
> sometimes
Martin v. Löwis wrote:
> One way (I think the only way) this could happen if:
> - the objects being allocated are all smaller than 256 bytes
> - when allocating new objects, the requested size was different
>from any other size previously deallocated.
>
> So if you first allocate 1,000,000 obj
Travis Oliphant wrote:
> So, I now believe that his code (plus the array scalar extension type)
> was actually exposing a real bug in the memory manager itself. In
> theory, the Python memory manager should have been able to re-use the
> memory for the array-scalar instances because they are al
Travis Oliphant wrote:
> As verified by removing usage of the Python PyObject_MALLOC function, it
> was the Python memory manager that was performing poorly. Even though
> the array-scalar objects were deleted, the memory manager would not
> re-use their memory for later object creation. Inste
Travis Oliphant wrote:
> In the long term, what is the status of plans to re-work the Python
> Memory manager to free memory that it acquires (or improve the detection
> of already freed memory locations).
The Python memory manager does reuse memory that has been deallocated
earlier. There are p
>>
>> Bingo. Yes, definitely allocating new _types_ (an awful lot of
>> them...)
>> --- that's what the "array scalars" are: new types created in C.
>
>
> Do you really mean that someArray[1] will create a new type to represent
> the second element of someArray? I would guess that you create an
Travis Oliphant <[EMAIL PROTECTED]> writes:
> Bingo. Yes, definitely allocating new _types_ (an awful lot of them...)
> --- that's what the "array scalars" are: new types created in C.
Ah! And, er, why?
> If they don't get properly collected then that would definitely have
> created the probl
Travis Oliphant wrote:
> The fact that it did happen is what I'm reporting on. If nothing will
> be done about it (which I can understand), at least this thread might
> help somebody else in a similar situation track down why their Python
> process consumes all of their memory even though their o
Travis Oliphant wrote:
> Bingo. Yes, definitely allocating new _types_ (an awful lot of them...)
> --- that's what the "array scalars" are: new types created in C.
are you allocating PyTypeObject structures dynamically?
why are you creating an awful lot of new type objects to represent the
cont
On 17-nov-2005, at 3:15, Travis Oliphant wrote:
> Jim Jewett wrote:
>
>>
>
>> (2) Is he allocating new _types_, which I think don't get properly
>>
>> collected.
>>
>>
>
> Bingo. Yes, definitely allocating new _types_ (an awful lot of
> them...)
> --- that's what the "array scalars" are: new
On 2005-11-16, Travis Oliphant <[EMAIL PROTECTED]> wrote:
> Josiah Carlson wrote:
>>I seemed to have misunderstood the discussion. Was the original user
>>accessing and saving copies of many millions of these doubles?
>>
> He *was* accessing them (therefore generating a call to an array-scalar
Jim Jewett wrote:
>Do you have the code that caused problems?
>
>
Yes. I was able to reproduce his trouble and was trying to debug it.
>The things I would check first are
>
>(1) Is he allocating (peak usage) a type (such as integers) that
>never gets returned to the free pool, in case you nee
>> [1] There *is* an array type for general PyObjects in scipy_core, but
>> that's not being used in the code that blows up and has nothing to do
>> with the problem Travis is talking about.
Josiah> I seemed to have misunderstood the discussion.
I'm sorry, but I'm confused as w
On 11/16/05, Travis Oliphant <[EMAIL PROTECTED]> wrote:
>
> As verified by removing usage of the Python PyObject_MALLOC function, it
> was the Python memory manager that was performing poorly. Even though
> the array-scalar objects were deleted, the memory manager would not
> re-use their memory
Josiah Carlson wrote:
>Robert Kern <[EMAIL PROTECTED]> wrote:
>
>
>>[1] There *is* an array type for general PyObjects in scipy_core, but
>>that's not being used in the code that blows up and has nothing to do
>>with the problem Travis is talking about.
>>
>>
>
>I seemed to have misunderstoo
Robert Kern <[EMAIL PROTECTED]> wrote:
>
> [1] There *is* an array type for general PyObjects in scipy_core, but
> that's not being used in the code that blows up and has nothing to do
> with the problem Travis is talking about.
I seemed to have misunderstood the discussion. Was the original us
Josiah Carlson wrote:
> Travis Oliphant <[EMAIL PROTECTED]> wrote:
>>I think definitely, his usage pattern represented a "bad" corner case.
>>An unusable "corner" case in fact. At any rate, moving to use the
>>system free and malloc fixed the immediate problem. I mainly wanted to
>>report t
Travis Oliphant <[EMAIL PROTECTED]> wrote:
>
> [EMAIL PROTECTED] wrote:
>
> >Travis> More to the point, however, these scalar objects were allocated
> >Travis> using the standard PyObject_New and PyObject_Del functions which
> >Travis> of course use the Python memory manager. One us
[EMAIL PROTECTED] wrote:
>Travis> More to the point, however, these scalar objects were allocated
>Travis> using the standard PyObject_New and PyObject_Del functions which
>Travis> of course use the Python memory manager. One user ported his
>Travis> (long-running) code to the new
Travis> More to the point, however, these scalar objects were allocated
Travis> using the standard PyObject_New and PyObject_Del functions which
Travis> of course use the Python memory manager. One user ported his
Travis> (long-running) code to the new scipy core and found much to
I know (thanks to Google) that much has been said in the past about the
Python Memory Manager. My purpose in posting is simply to given a
use-case example of how the current memory manager (in Python 2.4.X) can
be problematic in scientific/engineering code.
Scipy core is a replacement for Num
29 matches
Mail list logo
