Thanks Istvan for this interesting report. I checked the GarbageCollector
implementation, added a VERBOSE mode and here is the result:
$ python example_GC_istvan_v2.py
running number 0
creating 2500 points
creating poly
discretizing
exit function and collect garbage
Before smart_purge
The Gabrbage contains:
100 handles
100 transients
705 misc
After smart_purge
The Gabrbage contains:
0 handles
100 transients
0 misc
running number 1
creating 2500 points
creating poly
discretizing
exit function and collect garbage
Before smart_purge
The Gabrbage contains:
100 handles
200 transients
705 misc
After smart_purge
The Gabrbage contains:
0 handles
200 transients
0 misc
running number 2
creating 2500 points
creating poly
discretizing
exit function and collect garbage
Before smart_purge
The Gabrbage contains:
100 handles
300 transients
705 misc
After smart_purge
The Gabrbage contains:
0 handles
300 transients
0 misc
running number 3
creating 2500 points
creating poly
discretizing
exit function and collect garbage
Before smart_purge
The Gabrbage contains:
100 handles
400 transients
705 misc
After smart_purge
The Gabrbage contains:
0 handles
400 transients
0 misc
There actually is a leak, some transient are not deleted (some issues with
ref counting).
Could you please edit site-packages/OCC/GarbageCollector.py and add this
method to the GarbageCollector class?
def purge(self):
while len(self._handles)>0:
handle = self._handles.pop()
handle.Nullify()
handle.was_purged = True
while len(self._transients)>0:
transient = self._transients.pop()
transient._kill_pointed()
transient.was_purged = True
while len(self._misc)>0:
misc = self._misc.pop()
misc._kill_pointed()
misc.was_purged = True
And test your code with purge() rather than smart_purge(). I tested that all
objects are deleted and the memory footprint does not increase.
Thomas
2011/9/29 István Csanády <istvancsan...@gmail.com>
> I attached the code. Experiences:
> 1. The C++ code is about 30 times faster. (the python profiler says the
> most costly function is _loop_topo in Topology.py, I will check it out
> maybe I can make it faster)
> 2. The memory footprint does not grow even if I creating random sized
> grids.
> 3. It does not crash.
> I tried to emulate the behaviour of the pythonocc garbage collector with an
> stl::list - I don't know if it makes any sense.
>
>
> On Thu, Sep 29, 2011 at 8:56 PM, István Csanády
> <istvancsan...@gmail.com>wrote:
>
>> OK forget the code I previously sent, I have installed ftgl, and now
>> testing it, and the one I sent is not working (what is not very stunning
>> since I could not run it :) )
>>
>> On Thu, Sep 29, 2011 at 8:09 PM, István Csanády
>> <istvancsan...@gmail.com>wrote:
>>
>>> I have installed the OCE-0.6.0 with the .mpkg. I can compile OCC
>>> executables, but crashes because it can't find libftgl.dylib... Now I am
>>> going to recompile it from sources without the GL stuff but it takes a while
>>> on my macbook :(. I have attached the code, but I don't know if it works,
>>> and I have not implemented something like pythonocc's garbage collection yet
>>> (it is maybe not even neccessary).
>>>
>>> Error message:
>>>
>>> *dyld: Library not loaded: /usr/local/lib/libftgl.2.dylib*
>>>
>>> * Referenced from:
>>> /Users/istvancsanady/Library/Developer/Xcode/DerivedData/oce_memtest-augtghxmaekbeydwmiliecjltrpc/Build/Products/Debug/libTKOpenGl.1.dylib
>>> *
>>>
>>> * Reason: image not found*
>>>
>>> *
>>> *
>>>
>>> *
>>> *
>>>
>>> On Thu, Sep 29, 2011 at 5:16 PM, Thomas Paviot <tpav...@gmail.com>wrote:
>>>
>>>> Can you please attach the code?
>>>>
>>>>
>>>> Thomas
>>>>
>>>> 2011/9/29 István Csanády <istvancsan...@gmail.com>
>>>>
>>>>> I am currently working on it, but I getting compile errors for private
>>>>> copy constructors in the GCPnts_UniformDeflection and BRepLib_MakeShape
>>>>> headers, for eg. "Field of type 'TColStd_SequenceOfReal' has private copy
>>>>> constructor"... I don't really understand why do I get these since until
>>>>> now
>>>>> I have never met such compile errors in OCC. What am I doing wrong?
>>>>>
>>>>>
>>>>> On Thu, Sep 29, 2011 at 3:43 PM, Thomas Paviot <tpav...@gmail.com>wrote:
>>>>>
>>>>>> Istvan,
>>>>>>
>>>>>> I'm a bit confused with your example. It appears that the MMGT_OPT env
>>>>>> var has an impact on the crash produced by your program. If I set
>>>>>> MMGT_OPT
>>>>>> to 1, I don't have any crash, crashes with MMGT_OPT=0 and MMGT_OPT=2
>>>>>> (Intel
>>>>>> TBB). Really difficult to say whether it comes from occ or pythonocc.
>>>>>>
>>>>>> If you achieve a C++ version of your program (the random part is not
>>>>>> necessary), I will add it to the OCE test suite.
>>>>>>
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>> 2011/9/29 Thomas Paviot <tpav...@gmail.com>
>>>>>>
>>>>>>> It will.
>>>>>>>
>>>>>>>
>>>>>>> 2011/9/29 István Csanády <istvancsan...@gmail.com>
>>>>>>>
>>>>>>>> OK, I going to port it today, and check the memory consumption. (I
>>>>>>>> am using opencascade 6.5, but it will work with OCE, won't it?)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 29, 2011 at 11:29 AM, Thomas Paviot
>>>>>>>> <tpav...@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> There are two different issues IMO:
>>>>>>>>> - the segfault. I think I have identified the origin of the
>>>>>>>>> problem, it comes from the GarbageCollector implementation
>>>>>>>>> - the leak. I also noticed this increase in memory consumption,
>>>>>>>>> however it's no more the case if the size of the vert list is
>>>>>>>>> constant. If
>>>>>>>>> you replace, for instance, the random loop with:
>>>>>>>>> for x in range(50):
>>>>>>>>> for y in range(50):
>>>>>>>>> [...]
>>>>>>>>> Then memory consumption does not increase anymore (can you check
>>>>>>>>> that point?). That means that the pythonocc garbage collector does
>>>>>>>>> the job
>>>>>>>>> as expected and I suspect rather OCC memory manager to be the cause
>>>>>>>>> of this
>>>>>>>>> behavior. To be sure about that, this example should first be ported
>>>>>>>>> to C++
>>>>>>>>> and included into the OCE testing suite.
>>>>>>>>>
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>> 2011/9/29 István Csanády <istvancsan...@gmail.com>
>>>>>>>>>
>>>>>>>>>> The program you sent is still leaking even if I comment out
>>>>>>>>>> gcurve.GetObject() but it seems to be better then it was. If I leave
>>>>>>>>>> the
>>>>>>>>>> gcurve.GetObject() line it also crashes.
>>>>>>>>>> Here is a memory consumption log:
>>>>>>>>>>
>>>>>>>>>> 76.015625, 76.015625, 76.19921875, 76.19921875, 76.19921875,
>>>>>>>>>> 83.55859375, 82.9140625, 82.9140625, 82.9140625, 84.13671875,
>>>>>>>>>> 85.296875, 84.99609375, 84.99609375, 84.99609375, 83.24609375,
>>>>>>>>>> 84.234375, 83.24609375, 84.140625, 89.64453125, 89.4140625,
>>>>>>>>>> 87.4140625, 87.4140625, 87.1640625, 87.1640625, 88.3125, 88.1640625,
>>>>>>>>>> 88.1640625, 88.73046875, 88.1640625, 88.1640625, 87.21484375,
>>>>>>>>>> 86.1640625,
>>>>>>>>>> 86.1640625, 86.1640625, 89.1328125, 89.10546875, 88.16796875,
>>>>>>>>>> 92.15234375,
>>>>>>>>>> 91.16796875, 90.16796875, 90.16796875, 90.16796875, 95.42578125,
>>>>>>>>>> 95.07421875, 94.71875, 93.76953125, 93.32421875, 92.07421875,
>>>>>>>>>> 92.921875, 92.10546875, 91.49609375, 91.07421875, 91.07421875,
>>>>>>>>>> 91.07421875, 91.07421875, 94.8515625, 94.07421875, 94.07421875,
>>>>>>>>>> 93.82421875, 93.82421875, 93.82421875, 93.82421875, 92.82421875,
>>>>>>>>>> 92.82421875, 93.38671875, 93.1328125, 93.1328125, 93.1328125,
>>>>>>>>>> 93.1328125, 94.26171875, 92.625, 92.625, 93.69921875, 94.5078125,
>>>>>>>>>> 93.578125, 92.578125, 92.578125, 92.078125, 91.078125, 91.078125,
>>>>>>>>>> 93.2109375, 93.078125, 92.078125, 92.078125, 94.12109375,
>>>>>>>>>> 93.4921875, 93.078125, 95.7890625, 93.078125, 93.45703125, 93.078125,
>>>>>>>>>> 93.078125, 94.41796875, 93.578125, 94.27734375, 93.578125, 93.578125,
>>>>>>>>>> 93.578125, 93.59375, 93.66796875, 93.328125, 93.41796875,
>>>>>>>>>> 93.98828125, 96.0625, 95.5703125, 95.328125, 95.44140625,
>>>>>>>>>> 95.328125, 95.328125, 95.078125, 95.078125, 93.92578125,
>>>>>>>>>> 93.078125, 93.1796875, 93.078125, 92.078125, 92.078125, 93.4921875,
>>>>>>>>>> 93.4921875, 93.078125, 93.078125, 93.078125, 93.4296875, 93.078125,
>>>>>>>>>> 94.2109375, 93.48828125, 94.1640625, 93.578125, 93.26953125,
>>>>>>>>>> 92.578125,
>>>>>>>>>> 91.578125, 90.828125, 90.828125, 90.828125, 90.828125,
>>>>>>>>>> 97.32421875, 96.7421875, 97.63671875, 96.7421875, 96.7421875,
>>>>>>>>>> 96.7421875, 96.7421875, 96.7421875, 96.7421875, 96.7421875,
>>>>>>>>>> 96.7421875,
>>>>>>>>>> 95.7421875, 95.7421875, 95.7421875, 95.7421875, 98.7890625,
>>>>>>>>>> 97.33203125, 97.59375, 97.33203125, 97.08203125, 99.3828125,
>>>>>>>>>> 97.33203125, 96.4453125, 96.08203125, 95.08203125, 95.08203125,
>>>>>>>>>> 95.08203125, 95.08203125, 96.890625, 96.08203125, 96.08203125,
>>>>>>>>>> 97.80078125,
>>>>>>>>>> 97.08203125, 98.56640625, 97.46875, 97.46875, 97.46875, 96.46875,
>>>>>>>>>> 96.08203125, 100.66015625, 100.33203125
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 29, 2011 at 7:00 AM, Thomas Paviot <tpav...@gmail.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> The issue comes from the line curve_object = gcurve.GetObject().
>>>>>>>>>>> Comment out this line to check that memory consumption is constant
>>>>>>>>>>> over
>>>>>>>>>>> time.
>>>>>>>>>>>
>>>>>>>>>>> Attached another implementation of your program, can you please
>>>>>>>>>>> test it and report.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thomas
>>>>>>>>>>>
>>>>>>>>>>> 2011/9/28 István Csanády <istvancsan...@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The corrected code does not crash, but it leaks like sieve
>>>>>>>>>>>> (0-3MB/s for me)... And if you try to call smart_purge() after
>>>>>>>>>>>> pop_context()
>>>>>>>>>>>> it segfaults.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 28, 2011 at 10:22 PM, Thomas Paviot <
>>>>>>>>>>>> tpav...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> 2011/9/28 István Csanády <istvancsan...@gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Thomas,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 9:19 PM, Thomas Paviot <
>>>>>>>>>>>>>> tpav...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Istvan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Not yet. Trying to run your example actually helped me to
>>>>>>>>>>>>>>> figure out/fix a serious regression in the current pythonocc
>>>>>>>>>>>>>>> master branch.
>>>>>>>>>>>>>>> There is still another bug to fix. I'm currently running
>>>>>>>>>>>>>>> pythonocc master (
>>>>>>>>>>>>>>> https://github.com/tpaviot/pythonocc)/OCE-0.6.0-rc3.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do you run pythonocc-0.5/OCC630?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So you say that my code does not crash for you? If I remember
>>>>>>>>>>>>>> correctly you are a Mac user to, aren't you? Are you using
>>>>>>>>>>>>>> 10.6.7 too with
>>>>>>>>>>>>>> Python 2.6.1 ? May I try to recompile pythonocc with OCE? Can it
>>>>>>>>>>>>>> fix this
>>>>>>>>>>>>>> error? This bug is incredibly hard to track down since the
>>>>>>>>>>>>>> python debugger
>>>>>>>>>>>>>> does not stop when the C++ code crashes... Do you use any
>>>>>>>>>>>>>> special tools to
>>>>>>>>>>>>>> debug such errors when developing pythonocc?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I didn't say it crashes, I was running the test.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's done now, I can reproduce the crash, the program segfaults
>>>>>>>>>>>>> after a few loops are performed (I run OSX 10.6.8). I can't
>>>>>>>>>>>>> explain so far
>>>>>>>>>>>>> what happens, I need to dive into the example (make it even
>>>>>>>>>>>>> simpler) and
>>>>>>>>>>>>> check what's going on.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I also suggest you use garbage.push_context() and
>>>>>>>>>>>>> pop_context(): only objects created between push/pop will be
>>>>>>>>>>>>> killed by the
>>>>>>>>>>>>> smart_purge method. The corrected program attached works properly.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> István
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2011/9/28 István Csanády <istvancsan...@gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Thomas,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Have you managed to reproduce the error? I've been trying to
>>>>>>>>>>>>>>>> track down the bug with valgrind - without any success...
>>>>>>>>>>>>>>>> Probably I should
>>>>>>>>>>>>>>>> recompile python with the debug and --without-pymalloc flags
>>>>>>>>>>>>>>>> to make
>>>>>>>>>>>>>>>> valgrind work with it. I am using Mac OS X 10.6.7.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> István
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Sep 28, 2011 at 11:21 AM, Thomas Paviot <
>>>>>>>>>>>>>>>> tpav...@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2011/9/27 István Csanády <istvancsan...@gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Istvan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Last time when I wrote about this bug I thought it was
>>>>>>>>>>>>>>>>>> caused by some numpy buffer overflow. Unfortunately it was
>>>>>>>>>>>>>>>>>> not. Finally I
>>>>>>>>>>>>>>>>>> managed to create some code that can reproduce this bug. I
>>>>>>>>>>>>>>>>>> have attached the
>>>>>>>>>>>>>>>>>> code and some crash logs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've been trying to reproduce the issue.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think the problem is probably with OCC's
>>>>>>>>>>>>>>>>>> Standard_MMgrOpt class memory recycling for small blocks (Is
>>>>>>>>>>>>>>>>>> it possible
>>>>>>>>>>>>>>>>>> that smart_purge frees memory that Standard_MMgrOpt want to
>>>>>>>>>>>>>>>>>> reuse?).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> No.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Note that sometimes it takes a long time until the error
>>>>>>>>>>>>>>>>>> occurs, but usually after the 50th-100th iteration the code
>>>>>>>>>>>>>>>>>> crashes for me.
>>>>>>>>>>>>>>>>>> (The code is totally pointless, I have removed every
>>>>>>>>>>>>>>>>>> unneccessary parts)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> István
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Pythonocc-users mailing list
>>>>>>>>> Pythonocc-users@gna.org
>>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Pythonocc-users mailing list
>>>>>>>> Pythonocc-users@gna.org
>>>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Pythonocc-users mailing list
>>>>>> Pythonocc-users@gna.org
>>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pythonocc-users mailing list
>>>>> Pythonocc-users@gna.org
>>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pythonocc-users mailing list
>>>> Pythonocc-users@gna.org
>>>> https://mail.gna.org/listinfo/pythonocc-users
>>>>
>>>>
>>>
>>
>
> _______________________________________________
> Pythonocc-users mailing list
> Pythonocc-users@gna.org
> https://mail.gna.org/listinfo/pythonocc-users
>
>
_______________________________________________
Pythonocc-users mailing list
Pythonocc-users@gna.org
https://mail.gna.org/listinfo/pythonocc-users
