Re: [osg-users] Memory management in OSG
Hi J-S, Jean-Sébastien Guay wrote: I am currently chasing memory leaks in our application, which is like a modeling tool where the user will open/work on/save/close data files multiple times during the same application run. What we are seeing is that when a file is closed, not all memory related to scene graph objects seems to be released. First question: I enabled the DEBUG_OBJECT_ALLOCATION_DESTRUCTION define in src/osg/Referenced.cpp, and have set up a small test which just creates two viewers one after the other in different scopes. Pseudocode: [...] What I see is this (all in debug): Breakpoint #1 :8 Referenced objects, 12MB First viewer run : ~505 Referenced objects, 266MB Breakpoint #2 : 247 Referenced objects, 167MB Second viewer run : ~505 Referenced objects, 266MB Breakpoint #3 : 247 Referenced objects, 172MB But then, when I pass breakpoint #3, all the destructors for those Referenced objects are called, and at the end no objects are left allocated. (I can see that the final number of objects is 0) Other than seeing that there are no real leaks per se, my question about that is, what are those objects that are staying alive outside the scopes? You could perhaps hack on the DEBUG_OBJECT_ALLOCATION_DESTRUCTION code in src/osg/Referenced.cpp a bit and, for instances that derive from Object, print their libraryName()+className() as they get created/destoyed instead of just the total number of objects. Maybe that can give you some information on which objects stay around. Paul ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
Hi Jean-Sébastien, Jean-Sébastien Guay schrieb: However, running IBM Purify on the application reveals no massive leaks, only a few false positives. Are there any tools or techniques that someone could recommend to make finding and fixing memory leaks easy, and which work well with OSG? I tried Visual Leak Detector which is open source, but it reported so many false positives related to variables stored in ref_ptrs (over 400) that I gave up on it. For that type of debugging I am using a custom singleton-class called AllocationObserver which observes the refcount for osg::Referenced. You have to register every instance of objects you want to track and the AllocationObserver saves these instances as a std::mapstd::string, std::listosg::observer_ptrosg::Referenced I am using a custom NodeVisitor to feed the scene graph to the allocation-observer which prints out the refcounts / updates a line-graph for the different object-types and cleans up the std::lists of already deleted objects on a regular basis. This works quite well for my high dynamic scene graphs, but has a slight impact on performance and does not cover all referenced objects. I am using Memory Validator on Windows and Shark / MallocDebug on Mac OS X to track other memory-leaks, but these tools do not help me when using refcounted-objects or finding circular-references. If you are interested in the code (it's integrated with other code, but the principle should be clear) drop me a line. Cheers, Stephan ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
Paul Melis wrote: Hi J-S, Jean-Sébastien Guay wrote: I am currently chasing memory leaks in our application, which is like a modeling tool where the user will open/work on/save/close data files multiple times during the same application run. What we are seeing is that when a file is closed, not all memory related to scene graph objects seems to be released. First question: I enabled the DEBUG_OBJECT_ALLOCATION_DESTRUCTION define in src/osg/Referenced.cpp, and have set up a small test which just creates two viewers one after the other in different scopes. Pseudocode: [...] What I see is this (all in debug): Breakpoint #1 :8 Referenced objects, 12MB First viewer run : ~505 Referenced objects, 266MB Breakpoint #2 : 247 Referenced objects, 167MB Second viewer run : ~505 Referenced objects, 266MB Breakpoint #3 : 247 Referenced objects, 172MB But then, when I pass breakpoint #3, all the destructors for those Referenced objects are called, and at the end no objects are left allocated. (I can see that the final number of objects is 0) Other than seeing that there are no real leaks per se, my question about that is, what are those objects that are staying alive outside the scopes? You could perhaps hack on the DEBUG_OBJECT_ALLOCATION_DESTRUCTION code in src/osg/Referenced.cpp a bit and, for instances that derive from Object, print their libraryName()+className() as they get created/destoyed instead of just the total number of objects. Maybe that can give you some information on which objects stay around. Just tried to add this myself, but this is problematic as (part of) the debug info is printed from Referenced's constructor. At that point dynamic_cast and typeid don't give the actual type being constructed, but merely the constructor's type (i.e. Referenced). Browsing some more for that it seems the C++ spec says the results are actually undefined. So there's no way to dyncast a Referenced to an Object (to get to its className()), at least not at the point where currently the object tracking is done. Paul ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
2008/12/19 Pierre Bourdin (gmail) pierre.bour...@imerir.com: Hi, valgrind is very powerfull if you're on an Linux/Unix environnement, if you are using MSVC you can use this: add this bloc at the beginning of each file you want to trace (after all includes): #ifdef _WIN32 # ifdef _MSC_VER # ifdef _DEBUG # include crtdbg.h # undef THIS_FILE static char THIS_FILE[] = __FILE__; # define malloc(s) _malloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define calloc(s) _calloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define realloc(s) _recalloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define new new(_NORMAL_BLOCK, __FILE__, __LINE__) # define delete delete(_NORMAL_BLOCK, __FILE__, __LINE__) # endif /* _DEBUG */ # endif /* _MSC_VER */ #endif /* _WIN32 */ Doesn't crtDbg.h do what you want? #define _CRTDBG_MAP_ALLOC #define _CRTDBG_MAP_ALLOC_NEW #include crtDbg.h -- The truth is out there. Usually in header files. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
Le vendredi 19 décembre 2008 à 13:43 +, Simon Hammett a écrit : 2008/12/19 Pierre Bourdin (gmail) pierre.bour...@imerir.com: Hi, valgrind is very powerfull if you're on an Linux/Unix environnement, if you are using MSVC you can use this: add this bloc at the beginning of each file you want to trace (after all includes): #ifdef _WIN32 # ifdef _MSC_VER # ifdef _DEBUG # include crtdbg.h # undef THIS_FILE static char THIS_FILE[] = __FILE__; # define malloc(s) _malloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define calloc(s) _calloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define realloc(s) _recalloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define new new(_NORMAL_BLOCK, __FILE__, __LINE__) # define delete delete(_NORMAL_BLOCK, __FILE__, __LINE__) # endif /* _DEBUG */ # endif /* _MSC_VER */ #endif /* _WIN32 */ Doesn't crtDbg.h do what you want? #define _CRTDBG_MAP_ALLOC #define _CRTDBG_MAP_ALLOC_NEW #include crtDbg.h Yes it does except I don't know if it allows to go back to the allocated bloc when it finds some memory leaks... By the way, just putting before the # include crtdbg.h #define _CRTDBG_MAP_ALLOC #define _CRTDBG_MAP_ALLOC_NEW #include crtDbg.h in stead of after the # include crtdbg.h # define malloc(s) _malloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define calloc(s) _calloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define realloc(s) _recalloc_dbg(s, _NORMAL_BLOCK, __FILE__, __LINE__) # define new new(_NORMAL_BLOCK, __FILE__, __LINE__) # define delete delete(_NORMAL_BLOCK, __FILE__, __LINE__) makes a link error on msvc 2008, complaining operator new is already defined... So there's maybe something else missing ? Pierre BOURDIN I.M.E.R.I.R. Av. Pascot BP 90443 66004 PERPIGNAN tél: 04 68 56 80 18 fax: 04 68 55 03 86 email: bour...@imerir.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
Hey guys, Stephan Huber wrote: However, running IBM Purify on the application reveals no massive leaks, only a few false positives. Are there any tools or techniques that someone could recommend to make finding and fixing memory leaks easy, and which work well with OSG? I tried Visual Leak Detector which is open source, but it reported so many false positives related to variables stored in ref_ptrs (over 400) that I gave up on it. For that type of debugging I am using a custom singleton-class called AllocationObserver which observes the refcount for osg::Referenced. You That gave me an idea. You can't determine the actual type being constructed in either the Referenced constructor or destructor, but you can set a delete handler on Referenced that extracts this info before the destructor is called. See attached small diff to src/osg/Referenced.cpp (against the 2.6 branch, includes end-of-line whitespace trimming as my editor is set up like that and I'm too lazy to turn it off). This forces a line of output to be written to stdout every time a Referenced instance is allocated or deallocated, together with the instance's address. By using the deallocations you can figure out the allocations (which don't have type info), which is what the alloc.py script does. It filters the original output of a program run and replaces the unknown allocated instance types with the actual ones gotten from deallocations. This gives quite nice statistics on the number of objects created, but unfortunately there seems to be a group of objects that is never deallocated. Their type can therefore not be determined. See below for example (processed by alloc.py) output when running osgviewer on the cow, and doing some viewpoint interaction and stuff. What I don't understand is why osg::Matrix ends up in the list (at a count of a few thousand in this case), as that isn't a Referenced subclass. In J-S's original simple test case I get around 250 Referenced instances that are never deallocated when stopping at breakpoint 2, but sadly can't tell where they come from. Perhaps moving class/libraryName() up into Referenced might help here? Regards, Paul P.S. Stephan, you say AllocationObserver which observes the refcount for osg::Referenced, but it only observes the refcount going to 0 right? I don't seen any other 'events' for observers. SUMMARY osg::AlphaFunc 3 A 3 D (0) osg::AnimationPath 2 A 2 D (0) osg::AnimationPathCallback 1 A 1 D (0) osg::AutoTransform 1 A 1 D (0) osg::Billboard 1 A 1 D (0) osg::BlendColor 1 A 1 D (0) osg::BlendEquation 1 A 1 D (0) osg::BlendFunc 12 A 12 D (0) osg::Box 1 A 1 D (0) osg::Camera 13 A 13 D (0) osg::CameraView 1 A 1 D (0) osg::Capsule 1 A 1 D (0) osg::ClearNode 2 A 2 D (0) osg::ClipNode1 A 1 D (0) osg::ClipPlane 1 A 1 D (0) osg::ClusterCullingCallback 1 A 1 D (0) osg::ColorMask 5 A 5 D (0) osg::ColorMatrix 1 A 1 D (0) osg::CompositeShape 1 A 1 D (0) osg::Cone1 A 1 D (0) osg::ConvexPlanarOccluder1 A 1 D (0) osg::CoordinateSystemNode1 A 1 D (0) osg::CullFace1 A 1 D (0) osg::Cylinder1 A 1 D (0) osg::Depth 2 A 2 D (0) osg::DrawArrayLengths1 A 1 D (0) osg::DrawArrays 7 A 7 D (0) osg::DrawCallback 13 A 13 D (0) osg::DrawElementsUShort 2 A 2 D (0) osg::EllipsoidModel 1 A 1 D (0) osg::FloatArray 14 A 14 D
Re: [osg-users] Memory management in OSG
Hello Pierre, valgrind is very powerfull if you're on an Linux/Unix environnement, if you are using MSVC you can use this: ... when you stop debugging you program, it make a memory leaks report in the msvc output console... Does this work well with ref_ptrs? i.e. will it spit out hundreds of false positives for things that are ref counted and do not actually leak? What I'd really like is to compare the state of memory (preferably with where the memory was allocated) at two points of the program's run. If I can just get a dump of that, and then diff the two, I could see what leaks between when the first file is opened and when the second one is. The project we're working on currently does not run on Linux though it will eventually, so valgrind is not an alternative right now. Thanks, J-S -- __ Jean-Sebastien Guayjean-sebastien.g...@cm-labs.com http://www.cm-labs.com/ http://whitestar02.webhop.org/ ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
Paul Melis wrote: Hey guys, Stephan Huber wrote: However, running IBM Purify on the application reveals no massive leaks, only a few false positives. Are there any tools or techniques that someone could recommend to make finding and fixing memory leaks easy, and which work well with OSG? I tried Visual Leak Detector which is open source, but it reported so many false positives related to variables stored in ref_ptrs (over 400) that I gave up on it. For that type of debugging I am using a custom singleton-class called AllocationObserver which observes the refcount for osg::Referenced. You That gave me an idea. You can't determine the actual type being constructed in either the Referenced constructor or destructor, but you can set a delete handler on Referenced that extracts this info before the destructor is called. See attached small diff to src/osg/Referenced.cpp (against the 2.6 branch, includes end-of-line whitespace trimming as my editor is set up like that and I'm too lazy to turn it off). This forces a line of output to be written to stdout every time a Referenced instance is allocated or deallocated, together with the instance's address. By using the deallocations you can figure out the allocations (which don't have type info), which is what the alloc.py script does. It filters the original output of a program run and replaces the unknown allocated instance types with the actual ones gotten from deallocations. This gives quite nice statistics on the number of objects created, but unfortunately there seems to be a group of objects that is never deallocated. Their type can therefore not be determined. See below for example (processed by alloc.py) output when running osgviewer on the cow, and doing some viewpoint interaction and stuff. What I don't understand is why osg::Matrix ends up in the list (at a count of a few thousand in this case), as that isn't a Referenced subclass. Just figured it out that these must be RefMatrix's... Paul In J-S's original simple test case I get around 250 Referenced instances that are never deallocated when stopping at breakpoint 2, but sadly can't tell where they come from. Perhaps moving class/libraryName() up into Referenced might help here? Regards, Paul P.S. Stephan, you say AllocationObserver which observes the refcount for osg::Referenced, but it only observes the refcount going to 0 right? I don't seen any other 'events' for observers. SUMMARY osg::AlphaFunc 3 A 3 D (0) osg::AnimationPath 2 A 2 D (0) osg::AnimationPathCallback 1 A 1 D (0) osg::AutoTransform 1 A 1 D (0) osg::Billboard 1 A 1 D (0) osg::BlendColor 1 A 1 D (0) osg::BlendEquation 1 A 1 D (0) osg::BlendFunc 12 A 12 D (0) osg::Box 1 A 1 D (0) osg::Camera 13 A 13 D (0) osg::CameraView 1 A 1 D (0) osg::Capsule 1 A 1 D (0) osg::ClearNode 2 A 2 D (0) osg::ClipNode1 A 1 D (0) osg::ClipPlane 1 A 1 D (0) osg::ClusterCullingCallback 1 A 1 D (0) osg::ColorMask 5 A 5 D (0) osg::ColorMatrix 1 A 1 D (0) osg::CompositeShape 1 A 1 D (0) osg::Cone1 A 1 D (0) osg::ConvexPlanarOccluder1 A 1 D (0) osg::CoordinateSystemNode1 A 1 D (0) osg::CullFace1 A 1 D (0) osg::Cylinder1 A 1 D (0) osg::Depth 2 A 2 D (0) osg::DrawArrayLengths1 A 1 D (0) osg::DrawArrays 7 A 7 D (0) osg::DrawCallback 13 A 13 D (0) osg::DrawElementsUShort 2 A 2 D (0) osg::EllipsoidModel 1 A 1 D
Re: [osg-users] Memory management in OSG
Hi Stephan, If you are interested in the code (it's integrated with other code, but the principle should be clear) drop me a line. Yes, I am very interested. You mentioned circular references, and this is my current area of focus - does your code help finding these? For example, a node callback that would have a ref_ptr to a node somewhere above it in the graph, causing that subgraph never to be deleted? Thanks in advance, J-S -- __ Jean-Sebastien Guayjean-sebastien.g...@cm-labs.com http://www.cm-labs.com/ http://whitestar02.webhop.org/ ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
Hi Paul, Good sleuthing, I'll try your code out and see if I can improve on it a bit too... In J-S's original simple test case I get around 250 Referenced instances that are never deallocated when stopping at breakpoint 2, but sadly can't tell where they come from. Perhaps moving class/libraryName() up into Referenced might help here? I think those objects could be inspected by creating an atexit() handler, since they actually *are* deallocated at the end of the program (put a breakpoint in the Referenced destructor after breakpoint 2 and you'll see). I imagine they must be static objects or something, but ~250 seems like a lot to me. So getting info about what they are would be instructional if not that useful :-) J-S -- __ Jean-Sebastien Guayjean-sebastien.g...@cm-labs.com http://www.cm-labs.com/ http://whitestar02.webhop.org/ ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
Le vendredi 19 décembre 2008 à 10:31 -0500, Jean-Sébastien Guay a écrit : Hello Pierre, valgrind is very powerfull if you're on an Linux/Unix environnement, if you are using MSVC you can use this: ... when you stop debugging you program, it make a memory leaks report in the msvc output console... Does this work well with ref_ptrs? i.e. will it spit out hundreds of false positives for things that are ref counted and do not actually leak? What I'd really like is to compare the state of memory (preferably with where the memory was allocated) at two points of the program's run. If I can just get a dump of that, and then diff the two, I could see what leaks between when the first file is opened and when the second one is. The project we're working on currently does not run on Linux though it will eventually, so valgrind is not an alternative right now. Thanks, J-S I'm afraid you will have some false positive, but I've not tested... I understand pretty well what you would like to do, but I don't know if ti's possible... Maybe using the _CRTDBG_DELAY_FREE_MEM_DF (Keep freed memory blocks in the heap's linked list, assign them the _FREE_BLOCK type, and fill them with the byte value 0xDD. OFF: Do not keep freed blocks in the heap's linked list. ) The best way is probably to log every allocation and deletion to a file using a macro to override the normal new, delete, malloc or free... Then you will have a complete liste of allocated and liberated bloc... Maybe it can be of any help ? http://www.codeproject.com/KB/cpp/MLFDef.aspx Pierre. Pierre BOURDIN I.M.E.R.I.R. Av. Pascot BP 90443 66004 PERPIGNAN tél: 04 68 56 80 18 fax: 04 68 55 03 86 email: bour...@imerir.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
[osg-users] Memory management in OSG
Hello all,
I am currently chasing memory leaks in our application, which is like a
modeling tool where the user will open/work on/save/close data files
multiple times during the same application run. What we are seeing is
that when a file is closed, not all memory related to scene graph
objects seems to be released.
First question:
I enabled the DEBUG_OBJECT_ALLOCATION_DESTRUCTION define in
src/osg/Referenced.cpp, and have set up a small test which just creates
two viewers one after the other in different scopes. Pseudocode:
int main(int argc, char** argv)
{
int bob = 1;// breakpoint #1
{
osgViewer::Viewer viewer;
osg::ref_ptrosg::Node loadedModel =
osgDB::readNodeFile(cow.osg);
viewer.setSceneData( loadedModel.get() );
viewer.run();
}
bob = 2;// breakpoint #2
{
osgViewer::Viewer viewer;
osg::ref_ptrosg::Node loadedModel =
osgDB::readNodeFile(cow.osg);
viewer.setSceneData( loadedModel.get() );
viewer.run();
}
bob = 3;// breakpoint #3
}
What I see is this (all in debug):
Breakpoint #1 :8 Referenced objects, 12MB
First viewer run : ~505 Referenced objects, 266MB
Breakpoint #2 : 247 Referenced objects, 167MB
Second viewer run : ~505 Referenced objects, 266MB
Breakpoint #3 : 247 Referenced objects, 172MB
But then, when I pass breakpoint #3, all the destructors for those
Referenced objects are called, and at the end no objects are left
allocated. (I can see that the final number of objects is 0)
Other than seeing that there are no real leaks per se, my question
about that is, what are those objects that are staying alive outside the
scopes? I would have thought that most objects would only live in one of
the viewer scopes above, 247 seems pretty large to me...
Second question:
Our application apparently has memory leaks, because when closing a file
and opening the same one again, the application takes about 20MB more
each time (i.e. memory usage is ~100MB with a file open, then ~50MB when
it's closed, then ~120MB when it's opened again, then ~70MB when it's
closed again, then ~140MB when it's opened again, and so on).
However, running IBM Purify on the application reveals no massive leaks,
only a few false positives. Are there any tools or techniques that
someone could recommend to make finding and fixing memory leaks easy,
and which work well with OSG? I tried Visual Leak Detector which is open
source, but it reported so many false positives related to variables
stored in ref_ptrs (over 400) that I gave up on it.
Thanks in advance,
J-S
--
__
Jean-Sebastien Guayjean-sebastien.g...@cm-labs.com
http://www.cm-labs.com/
http://whitestar02.webhop.org/
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Memory management in OSG
Hi Jean Sebastion,
If your application run on linux plateform you can try valgrind, it will
give you a lot of informations about leaks and more.
Cheers,
Cedric
Jean-Sébastien Guay wrote:
Hello all,
I am currently chasing memory leaks in our application, which is like
a modeling tool where the user will open/work on/save/close data files
multiple times during the same application run. What we are seeing is
that when a file is closed, not all memory related to scene graph
objects seems to be released.
First question:
I enabled the DEBUG_OBJECT_ALLOCATION_DESTRUCTION define in
src/osg/Referenced.cpp, and have set up a small test which just
creates two viewers one after the other in different scopes. Pseudocode:
int main(int argc, char** argv)
{
int bob = 1;// breakpoint #1
{
osgViewer::Viewer viewer;
osg::ref_ptrosg::Node loadedModel =
osgDB::readNodeFile(cow.osg);
viewer.setSceneData( loadedModel.get() );
viewer.run();
}
bob = 2;// breakpoint #2
{
osgViewer::Viewer viewer;
osg::ref_ptrosg::Node loadedModel =
osgDB::readNodeFile(cow.osg);
viewer.setSceneData( loadedModel.get() );
viewer.run();
}
bob = 3;// breakpoint #3
}
What I see is this (all in debug):
Breakpoint #1 :8 Referenced objects, 12MB
First viewer run : ~505 Referenced objects, 266MB
Breakpoint #2 : 247 Referenced objects, 167MB
Second viewer run : ~505 Referenced objects, 266MB
Breakpoint #3 : 247 Referenced objects, 172MB
But then, when I pass breakpoint #3, all the destructors for those
Referenced objects are called, and at the end no objects are left
allocated. (I can see that the final number of objects is 0)
Other than seeing that there are no real leaks per se, my question
about that is, what are those objects that are staying alive outside
the scopes? I would have thought that most objects would only live in
one of the viewer scopes above, 247 seems pretty large to me...
Second question:
Our application apparently has memory leaks, because when closing a
file and opening the same one again, the application takes about 20MB
more each time (i.e. memory usage is ~100MB with a file open, then
~50MB when it's closed, then ~120MB when it's opened again, then ~70MB
when it's closed again, then ~140MB when it's opened again, and so on).
However, running IBM Purify on the application reveals no massive
leaks, only a few false positives. Are there any tools or techniques
that someone could recommend to make finding and fixing memory leaks
easy, and which work well with OSG? I tried Visual Leak Detector which
is open source, but it reported so many false positives related to
variables stored in ref_ptrs (over 400) that I gave up on it.
Thanks in advance,
J-S
--
+33 (0) 6 63 20 03 56 Cedric Pinson mailto:morni...@plopbyte.net
http://www.plopbyte.net
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
