Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Vincent, Could you post the whole modified file to osg-submissions. Thanks, Robert. On Thu, Mar 4, 2010 at 9:13 AM, Vincent Bourdier wrote: > Hi Robert, > > This is the memory management patch, without the callback. If you are > interested in... use it. :-) > > Regards, > Vincent. > > Le 19/02/2010 16:40, Vincent Bourdier a écrit : > > Hi Robert, >> >> osgcallback already implements a ReadFilecallback... so I don't know if it >> will be useful to make an example. >> If you want I can make a wiki page, but for that could you tell me were to >> put it ? >> >> Thanks. >> >> Regards, >> Vincent. >> >> Robert Osfield a écrit : >> >>> Hi Vincent, >>> >>> On Fri, Feb 19, 2010 at 2:09 PM, Vincent Bourdier >>> wrote: >>> I implemented my own ReadFile callback, working well. >>> >>> Yay! :-) >>> >>> Would you (and osg) be interested in having it ? I can submitt in directly in osg src. >>> >>> It's probably a bit too domain specific to be part of the OSG, perhaps >>> an example with it in would be good, or an example on the wiki so that >>> if others want to use the same techniques then they could deploy. >>> >>> I implemented if for WIN32 only, so someone should implement the other version... For the moment, it uses psapi.lib ... not very portable but maybe someone can help be making it portable and without other dependencies. >>> >>> >>> I don't even know was psapi.lib is... ;-) >>> >>> Robert. >>> ___ >>> osg-users mailing list >>> osg-users@lists.openscenegraph.org >>> http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org >>> >> >> >> __ Information from ESET NOD32 Antivirus, version of virus >> signature database 4880 (20100219) __ >> >> The message was checked by ESET NOD32 Antivirus. >> >> http://www.eset.com >> >> >> > ___ > osg-users mailing list > osg-users@lists.openscenegraph.org > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org > > ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Robert,
This is the memory management patch, without the callback. If you are
interested in... use it. :-)
Regards,
Vincent.
Le 19/02/2010 16:40, Vincent Bourdier a écrit :
Hi Robert,
osgcallback already implements a ReadFilecallback... so I don't know
if it will be useful to make an example.
If you want I can make a wiki page, but for that could you tell me
were to put it ?
Thanks.
Regards,
Vincent.
Robert Osfield a écrit :
Hi Vincent,
On Fri, Feb 19, 2010 at 2:09 PM, Vincent Bourdier
wrote:
I implemented my own ReadFile callback, working well.
Yay! :-)
Would you (and osg) be interested in having it ? I can submitt in
directly
in osg src.
It's probably a bit too domain specific to be part of the OSG, perhaps
an example with it in would be good, or an example on the wiki so that
if others want to use the same techniques then they could deploy.
I implemented if for WIN32 only, so someone should implement the other
version...
For the moment, it uses psapi.lib ... not very portable but maybe
someone
can help be making it portable and without
other dependencies.
I don't even know was psapi.lib is... ;-)
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
__ Information from ESET NOD32 Antivirus, version of virus
signature database 4880 (20100219) __
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
Index: include/osgDB/ReaderWriter
===
--- include/osgDB/ReaderWriter (revision 10288)
+++ include/osgDB/ReaderWriter (working copy)
@@ -208,6 +208,7 @@
FILE_LOADED, //!< File successfully found, loaded, and
converted into osg.
FILE_LOADED_FROM_CACHE, //!< File found in cache and
returned.
ERROR_IN_READING_FILE, //!< File found, loaded, but an
error was encountered during processing.
+ INSUFFICIENT_MEMORY_TO_LOAD, //!< File
found but not loaded because estimated memory usage is not enought
FILE_REQUESTED //!< Asyncronous file read has been
requested, but returning immediatiely, keep polling plugin till file read has
been completed.
};
@@ -248,6 +249,7 @@
bool error() const { return _status==ERROR_IN_READING_FILE; }
bool notHandled() const { return _status==FILE_NOT_HANDLED; }
bool notFound() const { return _status==FILE_NOT_FOUND; }
+ bool notEnoughMemory() const { return
_status==INSUFFICIENT_MEMORY_TO_LOAD; }
protected:
Index: src/osgDB/DatabasePager.cpp
===
--- src/osgDB/DatabasePager.cpp (revision 10288)
+++ src/osgDB/DatabasePager.cpp (working copy)
@@ -552,6 +552,7 @@
if (rr.validNode()) databaseRequest->_loadedModel = rr.getNode();
if (rr.error()) osg::notify(osg::WARN)<<"Error in reading file
"<_fileName<<" : "<_fileName << std::endl;
if (databaseRequest->_loadedModel.valid() &&
osgDB::containsServerAddress(databaseRequest->_fileName) &&
Index: src/osgDB/ReadFile.cpp
===
--- src/osgDB/ReadFile.cpp (revision 10288)
+++ src/osgDB/ReadFile.cpp (working copy)
@@ -69,6 +69,7 @@
ReaderWriter::ReadResult rr =
Registry::instance()->readNode(filename,options);
if (rr.validNode()) return rr.takeNode();
if (rr.error()) notify(WARN) << rr.message() << std::endl;
+ if (rr.notEnoughMemory()) notify (DEBUG_INFO) << "Not enought memory to
load file "<___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Robert, osgcallback already implements a ReadFilecallback... so I don't know if it will be useful to make an example. If you want I can make a wiki page, but for that could you tell me were to put it ? Thanks. Regards, Vincent. Robert Osfield a écrit : Hi Vincent, On Fri, Feb 19, 2010 at 2:09 PM, Vincent Bourdier wrote: I implemented my own ReadFile callback, working well. Yay! :-) Would you (and osg) be interested in having it ? I can submitt in directly in osg src. It's probably a bit too domain specific to be part of the OSG, perhaps an example with it in would be good, or an example on the wiki so that if others want to use the same techniques then they could deploy. I implemented if for WIN32 only, so someone should implement the other version... For the moment, it uses psapi.lib ... not very portable but maybe someone can help be making it portable and without other dependencies. I don't even know was psapi.lib is... ;-) Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org __ Information from ESET NOD32 Antivirus, version of virus signature database 4880 (20100219) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Vincent, On Fri, Feb 19, 2010 at 2:09 PM, Vincent Bourdier wrote: > I implemented my own ReadFile callback, working well. Yay! :-) > Would you (and osg) be interested in having it ? I can submitt in directly > in osg src. It's probably a bit too domain specific to be part of the OSG, perhaps an example with it in would be good, or an example on the wiki so that if others want to use the same techniques then they could deploy. > I implemented if for WIN32 only, so someone should implement the other > version... > For the moment, it uses psapi.lib ... not very portable but maybe someone > can help be making it portable and without > other dependencies. I don't even know was psapi.lib is... ;-) Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Robert, I implemented my own ReadFile callback, working well. Would you (and osg) be interested in having it ? I can submitt in directly in osg src. I implemented if for WIN32 only, so someone should implement the other version... For the moment, it uses psapi.lib ... not very portable but maybe someone can help be making it portable and without other dependencies. Regards, Vincent. Robert Osfield a écrit : On Thu, Feb 4, 2010 at 4:23 PM, Vincent Bourdier wrote: What difference will it makes if I return FILE_LOADED or ERROR_IN_READING_FILE ? If you didn't load the file you shouldn't return a FILE_LOADED, so since with don't have an INSUFFICIENT_MEMORY_TO_LOAD (or similar) enum then the closest would be ERROR_IN_READING_FILE. So If I decrease this value, some PagedLod will be released ? It won't release any right away, but as the frames progress it'll work to get the max down to your specified target when it's possible. What difference with cameraLODScale ? The osg::Camera::setLODScale() just changes which children are selected in the view, it doesn't change the max number of PagedLOD target, but it will directly affect how many PagedLOD that will be active in any view. I just made a test increasing/decreasing this value and I saw the effect on PagedLod and memory. Yes, this is exactly how one would expect it to work. Robert. __ Information from ESET NOD32 Antivirus, version of virus signature database 4869 (20100215) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
On Thu, Feb 4, 2010 at 4:23 PM, Vincent Bourdier wrote: > What difference will it makes if I return FILE_LOADED or > ERROR_IN_READING_FILE ? If you didn't load the file you shouldn't return a FILE_LOADED, so since with don't have an INSUFFICIENT_MEMORY_TO_LOAD (or similar) enum then the closest would be ERROR_IN_READING_FILE. > So If I decrease this value, some PagedLod will be released ? It won't release any right away, but as the frames progress it'll work to get the max down to your specified target when it's possible. > What difference with cameraLODScale ? The osg::Camera::setLODScale() just changes which children are selected in the view, it doesn't change the max number of PagedLOD target, but it will directly affect how many PagedLOD that will be active in any view. > I just made a test > increasing/decreasing this value and I saw the effect on PagedLod and > memory. Yes, this is exactly how one would expect it to work. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Robert Robert Osfield a écrit : So when the file size in memory estimation exceed the free memory estimated, the load is not done. But this is the point : if I return /ERROR_IN_READING_FILE, /won't the file be added to a black list ? What about /FILE_LOADED /? The OSG doesn't provide any blacklisting capability so it's something you'll have to implement yourself in your read callback. Sorry I was not enough precise in my explanations : I don't want any blacklist, I was asking if OSG gets one. So it is perfect for me that no file will be blacklisted. I was asking what error code should I return to make DatabasePager consider the file in not available, but if a new request is done later (when some memory has been released) its will try again. What difference will it makes if I return FILE_LOADED or ERROR_IN_READING_FILE ? Next, is there a way to force pagedLod to unload some nodes ? to implement a "force unload" to free a bit a memory. I'm using OSG 2.8.1 and I cannot use the last release... As the ReadFileCallback can be implemented, are there other possible personalization concerning the way to load/unload cached datas ? The DatabasePager now defaults to using a MaxPagedLOD scheme where it expires currently unused PageLOD children on demand, so that as lots of new PagedLOD come in to view ones that aren't in the view are deleted keeping a roughly constant load. You can adjust the target MaxPagedLOD in your app via the OSG_MAX_PAGED_LOD env var or the DatabasePager::setTargetMaximumNumberOfPageLOD(value). Please note this is a target that will be exceeded is more PagedLOD's are required for a single frame, so the view will not ever be constrained by the MaxPagedLOD target, once the high loaded view is moved away from the PagedLOD numbers will drop back down to the target. Robert. So If I decrease this value, some PagedLod will be released ? What difference with cameraLODScale ? I just made a test increasing/decreasing this value and I saw the effect on PagedLod and memory. Thanks for your answers. Regards, Vincent. __ Information from ESET NOD32 Antivirus, version of virus signature database 4835 (20100204) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Vincent, On Thu, Feb 4, 2010 at 2:16 PM, Vincent Bourdier wrote: > So when the file size in memory estimation exceed the free memory estimated, > the load is not done. > But this is the point : if I return /ERROR_IN_READING_FILE, /won't the file > be added to a black list ? > What about /FILE_LOADED /? The OSG doesn't provide any blacklisting capability so it's something you'll have to implement yourself in your read callback. > Next, is there a way to force pagedLod to unload some nodes ? to implement a > "force unload" to free a bit a memory. > I'm using OSG 2.8.1 and I cannot use the last release... > As the ReadFileCallback can be implemented, are there other possible > personalization concerning the way to load/unload cached datas ? The DatabasePager now defaults to using a MaxPagedLOD scheme where it expires currently unused PageLOD children on demand, so that as lots of new PagedLOD come in to view ones that aren't in the view are deleted keeping a roughly constant load. You can adjust the target MaxPagedLOD in your app via the OSG_MAX_PAGED_LOD env var or the DatabasePager::setTargetMaximumNumberOfPageLOD(value). Please note this is a target that will be exceeded is more PagedLOD's are required for a single frame, so the view will not ever be constrained by the MaxPagedLOD target, once the high loaded view is moved away from the PagedLOD numbers will drop back down to the target. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Robert,
I implemented my own ReadFileCallback and this sound to be a good
solution, but with some limits...
This is the code :
/if((_MaxAvailableMemoryInMBytes -
Common::MemoryAnalyzer::getProcWorkingSetSize() - (sizeInMBytes*2.1)) > 0 )
return
osgDB::Registry::instance()->readNodeImplementation(filename,options);
else
{
return
osgDB::ReaderWriter::ReadResult::FILE_LOADED/*ERROR_IN_READING_FILE*/;
}/
So when the file size in memory estimation exceed the free memory
estimated, the load is not done.
But this is the point : if I return /ERROR_IN_READING_FILE, /won't the
file be added to a black list ?
What about /FILE_LOADED /?
I don't what error code return to make the file being loaded later, when
some memory will been freed.
Next, is there a way to force pagedLod to unload some nodes ? to
implement a "force unload" to free a bit a memory.
I'm using OSG 2.8.1 and I cannot use the last release...
As the ReadFileCallback can be implemented, are there other possible
personalization concerning the way to load/unload cached datas ?
Thanks for your help.
Regard,
Vincent.
Vincent Bourdier a écrit :
Hi Robert,
I am trying to implement a new ReadFileCallback.
This callback will check the available memory, knowing the maximum
allowed du to JVM usage (c++ engine in Java application), and if there
is not enough memory the callback will not load the file.
I just ran some test to estimate the required memory space for each
ive size.
For big ive files, the memory used seems to be around 2*ive file
size. For little files (<5Mo) this is very fluctuating ... there is
not regularity (ratio from 1.8 to 5 ).
Do you have any idea of how to estimate better the required memory
space ? (only ive files as input datas)
Thanks.
Regards,
Vincent.
Robert Osfield a écrit :
Hi Vincent,
On Thu, Jan 14, 2010 at 5:50 PM, Vincent Bourdier
wrote:
I don't understand how this is working.
If a pagedLod is required it will be loaded... but we can cap the
number of
lod...
This sounds contradictory for me.
The max number of PagedLOD is a target that it aims for, but if there
are more PagedLOD required for rendering the current frame then the
max will exceed this trarget, and only once the view changes to
require less PagedLOD will the total number come down again.
If you wanted the LOD control then you just implement this yourself
using osgDB::Registry::ReadFileCallback that loads the tiles and then
modifies PagedLOD that it loads that are higher than a certain
resolution.
Do you mean that using my own readFileCallback is not sufficient and
I'll
need to modify PagedLod sources too ?
There is no need to modify any OSG sources to adjust the LOD ranges.
The ReadFileCallback can do the actual loading of the subgraph and
then catch the and modify the LOD ranges.
Personally I'd go go using the Camera LODScale though, it's a much
simpler tool.
Robert.
__ Information from ESET NOD32 Antivirus, version of virus
signature database 4805 (20100125) __
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
__ Information from ESET NOD32 Antivirus, version of virus signature
database 4835 (20100204) __
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Robert, I am trying to implement a new ReadFileCallback. This callback will check the available memory, knowing the maximum allowed du to JVM usage (c++ engine in Java application), and if there is not enough memory the callback will not load the file. I just ran some test to estimate the required memory space for each ive size. For big ive files, the memory used seems to be around 2*ive file size. For little files (<5Mo) this is very fluctuating ... there is not regularity (ratio from 1.8 to 5 ). Do you have any idea of how to estimate better the required memory space ? (only ive files as input datas) Thanks. Regards, Vincent. Robert Osfield a écrit : Hi Vincent, On Thu, Jan 14, 2010 at 5:50 PM, Vincent Bourdier wrote: I don't understand how this is working. If a pagedLod is required it will be loaded... but we can cap the number of lod... This sounds contradictory for me. The max number of PagedLOD is a target that it aims for, but if there are more PagedLOD required for rendering the current frame then the max will exceed this trarget, and only once the view changes to require less PagedLOD will the total number come down again. If you wanted the LOD control then you just implement this yourself using osgDB::Registry::ReadFileCallback that loads the tiles and then modifies PagedLOD that it loads that are higher than a certain resolution. Do you mean that using my own readFileCallback is not sufficient and I'll need to modify PagedLod sources too ? There is no need to modify any OSG sources to adjust the LOD ranges. The ReadFileCallback can do the actual loading of the subgraph and then catch the and modify the LOD ranges. Personally I'd go go using the Camera LODScale though, it's a much simpler tool. Robert. __ Information from ESET NOD32 Antivirus, version of virus signature database 4805 (20100125) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi, This is my stack trace : >nvoglnt.dll!69bf7af3() [Frames below may be incorrect and/or missing, no symbols loaded for nvoglnt.dll] nvoglnt.dll!69bf8f70() nvoglnt.dll!6965767a() nvoglnt.dll!69a410b8() nvoglnt.dll!69a0476b() nvoglnt.dll!69a0480b() nvoglnt.dll!69a3c870() nvoglnt.dll!69b0e0f2() nvoglnt.dll!6959a806() nvoglnt.dll!69ba523c() nvoglnt.dll!69bb9ae9() nvoglnt.dll!695b85c6() nvoglnt.dll!69bb65e1() kernel32.dll!7c80262a() kernel32.dll!7c802600() nvoglnt.dll!69a4166d() nvoglnt.dll!6960999f() nvoglnt.dll!69993d5f() nvoglnt.dll!69a3e499() nvoglnt.dll!69a3eeeb() nvoglnt.dll!69ab67e7() kernel32.dll!7c80b729() I'll try to recreate the bug with datas/code I can share with you. Thanks. Regards, Vincent Robert Osfield a écrit : Hi Vincent, No 3rd party will be able to glean what is going on your machine, with your app, with your data. You will at least have to provide a stack trace. Ideally try and recreate the crash on a standard OSG example with standard data such as the http://www.openscenegraph.osg/data/earth_bayarea/earth.ive database then other will then be to try and recreate the problem first hand. Robert. On Fri, Jan 15, 2010 at 1:56 PM, Vincent Bourdier wrote: Hi, Some more informations I do not understand. Each time I get a crash I have the following log : HANDLE_NON_HTTP: _pager->_requestList.size()= 11 to delete = 0 In DatabasePager thread readNodeFile(Z:\data\cache\s13/V53S9_H.ive) HANDLE_NON_HTTP: _pager->_requestList.size()= 10 to delete = 0 In DatabasePager thread readNodeFile(Z:\data\cache\s23/V57S4_H.ive) HANDLE_NON_HTTP: _pager->_requestList.size()= 9 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 8 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 7 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 6 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 5 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 4 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 3 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 2 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 1 to delete = 0 It finish every time with a request list size decreasing from a value to 0, and crash on requestList.size()= 1. ... I read the code but it it very complicated to understand for me. Does it helps ? are these informations useful ? Thanks. Regards, Vincent. Vincent Bourdier a écrit : Hi Robert, Robert Osfield a écrit : Hi Vincent, On Fri, Jan 8, 2010 at 2:30 PM, Vincent Bourdier wrote: Is there a way to manage memory usage with the load of models ? In my case I have a lot of PagedLod to load, and my question is : is there a way to know is memory is sufficient for the next PageLod higher level node ? I just would like to stop the data loading when memory usage exceed some point. The DatabasePager in svn/trunk + 2.9.x has the ability to cap the number of PagedLOD to a maximum, and automatically recycles PagedLOD children that aren't in the current view when new PagedLOD are required by the current view. What this capping doesn't do is stop the loading of new PagedLOD, so if a single view requires a PagedLOD it will be loaded. To set the cap level programmatically via: viewer.getDatabasePager()->setTargetMaximumNumberOfPageLOD(targetNumOfPaged); Or using env var: set OSG_MAX_PAGEDLOD=200 I don't understand how this is working. If a pagedLod is required it will be loaded... but we can cap the number of lod... This sounds contradictory for me. The number of PagedLod you were speaking about is the PagedLod children or the number of PagedLod node in the scene ? Another feature of svn/trunk + 2.9.x is that texture and buffer object's pools that also provide a maximum cap on the number of texture objects and vbo/pbo/ebo's that can be allocated at any one time. You can set it programmatically via: osg::Texture::getTextureObjectManager(contextID)->setNumberActiveTextureObjects(targetNumBytes); Or using env var: set OSG_TEXTURE_POOL_SIZE=64000 Like the PagedLOD target, the texture pool size is only a target, and if a single frame requires more memory than this it will allocated more memory to prevent any thrashing of memory within a single frame. I did do tests with a strict limit on memory size but when a single frame requires more than this the memory thrashing cut performance down to a fraction of it's normal, for instance going from a solid 60Hz to 1 or 2Hz. There are no textures in my scene... but this is good to know. If you wanted the LOD control then you just implement this yourself using osgDB::Registry::ReadFileCallback that loads the tiles and then modifies PagedLOD that it loads that are higher than a certain resolution.
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Vincent, No 3rd party will be able to glean what is going on your machine, with your app, with your data. You will at least have to provide a stack trace. Ideally try and recreate the crash on a standard OSG example with standard data such as the http://www.openscenegraph.osg/data/earth_bayarea/earth.ive database then other will then be to try and recreate the problem first hand. Robert. On Fri, Jan 15, 2010 at 1:56 PM, Vincent Bourdier wrote: > Hi, > > Some more informations I do not understand. > > Each time I get a crash I have the following log : > > HANDLE_NON_HTTP: _pager->_requestList.size()= 11 to delete = 0 > In DatabasePager thread readNodeFile(Z:\data\cache\s13/V53S9_H.ive) > HANDLE_NON_HTTP: _pager->_requestList.size()= 10 to delete = 0 > In DatabasePager thread readNodeFile(Z:\data\cache\s23/V57S4_H.ive) > HANDLE_NON_HTTP: _pager->_requestList.size()= 9 to delete = 0 > HANDLE_NON_HTTP: _pager->_requestList.size()= 8 to delete = 0 > HANDLE_NON_HTTP: _pager->_requestList.size()= 7 to delete = 0 > HANDLE_NON_HTTP: _pager->_requestList.size()= 6 to delete = 0 > HANDLE_NON_HTTP: _pager->_requestList.size()= 5 to delete = 0 > HANDLE_NON_HTTP: _pager->_requestList.size()= 4 to delete = 0 > HANDLE_NON_HTTP: _pager->_requestList.size()= 3 to delete = 0 > HANDLE_NON_HTTP: _pager->_requestList.size()= 2 to delete = 0 > HANDLE_NON_HTTP: _pager->_requestList.size()= 1 to delete = 0 > > It finish every time with a request list size decreasing from a value to 0, > and crash on requestList.size()= 1. ... > I read the code but it it very complicated to understand for me. > > Does it helps ? are these informations useful ? > > Thanks. > > Regards, > Vincent. > > Vincent Bourdier a écrit : >> >> Hi Robert, >> >> Robert Osfield a écrit : >>> >>> Hi Vincent, >>> >>> On Fri, Jan 8, 2010 at 2:30 PM, Vincent Bourdier >>> wrote: >>> Is there a way to manage memory usage with the load of models ? In my case I have a lot of PagedLod to load, and my question is : is there a way to know is memory is sufficient for the next PageLod higher level node ? I just would like to stop the data loading when memory usage exceed some point. >>> >>> The DatabasePager in svn/trunk + 2.9.x has the ability to cap the >>> number of PagedLOD to a maximum, and automatically recycles PagedLOD >>> children that aren't in the current view when new PagedLOD are >>> required by the current view. What this capping doesn't do is stop >>> the loading of new PagedLOD, so if a single view requires a PagedLOD >>> it will be loaded. To set the cap level programmatically via: >>> >>> >>> viewer.getDatabasePager()->setTargetMaximumNumberOfPageLOD(targetNumOfPaged); >>> >>> Or using env var: >>> >>> set OSG_MAX_PAGEDLOD=200 >>> >>> >> >> I don't understand how this is working. >> If a pagedLod is required it will be loaded... but we can cap the number >> of lod... >> This sounds contradictory for me. >> The number of PagedLod you were speaking about is the PagedLod children or >> the number of PagedLod node in the scene ? >> >>> Another feature of svn/trunk + 2.9.x is that texture and buffer >>> object's pools that also provide a maximum cap on the number of >>> texture objects and vbo/pbo/ebo's that can be allocated at any one >>> time. You can set it programmatically via: >>> >>> >>> osg::Texture::getTextureObjectManager(contextID)->setNumberActiveTextureObjects(targetNumBytes); >>> >>> Or using env var: >>> >>> set OSG_TEXTURE_POOL_SIZE=64000 >>> >>> Like the PagedLOD target, the texture pool size is only a target, and >>> if a single frame requires more memory than this it will allocated >>> more memory to prevent any thrashing of memory within a single frame. >>> I did do tests with a strict limit on memory size but when a single >>> frame requires more than this the memory thrashing cut performance >>> down to a fraction of it's normal, for instance going from a solid >>> 60Hz to 1 or 2Hz. >>> >>> >> >> There are no textures in my scene... but this is good to know. >> >>> If you wanted the LOD control then you just implement this yourself >>> using osgDB::Registry::ReadFileCallback that loads the tiles and then >>> modifies PagedLOD that it loads that are higher than a certain >>> resolution. >>> >> >> Do you mean that using my own readFileCallback is not sufficient and I'll >> need to modify PagedLod sources too ? >> >> >> Thanks. >> >> Regards, >> Vincent. >> >> >> __ Information from ESET NOD32 Antivirus, version of virus >> signature database 4771 (20100114) __ >> >> The message was checked by ESET NOD32 Antivirus. >> >> http://www.eset.com >> >> > > > __ Information from ESET NOD32 Antivirus, version of virus signature > database 4774 (20100115) __ > > The message was checked by ESET NOD32 Antivirus. > > http://www.eset.com > > > ___ > osg-users mailing list > osg-users@lists.openscenegraph.org > http
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Vincent, On Thu, Jan 14, 2010 at 5:50 PM, Vincent Bourdier wrote: > I don't understand how this is working. > If a pagedLod is required it will be loaded... but we can cap the number of > lod... > This sounds contradictory for me. The max number of PagedLOD is a target that it aims for, but if there are more PagedLOD required for rendering the current frame then the max will exceed this trarget, and only once the view changes to require less PagedLOD will the total number come down again. >> If you wanted the LOD control then you just implement this yourself >> using osgDB::Registry::ReadFileCallback that loads the tiles and then >> modifies PagedLOD that it loads that are higher than a certain >> resolution. >> > > Do you mean that using my own readFileCallback is not sufficient and I'll > need to modify PagedLod sources too ? There is no need to modify any OSG sources to adjust the LOD ranges. The ReadFileCallback can do the actual loading of the subgraph and then catch the and modify the LOD ranges. Personally I'd go go using the Camera LODScale though, it's a much simpler tool. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi, Some more informations I do not understand. Each time I get a crash I have the following log : HANDLE_NON_HTTP: _pager->_requestList.size()= 11 to delete = 0 In DatabasePager thread readNodeFile(Z:\data\cache\s13/V53S9_H.ive) HANDLE_NON_HTTP: _pager->_requestList.size()= 10 to delete = 0 In DatabasePager thread readNodeFile(Z:\data\cache\s23/V57S4_H.ive) HANDLE_NON_HTTP: _pager->_requestList.size()= 9 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 8 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 7 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 6 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 5 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 4 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 3 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 2 to delete = 0 HANDLE_NON_HTTP: _pager->_requestList.size()= 1 to delete = 0 It finish every time with a request list size decreasing from a value to 0, and crash on requestList.size()= 1. ... I read the code but it it very complicated to understand for me. Does it helps ? are these informations useful ? Thanks. Regards, Vincent. Vincent Bourdier a écrit : Hi Robert, Robert Osfield a écrit : Hi Vincent, On Fri, Jan 8, 2010 at 2:30 PM, Vincent Bourdier wrote: Is there a way to manage memory usage with the load of models ? In my case I have a lot of PagedLod to load, and my question is : is there a way to know is memory is sufficient for the next PageLod higher level node ? I just would like to stop the data loading when memory usage exceed some point. The DatabasePager in svn/trunk + 2.9.x has the ability to cap the number of PagedLOD to a maximum, and automatically recycles PagedLOD children that aren't in the current view when new PagedLOD are required by the current view. What this capping doesn't do is stop the loading of new PagedLOD, so if a single view requires a PagedLOD it will be loaded. To set the cap level programmatically via: viewer.getDatabasePager()->setTargetMaximumNumberOfPageLOD(targetNumOfPaged); Or using env var: set OSG_MAX_PAGEDLOD=200 I don't understand how this is working. If a pagedLod is required it will be loaded... but we can cap the number of lod... This sounds contradictory for me. The number of PagedLod you were speaking about is the PagedLod children or the number of PagedLod node in the scene ? Another feature of svn/trunk + 2.9.x is that texture and buffer object's pools that also provide a maximum cap on the number of texture objects and vbo/pbo/ebo's that can be allocated at any one time. You can set it programmatically via: osg::Texture::getTextureObjectManager(contextID)->setNumberActiveTextureObjects(targetNumBytes); Or using env var: set OSG_TEXTURE_POOL_SIZE=64000 Like the PagedLOD target, the texture pool size is only a target, and if a single frame requires more memory than this it will allocated more memory to prevent any thrashing of memory within a single frame. I did do tests with a strict limit on memory size but when a single frame requires more than this the memory thrashing cut performance down to a fraction of it's normal, for instance going from a solid 60Hz to 1 or 2Hz. There are no textures in my scene... but this is good to know. If you wanted the LOD control then you just implement this yourself using osgDB::Registry::ReadFileCallback that loads the tiles and then modifies PagedLOD that it loads that are higher than a certain resolution. Do you mean that using my own readFileCallback is not sufficient and I'll need to modify PagedLod sources too ? Thanks. Regards, Vincent. __ Information from ESET NOD32 Antivirus, version of virus signature database 4771 (20100114) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com __ Information from ESET NOD32 Antivirus, version of virus signature database 4774 (20100115) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Robert, Robert Osfield a écrit : Hi Vincent, On Fri, Jan 8, 2010 at 2:30 PM, Vincent Bourdier wrote: Is there a way to manage memory usage with the load of models ? In my case I have a lot of PagedLod to load, and my question is : is there a way to know is memory is sufficient for the next PageLod higher level node ? I just would like to stop the data loading when memory usage exceed some point. The DatabasePager in svn/trunk + 2.9.x has the ability to cap the number of PagedLOD to a maximum, and automatically recycles PagedLOD children that aren't in the current view when new PagedLOD are required by the current view. What this capping doesn't do is stop the loading of new PagedLOD, so if a single view requires a PagedLOD it will be loaded. To set the cap level programmatically via: viewer.getDatabasePager()->setTargetMaximumNumberOfPageLOD(targetNumOfPaged); Or using env var: set OSG_MAX_PAGEDLOD=200 I don't understand how this is working. If a pagedLod is required it will be loaded... but we can cap the number of lod... This sounds contradictory for me. The number of PagedLod you were speaking about is the PagedLod children or the number of PagedLod node in the scene ? Another feature of svn/trunk + 2.9.x is that texture and buffer object's pools that also provide a maximum cap on the number of texture objects and vbo/pbo/ebo's that can be allocated at any one time. You can set it programmatically via: osg::Texture::getTextureObjectManager(contextID)->setNumberActiveTextureObjects(targetNumBytes); Or using env var: set OSG_TEXTURE_POOL_SIZE=64000 Like the PagedLOD target, the texture pool size is only a target, and if a single frame requires more memory than this it will allocated more memory to prevent any thrashing of memory within a single frame. I did do tests with a strict limit on memory size but when a single frame requires more than this the memory thrashing cut performance down to a fraction of it's normal, for instance going from a solid 60Hz to 1 or 2Hz. There are no textures in my scene... but this is good to know. If you wanted the LOD control then you just implement this yourself using osgDB::Registry::ReadFileCallback that loads the tiles and then modifies PagedLOD that it loads that are higher than a certain resolution. Do you mean that using my own readFileCallback is not sufficient and I'll need to modify PagedLod sources too ? Thanks. Regards, Vincent. __ Information from ESET NOD32 Antivirus, version of virus signature database 4771 (20100114) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] PagedLod & dataBasePager memory managment
Hi Vincent, On Fri, Jan 8, 2010 at 2:30 PM, Vincent Bourdier wrote: > Is there a way to manage memory usage with the load of models ? > In my case I have a lot of PagedLod to load, and my question is : is there a > way to know is memory is sufficient for the next PageLod higher level node ? > I just would like to stop the data loading when memory usage exceed some > point. The DatabasePager in svn/trunk + 2.9.x has the ability to cap the number of PagedLOD to a maximum, and automatically recycles PagedLOD children that aren't in the current view when new PagedLOD are required by the current view. What this capping doesn't do is stop the loading of new PagedLOD, so if a single view requires a PagedLOD it will be loaded. To set the cap level programmatically via: viewer.getDatabasePager()->setTargetMaximumNumberOfPageLOD(targetNumOfPaged); Or using env var: set OSG_MAX_PAGEDLOD=200 Another feature of svn/trunk + 2.9.x is that texture and buffer object's pools that also provide a maximum cap on the number of texture objects and vbo/pbo/ebo's that can be allocated at any one time. You can set it programmatically via: osg::Texture::getTextureObjectManager(contextID)->setNumberActiveTextureObjects(targetNumBytes); Or using env var: set OSG_TEXTURE_POOL_SIZE=64000 Like the PagedLOD target, the texture pool size is only a target, and if a single frame requires more memory than this it will allocated more memory to prevent any thrashing of memory within a single frame. I did do tests with a strict limit on memory size but when a single frame requires more than this the memory thrashing cut performance down to a fraction of it's normal, for instance going from a solid 60Hz to 1 or 2Hz. If you wanted the LOD control then you just implement this yourself using osgDB::Registry::ReadFileCallback that loads the tiles and then modifies PagedLOD that it loads that are higher than a certain resolution. Or... just adjust the viewer's Camera::LODScale pre frame to manage the on screen screen complexity. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
[osg-users] PagedLod & dataBasePager memory managment
Hi all, Is there a way to manage memory usage with the load of models ? In my case I have a lot of PagedLod to load, and my question is : is there a way to know is memory is sufficient for the next PageLod higher level node ? I just would like to stop the data loading when memory usage exceed some point. Thanks. Regards, Vincent. __ Information from ESET NOD32 Antivirus, version of virus signature database 4753 (20100108) __ The message was checked by ESET NOD32 Antivirus. http://www.eset.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
