Re: [osg-users] Refactoring DatabasePagerNeedToRemovestringflagging technique
HI Wojtek, On Mon, Nov 23, 2009 at 5:22 PM, Wojciech Lewandowski lewandow...@ai.com.pl wrote: Our case is following: We mostly have to update camera position and some objects around camera. And ocassionaly we reposition a camera to new location in the world. We are doing our intersections in update traversal. I assume its allowed to use IntersectVisitor in update and load new tiles then. We load many objects in update traversal. The crucial part you don't talk about is the performance constraints. Can you pause a frame for hundreds of milliseseconds while the required high res tiles are pulled in off disk? Or are you trying to stick with a solid 60Hz? Loading any tiles in the update phase is not possible if you want to keep to a solid 60Hz. And believe it or not but our scheme was working. Cache kept nodes bit longer than DatabasePager wanted, but not used tiles were eventually freed. I know it did work, because we made quick dirty fix by renaming NeedToRemove nodes to empty string name when fetching PagedLODs back to activePageLOD list from cache (described in Pawel Ksiezopolski post I mentioned earlier). Unfortuantely this fix was not appropriate as elegant submission. Sharing loaded PageLOD tiles through cache was actually working as some preload for DatabasePager. DatabasePager still could load whatever it wanted. I think something that might provide a work around such as replacing theNeedToRemove is just a work around it's very unlikely to be a solution of trying to use the DatabasePager beyond it's current design assumptions. Using a cache to prevent a subgraph from being deleted defeats the load balancing that the DatabasePager will be attempting to do, so it's a dangerous thing to do - it's a recipe for relentless growth in memory usage. Well yes, but in our case PagedLOD usage timeframe should be extended to period when the tiles were used for intersections and cache maybe in bit hacky way provided this prolonged life time. And use of cache is not direct cause of leaks becaus osgDB cache is freed when node ref_count reaches zero so when DatabasePager IntersectionVisitor stopped using the tile it was effectively removed from the cache as well. Memory leaks were the result of extra NeedToRemove nodes that appeared in activePageLOD lists when some nodes were resurected from Cache because camera moved over them again. I know its sophistry, but in some way, it was more effective than native DatabasePager management because it loaded nodes in no time while DatabasePager would have to load them from disk in this case ;-). I can't understand why you want to do what you are trying to do, but it still doesn't change the fact that DatabasePager hasn't been designed with this in mind. We can certainly look to take this into account in another rev of DatabasePager. As a general note a good OS will actually cache file accesses, especially if you don't overload virtual memory/swap space too much so the cost of reloading a tile might not be quite as expensive as you think... I know that when I benchmark under Linux I have the file cache makes a huge difference, so I can to be careful about running test apps in sequence. Well I thought that if PagedLOD class is public and offers public interface and methods we could use it as long as we do not hit compiler errors. Perhaps different set of classes should be created for use in more VPB specific closed way. I don't think this is a VPB database specific issue. The problem you are trying to solve is really to do with trying to juggle user manage caching with how the DatabasePager manages expiry. When I get on to reviewing the multiple viewpoint issue with DatabasePager I'll have a think about the consequences of users caching subgraphs Robert. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Refactoring DatabasePagerNeedToRemovestringflagging technique
Hi Robert, How should we tackle it, if current approach is wrong ? [..] If you do want to mix the two then you need to ask questions about how you want to do intersections and how your thread them. Do you do intersections in the main loop? In a separate thread? Do you run the intersections multi-threaded? I.e. multiple intersections traversals at one time? Should the intersection traversal wait to load external tiles to get the result, or should then just return the intersections for what is already loaded into memory. What approach you will want to use will depend upon all of this. Our case is following: We mostly have to update camera position and some objects around camera. And ocassionaly we reposition a camera to new location in the world. We are doing our intersections in update traversal. I assume its allowed to use IntersectVisitor in update and load new tiles then. We load many objects in update traversal. And believe it or not but our scheme was working. Cache kept nodes bit longer than DatabasePager wanted, but not used tiles were eventually freed. I know it did work, because we made quick dirty fix by renaming NeedToRemove nodes to empty string name when fetching PagedLODs back to activePageLOD list from cache (described in Pawel Ksiezopolski post I mentioned earlier). Unfortuantely this fix was not appropriate as elegant submission. Sharing loaded PageLOD tiles through cache was actually working as some preload for DatabasePager. DatabasePager still could load whatever it wanted. Second important issue for me is usage of _name in scene graph. I always expected that _name is reserved for users and its a normal rule in all Scene Graph implementations that libraries do not change it. Names are ususlly used to identify certain portions of models and hook up the code properly. Thats something that provide standard linking mechanisms between artists and programmers works. I agree, but... in this instance the DatabasePager's algorithm was about deleting a subgraph that would no longer have any role to play in the applications life so the changing of name should never have got outside that algorithm as the subgraph would be just deleted. So it is in theory just a black box, how it does it's job shouldn't effect anything else. Alas in this case it looks like the algorithm in DatabasePager is flawed. Please also note that we use cache becuse we otherwise were loading PageLOD files twice. Is it reasonable ?. Using a cache to prevent a subgraph from being deleted defeats the load balancing that the DatabasePager will be attempting to do, so it's a dangerous thing to do - it's a recipe for relentless growth in memory usage. Well yes, but in our case PagedLOD usage timeframe should be extended to period when the tiles were used for intersections and cache maybe in bit hacky way provided this prolonged life time. And use of cache is not direct cause of leaks becaus osgDB cache is freed when node ref_count reaches zero so when DatabasePager IntersectionVisitor stopped using the tile it was effectively removed from the cache as well. Memory leaks were the result of extra NeedToRemove nodes that appeared in activePageLOD lists when some nodes were resurected from Cache because camera moved over them again. I know its sophistry, but in some way, it was more effective than native DatabasePager management because it loaded nodes in no time while DatabasePager would have to load them from disk in this case ;-). Even if we skip intersections, other situations are also possible. For example we may have few highly detailed special PageLOD tiles with ariports which we want to preload and keep in memory for whole application runtime. So we modify readFileCallback to work for such cache and return these preloaded models each time thery are requested. Use of the cache in conjunction with a paged database should be used very sparingly and for only very specific types of assets. It does also open the question of how DatabasePager should deal with such datasets, without lots of reflection on the issue I can't say. I can say in the design and development of DatabasePager I have made the assumption that it'd be the master of the PagedLOD's and manage all reading and expiring, and not have code on the outside managing things in a parallel. Well I thought that if PagedLOD class is public and offers public interface and methods we could use it as long as we do not hit compiler errors. Perhaps different set of classes should be created for use in more VPB specific closed way. It's worth noting that PagedLOD has settings that allow you to control what happens with expiry - so you can individually switch off the expiry. I have not thought about it. I will have to learn what we can do using this mechanism, perhaps we could do something smarter. Wojtek ___ osg-users mailing list
Re: [osg-users] Refactoring DatabasePagerNeedToRemovestringflagging technique
Thanks Glenn. I will look at this. Wojtek - Original Message - From: Glenn Waldron To: OpenSceneGraph Users Sent: Monday, November 23, 2009 5:41 PM Subject: Re: [osg-users] Refactoring DatabasePagerNeedToRemovestringflagging technique On Mon, Nov 23, 2009 at 11:10 AM, Robert Osfield robert.osfi...@gmail.com wrote: Hi Wojtek, On Mon, Nov 23, 2009 at 2:34 PM, Wojciech Lewandowski lewandow...@ai.com.pl wrote: But I would strongly defend merits of arguments in the post. You say we did wrong, but whats you recommendation on mixing many intersections with rendering ? Its very common scenario. How should we tackle it, if current approach is wrong ? The right approach is a difficult one. Getting a paged scene graph to work with intersections at highest resolutions and at the same time manage things for rendering with requires just the appropriate LOD child for the needs of visuals is awkward. I know often vis-sim apps don't even try to mix the two, and have a separate process entirely for dealing intersections as for doing the visuals. Some sims even run the visuals and intersection testing on entirely different machines. Other sims use entirely separate databases for intersection testing and visuals. Then there are others that use a height field for height above terrain testing... Wojtek, The keep it separate approach is what we use in osgEarth. The idea is to fetch terrain tiles directly, based on your target sampling resolution, instead of traversing the whole LOD hierarchy. Take a took at the ElevationManager utility. Perhaps it can provide some inspiration: http://wush.net/trac/osgearth/browser/trunk/src/osgEarthUtil/ElevationManager Glenn -- ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
