Re: [osg-users] Question on decreasing latency for database pagers...
Hi, I played with a variety of settings like varying LOD in the viewer, setting bound volume, range during the LOD building process, but is still unable to achieve my goal. In short, I wanted to aggressive load in new tiles with the DatabasePager. In this case, I wanted to pre-load the areas surrounding the area drawn in the view-port. Therefore, when the next frame is drawn, these areas will have already been loaded. Can somebody guide me in which part of OSG should I customise? Thanks in advance. Kind Regards, Boon -- Read this topic online here: http://forum.openscenegraph.org/viewtopic.php?p=42533#42533 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi Boon, On Thu, Sep 8, 2011 at 2:32 PM, Boon Wah boon...@gmail.com wrote: I played with a variety of settings like varying LOD in the viewer, setting bound volume, range during the LOD building process, but is still unable to achieve my goal. In short, I wanted to aggressive load in new tiles with the DatabasePager. In this case, I wanted to pre-load the areas surrounding the area drawn in the view-port. Therefore, when the next frame is drawn, these areas will have already been loaded. Can somebody guide me in which part of OSG should I customise? I'm afraid you have unrealistic expectations of just how fast your hardware can page data in. It won't be a simple case of customizing the OSG, you'll need to address the hardware issues that are at the core of the latency issue - you have to either put your tiles into ram or second best a solid state disk. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi Boon, On Wed, Sep 7, 2011 at 7:04 PM, Boon Wah boon...@gmail.com wrote: After some thought, I want to try to make the paging become more aggressive and load tiles further away from those in view. As my fly-pass is very fast, the paging mechanism must also 'cache' new tiles earlier. I am unfamiliar with OSG and require some help in doing so. Does modifying the setRange for the PagedLOD object help in achieving my objective? You can use Camera::setLODScale() to adjust the range used for PagedLOD child selection, this can be done at the viewer level rather than at the individual node level so it much more manageable . However, selecting high res children will result in a higher load on the CPU and GPU making it more likely that you'll drop frames. If you want to load tiles in quickly then best way is to use a solid state disk. You can also up the number of threads that are doing the database paging, see the DatabasePager API do details. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi Boon Wah, Loading shouldn't cause frame drops as the loading is done in back ground thread. Downloading the objects to graphics card is much more likely to cause frame drops, but for most VPB database on modern hardware this usually isn't an issue. From the sound of it you are setting the tile size to a very high value, you don't actually say whether you mean imagery or height field size, if it's imagery then it way too high, if your are referring to the height field then it's ridiculously high, and in both cases it's high enough to cause frame drops when downloading the data to GPU. Is there are reason why you aren't using VPB defaults for tile size? The defaults 256x256 for imagery, and 64x64 for the height field, and are chosen to provide a well balanced scene graph that loads and renders efficiently without framedrops. Robert. On Tue, Sep 6, 2011 at 3:54 AM, Boon Wah boon...@gmail.com wrote: Hi Robert, I will like to clarify on my question. I may not know the correct technical terms, but probably you are right, I am trying to minimize the frame drop rate. The scenario is like this: I have pre-compiled the data using VPB setting a tile size of 2048 by 2048. The entire 'terrain' image is over 1GB and this produces about 400+ tiles at the finest resolution. During rendering, I need to do a fly-pass at the finest resolution and maintaining a frame-rate of 60fps. I realise that the loading time for each tile is quite significant and this is causing a frame drop. A quick check is that I did two identical fly-pass consecutively and found that once the data has been loaded during the first fly-pass, there will not be frame drop for the second fly-pass. This makes me think of trying to pre-load all the tiles into the pager prior to commencing my fly-pass. What I wanted to know is how can I do this easily? I am looking at the direction of building a customiser pager to pre-load all the tiles (of course limited by my memory). Kind Regards, Boon Wah -- Read this topic online here: http://forum.openscenegraph.org/viewtopic.php?p=42435#42435 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi Robert, In VPB, I have set my tile-image-size to 2048 and tile-terrain-size to be 2048. The rationale is for me to have larger tiles so as to minimize the number of files. My development platform is Windows and it is pretty poor at handle large number of files. Is it possible to modify the pager such that it caches all my tiles prior to performing rendering? I understand it defeats the purpose of paging, but in this way the loading time of the tiles can be controlled. I hope somebody can guide me in doing this. Thanks for your help. Kind Regards, Boon Wah -- Read this topic online here: http://forum.openscenegraph.org/viewtopic.php?p=42443#42443 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi Jimmy, I also use VPB and osgTerrain with 500GB+ databases. Genereally you are right, NTFS isn't the best database for handling folders with lots of files. But I think you mix up two different aspects: In my opinion the handling capability of large file numbers is another topic than loading performance. My experience is: - NTFS can handle large numbers of files - Windows explorer has problems to move/copy so many files, the file management ist poor. Once the files are finally written to disk it's fine. To your original problem: I think you optimize your system too hard along that one parameter: the reduction of files to load. The small tiles are not default settings for fun: with large tiles you thwart the LOD algorithm an finally have much more vertices to render than with small tiles and good LOD'ing. As Robert said, 2000 x 2000 heighfield tiles is way to large, it makes 400 vertices per tile! The loading of you tiles shouldnt affect your rendering frame rate. As Robert said, OSG uses several seperate threads only for loading files from disk and preparing them to be rendered. If you order a tile and this tile is loaded to late (by whatever reason), it will be loaded simple in the next frame, but the rendering thread would not wait for it to be loaded. After the pager has loaded the tile and passes it to the rendering thread to render it, it has to deal with several potential performance bottle necks: - The first time the tile should be rendered, the geometryTechnique has to create a renderable geometry (triangle mesh) of the heightfield. This seperation of data storage in the tiles and data interpretation during rendering introduces a great flexibility but also some calculation needs which can not be preponed into the VPB processing stage. - Another bootleneck could be the grafic hardware. As robert said on some point in the rendering stage the data has to be transfered into the grafics hardware to be rendered, because the data is loaded but still in CPU and not GPU space. Unlikely, but that may be a booteneck too. If you fly and you have to load a lot of new tiles, one of the the bottlenecks may cause your lag. Remember, every calculation regarding a single tile is way more performance comsumptive if your tiles are so large. I think OSG provides a mechanism to control how many data is compiled in each frame. Also here using smaller tiles allows OSG a much finer grained control. If I'm wrong in some thoughts, please correct me :) Thank you! Cheers, Torben -- Read this topic online here: http://forum.openscenegraph.org/viewtopic.php?p=42444#42444 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi Boon Wah, On Tue, Sep 6, 2011 at 12:25 PM, Boon Wah boon...@gmail.com wrote: Hi Robert, In VPB, I have set my tile-image-size to 2048 and tile-terrain-size to be 2048. The rationale is for me to have larger tiles so as to minimize the number of files. My development platform is Windows and it is pretty poor at handle large number of files. Is it possible to modify the pager such that it caches all my tiles prior to performing rendering? I understand it defeats the purpose of paging, but in this way the loading time of the tiles can be controlled. I hope somebody can guide me in doing this. The problem is your settings of tile-image-size and tile-terrain-size, please discard these and use the defaults. Having a large number of tiles is normal and shouldn't cause any performance issues even under Windows. VPB/OSG users are using VPB generated databases that are multi-terrabyte in size with colossal numbers of tile, all running a solid 60Hz under windows. As for caching all your tiles prior to rendering, this rather defeats the purpose of doing paging, it's designed to be on-demand. The DatabasePager does have a cache which is populated as your moving around the database. Again leaving this in it's default setting is normally OK for most users. Robert. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi, for forcing load from disk to CPU mem you can check what osgposter or osgSim HeightAboveTerrain does. This does not however force tiles to GPU memory. jp On 06/09/2011 13:25, Boon Wah wrote: Hi Robert, In VPB, I have set my tile-image-size to 2048 and tile-terrain-size to be 2048. The rationale is for me to have larger tiles so as to minimize the number of files. My development platform is Windows and it is pretty poor at handle large number of files. Is it possible to modify the pager such that it caches all my tiles prior to performing rendering? I understand it defeats the purpose of paging, but in this way the loading time of the tiles can be controlled. I hope somebody can guide me in doing this. Thanks for your help. Kind Regards, Boon Wah -- Read this topic online here: http://forum.openscenegraph.org/viewtopic.php?p=42443#42443 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org -- This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard. The full disclaimer details can be found at http://www.csir.co.za/disclaimer.html. This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi all, Thanks for all of your replies. I will re-look and see how best to optimise my code. Thanks again! Kind Regards, Boon Wah -- Read this topic online here: http://forum.openscenegraph.org/viewtopic.php?p=42457#42457 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
[osg-users] Question on decreasing latency for database pagers...
I will like to ask for some help on decreasing the latency for database pagers. I have basically a fly-pass scene, with the terrain generated by VPB. I require every frame to be generated within a 30ms timing. I note that there are a portion of frames (about 15%, well-spread across) that misses this 30ms timing. This could be probably due to database reloading. I will like to try to minimise this. My IVE are generally set to be 2k tile and there are a few hundred tiles for my terrain data. Can somebody give me an idea on where to change to accomplish this? Thanks. -- Read this topic online here: http://forum.openscenegraph.org/viewtopic.php?p=42416#42416 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
HI Boo? Wah? Please sign your posts so we known how best to address you, thanks. On Mon, Sep 5, 2011 at 8:53 AM, Boon Wah boon...@gmail.com wrote: I will like to ask for some help on decreasing the latency for database pagers. I have basically a fly-pass scene, with the terrain generated by VPB. I require every frame to be generated within a 30ms timing. I note that there are a portion of frames (about 15%, well-spread across) that misses this 30ms timing. This could be probably due to database reloading. I will like to try to minimise this. My IVE are generally set to be 2k tile and there are a few hundred tiles for my terrain data. Can somebody give me an idea on where to change to accomplish this? Thanks. It doesn't sound like you are talking about straight latency but frame rate. Is is that your application is dropping frames? Is this problem? When you say 2k tile, what is 2k? Image size, height field size? Or do you mean 2000 tiles? Or 2000x2000 tiles? Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Question on decreasing latency for database pagers...
Hi Robert, I will like to clarify on my question. I may not know the correct technical terms, but probably you are right, I am trying to minimize the frame drop rate. The scenario is like this: I have pre-compiled the data using VPB setting a tile size of 2048 by 2048. The entire 'terrain' image is over 1GB and this produces about 400+ tiles at the finest resolution. During rendering, I need to do a fly-pass at the finest resolution and maintaining a frame-rate of 60fps. I realise that the loading time for each tile is quite significant and this is causing a frame drop. A quick check is that I did two identical fly-pass consecutively and found that once the data has been loaded during the first fly-pass, there will not be frame drop for the second fly-pass. This makes me think of trying to pre-load all the tiles into the pager prior to commencing my fly-pass. What I wanted to know is how can I do this easily? I am looking at the direction of building a customiser pager to pre-load all the tiles (of course limited by my memory). Kind Regards, Boon Wah -- Read this topic online here: http://forum.openscenegraph.org/viewtopic.php?p=42435#42435 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
