Re: [osg-users] Frame drops when using DB pager and compressed textures
Hi, On Monday 06 February 2012, Christiansen, Brad wrote: I am seeing some strange behaviour with the DatabasePager and compressed textures (compressed using the nvtt image processor) and cant get my head around what is happening. My setup is this: - A paged terrain very similar in nature to a VPB created terrain. - Images and height fields are stored in a database and read at runtime. As the db uses aggressive mem caching, many reads of 'files' are actually read straight from memory so images are loaded very quickly. To demonstrate the problem, I rapidly move the camera around the terrain to generate lots of database requests. If the textures are not compressed (or stored as jpegs etc, so they are not compressed once loaded) then the precompilation works perfectly and I get no frame drops. If I change the textures to use compressed dds images, I get terrible frame drops (down from completely solid 60hz to bellow 20). I figured something was going wrong with the precompiler but after much debugging this seems to be working as designed. If I set the max num of objects to compile to 1 then the problem is reduced, but this is obviously not a solution as it is just masking the problem. The only thing I can think of is that the precompiler isn't actually uploading / compiling the textures when they are compressed, but it is when they are not. As such, the precompilation stage races through lots of textures and then during the draw phase the actual work is done, causing the frame drops. However, I have no idea why this should be the case or how I could address issue. Has anyone come across this before or have any suggestions as to what might be going on? We see very similar things with the nvidia driver in flightgear. In the end it turns out to be the mipmapping code in the nvidia driver. They seem to have a horrible slow implementation of that in their code. People report that if you provide precomputed mipmaps in your dds files, compressed or uncompressed, you do not see any hangs of the rendering code. As far as I can tell you can put the mipmaps also into the dds files by your mentioned nvidia tool. Also not exactly all nvidia chips seem to suffer from that problem. But a wide range works really bad. For osg I think one reliable way to get around that problem would be to precompute the mipmaps on the cpu in the database pager thread. This seems ugly, since the hardware could in theory compute that way faster than we can on the cpu. But the nvidia driver behaves that unpredictible in that area that I consider doing that for flightgear in a image read callback. The open source driver stack really shows how fast computing mipmaps could be using the hardware. The internal implementation sets up an fbo for the texture, sets up a rendertarget of the next higher mipmap level, and draws a quad textured with the current mipmap level into this render target. This is repeated until you have reached the last level. Even if the official open source driver cannot support 3ds compression for patent reasons, you can trick the sourcecode for the amd chips to at least show that this method also performs equally fast for compressed textures. I have not tried the nvidia ones, but I assume that they also have on chip 3ds compression and decompression and could behave the same. And no, I have no clue why this described algorithm is probalby not used for in nvidias binary blob. Greetings Mathias -- Dr. Mathias Fröhlich, science + computing ag, Software Solutions Hagellocher Weg 71-75, D-72070 Tuebingen, Germany Phone: +49 7071 9457-268, Fax: +49 7071 9457-511 -- Vorstand/Board of Management: Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Roland Niemeier, Dr. Arno Steitz, Dr. Ingrid Zech Vorsitzender des Aufsichtsrats/ Chairman of the Supervisory Board: Philippe Miltin Sitz/Registered Office: Tuebingen Registergericht/Registration Court: Stuttgart Registernummer/Commercial Register No.: HRB 382196 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Frame drops when using DB pager and compressed textures
Thanks Mathias, Thanks for the response. Proves I am not going crazy : ) I will try generating mip maps and see if that resolves the issue. Cheers, Brad -Original Message- From: osg-users-boun...@lists.openscenegraph.org [mailto:osg-users-boun...@lists.openscenegraph.org] On Behalf Of Mathias Fröhlich Sent: Monday, 6 February 2012 6:06 PM To: OpenSceneGraph Users Subject: Re: [osg-users] Frame drops when using DB pager and compressed textures Hi, On Monday 06 February 2012, Christiansen, Brad wrote: I am seeing some strange behaviour with the DatabasePager and compressed textures (compressed using the nvtt image processor) and cant get my head around what is happening. My setup is this: - A paged terrain very similar in nature to a VPB created terrain. - Images and height fields are stored in a database and read at runtime. As the db uses aggressive mem caching, many reads of 'files' are actually read straight from memory so images are loaded very quickly. To demonstrate the problem, I rapidly move the camera around the terrain to generate lots of database requests. If the textures are not compressed (or stored as jpegs etc, so they are not compressed once loaded) then the precompilation works perfectly and I get no frame drops. If I change the textures to use compressed dds images, I get terrible frame drops (down from completely solid 60hz to bellow 20). I figured something was going wrong with the precompiler but after much debugging this seems to be working as designed. If I set the max num of objects to compile to 1 then the problem is reduced, but this is obviously not a solution as it is just masking the problem. The only thing I can think of is that the precompiler isn't actually uploading / compiling the textures when they are compressed, but it is when they are not. As such, the precompilation stage races through lots of textures and then during the draw phase the actual work is done, causing the frame drops. However, I have no idea why this should be the case or how I could address issue. Has anyone come across this before or have any suggestions as to what might be going on? We see very similar things with the nvidia driver in flightgear. In the end it turns out to be the mipmapping code in the nvidia driver. They seem to have a horrible slow implementation of that in their code. People report that if you provide precomputed mipmaps in your dds files, compressed or uncompressed, you do not see any hangs of the rendering code. As far as I can tell you can put the mipmaps also into the dds files by your mentioned nvidia tool. Also not exactly all nvidia chips seem to suffer from that problem. But a wide range works really bad. For osg I think one reliable way to get around that problem would be to precompute the mipmaps on the cpu in the database pager thread. This seems ugly, since the hardware could in theory compute that way faster than we can on the cpu. But the nvidia driver behaves that unpredictible in that area that I consider doing that for flightgear in a image read callback. The open source driver stack really shows how fast computing mipmaps could be using the hardware. The internal implementation sets up an fbo for the texture, sets up a rendertarget of the next higher mipmap level, and draws a quad textured with the current mipmap level into this render target. This is repeated until you have reached the last level. Even if the official open source driver cannot support 3ds compression for patent reasons, you can trick the sourcecode for the amd chips to at least show that this method also performs equally fast for compressed textures. I have not tried the nvidia ones, but I assume that they also have on chip 3ds compression and decompression and could behave the same. And no, I have no clue why this described algorithm is probalby not used for in nvidias binary blob. Greetings Mathias -- Dr. Mathias Fröhlich, science + computing ag, Software Solutions Hagellocher Weg 71-75, D-72070 Tuebingen, Germany Phone: +49 7071 9457-268, Fax: +49 7071 9457-511 -- Vorstand/Board of Management: Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Roland Niemeier, Dr. Arno Steitz, Dr. Ingrid Zech Vorsitzender des Aufsichtsrats/ Chairman of the Supervisory Board: Philippe Miltin Sitz/Registered Office: Tuebingen Registergericht/Registration Court: Stuttgart Registernummer/Commercial Register No.: HRB 382196 ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org - DISCLAIMER: This e-mail transmission and any documents, files and previous e-mail messages attached to it are private and confidential. They may contain proprietary or copyright material or information that is subject to legal professional privilege
Re: [osg-users] Frame drops when using DB pager and compressed textures
Hi, Just a follow up for anyone hitting this issue. I enabled the creation of mipmaps in my dds files and the issue vanished. Thanks so much for the tip Mathias. Luckly, thanks to the wonderful OSG API, making the change was as simple as changing a 'false' to a 'true' in my call to he image processor. Cheers, Brad - DISCLAIMER: This e-mail transmission and any documents, files and previous e-mail messages attached to it are private and confidential. They may contain proprietary or copyright material or information that is subject to legal professional privilege. They are for the use of the intended recipient only. Any unauthorised viewing, use, disclosure, copying, alteration, storage or distribution of, or reliance on, this message is strictly prohibited. No part may be reproduced, adapted or transmitted without the written permission of the owner. If you have received this transmission in error, or are not an authorised recipient, please immediately notify the sender by return email, delete this message and all copies from your e-mail system, and destroy any printed copies. Receipt by anyone other than the intended recipient should not be deemed a waiver of any privilege or protection. Thales Australia does not warrant or represent that this e-mail or any documents, files and previous e-mail messages attached are error or virus free. - ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
[osg-users] Frame drops when using DB pager and compressed textures
Hi, I am seeing some strange behaviour with the DatabasePager and compressed textures (compressed using the nvtt image processor) and cant get my head around what is happening. My setup is this: - A paged terrain very similar in nature to a VPB created terrain. - Images and height fields are stored in a database and read at runtime. As the db uses aggressive mem caching, many reads of 'files' are actually read straight from memory so images are loaded very quickly. To demonstrate the problem, I rapidly move the camera around the terrain to generate lots of database requests. If the textures are not compressed (or stored as jpegs etc, so they are not compressed once loaded) then the precompilation works perfectly and I get no frame drops. If I change the textures to use compressed dds images, I get terrible frame drops (down from completely solid 60hz to bellow 20). I figured something was going wrong with the precompiler but after much debugging this seems to be working as designed. If I set the max num of objects to compile to 1 then the problem is reduced, but this is obviously not a solution as it is just masking the problem. The only thing I can think of is that the precompiler isn't actually uploading / compiling the textures when they are compressed, but it is when they are not. As such, the precompilation stage races through lots of textures and then during the draw phase the actual work is done, causing the frame drops. However, I have no idea why this should be the case or how I could address issue. Has anyone come across this before or have any suggestions as to what might be going on? Cheers, Brad - DISCLAIMER: This e-mail transmission and any documents, files and previous e-mail messages attached to it are private and confidential. They may contain proprietary or copyright material or information that is subject to legal professional privilege. They are for the use of the intended recipient only. Any unauthorised viewing, use, disclosure, copying, alteration, storage or distribution of, or reliance on, this message is strictly prohibited. No part may be reproduced, adapted or transmitted without the written permission of the owner. If you have received this transmission in error, or are not an authorised recipient, please immediately notify the sender by return email, delete this message and all copies from your e-mail system, and destroy any printed copies. Receipt by anyone other than the intended recipient should not be deemed a waiver of any privilege or protection. Thales Australia does not warrant or represent that this e-mail or any documents, files and previous e-mail messages attached are error or virus free. - ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
