Re: [Flightgear-devel] Memory issues
On Mon, Sep 3, 2012 at 11:58 PM, Stuart Buchanan wrote: I've (finally) managed to get a prototype running using an instantiation of a shared geometry rather than a huge single object per tile. So far the results are promising. My standard test is at KSFO with the c172p, waiting until the intial set of tiles is loaded with standard weather conditions (e.g. no excessive visibility). With random buildings switched off FG uses ~1GB memory. With v2.8.0 random buildings, it's 2GB. With instantiated buildings at present it is 1.5GB. At KLAX, it's down even further, from 2.7GB to 1.6GB. I've not noticed any frame-rate impact. There's still quite a lot of work to do, but I hope to have this available before the next release. This work is now checked into next/master. You'll need an updated simgear and data to see any effect. Basically, I've moved to using a scheme very similar to that of the trees, where there are a small number of buildings created, and then instantiated multiple times. The technique relies on a vertex shader, as the final position and rotation of the building are passed in on the gl_Color vector. This in turn has meant that I've been forced to create new version of various vertex shaders. It might be possible to clean that up by having a new parameter to existing shaders indicating that position/rotation information is in gl_Color, but I haven't investigated that option in detail. There are three known regressions from the previous implementation: - The reflection map appears to be broken, and is currently switched off. I suspect this might be because the binormal or tangent are not being calculated correctly. - The buidings do not cast shadows in Rembrandt, though other objects _do_ cast shadows on the buildings. I don't know the reason for this. - At high building densities and certain angles of view, one can see gaps in the building placement where the edges of the scenery triangles go. This is unfortunately inevitable, as we have less sizing information during placement. If anyone with shader/effects knowledge has the time to look at the first two of these problems, that would be wonderful. Feedback/comments appreciated as always. -Stuart PS: With a building density of 4 at KSFO I have 1.6GB peak occupancy. That compares to 3GB previously. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Memory issues
On Thu, Aug 9, 2012 at 10:46 AM, Stuart Buchanan wrote: For performance reasons, all the buildings within a tile are a single object. The memory capacity could be addressed by changing the implementation to something closer to the trees, where a (relatively) small number of buildings are instantiated at different locations. I don't know what this would do for performance though. I've (finally) managed to get a prototype running using an instantiation of a shared geometry rather than a huge single object per tile. So far the results are promising. My standard test is at KSFO with the c172p, waiting until the intial set of tiles is loaded with standard weather conditions (e.g. no excessive visibility). With random buildings switched off FG uses ~1GB memory. With v2.8.0 random buildings, it's 2GB. With instantiated buildings at present it is 1.5GB. At KLAX, it's down even further, from 2.7GB to 1.6GB. I've not noticed any frame-rate impact. There's still quite a lot of work to do, but I hope to have this available before the next release. -Stuart -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Memory issues
On Mon, Aug 13, 2012 at 7:40 PM, James Turner zakal...@mac.com wrote: On 11 Aug 2012, at 18:21, Tim Moore wrote: This is what osg::PagedLOD does, though we often forget that the paging of the higher LODs is triggered in the cull phase. I can't recall what scene modifications are / are not permitted inside a cull-callbacl, however. You would want to let the OSG loader mechanism and the database pager do their thing. Geometry doesn't have to be loaded from files... Could you outline the pieces of machinery needed for this to work? Given that we already create LOD nodes, I assume it's switching those to be PagedLOD, and setting the filename / extension / reader-writerOptions to some magic, and creating a loader which matches that, which creates the buildings geometry. That is how PagedLODs work, without too much magic. Some data will need to be passed to the loader. A low-tech way to do that is to encode parameters in the file name passed to the loader, but in this case you will probably need access to the scenery to place buildings. You can subclass the DatabaseOptions object stored in the PagedLOD to store whatever you need. You would choose a file extension that doesn't correspond to a real file type, and then register a loader with that extension. If you could point to an example of such a loader, I'm pretty sure Stuart or I could adapt his code. James We register loading callbacks using both osgDB mechanisms and some SimGear goo to control optimizations, construction bvh trees, etc. grep for ModelRegistryCallbackProxy. For the body of the loader, there is nothing special; just don't read any files. Tim -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Memory issues
On 11 Aug 2012, at 18:21, Tim Moore wrote: This is what osg::PagedLOD does, though we often forget that the paging of the higher LODs is triggered in the cull phase. I can't recall what scene modifications are / are not permitted inside a cull-callbacl, however. You would want to let the OSG loader mechanism and the database pager do their thing. Geometry doesn't have to be loaded from files... Could you outline the pieces of machinery needed for this to work? Given that we already create LOD nodes, I assume it's switching those to be PagedLOD, and setting the filename / extension / reader-writerOptions to some magic, and creating a loader which matches that, which creates the buildings geometry. If you could point to an example of such a loader, I'm pretty sure Stuart or I could adapt his code. James -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Memory issues
On Fri, Aug 10, 2012 at 12:50 PM, James Turner zakal...@mac.com wrote: On 9 Aug 2012, at 21:49, Stuart Buchanan wrote: Do we have any ideas addressing these issues? Random thought, hopefully someone with more OSG knowledge can confirm or deny: Can we create the geometry (and unload it) in the cull-callback? I.e set an appropriate bounding volume for the LOD node, and only build the geometry when the node actually passes culling. And obviously after some number of frames of being culled, unload the geometry again. This is what osg::PagedLOD does, though we often forget that the paging of the higher LODs is triggered in the cull phase. I can't recall what scene modifications are / are not permitted inside a cull-callbacl, however. You would want to let the OSG loader mechanism and the database pager do their thing. Geometry doesn't have to be loaded from files... Tim -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Memory issues
On 9 Aug 2012, at 21:49, Stuart Buchanan wrote: Do we have any ideas addressing these issues? Random thought, hopefully someone with more OSG knowledge can confirm or deny: Can we create the geometry (and unload it) in the cull-callback? I.e set an appropriate bounding volume for the LOD node, and only build the geometry when the node actually passes culling. And obviously after some number of frames of being culled, unload the geometry again. I can't recall what scene modifications are / are not permitted inside a cull-callbacl, however. James -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
[Flightgear-devel] Memory issues
Of late, I have increasingly been running into some segfaults which seem to have to do with peak memory usage (I don't think memory leaks are a major issue for what I've been seeing, since some segfauls happened quite early on, and especially when I try pushing my limits of graphical goodies). Memory also appears to be an issue for other users, judging by forum response. For me, running into swap space means basically either immediate or imminent crash. Sometimes I get a 30 seconds grace with framerates below 5 fps, sometimes Flightgear dies immediately. Apparently other binaries (the 64 bit Jenkins) seem to be a bit more tolerant - see for instance user experiences here: http://www.flightgear.org/forums/viewtopic.php?f=68t=17114 None of this is a problem on a fundamental level for me personally - I largely know my 'safe' limits for visibility (250 km for default scenery and bare terrain on short suborbital hops, 120 km for long-distance flights in bare default scenery, 60 km in bare custom scenery or default scenery with buildings, 40 km in custum scenery with buildings, subtract another 10% for trees...), so I check a range of options I know by experience to be safe before the flight and am not usually troubled by crashes unless I try something new. But I know this stuff because I do a lot of benchmark testing, am active on this list and have some understanding of the inner workings of Flightgear. But there's nothing to inform the casual user what he should do. In fact, by chance the combination of options we offer is quite dangerous. * before the new edition of random buildings came out, I've played a lot with opening up visibility at high altitude. Even for a ground visibility of less than 8 km, you'd get 120 km at airliner cruise altitude if 'realistic visibility' is checked and the visibility limit shifted to max. in Advanced Weather * at the same time, random buildings (even in a rather unpopulated area like the Alps and with a density around 0.8) account for half of my memory consumption (I tested 1.0 GB for loading a 40 km radius without buildings against 1.9 GB for loading the same scene with buildings). That's before trees and/or higher densities. Hitting a major urban area changes the picture drastically. The combination of the two means instant death at high altitude, even when all you do on the ground looks sane - you can see a nice collection of random buildings nearby, have a safe visibility range, good momory consumption - and once you are up, the tile manager starts loading a city 120 km away you don't even see yet and populates it with buildings, memory consumption explodes and you are dead. Both options, random buildings and a realistic view range, are in my view extremely cool as long as you know what you are doing with them, and I don't really want to abandon either. But I suspect their combination (combined with other memory-hungry goodies) is not something we want to confront the casual user with. Do we have any ideas addressing these issues? For instance, should I agressively limit the maximally available visibility range in Advanced Weather if random buildings and trees are selected? Is it technically feasible to let Flightgear probe the remaining physical memory and if we run into trouble initiate an agressive unloading of terrain, e.g. by forcing visibility down (since most memory-hungry stuff scales with the square of the visibility range, I think adjusting that is most effective)? I know this is not realistic, but I'd rather have sudden haze than sudden segfault? Should we code a few warnings into the GUI? And other ideas? Or is this not an issue for the majority? * Thorsten -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Memory issues
On Thu, Aug 9, 2012 at 10:06 AM, Renk Thorsten wrote: Both options, random buildings and a realistic view range, are in my view extremely cool as long as you know what you are doing with them, and I don't really want to abandon either. But I suspect their combination (combined with other memory-hungry goodies) is not something we want to confront the casual user with. Do we have any ideas addressing these issues? Random buildings are switched off by default, but as you suggest, a warning in the GUI would be appropriate. I will do this. Unfortunately, the buildings are generated when the tiles are loaded, and it is this that causes the memory occupancy problems rather than visibility itself (subject to big view ranges causing the loading of more tiles). Aggressively limiting the maximum available visibility range might help, if it's causing more tiles to be loaded, but it is very dependent on the amount of Urban and Town terrain. For performance reasons, all the buildings within a tile are a single object. The memory capacity could be addressed by changing the implementation to something closer to the trees, where a (relatively) small number of buildings are instantiated at different locations. I don't know what this would do for performance though. I believe James and Matthias also had a plan to use some OpenGL/Shader technque to pass location information for instantiation, but I don't know the details. And other ideas? Or is this not an issue for the majority? I think it's a major issue, and one that I did not consider sufficiently in development. Sorry. -Stuart -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] Memory issues
On Thu, Aug 9, 2012 at 10:46 AM, Stuart Buchanan wrote: On Thu, Aug 9, 2012 at 10:06 AM, Renk Thorsten wrote: Both options, random buildings and a realistic view range, are in my view extremely cool as long as you know what you are doing with them, and I don't really want to abandon either. But I suspect their combination (combined with other memory-hungry goodies) is not something we want to confront the casual user with. Do we have any ideas addressing these issues? Random buildings are switched off by default, but as you suggest, a warning in the GUI would be appropriate. I will do this. This has now been done - commit ef93e36 Could someone please cherry-pick this to the release branch. Thanks. -Stuart -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
