Thorsten > I did a bit of testing of my own yesterday, and I have made a few other > observations as well. > > User feedback: > > I've largely come to ignore that (with few exceptions such as Vivian's > performance table), because trying to make sense of it is a path into > madness. Just a few examples: On my own system, Advanced Weather > tends to lead to better framerates than Basic Weather for similar cloud cover. > In fair weather, I'm getting 10% more, in overcast conditions sometimes even > a factor 2. I have some user feedback (4 cases by now) confirming that. Heiko > recently said that he previously got better framerates using Advanced, but > not any more. My problem - I didn't really change anything except parameter > values and such things between the two occasions. Logical conclusion: The > performance of Advanced Weather can change uncorrelated with changes in > the code, and it affects different users in a different way?
I too saw improved framerates in Advanced Weather a short while back, but they seems to have disappeared. It is quite hard to compare like-with-like between Basic and Advanced Weather. > A while ago, the user sgofferj experienced 'stutter' when running Flightgear. > Vivian was kind enough to immediately blame Advanced Weather on the > occasion, however the screenshots clearly showed (and sgofferj later > confirmed) that he had never loaded Advanced Weather (read the story > here http://www.flightgear.org/forums/viewtopic.php?f=68&t=15235). > Logical conclusion: Advanced Weather is able to cause stutter even if the > module is never loaded? > > So, for some it's faster and they gain a lot of performance, for some it's > slower, for some it causes stutter when loaded, for others the mere > presence on the harddisk causes stutter, for some it works fine... should any > of this really cause action from my side? Largely, this is anecdotal evidence. Yes. You have produced a very nice piece of work which gives variable results in different systems. We need to at least understand the cause and try to fix it > 'Bare Flightgear' > > My testcase was France Custom Scenery with the ufo, visibility probably 50 > km or so, no shaders. Removing *all* weather and just remaining stationary, > I get 70 fps with a worst frame delay of pretty constant 14 ms, since 14*70 = > 980 ~ 1000 this means I get very evenly spaced frames. However, just flying > in a circle, my worst frame delay starts to get jumpy every 2-3 seconds, it > goes to 35 - 45 ms, a bit dependent on the scenery to sky ratio. My > interpretation is that I'm seeing the scene being shuffled in and out of the > GPU memory - as I turn, the scene in the visual field changes, and the GPU > needs to compute the changed elements, causing delayed frames. > Pity you didn't use standard scenery. I don't think custom scenery and the ufo will produce representative results. > Just flying straight, the worst frame delay also changes and jumps to 45-55 > ms every few seconds. After a bit of experimenting, that's probably the > effect of loading and unloading of scenery tiles, objects and such like. Loading and unloading scenery tiles every few seconds? Any evidence for that? > That is to say, the bare structure of Flightgear has discontinuous workloads as > well, and they cause frame durations of up to 55 ms on my systems. There > are also genuine (but rare) pathologies like the first look back in the F-16 (a > few seconds of frame delay...) - I guess that's particularly large texture > sheets loading. Yes it probably does. And we should be trying to eliminating these wherever feasible. And don't forget that there's Nasal everywhere which is doing its Garbage Collection. > 'Full Flightgear' > > In my favourite setting, I have weather and lightfield shaders on and a > decent visibility. On the occasion, that delivers me a framerate of ~20 fps > with a constant worst frame delay of 50-55 ms. Since 20 *50 = 1000, the > frames come nice and like a clockwork whatever I do - scenery loading causes > upward spikes to 55 ms from 50 ms, but that's all, i.e. now the GPU is slow > enough to allow all CPU-related stuff to finish nicely. I can reproduce that here. If I throttle the framerate to 20, less than the worst framerate, then I get a steady framerate, but then I get very noticeable but even stutter on the display. I'm aware that there is a school of thought which suggests that regular flickering can trigger epilepsy, but I'm not at all sure if what we are producing falls into this category. Increasing the throttling framerate above the worst (23 fps) makes things gradually worse, reintroducing the spikes in framerate at 30hz. I think we need better than 50 hz for good visual appearance. 40Hz can be acceptable. > 'Shader impacts' > > Vivian always goes talking about smooth framerate, but for instance the > water shader is one of the worst offenders in this department, because on > my system it makes the framerate dramatically dependent on the view > angle. Looking up into the sky and looking down into the sea makes a > difference of a factor 4 (80 fps vs. 20 fps without lightfields, half of that with). > To me, this is quite natural, as the workload of the water shader per > fragment is way higher as the workload of any of the skydome-coloring > codes. Baseline - you don't need Nasal to get uneven framerates, GLSL (and > C++) can deliver that just fine. No that is not what I'm talking about. Altering the view changes the frame rate here, shaders or not, just try looking at KSFO from the air and then away from it. The issue I was on about is uneven framerates with big differences between the worst and average framerate. Are you seeing shaders causing that? However, I don't think you can have only C++ and GLSL in FG - there is always some Nasal running somewhere. I recently "optimised" the water shaders by including some conditions. I think this might have been a mistake: I must revisit this issue. > 'Advanced Weather and a diagnosis' > > Starting from the 'bare case' and running just advanced weather without > lightfields, I get a framerate of 60 fps and a pretty constant worst frame delay > of 50 ms. Since that is 3000 ms, the numbers don't match up and there are > indeed pathologically delayed frames occurring on a regular basis. De- > activating the heaviest per-frame loop (dynamics-loop-flag=0) gets me > almost back to the normal case, i.e. worst frame delay goes to a pretty > constant 22 (60*22 = 1320, not precisely smooth but much better) with > occasional spikes to 55 (note that they've been present without weather as > well - that's terrain loading). Note that this is somewhat supported by > Vivian's data (although not by his words) - deactivating mentioned loop > brings his standard-deviation of the frame duration down by 1/3, de- > activating all per-frame loops brings the standard deviation almost down to > normal. > As I said - here it never gets back to normal, but ThorstenB explained that we are not unloading object when disabling, and that a restart is needed. > The implication is that what is causing the problem is *not* loops which run > slow, it's garbage collection driven by loops which run on a per-frame basis. > Which means that I predict that Vivian's suggestion > > > Personally I would try trading in some framerate for smoothness, and > > let all loops run at maximum frame rate. > > is going to cause the opposite of what he intends - it will trigger even worse > garbage collection problems. > > > We know that "for" loops do > > not help, perhaps those loops can be advanced on index per frame. > > I have explained this before, but I can do it again: If you have 1000 objects, > and you need to visit them every second, you cannot use a loop that > advances index by frame, because you don't have 1000 fps. The solution to > the problem is to assume you have around 30 fps and do about 30 objects > per frame, which is technically realized by a for loop inside a per-frame loop. > I'm quite sure if you have n lifting surfaces for an aircraft, there is a for > loopfrom 0 to n-1 in the FDM code going over all of them per frame, not > calculating one lift coefficient per frame. Of course there are - it doesn't make for loops good. YASim has around 300 for loops. However, many of these are in obscure places such as hook, launchbar etc. and which probably rarely run, and most are small - 3 or 4 iterations. > For loops aren't inherently problematic, they are sometimes needed to get a > job done. They are inherently problematic, the only question is how much of a problem, but sometimes they're unavoidable. What's with this one? setsize(terrain_n,40); for(var j=0;j<40;j=j+1){terrain_n[j]=0;} can't this be: setsize(terrain_n=[],40); Mind you - I have no idea if this is an improvement, but it looks nicer. Do we need setsize() at all? I don't see it much used outside Advanced Weather. > Having said that, I agree that one can potentially split the loops with longer > intervals to generate a more even workload, unfortunately since my > diagnosis is that the bulk of delay is generated by garbage collection, at > present my opinion is that this would make matters even worse. Besides, > leaving GC aside, what counts is the execution time of the argument inside > the for loop - if that is essentially zero or unmeasurable, I can have loops with > very large indexes running asynchronously without generating frame delays. Can you? I have added a simple for loop to the Seahawk which only does j += 1.5; with a loop of 100 at a time interval of 5 sec. This loop shows up as an increased delay in events of 3-5 ms, and the 5 sec interval also shows up. It also coincides from time to time with something else to give a much increased delay. > In other cases, it needs to be accepted that a task simply *is* not continuous. > Loading a terrain tile isn't, generating a weather tile isn't either. I've heard > several people say on this list that they'd like to have weather tiles generated > 'as fast as possible' so that we can change to GUI from a launcher style to a > GUI which immediately reflects changes and re-builds weather whenever > the user changes something. If you want to be fast building weather tiles, > you have to accept that the system crams the calculations into 2-3 seconds > rather than placing clouds over the next minute (as was done previously) > with per-frame advancing loops. > > The discontinuity here is inherent in the design, fast tile generation was > formulated here as a design goal, thus the discontuinuity is in the algorithm > and has nothing to do with Nasal or C++. > So, my impression is that there are some discontinuous CPU tasks in > Flightgear which on my system seem to take about 55 ms in a single frame > (apart from pathologies which are rare). If the GPU works with the same > speed, everything is fine. If the GPU delivers frames faster, single frames still > get delayed to 55 ms due to the various discontinuous tasks (terrain > loading/unloading, garbage collection, possibly Nasal...), causing a stutter. > Probably, for higher framerate the GC problem gets even worse, since per- > frame Nasal is executed more often per second, and I'm guessing that GC is > triggered after a fixed amount of work, not after a fixed time. I have no idea how GC works, and I wouldn't speculate. ThorstenB says: "A significant part of Nasal-related frame rate impact is caused by garbage collection. Its delay/jitter only depends on the number of Nasal objects and their references which need to be searched. Increasing the number of Nasal objects (code+data) existing in memory also increases the delay." > In the short term, I can reduce the per-frame load somewhat. In the longer > term, I'm afraid my position is that Nasal is here to stay. It makes no sense to > me to port 12.000 lines of Nasal code (not to mention complex aircraft Nasal, > the Concorde probably has some 10.000 lines as well and would probably > cause similar problems if I could ever get it to much more than 20 fps) when > the problem is with the infrastructure of garbage collection. I mean, you can > of course port to C++ whatever you want, but as I tried to indicate above, > there are sometimes algorithmic rather than technical reasons for > discontinuities in the performance, so whatever you do it's at best going to > address part of the issues. Concorde has of the order of 6000 lines of active code, and yes, it displays exactly the same discontinuities as Advanced Weather (approx. 10,000 lines of code). So far, I have not found any other examples. > I know some here do not think that Nasal is a good thing, but the decision to > include it into Flightgear has been made before I ever became interested and > lots of lines of code have accumulated. When the fuse box in my house is > faulty, do I go around telling everyone not to use dishwasher and microwave > at the same time, do I devise rules for 'good cooking' that on that stove eye, > only half heat should be used, then the oven should cool for 2 minutes, and > under no circumstances should we use the light in the kitchen while the oven > is on? Or do I call the electrician and fix the fuse box? What an odd metaphor - you very definitely do the first , while waiting for the second. If the fuse box cannot be fixed, then you are stuck with the first, and that's where we are right now. Let's hope there is a solution. Yes, I was around when Nasal was introduced to Flightgear. Up to now it's been a few 100s of lines, not many 1000s. We have long been aware that GC could cause problems, but haven't fixed it because the effects weren't too bad while Nasal was small and framerates were low, and we had no obvious solution. Now Advanced Weather and Concorde have exposed the problem. > So, I'm willing to help addressing the issues from my side, but I'm not willing > to discuss on the level of 'Nasal is bad', 'Nasal is the only cause of uneven > framerate', 'for loops are bad' and these things. I think I have presented a > reasonable theory of what causes issues, it's backed up by my performance > tests, if you disagree then I'd like to see the evidence. I disagree. While Nasal isn't the only cause, it is a major cause. I accept that GC is probably at the bottom of it, combined with too many Nasal Objects and their references. While these remained small the inadequacies of GC were not exposed, but once these become large they are. I reiterate: Nasal is bad. Big Nasal is worse, small Nasal is not so bad. For loops are bad - in C++ as much as in Nasal. For loops hold up the execution of the main loop, even when they do very little. I managed to ruin the framerate by putting inappropriate for loops in C++ during my development of AG Radar. Sometimes they are a necessary evil. Execution of loops at other than main loop rate is bad. If the deferred loop does a lot then this will show up as a stutter, if they do not do much, then they aren't needed. I also suspect that deferred loops can interact unpredictably with other deferred loops (a bit like waves in the ocean) to produce a rogue stutter. I tried this in C++ during my development of AI stuff, I even tried starting at random intervals, but I discovered that letting Fg do its own thing was better. Let's hope James or someone comes up with a better GC algorithm. Regards Vivian ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
