the viewer discards small changes anyway if avatar imposters are enabled On Mon, Mar 28, 2011 at 11:54 AM, Melanie <[email protected]> wrote:
> No, we can't discard small changes. As the avatar comes closer, they > would be seen out of place, e.g. someone building in the distance > would move prims and then you come closer to look and all prims > would be out of place. > > Melanie > > Dahlia Trimble wrote: > > a couple thoughts.. > > > > Perhaps resend timeout period could be a function of throttle setting > and/or > > measured packet acknowledgement time per-client? (provided we measure > it). > > That may prevent excessive resend processing that may not be necessary. > > > > On the distance prioritization, could small changed in object > translations > > be discarded from the prioritization queues/resend buffers for distant > > objects when new updates occur for those objects? Small changes may not > be > > noticeable from the viewer perspective anyway. > > > > > > On Mon, Mar 28, 2011 at 10:48 AM, Teravus Ovares <[email protected]> > wrote: > > > >> Here are a few facts that I've personally discovered while working > >> with LLClientView. > >> > >> 1. It has been noted that people with poor connections to the > >> simulator do consume more bandwidth, cpu, and have a generally worse > >> experience. This has been tested and profiled extensively. This > >> may seem like a small issue because what it's doing is so basic... > >> however the frequency in which this occurs is a real cause of > >> performance issues. > >> > >> 2. It's also noted that the CPU used in these cases reduces the CPU > >> available to the rest of the simulator resulting in a lower quality of > >> service for the rest of the people on the simulator. > >> This has been seen in the profiling and has been qualitatively > >> observed by a large number of users connected and everything is OK and > >> then a 'problem connection' user connecting causing a wide range of > >> issues. > >> > >> 3. It's also noted that lowering the outgoing UDP packet throttles > >> beyond a certain point results in perpetual queuing and resends. > >> This was tested by using a throttle multiplier last year that was > >> implemented by justincc. I'm not sure if the multiplier is still > >> there. It's most easily seen with image packets. Again, I note > >> that the packets are not rebuilt going from the regular outbound queue > >> to the resend queue. The resend queue is /supposed/ to be used to > >> quickly get data that is essential to the client after attempting to > >> send once already. The UDP spec declares the maximum resend to be 2 > >> times, however there has been some considerable debate on whether or > >> not OpenSimulator should follow that specific specification item > >> leading to a configuration option to enable perpetual resends > >> (Implemented by Melanie). The configuration item was named similar > >> to, 'reliable is important' or something like that. I'm not sure if > >> the configuration item survived the many revisions however I suspect > >> that it did. > >> > >> 4. It's also noted that raising the packet throttles beyond what the > >> connection can support results in resending almost every packet the > >> maximum amount of times before the limit is reached. > >> This is easily reproducible by setting the connection (in the client) > >> to the maximum and connecting to a region that you've never been to > >> before on a sub par connection. Before the client adjusts and > >> requests a lower throttle setting there's massive data loss and > >> massive re-queuing. > >> > >> 5. The client tries to adjust the throttle settings based on network > >> conditions. This can be observed by monitoring the packet that sets > >> the throttles and dragging the bar to maximum. After a certain > >> amount of resends, the client will call the set throttle packet with > >> reduced settings (some argue that it doesn't do that fast enough). > >> > >> 6. A user who has connected previously to the simulator will use less > >> resources then a user who has never connected to the simulator. (this > >> is mostly because of the image cache on the client). Any client > >> that uses CAPS images will use less resources then one that uses > >> LLUDP. > >> > >> When working with the packet queues, it's essential to understand > >> those 6 observations. Even though, the place where you tend to see > >> the issues with queuing is the image queue over LLUDP, the principles > >> apply to all of the udp queues. > >> > >> Regards > >> > >> Teravus > >> > >> > >> On Mon, Mar 28, 2011 at 1:00 PM, Mic Bowman <[email protected]> wrote: > >> > Over the last several weeks, Dan Lake & I have been looking some of > the > >> > networking performance issues in opensim. As always, our concerns are > >> with > >> > the problems caused by very complex scenes with very large numbers of > >> > avatars. However, I think some of the issues we have found will > generally > >> > improve networking with OpenSim. Since the behavior represents a > fairly > >> > significant change in behavior (though the number of lines of code is > not > >> > great), I'm going to put this into a separate branch for testing > (called > >> > queuetest) in the opensim git repository. > >> > We've found several problems with the current > >> > networking/prioritization code. > >> > * Reprioritization is completely broken for SceneObjectParts. On > >> > reprioritization, the current code uses the localid stored in the > scene > >> > Entities list but since the scene does not store the localid for SOPs, > >> that > >> > attempt always fails. So the original priority of the SOP continues to > be > >> > used. This could be the cause of some problems since the initial > >> > prioritization assumes position 128,128. I don't understand all the > >> possible > >> > ramifications, but suffice it to say, using the localid is causing > >> > problems. > >> > Fix: The sceneentity is already stored in the update, just use that > >> instead > >> > of the localid. > >> > * We currently pull (by default) 100 entity updates from the > entityupdate > >> > queue and convert them into packets. Once converted into packets, they > >> are > >> > then queued again for transmissions. This is a bad thing. Under any > kind > >> of > >> > load, we've measured the time in the packet queue to be up to many > >> > hundreds/thousands of milliseconds (and to be highly variable). When > an > >> > object changes one property and then doesn't change it again, the time > in > >> > the packet queue is largely irrelevant. However, if the object is > >> > continuously changing (an avatar changing position, a physical object > >> > moving, etc) then the conversion from a entity update to a packet > >> "freezes" > >> > the properties to be sent. If the object is continuously changing, > then > >> with > >> > fairly high probability, the packet contains old data (the properties > of > >> the > >> > entity from the point at which it was converted into a packet). > >> > The real problem is that, in theory, to improve the efficiency of the > >> > packets (fill up each message) we are grabbing big chunks of updates. > >> Under > >> > load, that causes queuing at the packet layer which makes updates > stale. > >> > That is... queuing at the packet layer is BAD. > >> > Fix: We implemented an adaptive algorithm for the number of updates to > >> grab > >> > with each pass. We set a target time of 200ms for each iteration. That > >> > means, we are trying to bound the maximum age of any update in the > packet > >> > queue to 200ms. The adaptive algorithm looks a lot like a TCP slow > start: > >> > every time we complete an iteration (flush the packet queue) in less > than > >> > 200ms we increase linearly the number of updates we take in the next > >> > iteration (add 5 to the count) and when we don't make it back in > 200ms, > >> we > >> > drop the number we take quadratically (cut the number in half). In our > >> > experiments with large numbers of moving avatars, this algorithm works > >> > *very* well. The number of updates taken per iteration stabilizes very > >> > quickly and the response time is dramatically improved (no "snap back" > on > >> > avatars, for example). One difference from the traditional slow > start... > >> > since the number of "static" items in the queue is very high when a > >> client > >> > first enters a region, we start with the number of updates taken at > 500. > >> > that gets the static items out of the queue quickly (and delay doesn't > >> > matter as much) and the number taken is generally stable before the > >> > login/teleport screen even goes away. > >> > * The current prioritization queue can lead to update starvation. The > >> > prioritization algorithm dumps all entity updates into a single > ordered > >> > queue. Lets say you have several hundred avatars moving around in a > >> scene. > >> > Since we take a limited number of updates from the queue in each > >> iteration, > >> > we will take only the updates for the "closest" (highest priority) > >> avatars. > >> > However, since those avatars continue to move, they are re-inserted > into > >> the > >> > priority queue *ahead* of the updates that were already there. So... > >> unless > >> > the queue can be completely emptied each iteration or the priority of > the > >> > "distant" (low priority) avatars changes, those avatars will never be > >> > updated. > >> > Fix: We converted the single priority queue into multiple priority > queues > >> > and use fair queuing to retrieve updates from each. Here's how it > works > >> > (more or less)... the current metrics (all of the current > prioritization > >> > algorithms use distance at some point for prioritization) compute a > >> distance > >> > from the avatar/camera to an object. We take the log of that distance > and > >> > use that as the index for the queue where we place the update. So > close > >> > things go into the highest priority queue and distant things go into > the > >> > lowest priority queue. Since the area covered by a priority queue > grows > >> as > >> > the square of the radius, the distant (lowest priority queues) will > have > >> the > >> > most objects while the highest priority queues will have a small > number > >> of > >> > objects. Inside each priority queue, we order the updates by the time > in > >> > which they entered the queue. Then we pull a fixed number of updates > from > >> > each priority queue each iteration. The result is that local updates > get > >> a > >> > high fraction of the outgoing bandwidth but distant updates are > >> guaranteed > >> > to get at least "some" of the bandwidth. No starvation. The current > >> > prioritization algorithm we implemented is a modification of the "best > >> > avatar responsiveness" and "front back" in that we use root prim > location > >> > for child prims and the priority of updates "in back" of the avatar is > >> lower > >> > than updates "in front". Our experiments show that the fair queuing > does > >> > drain the update queue AND continues to provide a disproportionately > high > >> > percentage of the bw to "close" updates. > >> > One other note on this... we should be able to improve the performance > of > >> > reprioritization with this approach. If we know the distance an avatar > >> has > >> > moved, we only have to reprioritize objects that might have changed > >> priority > >> > queues. Haven't implemented this yet but have some ideas for how to do > >> it. > >> > * The resend queue is evil. When an update packet is sent (they are > >> marked > >> > reliable) it is moved to a queue to await acknowledgement. If no > >> > acknowledgement is received (in time), the packet is retransmitted and > >> the > >> > wait time is doubled and so on... What that means is that a resend > >> packets > >> > in a scene that is rapidly changing will often contain updates that > are > >> > outdated. That is, when we resend the packet, we are just resending > old > >> data > >> > (and if you're having a lot of resends that means you already have a > bad > >> > connection & now you're filling it up with useless data). > >> > Fix: this isn't implemented yet (help would be appreciated)... we > think > >> that > >> > instead of saving packets for resend... a better solution would be to > >> keep > >> > the entity updates that went into the packet. if we don't receive an > ack > >> in > >> > time, then put the entity updates back into the entity update queue > (with > >> > entry time from their original enqueuing). That would ensure that we > send > >> an > >> > update for the object & that the data sent is the most recent. > >> > * One final note... per client bandwidth throttles seem to work very > >> well. > >> > however, our experiments with per-simulator throttles was not > positive. > >> it > >> > appeared that a small number of clients was consuming all of the bw > >> > available to the simulator and the rest were starved. Haven't looked > into > >> > this any more. > >> > > >> > So... > >> > Feedback appreciated... there is some logging code (disabled) in the > >> branch; > >> > real data would be great. And help testing. there are a number of > >> > attachment, deletes and so on that i'm not sure work correctly. > >> > --mic > >> > > >> > > >> > > >> > > >> > > >> > _______________________________________________ > >> > Opensim-dev mailing list > >> > [email protected] > >> > https://lists.berlios.de/mailman/listinfo/opensim-dev > >> > > >> > > >> _______________________________________________ > >> Opensim-dev mailing list > >> [email protected] > >> https://lists.berlios.de/mailman/listinfo/opensim-dev > >> > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Opensim-dev mailing list > > [email protected] > > https://lists.berlios.de/mailman/listinfo/opensim-dev > _______________________________________________ > Opensim-dev mailing list > [email protected] > https://lists.berlios.de/mailman/listinfo/opensim-dev >
_______________________________________________ Opensim-dev mailing list [email protected] https://lists.berlios.de/mailman/listinfo/opensim-dev
