Hi, Thank you all for the responses:
@Robert: Yes, i forgot to say that it will be easier to manipulate a prefix using unix timestamps (aa234958494, ab23459059234) rather than a time structure + prefix. Which is tricky also, since most of the times you handle dates are either because ordering or expiring. So using the prefixed timestamps for indexing and the actual timestamp for business operations can lead to some unwanted behaviors or implementations (if not well described, i know). @Nick Well, the whole situation was really difficult for us due to the nature of the application: We handle a normal traffic around 500 requests per second constantly but there are peak times that traffic burst twice at least. We did ask for feedback later ( i joined the team later last year) and what we found is that some of the structures were designed following some relational patterns. For example, the most conflicting structure is a model that keeps track of current location of the user, geoboxes (we use geobox model for location proximity), some business information and the last activity of the user (timestamp) which is updated every time the user wants to load more content (sort of a bad online presence approach). Even worst, that structure was part of a group and transactions are constant to either sign in people (create or update) of sign off (delete => i know, deletes are expensive). So bottom line: entity in a group + contant updates of the timestamp + transactions + deletes + not usage of task queues = performance was really poor (and we are aware of the design problem) We have already taken actions, like some usage of memcache for some quick reads and theres an incoming patch that avoids so many updates. Following we are unplugging that structure to be parent and do not depend on hierarchies and signing people in and out using task queues (and avoiding deletes). While we were there we considered Ikai's suggestion for the timestamps (so we can take the most advantage of the patch) and realized during profiling that when our latency increases is due to this single update to timestamps (when it is in bad condition reaches up to 1450 ms of latency = no more sharding for us, when we normally handle a latency of 450 ms). Thats currently our best guess right now, since we can't go under the whole hood to see if thats actually a root cause for a small percentage of the whole performance decrease (so if you have something else to share in order to diagnose it, it is more than welcomed :) ) After some patches we were able to decrease the QPS to almost 1 per second (Really low) but thats just an aggregated number for the whole traffic and there are sometimes (and i know this) that hunders are executed per second when peak times are reached and people signs in/out. On 18 January 2011 22:12, Nick Johnson (Google) <[email protected]> wrote: > Hi David, > I was the Googler to originally observe this, in the form of 'hot tablets' > on Bigtable. The specific problem only occurs, in my experience, at write > rates in excess of a couple of hundred QPS, and in situations where all of > those writes are being directed to a single tablet (row range). This is a > level of write traffic that few, users are likely to encounter. > The issue also occurs at slightly lower traffic levels if the ID of the > entity is being auto-allocated, due to the higher overhead of writing the > entire entity instead of just an index row. This is usually trivially > alleviated by switching to key names - either randomly allocated (such as a > UUID) or based on something that isn't monotonic, such as email address. > What level of writes are you experiencing the issue at, and how did you > determine that this was the cause of your issue? > -Nick Johnson > > On Wed, Jan 19, 2011 at 5:30 AM, David Mora <[email protected]> wrote: >> >> Quoting Ikai >> >> "Just be aware of monotonically increasing indexes like timestamps: if >> you have an application with a high write rate that has a timestamp, >> this will cause the persistence unit storing the indexes to be unable >> to be autosplit easily across multiple hardware instances and you will >> take a performance hit. The solution here is, again, to shard the >> timestamp by prefixing a value to distribute the writes." >> >> This was in another threat and was left out of the conversation. >> >> From my perspective, this is a huge improvement since most of the >> times we have to deal timestamps. We handle an application that uses >> the problematic approach and it's just a bottle neck. We are starting >> to switch this and notice two things: >> >> - it will be nice if the data store implementation can supply the >> functionality needed to shard it (since seems like a change down deep >> in the layer). >> - What would be the best approach to solve this sharding? would unix >> timestamps would be enough for this? >> >> >> Appreciate your feedback >> >> >> -- >> http://about.me/david.mora >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > Nick Johnson, Developer Programs Engineer, App Engine > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- http://about.me/david.mora -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
