Thanks Nick, I understand now. So I guess the easiest thing to do is to have a random component in your keynames...at least for apps I'm considering I don't think I'd have any other way to reasonably ensure the range of keynames in a given (batch) update were well distributed.
On Feb 19, 11:58 am, "Nick Johnson (Google)" <[email protected]> wrote: > On Fri, Feb 19, 2010 at 12:28 AM, peterk <[email protected]> wrote: > > What about keynames like: > > > counter_standard_dbf > > counter_standard_clo > > > or would something like > > > dbfo01la_counter_standard > > clo091b_counter_standard > > > work better? > > > I'm thinking of cases where you may use keynames that can in some way > > be constructed/predicted for fast access later. > > like..<username>_counter_standard > > > Would the common pre-fix or post-fix make for close distribution? :| > > Either one will work fine - Bigtable will split tablets based on key to > ensure no tablet gets too big. Long identical prefixes just mean that the > split will be based on later characters in the string. > > What's important for key distribution for really high update rates is the > distribution of key names/IDs for those updates: If they all go to a single > tablet (eg, they make up a small proportion of the total range of IDs you're > employing), they will be limited by what that tablet server can support. If > they are widely spread out within the range you're using, regardless of what > that range is, you'll be fine. > > -Nick Johnson > > > > > > > > > On Feb 18, 5:59 pm, "Nick Johnson (Google)" <[email protected]> > > wrote: > > > Hi Eli, > > > > Using a randomly generated ID like a uuid is perfectly satisfactory to > > > achieve an even distribution. > > > > On Wed, Feb 17, 2010 at 7:02 PM, Eli Jones <[email protected]> wrote: > > > > I understand the process of evenly distributing IDs since they are > > Integer > > > > values.. is there a canonized appengine way to evenly distribute > > key_names? > > > > > Just make sure key_name1 and key_name2 don't have their i-th letters > > "too > > > > close" too eachother? How far is far enough? > > > > > Does doing even distribution matter if you aren't using auto-generated > > IDs? > > > > It certainly can - if you insert, in order, "aaaa", "aaab", "aaac", etc, > > > you'll encounter the same problem at very high volumes as you'd see with > > > auto generated IDs. > > > > -Nick Johnson > > > > > Thanks for information. > > > > > On Wed, Feb 17, 2010 at 1:32 PM, Nick Johnson (Google) < > > > > [email protected]> wrote: > > > > >> Hi Ulrich, > > > > >> On Wed, Feb 17, 2010 at 5:30 PM, Ulrich <[email protected] > > >wrote: > > > > >>> Hi, > > > > >>> I have read the following > > > >>> "Timeouts due to datastore issues --- [...] The most common example > > of > > > >>> this occurs when you are rapidly inserting a large number of entities > > > >>> of the same kind, with auto-generated IDs. In this case, most inserts > > > >>> hit the same range of the same tablet, and the single tablet server > > is > > > >>> overwhelmed with writes. [...] If this does affect your app, the > > > >>> easiest solution is to use more evenly distributed IDs instead of the > > > >>> auto-allocated ones [...]" > > > >>> ( > > >http://code.google.com/appengine/articles/handling_datastore_errors.html > > > >>> ) > > > > >>> Let's say I am having a model "Parent" and a model "Child". For > > Parent > > > >>> entities, I use key names that are evenly distributed. For Child > > > >>> entities, I use auto-generated key IDs and _no_ key names, but all > > > >>> Child entities are children of Parent entities, so the paths to the > > > >>> children contain the evenly distributes key names of the parents. > > > >>> If I have many write operations on children that are in the same > > > >>> entity group, the described error could occur. But what happens if my > > > >>> write operations are on children that are in different entity groups? > > > >>> Their IDs are auto-generated and not evenly distributed, but their > > > >>> paths contain the evenly distributed key names. > > > > >> Good question! The point being made in the article refers to the > > global > > > >> distribution of the complete key, so writes to these children will be > > well > > > >> distributed, and you won't have to worry about this source of > > contention. > > > > >> -Nick Johnson > > > > >>> -- > > > > >>> You received this message because you are subscribed to the Google > > Groups > > > >>> "Google App Engine" group. > > > >>> To post to this group, send email to > > [email protected]. > > > >>> To unsubscribe from this group, send email to > > > >>> [email protected]<google-appengine%2Bunsubscrib > > > >>> [email protected]><google-appengine%2Bunsubscrib > > [email protected]> > > > >>> . > > > >>> For more options, visit this group at > > > >>>http://groups.google.com/group/google-appengine?hl=en. > > > > >> -- > > > >> Nick Johnson, Developer Programs Engineer, App Engine > > > >> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > Number: > > > >> 368047 > > > > >> -- > > > >> You received this message because you are subscribed to the Google > > Groups > > > >> "Google App Engine" group. > > > >> To post to this group, send email to > > [email protected]. > > > >> To unsubscribe from this group, send email to > > > >> [email protected]<google-appengine%2Bunsubscrib > > > >> [email protected]><google-appengine%2Bunsubscrib > > [email protected]> > > > >> . > > > >> For more options, visit this group at > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to [email protected] > > . > > > > To unsubscribe from this group, send email to > > > > [email protected]<google-appengine%2Bunsubscrib > > > > [email protected]><google-appengine%2Bunsubscrib > > [email protected]> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > > Nick Johnson, Developer Programs Engineer, App Engine > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > > Number: > > > 368047 > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%2Bunsubscrib > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Nick Johnson, Developer Programs Engineer, App Engine > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
