I used to advise people to use shared entity groups as little as possible. That was on M/S. I have found that the HRD changes everything, and now I use hierarchical entity groups quite a lot.
>From previous discussions here, HRD eventuality is usually seconds, sometimes minutes, and in catastrophic scenarios could be hours. For a lot of situations, this isn't a big deal. For some (say, anything commerce-related) this could result in serious real-world problems. It's not ok to have someone's shopping cart go blank at a memcache flush, and under no circumstances should a customer's purchase be "lost", even for a few seconds. I've found that it's possible to work around this in most cases - it's not necessary to "accept" eventuality. The oft-repeated advice that an entity group should roughly correspond to a single user's data is good - a shopping cart will always be consistent that way. Another trick is to use lookup entities instead of indexed fields; for example, create an EmailLookup keyed by the normalized email address and which points at the relevant user record. Get by primary key is strongly consistent and you can use a transaction to ensure that any email address points at one and only one user record. Almost all of us came from RDBMS-land stuffed full of academic theories about the "correct" way to model data. I don't think there's an equivalent set of theories for the datastore. I've found that every application ends up with a pretty radically different schema depending on query profile and the specific shape of data. There's no way to say "entity groups are good" or "entity groups are bad" except for specific use cases, and any mature app on the HRD will probably have examples of each. Jeff On Mon, Oct 24, 2011 at 2:02 AM, Mos <[email protected]> wrote: > Thanks for the feedback. > Regarding eventually consistency one advice of this talk was: > "Accept it." > > I think for many use-cases in typical web-application this is ok. > For showing a use his own post immediately, you can remember the key of the > entity and add it manually to the query-list (if not yet available). > > But one question remains: > What does "eventually consistency" mean in practice? > > Does it only concern new written data? > For how long would this data not be shown up in queries? > Is it just a matter of milliseconds or a few seconds (or minutes)? > > > > > On Sat, Oct 22, 2011 at 10:43 PM, Gabriel Charette > <[email protected]>wrote: > >> If you design it right you can benefit from strong consistency in the >> situations you need it while still not hitting the 1 write/sec/group >> cap, see the Google IO talk about it (in particular the user centric >> design): >> http://www.youtube.com/watch?v=xO015C3R6dw >> >> And the slides of the talk: >> >> http://static.googleusercontent.com/external_content/untrusted_dlcp/www.google.com/en/us/events/io/2011/static/presofiles/more_9s_under_the_covers_of_the_high_replication_datastore.pdf >> >> As described in that talk, you want strong consistency in particular >> for a user's own posts (i.e. it's annoying as a user if you post and >> don't see your own post right away..) >> >> Cheers, >> Gabriel >> >> On Oct 22, 3:16 pm, Mos <[email protected]> wrote: >> > I'm still in a design phase of a new App Engine application and I'm >> > wondering >> > if it makes sense to go *without* using any Entity Groups. >> > >> > My application's data-model is similar to a discussion-forum. >> > It has Postings, Comments, Users, Ratings and so on. If the application >> > becomes successful, there will be significant write operations on >> > Postings, Comments and Ratings. >> > >> > If I would model hierarchical *Entity-Groups* (e.g. starting from a >> > User-Root), I would >> > have the following *advantages*: >> > +1: Transactional behavior during writes >> > +2: Queries are strongly consistent >> > +3: For hierachical data-structures, fast ancestor queries are >> available >> > >> > But *Entity-Groups* introduce also some *disadvantages*: >> > -1: Limitation in write throughput (in worst case just 1 write per >> > second on an entity group) >> > -2: Inflexible and a more complex data-model, which makes it harder to >> add >> > or re-factor relationships (forces "owned-relations" all over your >> > application) >> > >> > *I think all together the disadvantages outweigh the advantages for a >> > typical web application like mine.* >> > The limitation for writes seems to be a significant disadvantage. Ok, >> you >> > can work around this with sharding but this makes >> > you data-model even more complex. >> > And an hierarchical data-model which is technically motivated and not >> > related to your domain seems to be a bad idea. >> > >> > On the other hand transactions during write operations are often not >> > necessary ("the last one wins"). >> > Hierarchical searches are not needed most of the time. >> > >> > The only issue I'm not sure about is the advantage of "Queries are >> strongly >> > consistent "? >> > What does it mean if they are not strongly consistent? >> > Is this just an issue for a small time frame when new data are written? >> Once >> > the data is >> > stored and replicated this shouldn't be an issue at all, or? >> > >> > I'm happy about feedback! >> > >> > Thanks >> > Mos >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
