What I try to do in these cases is a combination of: - when a new xxxxx is added, say "Your xxxxx has been added successfully. It may take a few minutes to appear in the xxxxx list" - In the xxxxx listing page, only show the most basic info for each (Ideally a name that doesn't change, but see next bullet). So a vanilla query accross entity groups is fine here. Some users may see the new xxxxx sooner than others (no biggie, matter of seconds usually) - If the name _can_ get changed, say something to the changer like "The new name for your xxxxx may take a few minutes to appear in the xxxxx list" - Clicking on the "details" of a xxxxx does a get-by-key, ensuring the latest
And If you're going to cache the list page, have the cache maintained by a cron job, to ensure there are no collisions. Cron job needs to be a bit smart, since in theory it could also see a new xxxxx one run, then not see it on a query a minute later. But at least there is a single place to deal with that with no contention on the cache. /Tom On Sep 5, 1:58 pm, Joshua Smith <[email protected]> wrote: > Yeah, that'll work, too. Unless there are two changes to the list of boards > at about the same time. Then it will give a bizarre result. > > The general idea of augmenting queries with extra results that you suspect it > might not get seems to come up a lot as a "solution" to the eventual > consistency problem. Other than entity groups (which doesn't seem to fit any > of my real-world data models), this is the only suggestion in the docs. > > However, it really kinda sucks. Since query results are usually sorted in > some way, you need to re-sort to get the fixed record into the right place. > But, of course, you can't do that if you are using the query as an iterator. > And, of course, the change may have made that record no longer fit the query > criteria, or it may have caused it to fit where it didn't before, so you need > to deal with all three cases: remove the record from the list of results, add > the record to the list of results, and replace the record in a list of > results. And then there is the question of where you put the record. > Putting it into a cookie or query parameter is weird, and fragile. Putting > it into memcache is fragile. > > I suppose it's possible that there just is no good design pattern to use > here. In particular, I'm having trouble coming up with a way that doesn't > require me to wrap 3 old lines of code with 20 lines of crap, then repeating > that in 50 different places. I don't mind having a bunch of crap if there is > some way to reuse it. But I'm struggling with that. > > Is ANYBODY out there happy with the way they solved this problem? > > -Joshua > > On Sep 4, 2011, at 5:51 PM, Tom Phillips wrote: > > > > > "I clear the cache whenever the list of boards changes in some way" > > > How about update the cache at that point instead of clearing it? > > > Need be you could even generate the HTML for the cache update with a > > URLFetch to your UI handler where you include the added/changed board > > key(s) as parameters, so they can be gotten with strong consistency on > > that request and merge into the query result. > > > /Tom > > > On Sep 4, 11:16 am, Joshua Smith <[email protected]> wrote: > >> My monkeypatching solution (see my recent post in the -python group), > >> which Guido says I shouldn't use, but which is just so darned pretty I > >> can't help it, has gotten me through the first challenge of switching to > >> HR, which is dealing with google search results containing keys into my > >> old app's data store. > > >> So now I'm looking at the big Kahuna problem of consistency. Here's my > >> first messy challenge there: > > >> My app puts a list of boards on the home page for a town, along with the > >> list of meetings. Generating that list of boards was taking a lot of CPU, > >> but they hardly ever change, so I put in a memcache system that built the > >> HTML when it wasn't in the cache, and then cached it before serving. I > >> clear the cache whenever the list of boards changes in some way. > > >> Well that ain't gonna work in HR. It's quite possible that I update a > >> board, clear the cache, and someone comes and hits that page before > >> "eventually consistent" comes to pass. So now I've got a cached copy of > >> the stale data. > > >> (Note that I cannot use entity groups to solve this because some boards > >> are municipal agencies, and therefore cannot be parented to the town that > >> is building its list. I could parent all boards to some global parent, > >> but, well, yuck.) > > >> I have some different ideas about how to fix this, but I'm wondering if > >> anyone else who's done the port to HR has come up with a solution they > >> find particularly elegant? I assume this is a pretty common problem, so > >> there must be a design pattern out there… somewhere. > > >> Here are my ideas: > > >> - Clear the cache with a periodic task that re-clears it several times. > >> I'm thinking a recurring geometric retry would be prudent (1, 2, 4, 8, 16, > >> 32, 64, 128, 256, 512 seconds, and then pray that we have consistency) > > >> - Checksum the modified or new board, and put that sum into memcache. > >> When generating the new board, confirm that any checksums are good. This > >> seems more deterministic, except I don't trust memache not to squelch the > >> checksum record. So perhaps I should do something in the datastore. This > >> feels like it's be about 10x as much code as the stupid geometric flush. > > >> Any suggestions? > > >> -Joshua > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
