You can transactionally launch a deferred task with a future ETA that will clear the Memcache.
johnP On Sep 6, 9:55 am, Joshua Smith <[email protected]> wrote: > And you and your users are happy with this, Tom? > > I've been thinking more about entity groups, since they make this consistency > mess go away. In fact, while Entity Groups are not a great solution for all > my data, they do seem to fit for most of my models. Meetings can be parented > by Boards, Meeting Changes can be parented by Meetings, etc.. So it's really > only the top-level (Town) view where the consistency will be all over the > place, and doing what you suggested for that seems like it could work. > > However, I'm now realizing that adding entity groups to an existing > application (even during the transition to HR) is a real problem. > > - I won't be able to use the migration tools, since they are going to create > the new entities matching the old ones, and you can't change a parent once > the model is created. > > - The trick of making a new key from the old key (to compensate for the app > ID changing) won't work, because the parent paths will be wrong. > > So if I wanted to use entity groups, the best I can come up with is: > > - Do the migration from M/S to HR completely manually, with lots of > receiving-side logic to build up entity groups where none existed before > > - Create a table mapping old app keys to new app keys, so that when I get a > request containing a key with the old app ID, I can look it up in the table > to find out the new key (which includes a parent path) > > Also, there are a few places where I use IDs in URLs instead of keys (like in > an URL that might be sent by email, to keep it short). By my reading, that > won't work, because I can't call Model.get_by_id() unless I know the parent > already. So once you go to entity groups, id's are pretty much useless. (Is > that right?) I guess I could create id-like things that combine the IDs of > all the things up the chain of parenting (1234_5678_9012). The long strings > of digits might still be nicer than full keys. > > I'm feeling like I have a choice between a non-entity-group solution with a > ton of problems cause by consistency, or an entity-group solution with a ton > of problems cause by migration (for those models where entity groups are > feasible at all). Lesser of two evils design choices. Lovely. > > I'm really starting to hate HR. > > -Joshua > > On Sep 5, 2011, at 4:13 PM, Tom Phillips wrote: > > > > > > > > > What I try to do in these cases is a combination of: > > > - when a new xxxxx is added, say "Your xxxxx has been added > > successfully. It may take a few minutes to appear in the xxxxx list" > > - In the xxxxx listing page, only show the most basic info for each > > (Ideally a name that doesn't change, but see next bullet). So a > > vanilla query accross entity groups is fine here. Some users may see > > the new xxxxx sooner than others (no biggie, matter of seconds > > usually) > > - If the name _can_ get changed, say something to the changer like > > "The new name for your xxxxx may take a few minutes to appear in the > > xxxxx list" > > - Clicking on the "details" of a xxxxx does a get-by-key, ensuring the > > latest > > > And If you're going to cache the list page, have the cache maintained > > by a cron job, to ensure there are no collisions. Cron job needs to be > > a bit smart, since in theory it could also see a new xxxxx one run, > > then not see it on a query a minute later. But at least there is a > > single place to deal with that with no contention on the cache. > > > /Tom > > > On Sep 5, 1:58 pm, Joshua Smith <[email protected]> wrote: > >> Yeah, that'll work, too. Unless there are two changes to the list of > >> boards at about the same time. Then it will give a bizarre result. > > >> The general idea of augmenting queries with extra results that you suspect > >> it might not get seems to come up a lot as a "solution" to the eventual > >> consistency problem. Other than entity groups (which doesn't seem to fit > >> any of my real-world data models), this is the only suggestion in the docs. > > >> However, it really kinda sucks. Since query results are usually sorted in > >> some way, you need to re-sort to get the fixed record into the right > >> place. But, of course, you can't do that if you are using the query as an > >> iterator. And, of course, the change may have made that record no longer > >> fit the query criteria, or it may have caused it to fit where it didn't > >> before, so you need to deal with all three cases: remove the record from > >> the list of results, add the record to the list of results, and replace > >> the record in a list of results. And then there is the question of where > >> you put the record. Putting it into a cookie or query parameter is weird, > >> and fragile. Putting it into memcache is fragile. > > >> I suppose it's possible that there just is no good design pattern to use > >> here. In particular, I'm having trouble coming up with a way that doesn't > >> require me to wrap 3 old lines of code with 20 lines of crap, then > >> repeating that in 50 different places. I don't mind having a bunch of > >> crap if there is some way to reuse it. But I'm struggling with that. > > >> Is ANYBODY out there happy with the way they solved this problem? > > >> -Joshua > > >> On Sep 4, 2011, at 5:51 PM, Tom Phillips wrote: > > >>> "I clear the cache whenever the list of boards changes in some way" > > >>> How about update the cache at that point instead of clearing it? > > >>> Need be you could even generate the HTML for the cache update with a > >>> URLFetch to your UI handler where you include the added/changed board > >>> key(s) as parameters, so they can be gotten with strong consistency on > >>> that request and merge into the query result. > > >>> /Tom > > >>> On Sep 4, 11:16 am, Joshua Smith <[email protected]> wrote: > >>>> My monkeypatching solution (see my recent post in the -python group), > >>>> which Guido says I shouldn't use, but which is just so darned pretty I > >>>> can't help it, has gotten me through the first challenge of switching to > >>>> HR, which is dealing with google search results containing keys into my > >>>> old app's data store. > > >>>> So now I'm looking at the big Kahuna problem of consistency. Here's my > >>>> first messy challenge there: > > >>>> My app puts a list of boards on the home page for a town, along with the > >>>> list of meetings. Generating that list of boards was taking a lot of > >>>> CPU, but they hardly ever change, so I put in a memcache system that > >>>> built the HTML when it wasn't in the cache, and then cached it before > >>>> serving. I clear the cache whenever the list of boards changes in some > >>>> way. > > >>>> Well that ain't gonna work in HR. It's quite possible that I update a > >>>> board, clear the cache, and someone comes and hits that page before > >>>> "eventually consistent" comes to pass. So now I've got a cached copy of > >>>> the stale data. > > >>>> (Note that I cannot use entity groups to solve this because some boards > >>>> are municipal agencies, and therefore cannot be parented to the town > >>>> that is building its list. I could parent all boards to some global > >>>> parent, but, well, yuck.) > > >>>> I have some different ideas about how to fix this, but I'm wondering if > >>>> anyone else who's done the port to HR has come up with a solution they > >>>> find particularly elegant? I assume this is a pretty common problem, so > >>>> there must be a design pattern out there… somewhere. > > >>>> Here are my ideas: > > >>>> - Clear the cache with a periodic task that re-clears it several times. > >>>> I'm thinking a recurring geometric retry would be prudent (1, 2, 4, 8, > >>>> 16, 32, 64, 128, 256, 512 seconds, and then pray that we have > >>>> consistency) > > >>>> - Checksum the modified or new board, and put that sum into memcache. > >>>> When generating the new board, confirm that any checksums are good. > >>>> This seems more deterministic, except I don't trust memache not to > >>>> squelch the checksum record. So perhaps I should do something in the > >>>> datastore. This feels like it's be about 10x as much code as the stupid > >>>> geometric flush. > > >>>> Any suggestions? > > >>>> -Joshua > > >>> -- > >>> You received this message because you are subscribed to the Google Groups > >>> "Google App Engine" group. > >>> To post to this group, send email to [email protected]. > >>> To unsubscribe from this group, send email to > >>> [email protected]. > >>> For more options, visit this group > >>> athttp://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
