Sorry, I wasn't particularly clear. We're doing a rewrite of the db module:
http://code.google.com/p/appengine-ndb-experiment/ Hopefully we'll find that we can build it better. If you have any feedback for how db should look, that's probably the best place to start. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Wed, Sep 7, 2011 at 6:02 PM, Joshua Smith <[email protected]>wrote: > > > On Sep 7, 2011, at 8:38 PM, Ikai Lan (Google) wrote: > > We're rewriting the db module in ndb. > > > I don't know what this sentence means. > > Wrapping a get around keys only queries does not guarantee up to date > information. The indexes may not be up to date. When you do a get on the > keys themselves, those gets should be up to date, but because the indexes > might be stale, it's possible you get back bad data. Suppose: > > 1. You write a Person with a name "ikai" > 2. Ikai changes his name to "superman" > 3. You query for everyone whose name is "ikai" > 4. You get back a Person whose name is "superman" (get by key is > transactional and always returns the latest data) > > > Yes, this is not a panacea. As I said, "it doesn't fix the index." > > But the most frequent place I'm seeing an issue with HR so far is when a > user has a list of things, they click "Edit" next to one of those things, > change it, hit submit, and their change is not reflected in the list. With > this quick fix, what might happen is: > > - Everything looks exactly as expected, or > - Their change results in a record not being display in the right order in > the list for a moment (no big deal) > > When I add a new record, the pattern I've been using is to add it with > blank fields, and then open the editor on it. (And delete it if they just > click "Cancel".) So that means my list-of-all will certainly contain the > new record, since they spent some time editing it. So the fix works for > this, too. > > In the case of deleting a record, I can end up with a None in the list, but > there's a quick fix for that: > > @classmethod > def gql_with_get(cls, query_string, *args, **kwds): > return filter(None, db.get(db.GqlQuery('SELECT __key__ FROM %s %s' % > (cls.kind(), query_string), *args, **kwds))) > > > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > On Wed, Sep 7, 2011 at 5:28 PM, Joshua Smith <[email protected]>wrote: > >> Continuing the dialog with myself :) >> >> I've added this method to one of my classes that extends db.Model() and it >> is working well with the dev appserver in --high_replication mode: >> >> @classmethod >> def gql_with_get(cls, query_string, *args, **kwds): >> return db.get(db.GqlQuery('SELECT __key__ FROM %s %s' % (cls.kind(), >> query_string), *args, **kwds)) >> >> You use it just like gql().fetch(). For example: >> >> boards = BoardModel.gql_with_get("WHERE towns = :1 ORDER BY name", tid) >> >> It doesn't fix the index (things might be out of order, for instance), but >> otherwise, it cures the problem of seeing stale data in HR. >> >> On Sep 7, 2011, at 12:22 PM, Joshua Smith wrote: >> >> > Another thought: The reason I was doing only one meeting per request was >> because of the old 30 second limit on crons. But cron handlers can be 10 >> minutes now, which is plenty of time to schedule all the meetings. >> Therefore, I suppose I could do this, right? >> > >> > now = datetime.datetime.now() >> > for schedule in db.get(db.gql("SELECT __key__ FROM ScheduleModel WHERE >> next != :1 AND next < :2", None, now)): >> > if schedule.next and schedule.next < now: >> > schedule.cronAuto() >> > >> > Is wrapping a GET around a KEYS-ONLY query guaranteed to get me the >> real-deal results (except, of course, for the fact that the index might be >> out-of-date, so I might miss recent changes to who is in/out of the query >> parameters)? Is this an efficient way to express this, or should I be doing >> a fetch() on the gql first? >> > >> > It seems like it's possible to use a technique like this to get a >> more-consistent result in cases where that's desirable. It at least would >> get you a consistent data for a subset of things matching your query. In >> principle, you could even re-sort the results if there is an ORDER clause. >> Seems like this would be something useful in the db API... >> > >> > -Joshua >> > >> > On Sep 7, 2011, at 11:18 AM, Joshua Smith wrote: >> > >> >> >> >> I'm trying to port my existing M/S app to HR because I have a gun to my >> head with "Threaded Python Only for HR Apps" written on the bullets. >> >> >> >> My system will schedule meetings automatically. Scheduling a meeting >> can take some time, because a bunch of records are created, and a bunch of >> emails need to go out. So the code to schedule one looked like this: >> >> >> >> class MeetingAutoHandler(webapp.RequestHandler): >> >> def get(self): >> >> schedule = ScheduleModel.gql("WHERE next != :1 AND next < :2", None, >> datetime.datetime.now()).get() >> >> if schedule: >> >> schedule.cronAuto() >> >> taskqueue.add(url='/admin/meetingAuto', method='GET', countdown=1) >> >> >> >> The query looks for a schedule object that needs a meeting to to be >> scheduled now. There might be a few of these when the cron runs. So it >> does the hard work for one of them (in cronAuto()), and schedules another >> call to itself to get the next one using the task queue. >> >> >> >> This isn't going to work in HR because that query is going to keep >> finding the same meeting. I could trivially tweak this by setting the >> countdown=60, but I've yet to hear any of our google overlords commit to a >> maximum value of when "eventually" happens in "eventually consistent". I >> presume there might be cases, like during data center transitions, when >> "eventually" could be a very long time indeed. It is essentially unbounded. >> Right? >> >> >> >> But I like the pattern I'm using here, and I'm trying to change as >> little code as possible, so I want to put together a HR-resilient version. >> Here's what I came up with: >> >> >> >> class MeetingAutoHandler(webapp.RequestHandler): >> >> def get(self): >> >> now = datetime.datetime.now() >> >> for s in db.gql("SELECT __key__ FROM ScheduleModel WHERE next != :1 >> AND next < :2", None, now): >> >> schedule = db.get(s) >> >> if schedule.next and schedule.next < now: >> >> schedule.cronAuto() >> >> taskqueue.add(url='/admin/meetingAuto', method='GET', countdown=5) >> >> return >> >> >> >> So I'm doing a keys-only query and then doing a get() on the key. >> (I've never done a keys-only GQL query before, but I think I got it right. >> Note to google: There should be an option to Model.gql() to do keys-only >> queries!) >> >> >> >> The way I understand HR, that get is going to get the real Model, which >> might not meet the criteria in the gql, because the index might be out of >> date. Right? >> >> >> >> So I check that the model meets the criteria that I just specified. >> (Note to google: It'd be cool if there was a way to test an object against >> a query, so I don't have to write the same code twice!) >> >> >> >> Finally, I pushed the next task out a bit, to make it less likely that >> I'll have to look at the same objects over and over. >> >> >> >> So what do you think? Any suggestions? (I have a couple things that >> work this way, so I want to choose a good design pattern to apply to each of >> them.) >> >> >> >> The complexity would be lessened if I could to this: >> >> >> >> class MeetingAutoHandler(webapp.RequestHandler): >> >> def get(self): >> >> q = ScheduleModel.gql_keys_only("WHERE next != :1 AND next < :2", >> None, datetime.datetime.now()) >> >> for s in q: >> >> schedule = db.get(s) >> >> if q.matches(schedule): >> >> schedule.cronAuto() >> >> taskqueue.add(url='/admin/meetingAuto', method='GET', countdown=5) >> >> return >> >> >> >> This would require two changes: the db.Model would need to support >> gql_keys_only (that's probably trivial); GqlQuery would need a matches() >> method (that's probably not trivial). >> >> >> >> It's still a few more lines, but the complexity is about the same as >> the old one. >> >> >> >> Worth the trouble of a couple feature request issues? >> >> >> >> -Joshua >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> Groups "Google App Engine" group. >> >> To post to this group, send email to [email protected] >> . >> >> To unsubscribe from this group, send email to >> [email protected]. >> >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> [email protected]. >> > For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
