"We get a list of Keys back and figure out keys." I just realized how confusing this is - we get a list of INDEX keys back. These don't have any value; we do all storage inside the Key. We extract the keys of the Entity from the returned Index keys and do a query.
On Tue, Apr 6, 2010 at 11:42 AM, Ikai L (Google) <[email protected]> wrote: > Offset doesn't fetch data - it does an index scan. Going to the 10,000th > result using offset will require us to pass 10,000 results first in our > index, but we won't be retrieving those objects. Here's a bit of a > simplification of indexes, entities and how these queries work. As you know, > Bigtable is a key-value store with the ability to perform range scans. > Suppose you have a Person entity. Your entries may look like this: > > Key > Value > > -------------------------------------------------------------------------------------------------------------- > APPID:Person:1 { > name: Ikai, favorite_food: donuts } > APPID:Person:2 { name: > Wesley, favorite_food: tacos } > Index:APPID:Person:favorite_food:ASC:donuts:1 [ empty ] > Index:APPID:Person:favorite_food:ASC:tacos:2 [ empty ] > > *order of values in keys may not match how we actually do it, it's just to > illustrate my point > ** argh, non-monospaced fonts. Sorry for the hideous table > > When we look for Person entities sorted by favorite_food, we essentially > ask Bigtable to return all Keys that match: > > Index:APPID:Person:favorite_food:ASC* > > We get a list of Keys back and figure out keys. Using offsets, we would > just pass the first N entries per the supplied offset. > > Cursors work by serializing the query position via the last index, allow us > to go straight to the index in constant time and continue the range query. > > We've got a lot of content about how this works, though we probably haven't > done the best job organizing it. I'd start here: > http://code.google.com/appengine/articles/storage_breakdown.html > > I've been meaning to collect all of our talks and place them into a set of > YouTube playlists. Why don't I do that right now ... > > On Sat, Apr 3, 2010 at 8:16 AM, John Patterson <[email protected]>wrote: > >> How many pages do your users really want to see? Even Google search sets >> a limit - I think 1000 results. Although the number of results can exceed >> 1000 the offset is still limited so you would need to filter out results to >> continue past 1K. That would be slow. >> >> If you really need to return more than 1000 results and access them in >> forward and backward directions you could sort them by __key__ or some other >> unique combination and use that as you own "cursor". You could define both >> ascending and descending indexes on your chosen property(s) to let you >> iterate in both directions. >> >> >> On 3 Apr 2010, at 19:32, Arny wrote: >> >> But isn't it getting slower and slower on higher pages, since it >>> fetches ALL data (according to docs) and discards the offset value? >>> So Range(0,10) is faster than Range(10000,10) ? >>> >>> Anyone did some performance tests? >>> Regards >>> >>> On Apr 3, 7:49 am, John Patterson <[email protected]> wrote: >>> >>>> Probably you should set an offset and limit instead for your paging. >>>> I believe cursors are really intended for processing a lot of data off- >>>> line. Remember there is no longer a 1000 result limit on fetches. >>>> >>>> On 3 Apr 2010, at 02:57, Arny wrote: >>>> >>>> Hi, >>>>> >>>> >>>> Is there a way to get a cursor to page back to a previous page? >>>>> If not, whats the point of paging forward only? I'm not building an >>>>> ajax page. >>>>> >>>> >>>> Regards >>>>> >>>> >>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine for Java" group. >>>>> To post to this group, send email to >>>>> [email protected] >>>>> . >>>>> To unsubscribe from this group, send email to >>>>> [email protected]<google-appengine-java%[email protected]> >>>>> . >>>>> For more options, visit this group athttp:// >>>>> groups.google.com/group/google-appengine-java?hl=en >>>>> . >>>>> >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine for Java" group. >>> To post to this group, send email to >>> [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]<google-appengine-java%[email protected]> >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine-java?hl=en. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine for Java" group. >> To post to this group, send email to >> [email protected]. >> To unsubscribe from this group, send email to >> [email protected]<google-appengine-java%[email protected]> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine-java?hl=en. >> >> > > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > http://googleappengine.blogspot.com | http://twitter.com/app_engine > -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
