Hi Folks, great discussion - I'm still unable to understand if cursors can be used to implement a clear cut pagination solution (assuming we encrypt the cursor key) and send it to the html page. As of now, the offset and the total number of results seems like the only way to go.
By pagination, I mean (next and previous) and that being available at any point: a' la http://www.google.com/search?hl=en&q=test+internet+speed&start=10&sa=N On Feb 10, 3:10 am, "Nick Johnson (Google)" <[email protected]> wrote: > Hi Nickolas, > > 2010/2/10 Nickolas Daskalou <[email protected]> > > > > > > > What Jeff suggests would also be great. So one page 1 we could do something > > like: > > > .... > > next_page_url = 'http://myblog.com/comments/?q=the&first=%d'% > > cursor.start_key().id() > > .... > > > and then reconstruct the cursor on page 2: > > > .... > > cursor = > > db.Cursor(start_key=db.Key('Comment',int(self.request.get('first'))), ....) > > .... > > > How about it Google? Will we be able to do this? > > The cursor format is internal, and it's not really amenable to being parsed > like this, since it will vary depending on the type of query you're > executing. > > -Nick Johnson > > > > > > > 2010/2/10 Jeff Schnitzer <[email protected]> > > > In case it wasn't completely clear - 1234 in this example is the > >> object's id, not an offset. > > >> Jeff > > >> On Tue, Feb 9, 2010 at 9:02 AM, Jeff Schnitzer <[email protected]> > >> wrote: > >> > Still, a slightly modified version of the original request does not > >> > seem unreasonable. He would have to formulate his URLs something like > >> > this: > > >> > myblog.com/comments/?q=the&first=1234 > > >> > or maybe: > > >> > myblog.com/comments/?q=the&after=1234 > > >> > I could see this being really useful, since encrypting (or worse, > >> > storing on the server) the cursor is pretty painful. Furthermore, it > >> > seems highly probable that as things are, many people will obliviously > >> > write public webapps that take a raw cursor as a parameter. This > >> > could be the new SQL injection attack. > > >> > Jeff > > >> > 2010/2/9 Alkis Evlogimenos ('Αλκης Ευλογημένος) <[email protected] > >> >: > >> >> If the cursor had to skip entries by using an offset, its performance > >> would > >> >> depend on the size of the offset. This is what the current > >> Query.fetch() api > >> >> is doing when you give it an offset. A cursor is a pointer to the entry > >> from > >> >> which the next query will start. It has no notion of offset. > >> >> On Tue, Feb 9, 2010 at 4:07 PM, Nickolas Daskalou <[email protected]> > >> wrote: > > >> >>> Does the production cursor string contain information about the app > >> id, > >> >>> kind, any filter()s or order()s, and (more importantly) some sort of > >> >>> numerical value that indicates how many records the next query should > >> >>> "skip"? If so, and if we could extract this information (and then use > >> it > >> >>> again to the reconstruct the cursor), that would make for much > >> cleaner, > >> >>> safer and intuitive URLs than including the entire cursor string (or > >> some > >> >>> sort of encrypted/encoded cursor string replacement). > > >> >>> 2010/2/10 Nick Johnson (Google) <[email protected]> > > >> >>>> Hi Nickolas, > > >> >>>> 2010/2/9 Nickolas Daskalou <[email protected]> > > >> >>>>> I'd want to do this so that I could include parts of the cursor > >> (such as > >> >>>>> the offset) into a URL without including other parts (eg. the model > >> kind and > >> >>>>> filters). I could then reconstruct the cursor on the server side > >> based on > >> >>>>> what was passed into the URL. > > >> >>>> The offset argument you're talking about is specific to the > >> >>>> dev_appserver's implementation of cursors. In production, offsets are > >> not > >> >>>> used, so this won't work. > >> >>>> -Nick Johnson > > >> >>>>> For example, if I was searching for blog comments that contained the > >> >>>>> word "the" (with the default order being the creation time, > >> descending), the > >> >>>>> URL might look like this: > > >> >>>>> myblog.com/comments/?q=the > > >> >>>>> With model: > > >> >>>>> class Comment(db.Model): > >> >>>>> .... > >> >>>>> created_at = db.DateTimeProperty(auto_now_add=True) > >> >>>>> words = db.StringListProperty() # A list of all the words in a > >> comment > >> >>>>> (forget about exploding indexes for now) > >> >>>>> ... > > >> >>>>> The query object for this URL might look something like: > > >> >>>>> .... > >> >>>>> q = > > >> Comment.all().filter('words',self.request.get('q')).order('-created_at') > >> >>>>> .... > > >> >>>>> To get to the 1001st comment, it'd be good if the URL looked > >> something > >> >>>>> like this: > > >> >>>>> myblog.com/comments/?q=the&skip=1000 > > >> >>>>> instead of: > > >> >>>>> myblog.com/comments/?q=the&cursor=[something ugly] > > >> >>>>> so that when the request comes in, I can do this: > > >> >>>>> .... > >> >>>>> q = > > >> Comment.all().filter('words',self.request.get('q')).order('-created_at') > >> >>>>> cursor_template = q.cursor_template() > >> >>>>> cursor = > > >> db.Cursor.from_template(cursor_template,offset=int(self.request.get('skip') > >> )) > >> >>>>> .... > >> >>>>> (or something along these lines) > > >> >>>>> Does that make sense? > > >> >>>>> On 10 February 2010 01:03, Nick Johnson (Google) > >> >>>>> <[email protected]> wrote: > > >> >>>>>> Hi Nickolas, > > >> >>>>>> 2010/2/9 Nickolas Daskalou <[email protected]> > > >> >>>>>>> Will we be able to construct our own cursors much the same way > >> that we > >> >>>>>>> are able to construct our own Datastore keys (Key.from_path())? > > >> >>>>>> No, not practically speaking. > > >> >>>>>>> Also along the same lines, will we be able to "deconstruct" a > >> cursor > >> >>>>>>> to get its components (offset, start_inclusive etc.), as we can > >> now do with > >> >>>>>>> keys (key.name(), key.id(), key.kind() etc.)? > > >> >>>>>> While you could do this, there's no guarantees that it'll work (or > >> >>>>>> continue to work), as you'd be digging into internal implementation > >> details. > >> >>>>>> Why do you want to do this? > >> >>>>>> -Nick Johnson > > >> >>>>>>> 2010/2/9 Nick Johnson (Google) <[email protected]> > > >> >>>>>>>> 2010/2/9 Stephen <[email protected]> > > >> >>>>>>>>> I'm asking if it's wise to store it as a query parameter > >> embedded in > >> >>>>>>>>> a > >> >>>>>>>>> web page. > > >> >>>>>>>> You're right that it's unwise. Depending on how you construct > >> your > >> >>>>>>>> query, a user could potentially modify the cursor they send to > >> you to return > >> >>>>>>>> results from any query your datastore is capable of performing, > >> which could > >> >>>>>>>> result in you revealing information to the user that they > >> shouldn't know. > >> >>>>>>>> You should either store the cursor on the server-side, or encrypt > >> it before > >> >>>>>>>> sending it to the client. > >> >>>>>>>> I was going to mention something about this in my post, but it > >> >>>>>>>> slipped my mind. > >> >>>>>>>> -Nick Johnson > > >> >>>>>>>>> On Feb 9, 12:26 am, "Ikai L (Google)" <[email protected]> > >> wrote: > >> >>>>>>>>> > A cursor serializes to a Base64 encoded String, so you can > >> store > >> >>>>>>>>> > it anywhere > >> >>>>>>>>> > you want to store strings: Memcached, Datastore, etc. You can > >> even > >> >>>>>>>>> > pass it > >> >>>>>>>>> > as an URL parameter to task queues. > > >> >>>>>>>>> > 2010/2/8 Stephen <[email protected]> > > >> >>>>>>>>> > > Ah right, Nick's blog does say start_key and not offset. My > >> bad. > > >> >>>>>>>>> > > Maybe there will be warnings in the upcoming documentation, > >> but > >> >>>>>>>>> > > my > >> >>>>>>>>> > > first instinct was to embed the serialised cursor in the > >> HTML as > >> >>>>>>>>> > > the > >> >>>>>>>>> > > 'next' link. But that doesn't look like a good idea as > >> Nick's > >> >>>>>>>>> > > decoded > >> >>>>>>>>> > > query shows what's embedded: > > >> >>>>>>>>> > > PrimaryScan { > >> >>>>>>>>> > > start_key: "shell\000TestModel\000foo\000\232bar\000\200" > >> >>>>>>>>> > > start_inclusive: true > >> >>>>>>>>> > > } > >> >>>>>>>>> > > keys_only: false > > >> >>>>>>>>> > > First, you may or may not want to leak this info. Second, > >> could > >> >>>>>>>>> > > this > >> >>>>>>>>> > > be altered on the client to change the query in any way > >> that's > >> >>>>>>>>> > > undesirable? > > >> >>>>>>>>> > > Once you have a cursor, where do you store it so you can use > >> it > >> >>>>>>>>> > > again? > > >> >>>>>>>>> > > On Feb 8, 10:17 pm, "Ikai L (Google)" <[email protected]> > >> wrote: > >> >>>>>>>>> > > > I got beaten to this answer. No, there is no traversal to > >> get > >> >>>>>>>>> > > > to the > >> >>>>>>>>> > > offset. > > >> >>>>>>>>> > > > BigTable has an underlying mechanism for range queries on > >> >>>>>>>>> > > > keys. Indexes > >> >>>>>>>>> > > are > >> >>>>>>>>> > > > essentially a key comprised of a concatenation of > >> application > >> >>>>>>>>> > > > ID, entity > >> >>>>>>>>> > > > type, column, value. When a filter operation is performed, > >> the > >> >>>>>>>>> > > > datastore > >> >>>>>>>>> > > > looks for a range matching this criteria, returning the > >> set of > >> >>>>>>>>> > > > keys. A > >> >>>>>>>>> > > > cursor also adds the datastore key of the entity so it is > >> >>>>>>>>> > > > possible to > >> >>>>>>>>> > > > serialize where to begin the query. This is actually a bit > >> >>>>>>>>> > > > awkward to > >> >>>>>>>>> > > > explain without visuals. You can watch Ryan Barrett's talk > >> >>>>>>>>> > > > here: > > >> >>>>>>>>> > > >http://www.youtube.com/watch?v=tx5gdoNpcZM > > >> >>>>>>>>> > > > Hopefully, we'll be able to post an article at some point > >> in > >> >>>>>>>>> > > > the future > >> >>>>>>>>> > > > explaining how cursors work. > > >> >>>>>>>>> > > > 2010/2/8 Alkis Evlogimenos ('Αλκης Ευλογημένος) > >> >>>>>>>>> > > > <[email protected]> > > >> >>>>>>>>> > > > > There is no offset. The protocol buffer stores a > >> start_key > >> >>>>>>>>> > > > > and a > >> >>>>>>>>> > > boolean > >> >>>>>>>>> > > > > denoting if this start key is inclusive or not. The > >> >>>>>>>>> > > > > performance of > >> >>>>>>>>> > > > > continuing the fetch from a cursor should be the same as > >> the > >> >>>>>>>>> > > performance of > >> >>>>>>>>> > > > > the first entities you got > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
