They are faster - use less system resources - but I do not see an effective way to paginate using the current implementation of cursors which are "forward only" unless YOU store the cursor . Given that cursors are "opaque", there is no guarantee if that will work over time.
2010/2/16 Waldemar Kornewald <[email protected]> > Hi, > what's the advantage of cursors compared to key-based pagination? The > latter at least allows for paginating backwards from any point. Why > don't cursors just build on the same principle? > > Bye, > Waldemar Kornewald > > On Feb 16, 5:46 pm, "Nick Johnson (Google)" <[email protected]> > wrote: > > Hi Andy, > > > > 2010/2/16 Andy Freeman <[email protected]> > > > > > > Furthermore, it > > > > seems highly probable that as things are, many people will > obliviously > > > > write public webapps that take a raw cursor as a parameter. This > > > > could be the new SQL injection attack. > > > > > Can you comment a bit more on the security issues? > > > > > AFAIK, cursors can not be used to write anything. The cursor still > > > has to match the query with its parameters, so I don't see how they > > > can synthesize a cursor to see anything that they haven't already seen > > > (replay) or that they'd see by requesting more and more pages (skip > > > ahead). > > > > I was mistaken when I stated that they shouldn't be sent to the user in > the > > clear. As you point out, in order to use a cursor, you still have to > > reconstruct the original query, so a user could not modify a cursor to > cause > > you to display records they should not have access to. > > > > > The cursor may, as part of its "is this the right query" content, > > > reveal something about the query. > > > > > Hmm - the latter seems somewhat serious. It isn't data modification, > > > but it is a data reveal. > > > > This is true, though I wouldn't personally consider it a serious issue. > That > > is up to you, naturally. > > > > > > > > > What information can someone extract from a production cursor? Does > > > it contain the parameters (bad) or signatures (okay if someone can't > > > derive one parameter given the other parameters). > > > > It contains the complete key of the next record to be returned, along > with > > some extra information about the query. Feel free to experiment and see > for > > yourself, of course. :) > > > > -Nick Johnson > > > > > > > > > -andy > > > > > On Feb 9, 9:02 am, Jeff Schnitzer <[email protected]> wrote: > > > > Still, a slightly modified version of the original request does not > > > > seem unreasonable. He would have to formulate his URLs something > like > > > > this: > > > > > > myblog.com/comments/?q=the&first=1234 > > > > > > or maybe: > > > > > > myblog.com/comments/?q=the&after=1234 > > > > > > I could see this being really useful, since encrypting (or worse, > > > > storing on the server) the cursor is pretty painful. Furthermore, it > > > > seems highly probable that as things are, many people will > obliviously > > > > write public webapps that take a raw cursor as a parameter. This > > > > could be the new SQL injection attack. > > > > > > Jeff > > > > > > 2010/2/9 Alkis Evlogimenos ('Αλκης Ευλογημένος) < > [email protected]>:> > > > If the cursor had to skip entries by using an offset, its performance > would > > > > > depend on the size of the offset. This is what the current > > > Query.fetch() api > > > > > is doing when you give it an offset. A cursor is a pointer to the > entry > > > from > > > > > which the next query will start. It has no notion of offset. > > > > > On Tue, Feb 9, 2010 at 4:07 PM, Nickolas Daskalou < > [email protected]> > > > wrote: > > > > > > >> Does the production cursor string contain information about the > app > > > id, > > > > >> kind, any filter()s or order()s, and (more importantly) some sort > of > > > > >> numerical value that indicates how many records the next query > should > > > > >> "skip"? If so, and if we could extract this information (and then > use > > > it > > > > >> again to the reconstruct the cursor), that would make for much > > > cleaner, > > > > >> safer and intuitive URLs than including the entire cursor string > (or > > > some > > > > >> sort of encrypted/encoded cursor string replacement). > > > > > > >> 2010/2/10 Nick Johnson (Google) <[email protected]> > > > > > > >>> Hi Nickolas, > > > > > > >>> 2010/2/9 Nickolas Daskalou <[email protected]> > > > > > > >>>> I'd want to do this so that I could include parts of the cursor > > > (such as > > > > >>>> the offset) into a URL without including other parts (eg. the > model > > > kind and > > > > >>>> filters). I could then reconstruct the cursor on the server side > > > based on > > > > >>>> what was passed into the URL. > > > > > > >>> The offset argument you're talking about is specific to the > > > > >>> dev_appserver's implementation of cursors. In production, offsets > are > > > not > > > > >>> used, so this won't work. > > > > >>> -Nick Johnson > > > > > > >>>> For example, if I was searching for blog comments that contained > the > > > > >>>> word "the" (with the default order being the creation time, > > > descending), the > > > > >>>> URL might look like this: > > > > > > >>>> myblog.com/comments/?q=the > > > > > > >>>> With model: > > > > > > >>>> class Comment(db.Model): > > > > >>>> .... > > > > >>>> created_at = db.DateTimeProperty(auto_now_add=True) > > > > >>>> words = db.StringListProperty() # A list of all the words in a > > > comment > > > > >>>> (forget about exploding indexes for now) > > > > >>>> ... > > > > > > >>>> The query object for this URL might look something like: > > > > > > >>>> .... > > > > >>>> q = > > > > > > Comment.all().filter('words',self.request.get('q')).order('-created_at') > > > > >>>> .... > > > > > > >>>> To get to the 1001st comment, it'd be good if the URL looked > > > something > > > > >>>> like this: > > > > > > >>>> myblog.com/comments/?q=the&skip=1000 > > > > > > >>>> instead of: > > > > > > >>>> myblog.com/comments/?q=the&cursor=[something ugly] > > > > > > >>>> so that when the request comes in, I can do this: > > > > > > >>>> .... > > > > >>>> q = > > > > > > Comment.all().filter('words',self.request.get('q')).order('-created_at') > > > > >>>> cursor_template = q.cursor_template() > > > > >>>> cursor = > > > > > > db.Cursor.from_template(cursor_template,offset=int(self.request.get('skip') > )) > > > > >>>> .... > > > > >>>> (or something along these lines) > > > > > > >>>> Does that make sense? > > > > > > >>>> On 10 February 2010 01:03, Nick Johnson (Google) > > > > >>>> <[email protected]> wrote: > > > > > > >>>>> Hi Nickolas, > > > > > > >>>>> 2010/2/9 Nickolas Daskalou <[email protected]> > > > > > > >>>>>> Will we be able to construct our own cursors much the same way > > > that we > > > > >>>>>> are able to construct our own Datastore keys > (Key.from_path())? > > > > > > >>>>> No, not practically speaking. > > > > > > >>>>>> Also along the same lines, will we be able to "deconstruct" a > > > cursor > > > > >>>>>> to get its components (offset, start_inclusive etc.), as we > can > > > now do with > > > > >>>>>> keys (key.name(), key.id(), key.kind() etc.)? > > > > > > >>>>> While you could do this, there's no guarantees that it'll work > (or > > > > >>>>> continue to work), as you'd be digging into internal > implementation > > > details. > > > > >>>>> Why do you want to do this? > > > > >>>>> -Nick Johnson > > > > > > >>>>>> 2010/2/9 Nick Johnson (Google) <[email protected]> > > > > > > >>>>>>> 2010/2/9 Stephen <[email protected]> > > > > > > >>>>>>>> I'm asking if it's wise to store it as a query parameter > > > embedded in > > > > >>>>>>>> a > > > > >>>>>>>> web page. > > > > > > >>>>>>> You're right that it's unwise. Depending on how you construct > > > your > > > > >>>>>>> query, a user could potentially modify the cursor they send > to > > > you to return > > > > >>>>>>> results from any query your datastore is capable of > performing, > > > which could > > > > >>>>>>> result in you revealing information to the user that they > > > shouldn't know. > > > > >>>>>>> You should either store the cursor on the server-side, or > encrypt > > > it before > > > > >>>>>>> sending it to the client. > > > > >>>>>>> I was going to mention something about this in my post, but > it > > > > >>>>>>> slipped my mind. > > > > >>>>>>> -Nick Johnson > > > > > > >>>>>>>> On Feb 9, 12:26 am, "Ikai L (Google)" <[email protected]> > > > wrote: > > > > >>>>>>>> > A cursor serializes to a Base64 encoded String, so you can > > > store > > > > >>>>>>>> > it anywhere > > > > >>>>>>>> > you want to store strings: Memcached, Datastore, etc. You > can > > > even > > > > >>>>>>>> > pass it > > > > >>>>>>>> > as an URL parameter to task queues. > > > > > > >>>>>>>> > 2010/2/8 Stephen <[email protected]> > > > > > > >>>>>>>> > > Ah right, Nick's blog does say start_key and not offset. > My > > > bad. > > > > > > >>>>>>>> > > Maybe there will be warnings in the upcoming > documentation, > > > but > > > > >>>>>>>> > > my > > > > >>>>>>>> > > first instinct was to embed the serialised cursor in the > > > HTML as > > > > >>>>>>>> > > the > > > > >>>>>>>> > > 'next' link. But that doesn't look like a good idea as > > > Nick's > > > > >>>>>>>> > > decoded > > > > >>>>>>>> > > query shows what's embedded: > > > > > > >>>>>>>> > > PrimaryScan { > > > > >>>>>>>> > > start_key: > "shell\000TestModel\000foo\000\232bar\000\200" > > > > >>>>>>>> > > start_inclusive: true > > > > >>>>>>>> > > } > > > > >>>>>>>> > > keys_only: false > > > > > > >>>>>>>> > > First, you may or may not want to leak this info. > Second, > > > could > > > > >>>>>>>> > > this > > > > >>>>>>>> > > be altered on the client to change the query in any way > > > that's > > > > >>>>>>>> > > undesirable? > > > > > > >>>>>>>> > > Once you have a cursor, where do you store it so you can > use > > > it > > > > >>>>>>>> > > again? > > > > > > >>>>>>>> > > On Feb 8, 10:17 pm, "Ikai L (Google)" < > [email protected]> > > > wrote: > > > > >>>>>>>> > > > I got beaten to this answer. No, there is no traversal > to > > > get > > > > >>>>>>>> > > > to the > > > > >>>>>>>> > > offset. > > > > > > >>>>>>>> > > > BigTable has an underlying mechanism for range queries > on > > > > >>>>>>>> > > > keys. Indexes > > > > >>>>>>>> > > are > > > > >>>>>>>> > > > essentially a key comprised of a concatenation of > > > application > > > > >>>>>>>> > > > ID, entity > > > > >>>>>>>> > > > type, column, value. When a filter operation is > performed, > > > the > > > > >>>>>>>> > > > datastore > > > > >>>>>>>> > > > looks for a range matching this criteria, returning > the > > > set of > > > > >>>>>>>> > > > keys. A > > > > >>>>>>>> > > > cursor also adds the datastore key of the entity so it > is > > > > >>>>>>>> > > > possible to > > > > >>>>>>>> > > > serialize where to begin the query. This is actually a > bit > > > > >>>>>>>> > > > awkward to > > > > >>>>>>>> > > > explain without visuals. You can watch Ryan Barrett's > talk > > > > >>>>>>>> > > > here: > > > > > > >>>>>>>> > > >http://www.youtube.com/watch?v=tx5gdoNpcZM > > > > > > >>>>>>>> > > > Hopefully, we'll be able to post an article at some > point > > > in > > > > >>>>>>>> > > > the future > > > > >>>>>>>> > > > explaining how cursors work. > > > > > > >>>>>>>> > > > 2010/2/8 Alkis Evlogimenos ('Αλκης Ευλογημένος) > > > > >>>>>>>> > > > <[email protected]> > > > > > > >>>>>>>> > > > > There is no offset. The protocol buffer stores a > > > start_key... > > > > read more » > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
