In case it wasn't completely clear - 1234 in this example is the object's id, not an offset.
Jeff On Tue, Feb 9, 2010 at 9:02 AM, Jeff Schnitzer <[email protected]> wrote: > Still, a slightly modified version of the original request does not > seem unreasonable. He would have to formulate his URLs something like > this: > > myblog.com/comments/?q=the&first=1234 > > or maybe: > > myblog.com/comments/?q=the&after=1234 > > I could see this being really useful, since encrypting (or worse, > storing on the server) the cursor is pretty painful. Furthermore, it > seems highly probable that as things are, many people will obliviously > write public webapps that take a raw cursor as a parameter. This > could be the new SQL injection attack. > > Jeff > > 2010/2/9 Alkis Evlogimenos ('Αλκης Ευλογημένος) <[email protected]>: >> If the cursor had to skip entries by using an offset, its performance would >> depend on the size of the offset. This is what the current Query.fetch() api >> is doing when you give it an offset. A cursor is a pointer to the entry from >> which the next query will start. It has no notion of offset. >> On Tue, Feb 9, 2010 at 4:07 PM, Nickolas Daskalou <[email protected]> wrote: >>> >>> Does the production cursor string contain information about the app id, >>> kind, any filter()s or order()s, and (more importantly) some sort of >>> numerical value that indicates how many records the next query should >>> "skip"? If so, and if we could extract this information (and then use it >>> again to the reconstruct the cursor), that would make for much cleaner, >>> safer and intuitive URLs than including the entire cursor string (or some >>> sort of encrypted/encoded cursor string replacement). >>> >>> >>> 2010/2/10 Nick Johnson (Google) <[email protected]> >>>> >>>> Hi Nickolas, >>>> >>>> 2010/2/9 Nickolas Daskalou <[email protected]> >>>>> >>>>> I'd want to do this so that I could include parts of the cursor (such as >>>>> the offset) into a URL without including other parts (eg. the model kind >>>>> and >>>>> filters). I could then reconstruct the cursor on the server side based on >>>>> what was passed into the URL. >>>> >>>> The offset argument you're talking about is specific to the >>>> dev_appserver's implementation of cursors. In production, offsets are not >>>> used, so this won't work. >>>> -Nick Johnson >>>> >>>>> >>>>> For example, if I was searching for blog comments that contained the >>>>> word "the" (with the default order being the creation time, descending), >>>>> the >>>>> URL might look like this: >>>>> >>>>> myblog.com/comments/?q=the >>>>> >>>>> With model: >>>>> >>>>> class Comment(db.Model): >>>>> .... >>>>> created_at = db.DateTimeProperty(auto_now_add=True) >>>>> words = db.StringListProperty() # A list of all the words in a comment >>>>> (forget about exploding indexes for now) >>>>> ... >>>>> >>>>> The query object for this URL might look something like: >>>>> >>>>> .... >>>>> q = >>>>> Comment.all().filter('words',self.request.get('q')).order('-created_at') >>>>> .... >>>>> >>>>> To get to the 1001st comment, it'd be good if the URL looked something >>>>> like this: >>>>> >>>>> myblog.com/comments/?q=the&skip=1000 >>>>> >>>>> instead of: >>>>> >>>>> myblog.com/comments/?q=the&cursor=[something ugly] >>>>> >>>>> so that when the request comes in, I can do this: >>>>> >>>>> .... >>>>> q = >>>>> Comment.all().filter('words',self.request.get('q')).order('-created_at') >>>>> cursor_template = q.cursor_template() >>>>> cursor = >>>>> db.Cursor.from_template(cursor_template,offset=int(self.request.get('skip'))) >>>>> .... >>>>> (or something along these lines) >>>>> >>>>> Does that make sense? >>>>> >>>>> >>>>> On 10 February 2010 01:03, Nick Johnson (Google) >>>>> <[email protected]> wrote: >>>>>> >>>>>> Hi Nickolas, >>>>>> >>>>>> 2010/2/9 Nickolas Daskalou <[email protected]> >>>>>>> >>>>>>> Will we be able to construct our own cursors much the same way that we >>>>>>> are able to construct our own Datastore keys (Key.from_path())? >>>>>> >>>>>> No, not practically speaking. >>>>>> >>>>>>> >>>>>>> Also along the same lines, will we be able to "deconstruct" a cursor >>>>>>> to get its components (offset, start_inclusive etc.), as we can now do >>>>>>> with >>>>>>> keys (key.name(), key.id(), key.kind() etc.)? >>>>>> >>>>>> While you could do this, there's no guarantees that it'll work (or >>>>>> continue to work), as you'd be digging into internal implementation >>>>>> details. >>>>>> Why do you want to do this? >>>>>> -Nick Johnson >>>>>>> >>>>>>> >>>>>>> 2010/2/9 Nick Johnson (Google) <[email protected]> >>>>>>>> >>>>>>>> 2010/2/9 Stephen <[email protected]> >>>>>>>>> >>>>>>>>> I'm asking if it's wise to store it as a query parameter embedded in >>>>>>>>> a >>>>>>>>> web page. >>>>>>>> >>>>>>>> You're right that it's unwise. Depending on how you construct your >>>>>>>> query, a user could potentially modify the cursor they send to you to >>>>>>>> return >>>>>>>> results from any query your datastore is capable of performing, which >>>>>>>> could >>>>>>>> result in you revealing information to the user that they shouldn't >>>>>>>> know. >>>>>>>> You should either store the cursor on the server-side, or encrypt it >>>>>>>> before >>>>>>>> sending it to the client. >>>>>>>> I was going to mention something about this in my post, but it >>>>>>>> slipped my mind. >>>>>>>> -Nick Johnson >>>>>>>>> >>>>>>>>> On Feb 9, 12:26 am, "Ikai L (Google)" <[email protected]> wrote: >>>>>>>>> > A cursor serializes to a Base64 encoded String, so you can store >>>>>>>>> > it anywhere >>>>>>>>> > you want to store strings: Memcached, Datastore, etc. You can even >>>>>>>>> > pass it >>>>>>>>> > as an URL parameter to task queues. >>>>>>>>> > >>>>>>>>> > 2010/2/8 Stephen <[email protected]> >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > > Ah right, Nick's blog does say start_key and not offset. My bad. >>>>>>>>> > >>>>>>>>> > > Maybe there will be warnings in the upcoming documentation, but >>>>>>>>> > > my >>>>>>>>> > > first instinct was to embed the serialised cursor in the HTML as >>>>>>>>> > > the >>>>>>>>> > > 'next' link. But that doesn't look like a good idea as Nick's >>>>>>>>> > > decoded >>>>>>>>> > > query shows what's embedded: >>>>>>>>> > >>>>>>>>> > > PrimaryScan { >>>>>>>>> > > start_key: "shell\000TestModel\000foo\000\232bar\000\200" >>>>>>>>> > > start_inclusive: true >>>>>>>>> > > } >>>>>>>>> > > keys_only: false >>>>>>>>> > >>>>>>>>> > > First, you may or may not want to leak this info. Second, could >>>>>>>>> > > this >>>>>>>>> > > be altered on the client to change the query in any way that's >>>>>>>>> > > undesirable? >>>>>>>>> > >>>>>>>>> > > Once you have a cursor, where do you store it so you can use it >>>>>>>>> > > again? >>>>>>>>> > >>>>>>>>> > > On Feb 8, 10:17 pm, "Ikai L (Google)" <[email protected]> wrote: >>>>>>>>> > > > I got beaten to this answer. No, there is no traversal to get >>>>>>>>> > > > to the >>>>>>>>> > > offset. >>>>>>>>> > >>>>>>>>> > > > BigTable has an underlying mechanism for range queries on >>>>>>>>> > > > keys. Indexes >>>>>>>>> > > are >>>>>>>>> > > > essentially a key comprised of a concatenation of application >>>>>>>>> > > > ID, entity >>>>>>>>> > > > type, column, value. When a filter operation is performed, the >>>>>>>>> > > > datastore >>>>>>>>> > > > looks for a range matching this criteria, returning the set of >>>>>>>>> > > > keys. A >>>>>>>>> > > > cursor also adds the datastore key of the entity so it is >>>>>>>>> > > > possible to >>>>>>>>> > > > serialize where to begin the query. This is actually a bit >>>>>>>>> > > > awkward to >>>>>>>>> > > > explain without visuals. You can watch Ryan Barrett's talk >>>>>>>>> > > > here: >>>>>>>>> > >>>>>>>>> > > >http://www.youtube.com/watch?v=tx5gdoNpcZM >>>>>>>>> > >>>>>>>>> > > > Hopefully, we'll be able to post an article at some point in >>>>>>>>> > > > the future >>>>>>>>> > > > explaining how cursors work. >>>>>>>>> > >>>>>>>>> > > > 2010/2/8 Alkis Evlogimenos ('Αλκης Ευλογημένος) >>>>>>>>> > > > <[email protected]> >>>>>>>>> > >>>>>>>>> > > > > There is no offset. The protocol buffer stores a start_key >>>>>>>>> > > > > and a >>>>>>>>> > > boolean >>>>>>>>> > > > > denoting if this start key is inclusive or not. The >>>>>>>>> > > > > performance of >>>>>>>>> > > > > continuing the fetch from a cursor should be the same as the >>>>>>>>> > > performance of >>>>>>>>> > > > > the first entities you got from a query. >>>>>>>>> > >>>>>>>>> > > > > On Mon, Feb 8, 2010 at 4:33 PM, Stephen <[email protected]> >>>>>>>>> > > > > wrote: >>>>>>>>> > >>>>>>>>> > > > >> On Feb 8, 7:06 pm, "Ikai L (Google)" <[email protected]> >>>>>>>>> > > > >> wrote: >>>>>>>>> > > > >> > The official docs are pending, but here's Nick Johnson to >>>>>>>>> > > > >> > the >>>>>>>>> > > rescue: >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > > >http://blog.notdot.net/2010/02/New-features-in-1-3-1-prerelease-Cursors >>>>>>>>> > >>>>>>>>> > > > >> What are the performance characteristics of cursors? >>>>>>>>> > >>>>>>>>> > > > >> The serialised cursor shows that it stores an offset. Does >>>>>>>>> > > > >> that mean >>>>>>>>> > > > >> that if the offset is one million, one million rows will >>>>>>>>> > > > >> have to be >>>>>>>>> > > > >> skipped before the next 10 are returned? This will be >>>>>>>>> > > > >> faster than >>>>>>>>> > > > >> doing it in your app, but not as quick as the existing >>>>>>>>> > > > >> bookmark >>>>>>>>> > > > >> techniques which use the primary key index. >>>>>>>>> > >>>>>>>>> > > > >> Or is the server-side stateful, like a typical SQL >>>>>>>>> > > > >> implementation of >>>>>>>>> > > > >> cursors? In which case, are there any limits to the number >>>>>>>>> > > > >> of active >>>>>>>>> > > > >> cursors? Or what if a cursor is resumed some time in the >>>>>>>>> > > > >> future; will >>>>>>>>> > > > >> it work at all, or work slower? >>>>>>>>> > >>>>>>>>> > > > >> -- >>>>>>>>> > > > >> You received this message because you are subscribed to the >>>>>>>>> > > > >> Google >>>>>>>>> > > Groups >>>>>>>>> > > > >> "Google App Engine" group. >>>>>>>>> > > > >> To post to this group, send email to >>>>>>>>> > > [email protected]. >>>>>>>>> > > > >> To unsubscribe from this group, send email to >>>>>>>>> > > > >> >>>>>>>>> > > > >> [email protected]<google-appengine%[email protected]> >>>>>>>>> > > >>>>>>>>> > > <google-appengine%[email protected]<google-appengine%[email protected]> >>>>>>>>> > >>>>>>>>> > > > >> . >>>>>>>>> > > > >> For more options, visit this group at >>>>>>>>> > > > >>http://groups.google.com/group/google-appengine?hl=en. >>>>>>>>> > >>>>>>>>> > > > > -- >>>>>>>>> > >>>>>>>>> > > > > Alkis >>>>>>>>> > >>>>>>>>> > > > > -- >>>>>>>>> > > > > You received this message because you are subscribed to the >>>>>>>>> > > > > Google >>>>>>>>> > > Groups >>>>>>>>> > > > > "Google App Engine" group. >>>>>>>>> > > > > To post to this group, send email to >>>>>>>>> > > > > [email protected] >>>>>>>>> > > . >>>>>>>>> > > > > To unsubscribe from this group, send email to >>>>>>>>> > > > > >>>>>>>>> > > > > [email protected]<google-appengine%[email protected]> >>>>>>>>> > > >>>>>>>>> > > <google-appengine%[email protected]<google-appengine%[email protected]> >>>>>>>>> > >>>>>>>>> > > > > . >>>>>>>>> > > > > For more options, visit this group at >>>>>>>>> > > > >http://groups.google.com/group/google-appengine?hl=en. >>>>>>>>> > >>>>>>>>> > > > -- >>>>>>>>> > > > Ikai Lan >>>>>>>>> > > > Developer Programs Engineer, Google App Enginehttp:// >>>>>>>>> > > googleappengine.blogspot.com|http://twitter.com/app_engine >>>>>>>>> > >>>>>>>>> > > -- >>>>>>>>> > > You received this message because you are subscribed to the >>>>>>>>> > > Google Groups >>>>>>>>> > > "Google App Engine" group. >>>>>>>>> > > To post to this group, send email to >>>>>>>>> > > [email protected]. >>>>>>>>> > > To unsubscribe from this group, send email to >>>>>>>>> > > >>>>>>>>> > > [email protected]<google-appengine%[email protected]> >>>>>>>>> > > . >>>>>>>>> > > For more options, visit this group at >>>>>>>>> > >http://groups.google.com/group/google-appengine?hl=en. >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > Ikai Lan >>>>>>>>> > Developer Programs Engineer, Google App >>>>>>>>> > Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Google App Engine" group. >>>>>>>>> To post to this group, send email to >>>>>>>>> [email protected]. >>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>> [email protected]. >>>>>>>>> For more options, visit this group at >>>>>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Nick Johnson, Developer Programs Engineer, App Engine >>>>>>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >>>>>>>> Number: 368047 >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Google App Engine" group. >>>>>>>> To post to this group, send email to >>>>>>>> [email protected]. >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit this group at >>>>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Google App Engine" group. >>>>>>> To post to this group, send email to >>>>>>> [email protected]. >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected]. >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nick Johnson, Developer Programs Engineer, App Engine >>>>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >>>>>> Number: 368047 >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google App Engine" group. >>>>>> To post to this group, send email to [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/google-appengine?hl=en. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> >>>> >>>> -- >>>> Nick Johnson, Developer Programs Engineer, App Engine >>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >>>> Number: 368047 >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> >> Alkis >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
