Does the production cursor string contain information about the app id,
kind, any filter()s or order()s, and (more importantly) some sort of
numerical value that indicates how many records the next query should
"skip"? If so, and if we could extract this information (and then use it
again to the reconstruct the cursor), that would make for much cleaner,
safer and intuitive URLs than including the entire cursor string (or some
sort of encrypted/encoded cursor string replacement).
2010/2/10 Nick Johnson (Google) <nick.john...@google.com>
> Hi Nickolas,
>
> 2010/2/9 Nickolas Daskalou <n...@daskalou.com>
>
>> I'd want to do this so that I could include parts of the cursor (such as
>> the offset) into a URL without including other parts (eg. the model kind and
>> filters). I could then reconstruct the cursor on the server side based on
>> what was passed into the URL.
>>
>
> The offset argument you're talking about is specific to the dev_appserver's
> implementation of cursors. In production, offsets are not used, so this
> won't work.
>
> -Nick Johnson
>
>
>>
>> For example, if I was searching for blog comments that contained the word
>> "the" (with the default order being the creation time, descending), the URL
>> might look like this:
>>
>> myblog.com/comments/?q=the
>>
>> With model:
>>
>> class Comment(db.Model):
>> ....
>> created_at = db.DateTimeProperty(auto_now_add=True)
>> words = db.StringListProperty() # A list of all the words in a comment
>> (forget about exploding indexes for now)
>> ...
>>
>> The query object for this URL might look something like:
>>
>> ....
>> q =
>> Comment.all().filter('words',self.request.get('q')).order('-created_at')
>> ....
>>
>> To get to the 1001st comment, it'd be good if the URL looked something
>> like this:
>>
>> myblog.com/comments/?q=the&skip=1000
>>
>> instead of:
>>
>> myblog.com/comments/?q=the&cursor=[something ugly]
>>
>> so that when the request comes in, I can do this:
>>
>> ....
>> q =
>> Comment.all().filter('words',self.request.get('q')).order('-created_at')
>> cursor_template = q.cursor_template()
>> cursor =
>> db.Cursor.from_template(cursor_template,offset=int(self.request.get('skip')))
>> ....
>> (or something along these lines)
>>
>> Does that make sense?
>>
>>
>>
>> On 10 February 2010 01:03, Nick Johnson (Google) <nick.john...@google.com
>> > wrote:
>>
>>> Hi Nickolas,
>>>
>>> 2010/2/9 Nickolas Daskalou <n...@daskalou.com>
>>>
>>> Will we be able to construct our own cursors much the same way that we
>>>> are able to construct our own Datastore keys (Key.from_path())?
>>>>
>>>
>>> No, not practically speaking.
>>>
>>>
>>>>
>>>> Also along the same lines, will we be able to "deconstruct" a cursor to
>>>> get its components (offset, start_inclusive etc.), as we can now do with
>>>> keys (key.name(), key.id(), key.kind() etc.)?
>>>>
>>>
>>> While you could do this, there's no guarantees that it'll work (or
>>> continue to work), as you'd be digging into internal implementation details.
>>> Why do you want to do this?
>>>
>>> -Nick Johnson
>>>
>>>
>>>>
>>>> 2010/2/9 Nick Johnson (Google) <nick.john...@google.com>
>>>>
>>>>> 2010/2/9 Stephen <sdea...@gmail.com>
>>>>>
>>>>>
>>>>>> I'm asking if it's wise to store it as a query parameter embedded in a
>>>>>> web page.
>>>>>>
>>>>>
>>>>> You're right that it's unwise. Depending on how you construct your
>>>>> query, a user could potentially modify the cursor they send to you to
>>>>> return
>>>>> results from any query your datastore is capable of performing, which
>>>>> could
>>>>> result in you revealing information to the user that they shouldn't know.
>>>>> You should either store the cursor on the server-side, or encrypt it
>>>>> before
>>>>> sending it to the client.
>>>>>
>>>>> I was going to mention something about this in my post, but it slipped
>>>>> my mind.
>>>>>
>>>>> -Nick Johnson
>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 9, 12:26 am, "Ikai L (Google)" <ika...@google.com> wrote:
>>>>>> > A cursor serializes to a Base64 encoded String, so you can store it
>>>>>> anywhere
>>>>>> > you want to store strings: Memcached, Datastore, etc. You can even
>>>>>> pass it
>>>>>> > as an URL parameter to task queues.
>>>>>> >
>>>>>> > 2010/2/8 Stephen <sdea...@gmail.com>
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > > Ah right, Nick's blog does say start_key and not offset. My bad.
>>>>>> >
>>>>>> > > Maybe there will be warnings in the upcoming documentation, but my
>>>>>> > > first instinct was to embed the serialised cursor in the HTML as
>>>>>> the
>>>>>> > > 'next' link. But that doesn't look like a good idea as Nick's
>>>>>> decoded
>>>>>> > > query shows what's embedded:
>>>>>> >
>>>>>> > > PrimaryScan {
>>>>>> > > start_key: "shell\000TestModel\000foo\000\232bar\000\200"
>>>>>> > > start_inclusive: true
>>>>>> > > }
>>>>>> > > keys_only: false
>>>>>> >
>>>>>> > > First, you may or may not want to leak this info. Second, could
>>>>>> this
>>>>>> > > be altered on the client to change the query in any way that's
>>>>>> > > undesirable?
>>>>>> >
>>>>>> > > Once you have a cursor, where do you store it so you can use it
>>>>>> again?
>>>>>> >
>>>>>> > > On Feb 8, 10:17 pm, "Ikai L (Google)" <ika...@google.com> wrote:
>>>>>> > > > I got beaten to this answer. No, there is no traversal to get to
>>>>>> the
>>>>>> > > offset.
>>>>>> >
>>>>>> > > > BigTable has an underlying mechanism for range queries on keys.
>>>>>> Indexes
>>>>>> > > are
>>>>>> > > > essentially a key comprised of a concatenation of application
>>>>>> ID, entity
>>>>>> > > > type, column, value. When a filter operation is performed, the
>>>>>> datastore
>>>>>> > > > looks for a range matching this criteria, returning the set of
>>>>>> keys. A
>>>>>> > > > cursor also adds the datastore key of the entity so it is
>>>>>> possible to
>>>>>> > > > serialize where to begin the query. This is actually a bit
>>>>>> awkward to
>>>>>> > > > explain without visuals. You can watch Ryan Barrett's talk here:
>>>>>> >
>>>>>> > > >http://www.youtube.com/watch?v=tx5gdoNpcZM
>>>>>> >
>>>>>> > > > Hopefully, we'll be able to post an article at some point in the
>>>>>> future
>>>>>> > > > explaining how cursors work.
>>>>>> >
>>>>>> > > > 2010/2/8 Alkis Evlogimenos ('Αλκης Ευλογημένος) <
>>>>>> evlogime...@gmail.com>
>>>>>> >
>>>>>> > > > > There is no offset. The protocol buffer stores a start_key and
>>>>>> a
>>>>>> > > boolean
>>>>>> > > > > denoting if this start key is inclusive or not. The
>>>>>> performance of
>>>>>> > > > > continuing the fetch from a cursor should be the same as the
>>>>>> > > performance of
>>>>>> > > > > the first entities you got from a query.
>>>>>> >
>>>>>> > > > > On Mon, Feb 8, 2010 at 4:33 PM, Stephen <sdea...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > > > >> On Feb 8, 7:06 pm, "Ikai L (Google)" <ika...@google.com>
>>>>>> wrote:
>>>>>> > > > >> > The official docs are pending, but here's Nick Johnson to
>>>>>> the
>>>>>> > > rescue:
>>>>>> >
>>>>>> > >
>>>>>> http://blog.notdot.net/2010/02/New-features-in-1-3-1-prerelease-Cursors
>>>>>> >
>>>>>> > > > >> What are the performance characteristics of cursors?
>>>>>> >
>>>>>> > > > >> The serialised cursor shows that it stores an offset. Does
>>>>>> that mean
>>>>>> > > > >> that if the offset is one million, one million rows will have
>>>>>> to be
>>>>>> > > > >> skipped before the next 10 are returned? This will be faster
>>>>>> than
>>>>>> > > > >> doing it in your app, but not as quick as the existing
>>>>>> bookmark
>>>>>> > > > >> techniques which use the primary key index.
>>>>>> >
>>>>>> > > > >> Or is the server-side stateful, like a typical SQL
>>>>>> implementation of
>>>>>> > > > >> cursors? In which case, are there any limits to the number of
>>>>>> active
>>>>>> > > > >> cursors? Or what if a cursor is resumed some time in the
>>>>>> future; will
>>>>>> > > > >> it work at all, or work slower?
>>>>>> >
>>>>>> > > > >> --
>>>>>> > > > >> You received this message because you are subscribed to the
>>>>>> Google
>>>>>> > > Groups
>>>>>> > > > >> "Google App Engine" group.
>>>>>> > > > >> To post to this group, send email to
>>>>>> > > google-appeng...@googlegroups.com.
>>>>>> > > > >> To unsubscribe from this group, send email to
>>>>>> > > > >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>>>> <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>>>> >
>>>>>> > > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>>>> <google-appengine%252bunsubscr...@googlegroups.com<google-appengine%25252bunsubscr...@googlegroups.com>
>>>>>> >
>>>>>> >
>>>>>> > > > >> .
>>>>>> > > > >> For more options, visit this group at
>>>>>> > > > >>http://groups.google.com/group/google-appengine?hl=en.
>>>>>> >
>>>>>> > > > > --
>>>>>> >
>>>>>> > > > > Alkis
>>>>>> >
>>>>>> > > > > --
>>>>>> > > > > You received this message because you are subscribed to the
>>>>>> Google
>>>>>> > > Groups
>>>>>> > > > > "Google App Engine" group.
>>>>>> > > > > To post to this group, send email to
>>>>>> google-appengine@googlegroups.com
>>>>>> > > .
>>>>>> > > > > To unsubscribe from this group, send email to
>>>>>> > > > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>>>> <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>>>> >
>>>>>> > > <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>>>> <google-appengine%252bunsubscr...@googlegroups.com<google-appengine%25252bunsubscr...@googlegroups.com>
>>>>>> >
>>>>>> >
>>>>>> > > > > .
>>>>>> > > > > For more options, visit this group at
>>>>>> > > > >http://groups.google.com/group/google-appengine?hl=en.
>>>>>> >
>>>>>> > > > --
>>>>>> > > > Ikai Lan
>>>>>> > > > Developer Programs Engineer, Google App Enginehttp://
>>>>>> > > googleappengine.blogspot.com|http://twitter.com/app_engine
>>>>>> >
>>>>>> > > --
>>>>>> > > You received this message because you are subscribed to the Google
>>>>>> Groups
>>>>>> > > "Google App Engine" group.
>>>>>> > > To post to this group, send email to
>>>>>> google-appeng...@googlegroups.com.
>>>>>> > > To unsubscribe from this group, send email to
>>>>>> > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>>>> <google-appengine%2bunsubscr...@googlegroups.com<google-appengine%252bunsubscr...@googlegroups.com>
>>>>>> >
>>>>>> > > .
>>>>>> > > For more options, visit this group at
>>>>>> > >http://groups.google.com/group/google-appengine?hl=en.
>>>>>> >
>>>>>> > --
>>>>>> > Ikai Lan
>>>>>> > Developer Programs Engineer, Google App Enginehttp://
>>>>>> googleappengine.blogspot.com|http://twitter.com/app_engine
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Google App Engine" group.
>>>>>> To post to this group, send email to
>>>>>> google-appeng...@googlegroups.com.
>>>>>> To unsubscribe from this group, send email to
>>>>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>>>> .
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nick Johnson, Developer Programs Engineer, App Engine
>>>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
>>>>> Number: 368047
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Google App Engine" group.
>>>>> To post to this group, send email to google-appengine@googlegroups.com
>>>>> .
>>>>> To unsubscribe from this group, send email to
>>>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>>> .
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Google App Engine" group.
>>>> To post to this group, send email to google-appeng...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>>> .
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>
>>>
>>>
>>>
>>> --
>>> Nick Johnson, Developer Programs Engineer, App Engine
>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
>>> Number: 368047
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appeng...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appeng...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
>
>
> --
> Nick Johnson, Developer Programs Engineer, App Engine
> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
> 368047
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appeng...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
