Hi Nickolas,
2010/2/9 Nickolas Daskalou <[email protected]>
> I'd want to do this so that I could include parts of the cursor (such as
> the offset) into a URL without including other parts (eg. the model kind and
> filters). I could then reconstruct the cursor on the server side based on
> what was passed into the URL.
>
The offset argument you're talking about is specific to the dev_appserver's
implementation of cursors. In production, offsets are not used, so this
won't work.
-Nick Johnson
>
> For example, if I was searching for blog comments that contained the word
> "the" (with the default order being the creation time, descending), the URL
> might look like this:
>
> myblog.com/comments/?q=the
>
> With model:
>
> class Comment(db.Model):
> ....
> created_at = db.DateTimeProperty(auto_now_add=True)
> words = db.StringListProperty() # A list of all the words in a comment
> (forget about exploding indexes for now)
> ...
>
> The query object for this URL might look something like:
>
> ....
> q =
> Comment.all().filter('words',self.request.get('q')).order('-created_at')
> ....
>
> To get to the 1001st comment, it'd be good if the URL looked something like
> this:
>
> myblog.com/comments/?q=the&skip=1000
>
> instead of:
>
> myblog.com/comments/?q=the&cursor=[something ugly]
>
> so that when the request comes in, I can do this:
>
> ....
> q =
> Comment.all().filter('words',self.request.get('q')).order('-created_at')
> cursor_template = q.cursor_template()
> cursor =
> db.Cursor.from_template(cursor_template,offset=int(self.request.get('skip')))
> ....
> (or something along these lines)
>
> Does that make sense?
>
>
>
> On 10 February 2010 01:03, Nick Johnson (Google)
> <[email protected]>wrote:
>
>> Hi Nickolas,
>>
>> 2010/2/9 Nickolas Daskalou <[email protected]>
>>
>> Will we be able to construct our own cursors much the same way that we are
>>> able to construct our own Datastore keys (Key.from_path())?
>>>
>>
>> No, not practically speaking.
>>
>>
>>>
>>> Also along the same lines, will we be able to "deconstruct" a cursor to
>>> get its components (offset, start_inclusive etc.), as we can now do with
>>> keys (key.name(), key.id(), key.kind() etc.)?
>>>
>>
>> While you could do this, there's no guarantees that it'll work (or
>> continue to work), as you'd be digging into internal implementation details.
>> Why do you want to do this?
>>
>> -Nick Johnson
>>
>>
>>>
>>> 2010/2/9 Nick Johnson (Google) <[email protected]>
>>>
>>>> 2010/2/9 Stephen <[email protected]>
>>>>
>>>>
>>>>> I'm asking if it's wise to store it as a query parameter embedded in a
>>>>> web page.
>>>>>
>>>>
>>>> You're right that it's unwise. Depending on how you construct your
>>>> query, a user could potentially modify the cursor they send to you to
>>>> return
>>>> results from any query your datastore is capable of performing, which could
>>>> result in you revealing information to the user that they shouldn't know.
>>>> You should either store the cursor on the server-side, or encrypt it before
>>>> sending it to the client.
>>>>
>>>> I was going to mention something about this in my post, but it slipped
>>>> my mind.
>>>>
>>>> -Nick Johnson
>>>>
>>>>>
>>>>>
>>>>> On Feb 9, 12:26 am, "Ikai L (Google)" <[email protected]> wrote:
>>>>> > A cursor serializes to a Base64 encoded String, so you can store it
>>>>> anywhere
>>>>> > you want to store strings: Memcached, Datastore, etc. You can even
>>>>> pass it
>>>>> > as an URL parameter to task queues.
>>>>> >
>>>>> > 2010/2/8 Stephen <[email protected]>
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > > Ah right, Nick's blog does say start_key and not offset. My bad.
>>>>> >
>>>>> > > Maybe there will be warnings in the upcoming documentation, but my
>>>>> > > first instinct was to embed the serialised cursor in the HTML as
>>>>> the
>>>>> > > 'next' link. But that doesn't look like a good idea as Nick's
>>>>> decoded
>>>>> > > query shows what's embedded:
>>>>> >
>>>>> > > PrimaryScan {
>>>>> > > start_key: "shell\000TestModel\000foo\000\232bar\000\200"
>>>>> > > start_inclusive: true
>>>>> > > }
>>>>> > > keys_only: false
>>>>> >
>>>>> > > First, you may or may not want to leak this info. Second, could
>>>>> this
>>>>> > > be altered on the client to change the query in any way that's
>>>>> > > undesirable?
>>>>> >
>>>>> > > Once you have a cursor, where do you store it so you can use it
>>>>> again?
>>>>> >
>>>>> > > On Feb 8, 10:17 pm, "Ikai L (Google)" <[email protected]> wrote:
>>>>> > > > I got beaten to this answer. No, there is no traversal to get to
>>>>> the
>>>>> > > offset.
>>>>> >
>>>>> > > > BigTable has an underlying mechanism for range queries on keys.
>>>>> Indexes
>>>>> > > are
>>>>> > > > essentially a key comprised of a concatenation of application ID,
>>>>> entity
>>>>> > > > type, column, value. When a filter operation is performed, the
>>>>> datastore
>>>>> > > > looks for a range matching this criteria, returning the set of
>>>>> keys. A
>>>>> > > > cursor also adds the datastore key of the entity so it is
>>>>> possible to
>>>>> > > > serialize where to begin the query. This is actually a bit
>>>>> awkward to
>>>>> > > > explain without visuals. You can watch Ryan Barrett's talk here:
>>>>> >
>>>>> > > >http://www.youtube.com/watch?v=tx5gdoNpcZM
>>>>> >
>>>>> > > > Hopefully, we'll be able to post an article at some point in the
>>>>> future
>>>>> > > > explaining how cursors work.
>>>>> >
>>>>> > > > 2010/2/8 Alkis Evlogimenos ('Αλκης Ευλογημένος) <
>>>>> [email protected]>
>>>>> >
>>>>> > > > > There is no offset. The protocol buffer stores a start_key and
>>>>> a
>>>>> > > boolean
>>>>> > > > > denoting if this start key is inclusive or not. The performance
>>>>> of
>>>>> > > > > continuing the fetch from a cursor should be the same as the
>>>>> > > performance of
>>>>> > > > > the first entities you got from a query.
>>>>> >
>>>>> > > > > On Mon, Feb 8, 2010 at 4:33 PM, Stephen <[email protected]>
>>>>> wrote:
>>>>> >
>>>>> > > > >> On Feb 8, 7:06 pm, "Ikai L (Google)" <[email protected]>
>>>>> wrote:
>>>>> > > > >> > The official docs are pending, but here's Nick Johnson to
>>>>> the
>>>>> > > rescue:
>>>>> >
>>>>> > >
>>>>> http://blog.notdot.net/2010/02/New-features-in-1-3-1-prerelease-Cursors
>>>>> >
>>>>> > > > >> What are the performance characteristics of cursors?
>>>>> >
>>>>> > > > >> The serialised cursor shows that it stores an offset. Does
>>>>> that mean
>>>>> > > > >> that if the offset is one million, one million rows will have
>>>>> to be
>>>>> > > > >> skipped before the next 10 are returned? This will be faster
>>>>> than
>>>>> > > > >> doing it in your app, but not as quick as the existing
>>>>> bookmark
>>>>> > > > >> techniques which use the primary key index.
>>>>> >
>>>>> > > > >> Or is the server-side stateful, like a typical SQL
>>>>> implementation of
>>>>> > > > >> cursors? In which case, are there any limits to the number of
>>>>> active
>>>>> > > > >> cursors? Or what if a cursor is resumed some time in the
>>>>> future; will
>>>>> > > > >> it work at all, or work slower?
>>>>> >
>>>>> > > > >> --
>>>>> > > > >> You received this message because you are subscribed to the
>>>>> Google
>>>>> > > Groups
>>>>> > > > >> "Google App Engine" group.
>>>>> > > > >> To post to this group, send email to
>>>>> > > [email protected].
>>>>> > > > >> To unsubscribe from this group, send email to
>>>>> > > > >> [email protected]<google-appengine%[email protected]>
>>>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>>>> >
>>>>> > > <google-appengine%[email protected]<google-appengine%[email protected]>
>>>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>>>> >
>>>>> >
>>>>> > > > >> .
>>>>> > > > >> For more options, visit this group at
>>>>> > > > >>http://groups.google.com/group/google-appengine?hl=en.
>>>>> >
>>>>> > > > > --
>>>>> >
>>>>> > > > > Alkis
>>>>> >
>>>>> > > > > --
>>>>> > > > > You received this message because you are subscribed to the
>>>>> Google
>>>>> > > Groups
>>>>> > > > > "Google App Engine" group.
>>>>> > > > > To post to this group, send email to
>>>>> [email protected]
>>>>> > > .
>>>>> > > > > To unsubscribe from this group, send email to
>>>>> > > > > [email protected]<google-appengine%[email protected]>
>>>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>>>> >
>>>>> > > <google-appengine%[email protected]<google-appengine%[email protected]>
>>>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>>>> >
>>>>> >
>>>>> > > > > .
>>>>> > > > > For more options, visit this group at
>>>>> > > > >http://groups.google.com/group/google-appengine?hl=en.
>>>>> >
>>>>> > > > --
>>>>> > > > Ikai Lan
>>>>> > > > Developer Programs Engineer, Google App Enginehttp://
>>>>> > > googleappengine.blogspot.com|http://twitter.com/app_engine
>>>>> >
>>>>> > > --
>>>>> > > You received this message because you are subscribed to the Google
>>>>> Groups
>>>>> > > "Google App Engine" group.
>>>>> > > To post to this group, send email to
>>>>> [email protected].
>>>>> > > To unsubscribe from this group, send email to
>>>>> > > [email protected]<google-appengine%[email protected]>
>>>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>>>> >
>>>>> > > .
>>>>> > > For more options, visit this group at
>>>>> > >http://groups.google.com/group/google-appengine?hl=en.
>>>>> >
>>>>> > --
>>>>> > Ikai Lan
>>>>> > Developer Programs Engineer, Google App Enginehttp://
>>>>> googleappengine.blogspot.com|http://twitter.com/app_engine
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Google App Engine" group.
>>>>> To post to this group, send email to [email protected]
>>>>> .
>>>>> To unsubscribe from this group, send email to
>>>>> [email protected]<google-appengine%[email protected]>
>>>>> .
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Nick Johnson, Developer Programs Engineer, App Engine
>>>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
>>>> Number: 368047
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Google App Engine" group.
>>>> To post to this group, send email to [email protected].
>>>> To unsubscribe from this group, send email to
>>>> [email protected]<google-appengine%[email protected]>
>>>> .
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/google-appengine?hl=en.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to [email protected].
>>> To unsubscribe from this group, send email to
>>> [email protected]<google-appengine%[email protected]>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>
>>
>>
>> --
>> Nick Johnson, Developer Programs Engineer, App Engine
>> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
>> 368047
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected]<google-appengine%[email protected]>
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected]<google-appengine%[email protected]>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
