Hi Manny,
On Wed, Feb 17, 2010 at 6:57 AM, Manny S <[email protected]> wrote:
> Ryan,
> I just made the pagination code work using cursors. One thing I noticed in
> the documentation is this:
>
>
> - A query with a cursor does not always work as expected when the query
> uses an inequality filter or a sort order on a property with multiple
> values. Logic that de-duplicates the multivalued property for the query
> does
> not persist between queries, and may return a result more than once.
>
>
> I am sorting the results of the query (simple query that returns data below
> a threshold value which is a long) by post date. The sort order is post date
> and might have duplicate values. Does this mean that the cursors will return
> duplicates?
>
No, that's not (quite) what that bullet point means. An example may help.
Suppose you have three entities:
BlogPost(key_name="one", tags=["abacus", "dog"])
BlogPost(key_name="two", tags=["banana", "elephant"])
BlogPost(key_name=three", tags=["cat", "zebra"])
And a query like this:
q = BlogPost.all().order("tags")
The index used for this query will look something like this:
(BlogPost, tags, abacus) -> Key(BlogPost, one)
(BlogPost, tags, banana) -> Key(BlogPost, two)
(BlogPost, tags, cat) -> Key(BlogPost, three)
(BlogPost, tags, dog) -> Key(BlogPost, one)
(BlogPost, tags, elephant) -> Key(BlogPost, two)
(BlogPost, tags, zebra) -> Key(BlogPost, three)
Calling q.fetch(2) will scan over that index to return the first two entries
and return them - entities "one" and "two". Calling q.fetch(3) will return
all three posts, in that order, but without duplicates, as the query
infrastructure keeps a list of previously seen keys to avoid returning
duplicates.
If, however, you call q.fetch(2), then get a cursor with q.cursor(), and
later re-use it again:
q.fetch(2) # Returns "one", "two"
cursor = q.cursor()
q = BlogPost.all().order("tags").with_cursor(cursor)
q.fetch(2) # Returns "three", "one"
The resumed fetch returns "one" again, because the cursor doesn't contain
enough information to know what results were previously returned.
As you can see, this only happens if you're using multiply valued properties
(ListProperties), and sorting or doing an inequality filter on them.
-Nick Johnson
>
> Manny
>
> On Tue, Feb 16, 2010 at 4:01 PM, Manny S <[email protected]> wrote:
>
>> Ryan - I will check out query cursors. Took a quick look and I think this
>> would solve my problems. Thanks.
>> Jeff - thanks for the input. I have included a createdDate and a
>> modifiedDate in all my entities.
>>
>> manny
>>
>> On Tue, Feb 16, 2010 at 12:11 AM, Jeff Schnitzer <[email protected]>wrote:
>>
>>> You will likely find it good practice to keep a createdDate (and
>>> possibly a modifiedDate) in all of your entities. Makes debugging
>>> easier, and if you're looking for this kind of pagination, there you
>>> go. Trying to abuse the key system for these purposes seems a bit
>>> hazardous.
>>>
>>> That said, you can always generate your own String name as part of the
>>> key. This is not a trivial task given the distributed nature of
>>> appengine.
>>>
>>> Jeff
>>>
>>> On Sat, Feb 13, 2010 at 8:12 AM, Manny S <[email protected]> wrote:
>>> > Hi Tim,
>>> > Thanks again Tim. A naive question - if I use the datastore generated
>>> keys
>>> > and sort by the keys desc (for pagination) the key value of new data
>>> > inserted will be greater than that of previously entered data (if I do
>>> a
>>> > keytostring and a string compare?).
>>> > In other words does the datastore generate keys in a order or at
>>> random. I
>>> > did some experiments and I found that pagination works according to
>>> > date/time of insert if I just sort keys by descending but just wanted
>>> to
>>> > make sure.
>>> > Regards,
>>> > Manny
>>> > On Sat, Feb 13, 2010 at 1:17 PM, Tim Hoffman <[email protected]>
>>> wrote:
>>> >>
>>> >> Hi Manny
>>> >>
>>> >> Understood.
>>> >>
>>> >> You know you can let the system generate the entity identifiers for
>>> >> you, and pagination will
>>> >> just work (by __key__) any way. So you don't have to the key_name or
>>> >> id at all
>>> >> if you don't care what the key is.
>>> >>
>>> >> Rgds
>>> >>
>>> >> T
>>> >>
>>> >> On Feb 13, 1:00 am, Manny S <[email protected]> wrote:
>>> >> > Thanks a bunch Tim for your inputs,
>>> >> >
>>> >> > My rationale for adding the date to the appstore generated key is
>>> to
>>> >> > make
>>> >> > pagination easier. I would do pagination on the key and not add a
>>> >> > separate
>>> >> > column for that. (Pagination by date alone will also not solve my
>>> >> > problem
>>> >> > since it can have duplicates and hence couple it with the key)
>>> >> >
>>> >> > I understand fetching by keys is much faster. Though I don't see a
>>> >> > scenario
>>> >> > where I would have to do that now I would like to architect my app
>>> where
>>> >> > that would be possible. However, I do not have anything unique in my
>>> >> > record
>>> >> > with which I can set the key. It just contains city name, locality
>>> >> > details
>>> >> > and a series of other fields all of which could have duplicates. Any
>>> >> > ideas
>>> >> > as to how I can generate unique ids from these or any general
>>> pointers
>>> >> > towards generating unique Ids from data where the data itself does
>>> not
>>> >> > have
>>> >> > a unique field?
>>> >> >
>>> >> > Manny
>>> >> >
>>> >> > On Thu, Feb 11, 2010 at 4:49 PM, Tim Hoffman <[email protected]>
>>> wrote:
>>> >> > > Hi Manny
>>> >> >
>>> >> > > Do you really want to do that for a key. One if the big
>>> advantages of
>>> >> > > creating your own keys
>>> >> > > is being able to explicitly get entities by key (1 or more with
>>> >> > > db.get(list_of_keys) which is much
>>> >> > > quicker than a gql or filter. Making your keys include dates mean
>>> you
>>> >> > > will be unlikely to
>>> >> > > guess/know what the keys are in advance.
>>> >> >
>>> >> > > This of course may not be useful for what you are doing, but worth
>>> >> > > keeping in mind.
>>> >> >
>>> >> > > Rgds
>>> >> >
>>> >> > > T
>>> >> >
>>> >> > > On Feb 11, 2:12 pm, Manny S <[email protected]> wrote:
>>> >> > > > Hi Ikai,
>>> >> >
>>> >> > > > I did read the documentation and now I have my data structures
>>> in
>>> >> > > > place.
>>> >> > > One
>>> >> > > > thing I wanted to do and that was not clear from my previous
>>> post
>>> >> > > > was to
>>> >> > > > append a app generated string (not unique) as a prefix to a
>>> >> > > > datastore
>>> >> > > > generated key. For instance, I want to generate a key that has
>>> the
>>> >> > > > date
>>> >> > > (of
>>> >> > > > record creation) as a prefix to the datastore generated unique
>>> key.
>>> >> > > > Is
>>> >> > > > there a way to do this? I do not want my application to generate
>>> >> > > > unique
>>> >> > > Ids.
>>> >> >
>>> >> > > > From reading through the literature so far, I am guessing that
>>> will
>>> >> > > > not
>>> >> > > be
>>> >> > > > possible since the datastore keys are generated only at the time
>>> >> > > > when the
>>> >> > > > objects are being made persistent.
>>> >> >
>>> >> > > > Manny
>>> >> >
>>> >> > > > On Wed, Feb 10, 2010 at 3:30 AM, Ikai L (Google) <
>>> [email protected]>
>>> >> > > wrote:
>>> >> >
>>> >> > > > > Have you read our documentation on KeyFactory?
>>> >> >
>>> >> > > >
>>> >> > > > > >
>>> http://code.google.com/appengine/docs/java/datastore/relationships.html
>>> >> >
>>> >> > > > > <
>>> >> > >
>>> http://code.google.com/appengine/docs/java/datastore/relationships.html
>>> >> > > >I'd
>>> >> > > > > try to understand what's going on there. It sounds like you're
>>> >> > > > > doing it
>>> >> > > the
>>> >> > > > > right way, but it's up to you to benchmark and find the best
>>> >> > > > > approach
>>> >> > > for
>>> >> > > > > what works for you. The usage characteristics of your
>>> application
>>> >> > > should
>>> >> > > > > determine the way your store your data.
>>> >> >
>>> >> > > > > On Wed, Feb 3, 2010 at 3:42 AM, Manny S <[email protected]
>>> >
>>> >> > > > > wrote:
>>> >> >
>>> >> > > > >> Ikai,
>>> >> > > > >> Based on your inputs I created two data classes that have a
>>> >> > > unidirectional
>>> >> > > > >> one-to-one relationship
>>> >> > > > >> Now, I have two data classes simpledata and detailscol.
>>> >> > > > >> simpledata contains fields A, B, C (and a Key field)
>>> >> > > > >> detailscol just contains field D.
>>> >> >
>>> >> > > > >> simpledata imports detailscol that contains field D (and a
>>> Key
>>> >> > > > >> field).
>>> >> > > It
>>> >> > > > >> also contains an accessor for the detailscol.
>>> >> > > > >> Code:
>>> >> > > > >> simpledata sdata = new simpledata(A,B,C);
>>> >> > > > >> sdata.setKey(null);
>>> >> > > > >> detailscol obj = new detailscol(D);
>>> >> > > > >> sdata.setD(obj);
>>> >> >
>>> >> > > > >> The keys are generated by the application and then I make the
>>> >> > > > >> data
>>> >> > > > >> persistent.
>>> >> >
>>> >> > > > >> Now, I display just the data in simpledata and if the user
>>> clicks
>>> >> > > > >> on a
>>> >> > > > >> details link I get the data stored in detailscol
>>> >> > > > >> To get to that data I just do
>>> >> >
>>> >> > > > >> detailscol d = sdata.getDetails();
>>> >> >
>>> >> > > > >> Two questions:
>>> >> >
>>> >> > > > >> 1) Is this the right approach?
>>> >> >
>>> >> > > > >> 2) If I want to get the child data using just the parent
>>> keyhow
>>> >> > > > >> do I
>>> >> > > go
>>> >> > > > >> about it?
>>> >> >
>>> >> > > > >> E.g, user clicks details and I use some AJAX to redirect to a
>>> >> > > different
>>> >> > > > >> servlet with just parent key as a parameter (since I don't
>>> access
>>> >> > > > >> the
>>> >> > > child
>>> >> > > > >> object yet). I get the parent key using
>>> >> > > > >> KeyFactory.keyToString(sdata.getKey());
>>> >> >
>>> >> > > > >> Now, that I have the parent's key should I do a getObjectbyID
>>> on
>>> >> > > > >> the
>>> >> > > > >> parent data again using this and then get the child using the
>>> >> > > > >> accessor
>>> >> > > > >> method or is there a direct way to construct the child key
>>> and
>>> >> > > > >> get to
>>> >> > > the
>>> >> > > > >> child data.
>>> >> >
>>> >> > > > >> Due to the nature of my application I would like to have the
>>> key
>>> >> > > generated
>>> >> > > > >> automatically (using setKey(null)).
>>> >> >
>>> >> > > > >> Apologies for the confusion in advance :)
>>> >> >
>>> >> > > > >> Manny
>>> >> >
>>> >> > > > >> On Sat, Jan 30, 2010 at 12:16 AM, Ikai L (Google)
>>> >> > > > >> <[email protected]
>>> >> > > >wrote:
>>> >> >
>>> >> > > > >>> Hi Manny,
>>> >> >
>>> >> > > > >>> A few things to first remember - App Engine's datastore is
>>> not a
>>> >> > > > >>> database, but a distributed key value store with additional
>>> >> > > > >>> features.
>>> >> > > Thus,
>>> >> > > > >>> we should be careful not to frame our thinking in terms of
>>> RDBMS
>>> >> > > schemas.
>>> >> > > > >>> For this reason, I like to avoid using database terminology
>>> that
>>> >> > > > >>> can
>>> >> > > > >>> confound the design process like "table" or "column". App
>>> Engine
>>> >> > > stores
>>> >> > > > >>> objects serialized ("entities") and indexes on the values.
>>> It'd
>>> >> > > > >>> be
>>> >> > > similar
>>> >> > > > >>> to an approach of creating a MySQL table with a String ID
>>> and a
>>> >> > > > >>> blob
>>> >> > > value,
>>> >> > > > >>> storing serialized Objects in the blob column, or using
>>> Memcache
>>> >> > > > >>> and
>>> >> > > storing
>>> >> > > > >>> JSON values.
>>> >> >
>>> >> > > > >>> When you retrieve a single value from the key value store,
>>> we
>>> >> > > > >>> have to
>>> >> > > > >>> retrieve everything at once. In most scenarios, unlike SQL
>>> >> > > > >>> databases
>>> >> > > you may
>>> >> > > > >>> be used to, retrieving large binary or text data does not
>>> add
>>> >> > > > >>> serious
>>> >> > > > >>> overhead. Of course, this changes if you start storing data
>>> on
>>> >> > > > >>> the
>>> >> > > scale of
>>> >> > > > >>> 1mb and are retrieving it unnecessarily. How large is the
>>> data
>>> >> > > > >>> you
>>> >> > > are
>>> >> > > > >>> retrieving?
>>> >> >
>>> >> > > > >>> Here's the way I would model your scenario if I was positive
>>> the
>>> >> > > > >>> text/binary field had a 1:1 relationship with the parent
>>> class:
>>> >> >
>>> >> > > > >>> * on your main entity, define the properties.
>>> >> > > > >>> * define a new entity with a text/binary field, and encode
>>> the
>>> >> > > > >>> parent
>>> >> > > key
>>> >> > > > >>> information in this key such that generating the key for
>>> this
>>> >> > > > >>> child
>>> >> > > field is
>>> >> > > > >>> very cheap. KeyFactory.stringToKey and
>>> KeyFactory.keyToString
>>> >> > > > >>> are
>>> >> > > crucial
>>> >> > > > >>> here. Read more about them here:
>>> >> >
>>> >> >
>>> >> > > >
>>> http://code.google.com/appengine/docs/java/javadoc/com/google/appengi..
>>> ..
>>> >> > > > >>> You can call your child property "parent_id:additional_info"
>>> or
>>> >> > > whatever
>>> >> > > > >>> makes sense to you.
>>> >> >
>>> >> > > > >>> Robert's solution of using a child key is basically just a
>>> >> > > > >>> variation
>>> >> > > on
>>> >> > > > >>> this, as parent key information is encoded in a child key.
>>> >> >
>>> >> > > > >>> A lot of this stuff can be a bit different to get used to. I
>>> >> > > > >>> suggest
>>> >> > > > >>> becoming familiar with keys and how they are used in App
>>> Engine:
>>> >> >
>>> >> > > > >>> Basic documentation about relationships:
>>> >> >
>>> >> > >
>>> http://code.google.com/appengine/docs/java/datastore/relationships.html
>>> >> > > > >>> A more advanced article:
>>> >> > > > >>>
>>> http://code.google.com/appengine/articles/storage_breakdown.html
>>> >> >
>>> >> > > > >>> On Thu, Jan 28, 2010 at 10:28 PM, Manny S
>>> >> > > > >>> <[email protected]
>>> >> > > >wrote:
>>> >> >
>>> >> > > > >>>> Hi All,
>>> >> >
>>> >> > > > >>>> First off, thanks for your time. A quick noob question on
>>> the
>>> >> > > > >>>> right
>>> >> > > way
>>> >> > > > >>>> to model data.
>>> >> >
>>> >> > > > >>>> I have a table with four columns A,B,C, D. D - the fourth
>>> is
>>> >> > > > >>>> of
>>> >> > > type
>>> >> > > > >>>> text (contains quite a bit of data).
>>> >> >
>>> >> > > > >>>> I wanted to ensure that the contents of the details column
>>> 'D'
>>> >> > > > >>>> is
>>> >> > > not
>>> >> > > > >>>> fetched during a query. A sample scenario
>>> >> > > > >>>> User does a search. Sees Columns A,B,C. If they need more
>>> >> > > > >>>> details
>>> >> > > for
>>> >> > > > >>>> that particular record Click on a link that fetches D for
>>> that
>>> >> > > particular
>>> >> > > > >>>> record.
>>> >> >
>>> >> > > > >>>> So I tried to do something like - Select A, B, C from
>>> >> > > > >>>> tablename.
>>> >> >
>>> >> > > > >>>> I found from the documentation that the GQL query returns
>>> full
>>> >> > > > >>>> data
>>> >> > > > >>>> objects and so all queries start with SELECT *. Is this
>>> true
>>> >> > > > >>>> for
>>> >> > > JDOQL on
>>> >> > > > >>>> the datastore as well? Does this mean everytime I query the
>>> >> > > > >>>> data
>>> >> > > store its
>>> >> > > > >>>> going to return all columns consuming bandwidth?
>>> >> >
>>> >> > > > >>>> Also since I want the content of COlumn D to be fetched on
>>> >> > > subsequent
>>> >> > > > >>>> user action so should I instead create two tables one with
>>> >> >
>>> >> > > > >>>> ID_TB1, A, B, C
>>> >> >
>>> >> > > > >>>> and the other one with
>>> >> >
>>> >> > > > >>>> ID, ID_TB1, D?
>>> >> >
>>> >> > > > >>>> Manny
>>> >> >
>>> >> > > > >>>> --
>>> >> > > > >>>> You received this message because you are subscribed to the
>>> >> > > > >>>> Google
>>> >> > > > >>>> Groups "Google App Engine" group.
>>> >> > > > >>>> To post to this group, send email to
>>> >> > > [email protected].
>>> >> > > > >>>> To unsubscribe from this group, send email to
>>> >> > > > >>>>
>>> >> > > > >>>> [email protected]<google-appengine%[email protected]>
>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>> >
>>> >> > >
>>> >> > > <google-appengine%[email protected]<google-appengine%[email protected]>
>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>> >
>>> >> >
>>> >> > > > >>>> .
>>> >> > > > >>>> For more options, visit this group at
>>> >> > > > >>>>http://groups.google.com/group/google-appengine?hl=en.
>>> >> >
>>> >> > > > >>> --
>>> >> > > > >>> Ikai Lan
>>> >> > > > >>> Developer Programs Engineer, Google App Engine
>>> >> > > > >>>http://googleappengine.blogspot.com|
>>> http://twitter.com/app_engine
>>> >> >
>>> >> > > > >>> --
>>> >> > > > >>> You received this message because you are subscribed to the
>>> >> > > > >>> Google
>>> >> > > > >>> Groups "Google App Engine" group.
>>> >> > > > >>> To post to this group, send email to
>>> >> > > [email protected].
>>> >> > > > >>> To unsubscribe from this group, send email to
>>> >> > > > >>>
>>> >> > > > >>> [email protected]<google-appengine%[email protected]>
>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>> >
>>> >> > >
>>> >> > > <google-appengine%[email protected]<google-appengine%[email protected]>
>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>> >
>>> >> >
>>> >> > > > >>> .
>>> >> > > > >>> For more options, visit this group at
>>> >> > > > >>>http://groups.google.com/group/google-appengine?hl=en.
>>> >> >
>>> >> > > > >> --
>>> >> > > > >> You received this message because you are subscribed to the
>>> >> > > > >> Google
>>> >> > > Groups
>>> >> > > > >> "Google App Engine" group.
>>> >> > > > >> To post to this group, send
>>> >> >
>>> >> > ...
>>> >> >
>>> >> > read more ยป
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> Groups
>>> >> "Google App Engine" group.
>>> >> To post to this group, send email to
>>> [email protected].
>>> >> To unsubscribe from this group, send email to
>>> >> [email protected]<google-appengine%[email protected]>
>>> .
>>> >> For more options, visit this group at
>>> >> http://groups.google.com/group/google-appengine?hl=en.
>>> >>
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "Google App Engine" group.
>>> > To post to this group, send email to [email protected]
>>> .
>>> > To unsubscribe from this group, send email to
>>> > [email protected]<google-appengine%[email protected]>
>>> .
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/google-appengine?hl=en.
>>> >
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to [email protected].
>>> To unsubscribe from this group, send email to
>>> [email protected]<google-appengine%[email protected]>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected]<google-appengine%[email protected]>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
