I think you should run AppStats and see what's going on. 2000 entities is never going to be particularly fast, but 3 seconds might be a bit on the high side. Run the tool to be sure.
Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Tue, Mar 22, 2011 at 7:25 AM, adhi <[email protected]> wrote: > Hi Ikai, is there a measure to say the entities are large? is it about > the number of > columns or blob or text? if so in my entities I have string and number > properties only. > > Yes. I need all the 2000 (some times bit more) entities at once for > reporting. > Currently I'm not seeing a way where I can split the keys and fetch. > > > On Mar 21, 11:54 pm, "Ikai Lan (Google)" <[email protected]> wrote: > > If that's the performance, then I'm inclined to say the entities are > large > > and that's about the performance you can expect. One more thing to keep > in > > mind: the cost of deserializing each entity is non-trivial. > > > > Is there a reason you need all 2000 entities? Perhaps there is a > different > > way to accomplish what you want without querying a (relatively) large > number > > of entities at once. > > > > One other possible solution that will at least make your user facing code > > run faster is to split the query up by key, perform multiple async > queries > > and join them at the end. It'll still consume the same (or more) CPU > time, > > but the user ms should be lower since you are doing things in parallel. > The > > tradeoff here, of course, is that you'd need to know how you can split up > > the keyspace. > > > > Ikai Lan > > Developer Programs Engineer, Google App Engine > > Blog:http://googleappengine.blogspot.com > > Twitter:http://twitter.com/app_engine > > Reddit:http://www.reddit.com/r/appengine > > > > > > > > > > > > > > > > On Mon, Mar 21, 2011 at 12:17 AM, adhi <[email protected]> wrote: > > > q = db.Query(PrimaryData) > > > q.filter('SheetMetadataId', metadata.getSheetId()) > > > return q.fetch(2000) > > > > > Hi Ikai, even the above query takes 3.3 ~ 3.6 seconds. The number of > > > entities returned by the query is 1261. > > > I'm using Expando model, the total number columns for this particular > > > set of entities are 25. > > > Yes. I've created composite index. Here is the index definition. > > > > > - kind: PrimaryData > > > properties: > > > - name: SheetMetadataId > > > - name: InstanceAssignedTo > > > > > please let me know if you need more information. > > > > > On Mar 17, 11:50 pm, "Ikai Lan (Google)" <[email protected]> wrote: > > > > If you have a composite index, the performance of this query should > be > > > > mostly linear (find start location of index and just return min(2000, > > > number > > > > of entities that satisfy query from there on out) number of keys and > > > items. > > > > If you only have individual indexes for SheetMetadataId as well as > > > > InstanceAssignedTo, this uses zigzag merge join, where performance > can > > > > greatly vary depending on the shape of the data. > > > > > > Just out of curiosity, what is the performance of this query? > > > > > > q = db.Query(PrimaryData) > > > > q.filter('SheetMetadataId', metadata.getSheetId()) > > > > return q.fetch(2000) > > > > > > If it's in a similar ballpark, my other guess is that the query takes > > > that > > > > amount of time because the entities are large. > > > > > > You mentioned created a composite index. Can you post that? > > > > > > I'll talk to the datastore team to see what we can do to provide > > > something > > > > similar to EXPLAIN plans for queries. > > > > > > Ikai Lan > > > > Developer Programs Engineer, Google App Engine > > > > Blog:http://googleappengine.blogspot.com > > > > Twitter:http://twitter.com/app_engine > > > > Reddit:http://www.reddit.com/r/appengine > > > > > > On Thu, Mar 17, 2011 at 10:07 AM, adhi <[email protected]> > wrote: > > > > > Hi Robert, > > > > > I've not used reference properties. Its a simple query and it'll be > > > > > like this. > > > > > > > q = db.Query(PrimaryData) > > > > > q.filter('SheetMetadataId', metadata.getSheetId()) > > > > > q.filter('InstanceAssignedTo IN', > > > > > [u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3', > > > > > u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a']) > > > > > return q.fetch(2000) > > > > > > > and I replaced the IN filter to '=' with single value. Still its > > > > > taking same time. > > > > > > > On Mar 17, 6:27 am, Robert Kluin <[email protected]> wrote: > > > > > > Use Appstats. It may not be the query that is slow. If you are > using > > > > > > reference properties, perhaps you are dereferencing them. If you > > > > > > should us the query and how you're using the results we might be > abel > > > > > > to give more suggestions. > > > > > > > >http://code.google.com/appengine/docs/python/tools/appstats.html > > > > > > > > Robert > > > > > > > > On Mon, Mar 14, 2011 at 10:54, adhi <[email protected]> > wrote: > > > > > > > Hi, I'm running a query in appengine which is returning just > 1200 > > > > > > > entities, but its taking 3.5 seconds. The query doesn't > contains > > > > > > > inequality filters, but anyway I added index for that. Can > anyone > > > tell > > > > > > > me how to analyse this and improve theperformance? > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the > Google > > > > > Groups "Google App Engine" group. > > > > > > > To post to this group, send email to > > > [email protected] > > > > > . > > > > > > > To unsubscribe from this group, send email to > > > > > [email protected]. > > > > > > > For more options, visit this group athttp:// > > > > > groups.google.com/group/google-appengine?hl=en. > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Google App Engine" group. > > > > > To post to this group, send email to > [email protected] > > > . > > > > > To unsubscribe from this group, send email to > > > > > [email protected]. > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected] > . > > > To unsubscribe from this group, send email to > > > [email protected]. > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
