I actually only need 2 of my 8 columns of those 1000 posts. (the rest is lazily loaded with ajax when clicked)
My line of thinking is that in a relational database, getting 2 vs 8 columns won't really matter. The database does the same query and would just dump the extra 6 rows on the table before handing them to me. Sure there is a tiny tiny bandwidth difference if it gets dumped out at the database level or the application level, but they are normally sitting right next to each other on gigabit, so it doesn't really matter. I see how the app engine isn't exactly like this, but I am still trying to figure out all of the performance characteristics of the service. For step 1 I will try cutting down to the 2 columns I care about in my query. If that doesn't help, maybe I will just start caching the result with the cache implementation.. it is a really small amount of data I get back from the query (maybe 50KB tops). Really insane it takes me 8 api and app cpu seconds every single time I get them. On Feb 10, 12:52 pm, "Ikai L (Google)" <[email protected]> wrote: > The datastore isn't a relational database, and creating an "index" isn't the > same as it is in SQL - you aren't creating a B-tree index that allows a fast > query, you're creating an actual index so the datastore knows which entities > have the value you are looking for. Basically, this index has been created > already. > > 8 seconds is pretty high, but in general, it depends on the size of your > entities and whether offset is being used. The datastore is a distributed > key-value store. Rather than query for 1000 entities, is there a way you can > limit what you are querying for to a key-value retrieval? Think of the > datastore as a giant distributed, persistent hash table with additional > functionality: how would you optimize data retrieval for this type of store? > My guess is the functionality you're looking for can be either cached, or > you can do a keys only query and lazy query, or you can try to store the > data in sharded "slabs" - that is, store a denormalized cache of the data > you need from 200 temporally sharded lists of posts - it's unlikely that you > need all the data from each post. > > A good project to study would be the Jaiku > engine:http://code.google.com/p/jaikuengine/ > > > > > > On Tue, Feb 9, 2010 at 9:10 PM, Brian <[email protected]> wrote: > > I am also getting and closing a new PersistenceManager around the > > query, if that matters > > > On Feb 9, 11:08 pm, Brian <[email protected]> wrote: > > > Hi guys, > > > > I was wondering if anyone has found a less CPU intensive way to query. > > > I basically have a bunch of small objects in the database I want out. > > > I tried a few ways to get them out, including an extent and a query. I > > > ended up settling on this: > > > > Query query = pm.newQuery(Post.class); > > > query.setOrdering("postedDate desc"); > > > query.setRange(start, end); > > > List<Post> posts = (List<Post>) query.execute(); > > > > Doing a range of 0 to 1000.. figuring the 1000 newest items is good > > > enough. > > > > I find that this tends to be pretty bad on my quota CPU.. like maybe 8 > > > seconds of CPU time per query, if there are 1000 posts. Is this just > > > how it is? Or do I need to add an index to postedDate maybe? > > > > I plan to play with it more tomorow, but if anyone has insight on > > > getting stuff out of the database fast (from both a time and CPU quota > > > standpoint), I would appreciate it. I have never seen a datbase take> .1 > > seconds for such a stupid silly query, I have to be doing > > > > something wrong. > > > > Thanks! > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine-java%2B > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App > Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
