I would suggest any debutante interested in how the datastore to watch some of the Google IO videos related to App Engine. You should also read all of the articles related to the datastore. The 'datastore' is not BigTable; it is a layer implemented on top of BigTable. You might find the discussions on merge-joins interesting.
Regarding the sort-order, my statement probably lacked verbosity and clarity. Read up on how the datastore handles composite indexes. They are stored, more-or-less, as I mentioned. So, the 'final property' in the index will determine the 'initial' sort order. Also, it is the only property you can do a range-scan on using indexes. If you read some of the docs and articles and watch some of the IO video's, you'll have a much better understanding of how App Engine's indexes work. I'm not 100% sure what you mean by not being able to parallelize with a composite index. If you want to work on the dataset in parallel, start workers at different points in the composite index. Using composite indexes saves the step of combining the results from multiple indexes in-code. Robert On Sat, Feb 26, 2011 at 10:31, philip <[email protected]> wrote: > It is not correct to say that 'the range-scan / order on must go last'. If > the requested range-scan/sort-order is on the same property as the primary > sort then there is no work for appengine to do. The results simply fall out > in the required order. > > A dilettante such as myself expects a performance-optimised system to > suggest indexes sorted in such a way that they can take advantage of this, > but appengine did not do so. It suggested the same index regardless of what > sort order my query requested. This suggests to me that appengine is either > very highly performance-optimised in this area, or is not > performance-optimised at all. > From reading the BigTable paper (bigtable-osdi06.pdf) I see that BigTable > files are found by traversing a B+ tree with a maximum of 3 levels. If a > web search query is handled by taking the intersection of n single-property > index scans all of which go through these B+ trees, then that's good > performance optimisation. I assume that this is how google does web search. > But composite indexes force you to work linearly. You can't parallelise. > My guess is that an un-optimised composite index is significantly inferior > to parallel single-property index scans. > > So I can refine my question a little: how does appengine traverse its > composite indexes when performing a query ? > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
