Hi, On Fri, Dec 4, 2009 at 11:53 AM, jpmorganuk <[email protected] > wrote:
> Hi, > > I have been reading "How Index Building Works" by Ryan Barrett, > published on November 24, 2008 at > http://code.google.com/appengine/articles/index_building.html > > The article suggests the performance of indexing depends on the > performance/ability of a worker to process the index on a shard of > data in a given time (lease): "if the lease expires before the worker > is done, it gives up, discards the partially completed work, and the > shard becomes available for other workers to retry." > I have a table that I expect to contain millions of rows. I plan to > update around 8000 rows per night working at a rate of 10 updates per > second (I know the limit is currently 5 - let's leave this out of this > discussion for now). > > My table has many indexes because there are many ways of filtering the > data. I'd like these indexes to be updated as quickly as possible. > > Is there anything I can do in the design of my table to enable > Google's Worker/Shard model to index most efficiently, and thus speed > up indexing? > The article describes the process of building indexes when they're first created - this doesn't apply to updates to already built indexes. Updates are handled synchronously with the row being updated. > > For example, if my table is: > * Company Name (StringProperty) > * Company Address (StringProperty) > * Product Name (StringProperty) > * Product Short Description (StringProperty) > * Product Long Description (BlobProperty) > * Product Image (BlobProperty) > * Product Price (IntegerProperty) > * Product Location (GeoPtProperty) > * Product Feature Type 1 (StringProperty) > * Product Feature Type 2 (StringProperty) > * Product Feature Type 3 (StringProperty) > * Product Feature Type 4 (StringProperty) > * Product Feature Type 5 (StringProperty) > * Product Feature Type 6 (StringProperty) > * Product Feature Type 7 (StringProperty) > * Product Feature Type 8 (StringProperty) > * Product Feature Type 9 (StringProperty) > * Product Feature Type 10 (StringProperty) > > and my indexes are different ascending and descending combinations of > two or more of the fields Price, Location and the Feature Type <n> > fields, would the index rebuild be more efficient if I moved the non- > indexed fields into a separate table which links back to the main many- > indexed table using a ReferenceProperty? > The presence of unindexed fields won't affect indexing speed, but it will affect retrieval overhead, so if you don't refer to them in some queries, separating them out is a good idea. For simplicity, you may want to make the 'product feature n' fields a list property. -Nick Johnson > > Regards, > jpmorganuk > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
