Cross-posting my comments from the earlier thread. The restriction that every index'able property must be indexed at put-time when put into the datastore (using setProperty as opposed to setUnindexedProperty), has 2 MAJOR disadvantages with far-reaching repercussions: - you cannot index entities after they have been put (even with a custom index). You will have to re-write the entity to index it. - Each put is 4X more expensive in latency/clock-time (potentially), CPU-cost and storage. This seems un-necessary, especially if a single custom index will suffice all your querying needs.
I was really hoping to use custom indexes to buy major performance savings (in latency, CPU-cost and storage), and also be able to re- index every entity after the fact. But this restriction is heavy (preventing direct solution to a problem everyone may end up facing), and very expensive (costing us big-time). A way to disable automatic single-property indexes for certain properties or entities will help to solve this issue. On Mar 18, 10:36 am, Jeff Schnitzer <[email protected]> wrote: > On Thu, Mar 18, 2010 at 9:55 AM, Nick Johnson (Google) > > <[email protected]> wrote: > > On Thu, Mar 18, 2010 at 4:49 PM, Jeff Schnitzer <[email protected]> wrote: > > >> This current arrangement has some particularly nasty consequences: > > >> 1) If you want a custom index across 3 properties, you are now forced > >> to maintain seven indexes total - the three single-property ASC > >> indexes, the three single-property DESC indexes, and the actual custom > >> index. > > > None of the built in indexes require 'maintenance'. There's also only a > > single desc, and a single asc index for all properties. > > Sorry, perhaps my terminology is faltering a bit here - by "maintain" > I mean "write the values of". While there are only two index tables, > our hypothetical put() requires three writes to each table, plus > another write to the custom index table. This seems rather painful. > > >> 2) If you want to create a custom index across a property that does > >> not currently have a single-property index, you must manually go > >> through and reprocess you entire dataset. The automatic index > >> building isn't automatic. > > > If you mean that you want to index a field that was previously listed as an > > unindexed field then yes, you do. > > I understand that this is how it works, and that it was deliberately > chosen. My question is, why does it work this way? > > I would like you to change this behavior: If you call > setUnindexedProperty(), leave the property in custom indexes. Or at > least add a setProperty() method that only creates custom indexes, not > single-property indexes. > > The reason for this is that it more accurately reflects how people > actually use the datastore: > > 1) In a write-heavy application, indexes are expensive to maintain. > Forcing single-property indexes for every custom index is particularly > onerous since custom indexes may contain large numbers of properties. > > 2) People add custom indexes to enable new features; requiring that > single-property indexes exist means a lot of tedious data reprocessing > whenever you want to add such a feature. You can't rely on automatic > indexing anymore. > > In contrast, the current scheme of setUnindexedProperty() skipping > custom indexes doesn't really buy us developers any advantage. > > Thanks, > Jeff -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
