Hey Richard, I believe indexes are written similar to what you've stated there. You can use namespaces to mitigate this problem. The other way to "shard" would be to prefix the value with some known value -- like a letter. That way you spread the load across multiple index shards. Both of these result in the same basic pain point though, to do a "global" query you need to run a query within each of the shards then join the results.
In the past I've worked around this problem in several different ways. The best is to see if there is a natural way to "shard" the index such that you won't need to do queries across the shards. If you can do that, you're done. It also depends on what the problem value is used for. In some cases it may not need to be so precise, so you can spread the load around a bit by randomizing the value within some acceptable range of error. I've also used the task-queue to control the rate at which the problem entities get written. This can help if you've got very bursty write rates. Robert On Wed, Feb 1, 2012 at 09:00, Richard Watson <[email protected]> wrote: > I haven't mixed it up with entity groups, I refer only to sharding in the > context of an index. Apologies if my use of the terminology confused the > issue in any way. > > My point was that the entries in the index are surely already sharded by > some kind of prefix. There's no way all apps that save current millis in a > field are being indexed on the same tablet. They must have some type of > separation, which could maybe be achieved by prefix. For example, the index > entry could be something like > [AppId][NamespaceId][EntityName][FieldName][123456789], which would remove > contention between apps. If [Namespace] is in there, then any apps that use > namespaces already have a built-in natural shard point for index entries. > > And I think the point about adding the indexed field value later is still > valid. Save the value in an unindexed field (e.g. unindexedTimestamp), and > then update the indexed field (indexedTimestamp) via a task later, > controlling the frequency of saves so it reduced impact on the index > tablets. That way you don't mess about with prefixes and adding result sets > together. > > Richard > > [1] Does an index entry look like this: > [AppId][NamespaceId][ > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/3Ltu7gGC8EsJ. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
