Nick, Please make sure this index definition gets added to the MapReduce documentation (unless it's not that big of deal). I just happened to stumble across the log entry; I'm sure many wouldn't notice it.
j On Jun 9, 8:33 am, "Nick Johnson (Google)" <[email protected]> wrote: > 2010/6/9 Jason C <[email protected]> > > > > > > Why do you need to walk your data in __key__ reverse order? > > > This seems to be something that the MapReduce for App Engine does. It > > looks like, when sharding, half the shards are from the asc index and > > half are from the desc index, presumably to spread the scan load over > > a pair of indexes instead of just hammering one. > > No, the reverse index is required in order to determine what the maximum key > is, so it can lexicographically shard the work between minimum and maximum > key. Without a descending index, the mapreduce is forced to do a binary > search to find the maximum key. > > -Nick Johnson > > > > > > > > > j > > > On Jun 9, 2:05 am, Alkis Evlogimenos ('Αλκης Ευλογημένος) > > <[email protected]> wrote: > > > Why do you need to walk your data in __key__ reverse order? > > > > - alkis > > > > 2010/6/9 Jason C <[email protected]> > > > > > Ok, thanks. > > > > > So it seems for MapReduce for App Engine to work optimally, I would > > > > need to manually add the requested index (and then, somewhat > > > > ironically, use MapReduce for App Engine to walk the entire data set > > > > and put each entity to get the new index populated). > > > > > j > > > > > On Jun 8, 5:19 pm, Alkis Evlogimenos ('Αλκης Ευλογημένος) > > > > <[email protected]> wrote: > > > > > There is no desc index on __key__. Each indexed property has an asc > > and > > > > desc > > > > > index on it but that doesn't apply for keys. > > > > > > - alkis > > > > > > On Wed, Jun 9, 2010 at 12:39 AM, Jason C <[email protected]> > > > > wrote: > > > > > > I'm attempting to use mapreduce for app engine as presented at I/O > > > > > > 2010. > > > > > > > I just noticed the following warning message in our logs (on / > > > > > > mapreduce/command/start_job): > > > > > > > Cannot create accurate approximation of keyspace, guessing instead. > > > > > > Please address this problem: no matching index found. > > > > > > This query needs this index: > > > > > > - kind: AccountKeyword > > > > > > properties: > > > > > > - name: __key__ > > > > > > direction: desc > > > > > > > This seems to suggest that there is no (desc) index for the key of > > our > > > > > > Model. However, isn't this created automatically? Indeed, I can't > > even > > > > > > think of a way to _not_ have it created (i.e., I can't think of a > > way > > > > > > to specify indexed=False for the key). > > > > > > > Is this an error in reporting, or is something very wrong with our > > > > > > Datastore? > > > > > > > Thanks, > > > > > > j > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > > > "Google App Engine" group. > > > > > > To post to this group, send email to > > [email protected] > > > > . > > > > > > To unsubscribe from this group, send email to > > > > > > [email protected]<google-appengine%2Bunsubscrib > > > > > > [email protected]><google-appengine%2Bunsubscrib > > [email protected]><google-appengine%2Bunsubscrib > > > > [email protected]> > > > > > > . > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to [email protected] > > . > > > > To unsubscribe from this group, send email to > > > > [email protected]<google-appengine%2Bunsubscrib > > > > [email protected]><google-appengine%2Bunsubscrib > > [email protected]> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%2Bunsubscrib > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: > Registered in Dublin, Ireland, Registration Number: 368047 > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
