2010/6/9 Jason C <[email protected]>
>
> > Why do you need to walk your data in __key__ reverse order?
>
> This seems to be something that the MapReduce for App Engine does. It
> looks like, when sharding, half the shards are from the asc index and
> half are from the desc index, presumably to spread the scan load over
> a pair of indexes instead of just hammering one.
>
No, the reverse index is required in order to determine what the maximum key
is, so it can lexicographically shard the work between minimum and maximum
key. Without a descending index, the mapreduce is forced to do a binary
search to find the maximum key.
-Nick Johnson
>
> j
>
>
> On Jun 9, 2:05 am, Alkis Evlogimenos ('Αλκης Ευλογημένος)
> <[email protected]> wrote:
> > Why do you need to walk your data in __key__ reverse order?
> >
> > - alkis
> >
> > 2010/6/9 Jason C <[email protected]>
> >
> >
> >
> > > Ok, thanks.
> >
> > > So it seems for MapReduce for App Engine to work optimally, I would
> > > need to manually add the requested index (and then, somewhat
> > > ironically, use MapReduce for App Engine to walk the entire data set
> > > and put each entity to get the new index populated).
> >
> > > j
> >
> > > On Jun 8, 5:19 pm, Alkis Evlogimenos ('Αλκης Ευλογημένος)
> > > <[email protected]> wrote:
> > > > There is no desc index on __key__. Each indexed property has an asc
> and
> > > desc
> > > > index on it but that doesn't apply for keys.
> >
> > > > - alkis
> >
> > > > On Wed, Jun 9, 2010 at 12:39 AM, Jason C <[email protected]>
> > > wrote:
> > > > > I'm attempting to use mapreduce for app engine as presented at I/O
> > > > > 2010.
> >
> > > > > I just noticed the following warning message in our logs (on /
> > > > > mapreduce/command/start_job):
> >
> > > > > Cannot create accurate approximation of keyspace, guessing instead.
> > > > > Please address this problem: no matching index found.
> > > > > This query needs this index:
> > > > > - kind: AccountKeyword
> > > > > properties:
> > > > > - name: __key__
> > > > > direction: desc
> >
> > > > > This seems to suggest that there is no (desc) index for the key of
> our
> > > > > Model. However, isn't this created automatically? Indeed, I can't
> even
> > > > > think of a way to _not_ have it created (i.e., I can't think of a
> way
> > > > > to specify indexed=False for the key).
> >
> > > > > Is this an error in reporting, or is something very wrong with our
> > > > > Datastore?
> >
> > > > > Thanks,
> > > > > j
> >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > Groups
> > > > > "Google App Engine" group.
> > > > > To post to this group, send email to
> [email protected]
> > > .
> > > > > To unsubscribe from this group, send email to
> > > > > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib
> [email protected]><google-appengine%2Bunsubscrib
> > > [email protected]>
> > > > > .
> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/google-appengine?hl=en.
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Google App Engine" group.
> > > To post to this group, send email to [email protected]
> .
> > > To unsubscribe from this group, send email to
> > > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib
> [email protected]>
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected]<google-appengine%[email protected]>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. ::
Registered in Dublin, Ireland, Registration Number: 368047
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
