Hi,
This is the quintessential exploding index problem that I am fixing. The
problem is that we don't currently support zigzag merge join and sorting at
the same time. You have several options to address this:
- Remove the sort order
- Slightly worse results until you add it back when I add support for
zigzag + sort
- Encode the sort order in the key_name
- since zigzag results are sorted in key order you can prefix the
key_name with a value that will replicate that order (key_names
are sorted
by lexicographically). This will only work if all the entities
in your query
either have no parent or the same parent.
- Restrict the # of tag= in your query
- If you try to sort today you have a upper bound of 5 tags per query
(which in turn restricts you to 5 tags per entity). You can get more
tags/entity if you further restrict the # of tags in the query.
- Alfred
On Mon, Sep 20, 2010 at 10:08 AM, DevShop <[email protected]> wrote:
> > Do you have 2 list properties? A single list property won't cause
> exploding
> > indexes. Can you post the whole property?
>
> Thanks for your help!
>
> Do you mean "Can you post the whole class?" ? Here it is, the list
> property that explodes is the first property:
>
> class Report(db.Model):
>
> # List of tags for this report, i.e. ['seattle', 'rockroll_rep',
> 'rock', 'zztop', 'qwest_field'].
> tags = db.ListProperty(basestring, required=True,
> validator=are_valid_tags)
>
> status = db.StringProperty(required=True, choices = ['PENDING',
> 'APPROVED', 'DECLINED', 'FLAGGED'], default = 'PENDING')
>
> # If status is PENDING, the reason is detailed here
> pending_reason = db.TextProperty(default='')
>
> # If status is FLAGGED, the reason is detailed here
> flagged_reason = db.TextProperty(default='')
>
> # The report, first line is the headline.
> text = db.TextProperty(default='')
>
> # URL with more info about this report.
> url = db.StringProperty(default=None, validator=is_valid_url)
>
> # Editor that approved this report. Could be None if report
> submitted
> from an 'APPROVED' reporter.
> editor = db.ReferenceProperty(Reporter,
> collection_name='approved_reports', default=None)
>
> # Date/time of insertion of this report into the system.
> creation_time = db.DateTimeProperty(auto_now_add=True)
>
> # Date/time of approval of this report, might be the same as
> creation_time if it was automatically approved.
> approval_time = db.DateTimeProperty(auto_now_add=True)
>
>
> On Sep 20, 9:35 am, "Ikai Lan (Google)"
> <[email protected]<ikai.l%[email protected]>
> >
> wrote:
> > Do you have 2 list properties? A single list property won't cause
> exploding
> > indexes. Can you post the whole property?
> >
> > --
> > Ikai Lan
> > Developer Programs Engineer, Google App Engine
> > Blogger:http://googleappengine.blogspot.com
> > Reddit:http://www.reddit.com/r/appengine
> > Twitter:http://twitter.com/app_engine
> >
> > On Mon, Sep 20, 2010 at 11:56 AM, DevShop <[email protected]> wrote:
> > > Ikai, thanks for the prompt response.
> >
> > > Here's our situation: records on entity 'Report' are tagged with
> > > multiple keywords ('tags') like: 'music', 'event', 'downtown',
> > > 'zz_top', 'concert', 'g_arena', 'september', '2010'. People might
> > > search for events downtown in September like this:
> >
> > > city.domain.com/tags/event/september/2010/downtown
> >
> > > So we are using a 'tags = db.ListProperty(basestring, required=True,
> > > validator=are_valid_tags)' in our table with queries like:
> >
> > > reports = db.Query(Report).filter('tags =', subdomain)
> > > for tag in tags:
> > > reports = reports.filter('tags =', tag)
> > > reports = reports.order('-approval_time').fetch(int(25))
> >
> > > Advice and/or insight would be much appreciated!
> >
> > > On Sep 20, 7:59 am, "Ikai Lan (Google)"
> > > <[email protected]<ikai.l%[email protected]>
> <ikai.l%[email protected] <ikai.l%[email protected]>>
> >
> > > wrote:
> > > > In general, we do not release ETAs for features. Alfred's
> enhancements
> > > are
> > > > coming in multiple small feature releases over the coming months.
> >
> > > > It'd be more practical to post what's causing an exploding index, as
> > > there
> > > > are probably ways to design around it and still meet your needs.
> >
> > > > --
> > > > Ikai Lan
> > > > Developer Programs Engineer, Google App Engine
> > > > Blogger:http://googleappengine.blogspot.com
> > > > Reddit:http://www.reddit.com/r/appengine
> > > > Twitter:http://twitter.com/app_engine
> >
> > > > On Mon, Sep 20, 2010 at 10:49 AM, DevShop <[email protected]>
> wrote:
> > > > > At the Google IO Conference in May, Alfred Fuller talked about his
> > > > > solution for the exploding indexes problem.
> >
> > > > > I cannot deploy my app due to an exploding index, just seeding the
> > > > > database with initial data triggers the problem.
> >
> > > > > Any updates as to when the solution will be deployed by the App
> Engine
> > > > > team?
> >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > Groups
> > > > > "Google App Engine" group.
> > > > > To post to this group, send email to
> [email protected]
> > > .
> > > > > To unsubscribe from this group, send email to
> > > > > [email protected]<google-appengine%[email protected]>
> <google-appengine%[email protected]<google-appengine%[email protected]>
> >
> > > <google-appengine%[email protected]<google-appengine%[email protected]>
> <google-appengine%[email protected]<google-appengine%[email protected]>
> >
> >
> > > > > .
> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/google-appengine?hl=en.
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Google App Engine" group.
> > > To post to this group, send email to [email protected]
> .
> > > To unsubscribe from this group, send email to
> > > [email protected]<google-appengine%[email protected]>
> <google-appengine%[email protected]<google-appengine%[email protected]>
> >
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected]<google-appengine%[email protected]>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
