Wow, thanks Alfred, this is good info.
> - Remove the sort order
> - Restrict the # of tag= in your query
OR (choose one) or AND (we have to do both)?
If I just have to remove the sort order from the query and manipulate
the keys value then I can live with that. But if I have to also
restrict the number of tags to 5 then I guess I'll have to wait until
you complete your work since our spec requires up to 10 tags.
Error message when uploading data through the remote API:
google.appengine.api.datastore_errors.BadRequestError: Too many
indexed
properties for entity: app: "sampleapp",path < Element { type:
"Report", id: 6001 }>
This index put it over the limit: entity_type: "Report",ancestor:
false,
Property { name: "tags", direction: ASCENDING},Property { name:
"tags",
direction: ASCENDING},Property { name: "tags", direction:
ASCENDING},
Property { name: "tags", direction: ASCENDING},Property { name:
"approval_time", direction: DESCENDING}
On Sep 20, 11:52 am, Alfred Fuller <[email protected]>
wrote:
> Hi,
>
> This is the quintessential exploding index problem that I am fixing. The
> problem is that we don't currently support zigzag merge join and sorting at
> the same time. You have several options to address this:
>
> - Remove the sort order
> - Slightly worse results until you add it back when I add support for
> zigzag + sort
> - Encode the sort order in the key_name
> - since zigzag results are sorted in key order you can prefix the
> key_name with a value that will replicate that order (key_names
> are sorted
> by lexicographically). This will only work if all the entities
> in your query
> either have no parent or the same parent.
> - Restrict the # of tag= in your query
> - If you try to sort today you have a upper bound of 5 tags per query
> (which in turn restricts you to 5 tags per entity). You can get more
> tags/entity if you further restrict the # of tags in the query.
>
> - Alfred
>
> On Mon, Sep 20, 2010 at 10:08 AM, DevShop <[email protected]> wrote:
> > > Do you have 2 list properties? A single list property won't cause
> > exploding
> > > indexes. Can you post the whole property?
>
> > Thanks for your help!
>
> > Do you mean "Can you post the whole class?" ? Here it is, the list
> > property that explodes is the first property:
>
> > class Report(db.Model):
>
> > # List of tags for this report, i.e. ['seattle', 'rockroll_rep',
> > 'rock', 'zztop', 'qwest_field'].
> > tags = db.ListProperty(basestring, required=True,
> > validator=are_valid_tags)
>
> > status = db.StringProperty(required=True, choices = ['PENDING',
> > 'APPROVED', 'DECLINED', 'FLAGGED'], default = 'PENDING')
>
> > # If status is PENDING, the reason is detailed here
> > pending_reason = db.TextProperty(default='')
>
> > # If status is FLAGGED, the reason is detailed here
> > flagged_reason = db.TextProperty(default='')
>
> > # The report, first line is the headline.
> > text = db.TextProperty(default='')
>
> > # URL with more info about this report.
> > url = db.StringProperty(default=None, validator=is_valid_url)
>
> > # Editor that approved this report. Could be None if report
> > submitted
> > from an 'APPROVED' reporter.
> > editor = db.ReferenceProperty(Reporter,
> > collection_name='approved_reports', default=None)
>
> > # Date/time of insertion of this report into the system.
> > creation_time = db.DateTimeProperty(auto_now_add=True)
>
> > # Date/time of approval of this report, might be the same as
> > creation_time if it was automatically approved.
> > approval_time = db.DateTimeProperty(auto_now_add=True)
>
> > On Sep 20, 9:35 am, "Ikai Lan (Google)"
> > <[email protected]<ikai.l%[email protected]>
>
> > wrote:
> > > Do you have 2 list properties? A single list property won't cause
> > exploding
> > > indexes. Can you post the whole property?
>
> > > --
> > > Ikai Lan
> > > Developer Programs Engineer, Google App Engine
> > > Blogger:http://googleappengine.blogspot.com
> > > Reddit:http://www.reddit.com/r/appengine
> > > Twitter:http://twitter.com/app_engine
>
> > > On Mon, Sep 20, 2010 at 11:56 AM, DevShop <[email protected]> wrote:
> > > > Ikai, thanks for the prompt response.
>
> > > > Here's our situation: records on entity 'Report' are tagged with
> > > > multiple keywords ('tags') like: 'music', 'event', 'downtown',
> > > > 'zz_top', 'concert', 'g_arena', 'september', '2010'. People might
> > > > search for events downtown in September like this:
>
> > > > city.domain.com/tags/event/september/2010/downtown
>
> > > > So we are using a 'tags = db.ListProperty(basestring, required=True,
> > > > validator=are_valid_tags)' in our table with queries like:
>
> > > > reports = db.Query(Report).filter('tags =', subdomain)
> > > > for tag in tags:
> > > > reports = reports.filter('tags =', tag)
> > > > reports = reports.order('-approval_time').fetch(int(25))
>
> > > > Advice and/or insight would be much appreciated!
>
> > > > On Sep 20, 7:59 am, "Ikai Lan (Google)"
> > > > <[email protected]<ikai.l%[email protected]>
> > <ikai.l%[email protected] <ikai.l%[email protected]>>
>
> > > > wrote:
> > > > > In general, we do not release ETAs for features. Alfred's
> > enhancements
> > > > are
> > > > > coming in multiple small feature releases over the coming months.
>
> > > > > It'd be more practical to post what's causing an exploding index, as
> > > > there
> > > > > are probably ways to design around it and still meet your needs.
>
> > > > > --
> > > > > Ikai Lan
> > > > > Developer Programs Engineer, Google App Engine
> > > > > Blogger:http://googleappengine.blogspot.com
> > > > > Reddit:http://www.reddit.com/r/appengine
> > > > > Twitter:http://twitter.com/app_engine
>
> > > > > On Mon, Sep 20, 2010 at 10:49 AM, DevShop <[email protected]>
> > wrote:
> > > > > > At the Google IO Conference in May, Alfred Fuller talked about his
> > > > > > solution for the exploding indexes problem.
>
> > > > > > I cannot deploy my app due to an exploding index, just seeding the
> > > > > > database with initial data triggers the problem.
>
> > > > > > Any updates as to when the solution will be deployed by the App
> > Engine
> > > > > > team?
>
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > Groups
> > > > > > "Google App Engine" group.
> > > > > > To post to this group, send email to
> > [email protected]
> > > > .
> > > > > > To unsubscribe from this group, send email to
> > > > > > [email protected]<google-appengine%[email protected]>
> > <google-appengine%[email protected]<google-appengine%[email protected]>
>
> > > > <google-appengine%[email protected]<google-appengine%[email protected]>
> > <google-appengine%[email protected]<google-appengine%[email protected]>
>
> > > > > > .
> > > > > > For more options, visit this group at
> > > > > >http://groups.google.com/group/google-appengine?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Google App Engine" group.
> > > > To post to this group, send email to [email protected]
> > .
> > > > To unsubscribe from this group, send email to
> > > > [email protected]<google-appengine%[email protected]>
> > <google-appengine%[email protected]<google-appengine%[email protected]>
>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to [email protected].
> > To unsubscribe from this group, send email to
> > [email protected]<google-appengine%[email protected]>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
