hi guys,
im having the same problem with multiple tags and an order in the query.
so you are saying that without order i wont have exploding indexes even if i
have 20 keywords im filtering on?
when do you think will you finish your work?
thank you!
On Sep 20, 2010, at 3:57 PM, DevShop wrote:
> Wow, thanks Alfred, this is good info.
>
>> - Remove the sort order
>> - Restrict the # of tag= in your query
>
> OR (choose one) or AND (we have to do both)?
>
> If I just have to remove the sort order from the query and manipulate
> the keys value then I can live with that. But if I have to also
> restrict the number of tags to 5 then I guess I'll have to wait until
> you complete your work since our spec requires up to 10 tags.
>
> Error message when uploading data through the remote API:
>
> google.appengine.api.datastore_errors.BadRequestError: Too many
> indexed
> properties for entity: app: "sampleapp",path < Element { type:
> "Report", id: 6001 }>
>
> This index put it over the limit: entity_type: "Report",ancestor:
> false,
> Property { name: "tags", direction: ASCENDING},Property { name:
> "tags",
> direction: ASCENDING},Property { name: "tags", direction:
> ASCENDING},
> Property { name: "tags", direction: ASCENDING},Property { name:
> "approval_time", direction: DESCENDING}
>
>
> On Sep 20, 11:52 am, Alfred Fuller <[email protected]>
> wrote:
>> Hi,
>>
>> This is the quintessential exploding index problem that I am fixing. The
>> problem is that we don't currently support zigzag merge join and sorting at
>> the same time. You have several options to address this:
>>
>> - Remove the sort order
>> - Slightly worse results until you add it back when I add support for
>> zigzag + sort
>> - Encode the sort order in the key_name
>> - since zigzag results are sorted in key order you can prefix the
>> key_name with a value that will replicate that order (key_names
>> are sorted
>> by lexicographically). This will only work if all the entities
>> in your query
>> either have no parent or the same parent.
>> - Restrict the # of tag= in your query
>> - If you try to sort today you have a upper bound of 5 tags per query
>> (which in turn restricts you to 5 tags per entity). You can get more
>> tags/entity if you further restrict the # of tags in the query.
>>
>> - Alfred
>>
>> On Mon, Sep 20, 2010 at 10:08 AM, DevShop <[email protected]> wrote:
>>>> Do you have 2 list properties? A single list property won't cause
>>> exploding
>>>> indexes. Can you post the whole property?
>>
>>> Thanks for your help!
>>
>>> Do you mean "Can you post the whole class?" ? Here it is, the list
>>> property that explodes is the first property:
>>
>>> class Report(db.Model):
>>
>>> # List of tags for this report, i.e. ['seattle', 'rockroll_rep',
>>> 'rock', 'zztop', 'qwest_field'].
>>> tags = db.ListProperty(basestring, required=True,
>>> validator=are_valid_tags)
>>
>>> status = db.StringProperty(required=True, choices = ['PENDING',
>>> 'APPROVED', 'DECLINED', 'FLAGGED'], default = 'PENDING')
>>
>>> # If status is PENDING, the reason is detailed here
>>> pending_reason = db.TextProperty(default='')
>>
>>> # If status is FLAGGED, the reason is detailed here
>>> flagged_reason = db.TextProperty(default='')
>>
>>> # The report, first line is the headline.
>>> text = db.TextProperty(default='')
>>
>>> # URL with more info about this report.
>>> url = db.StringProperty(default=None, validator=is_valid_url)
>>
>>> # Editor that approved this report. Could be None if report
>>> submitted
>>> from an 'APPROVED' reporter.
>>> editor = db.ReferenceProperty(Reporter,
>>> collection_name='approved_reports', default=None)
>>
>>> # Date/time of insertion of this report into the system.
>>> creation_time = db.DateTimeProperty(auto_now_add=True)
>>
>>> # Date/time of approval of this report, might be the same as
>>> creation_time if it was automatically approved.
>>> approval_time = db.DateTimeProperty(auto_now_add=True)
>>
>>> On Sep 20, 9:35 am, "Ikai Lan (Google)"
>>> <[email protected]<ikai.l%[email protected]>
>>
>>> wrote:
>>>> Do you have 2 list properties? A single list property won't cause
>>> exploding
>>>> indexes. Can you post the whole property?
>>
>>>> --
>>>> Ikai Lan
>>>> Developer Programs Engineer, Google App Engine
>>>> Blogger:http://googleappengine.blogspot.com
>>>> Reddit:http://www.reddit.com/r/appengine
>>>> Twitter:http://twitter.com/app_engine
>>
>>>> On Mon, Sep 20, 2010 at 11:56 AM, DevShop <[email protected]> wrote:
>>>>> Ikai, thanks for the prompt response.
>>
>>>>> Here's our situation: records on entity 'Report' are tagged with
>>>>> multiple keywords ('tags') like: 'music', 'event', 'downtown',
>>>>> 'zz_top', 'concert', 'g_arena', 'september', '2010'. People might
>>>>> search for events downtown in September like this:
>>
>>>>> city.domain.com/tags/event/september/2010/downtown
>>
>>>>> So we are using a 'tags = db.ListProperty(basestring, required=True,
>>>>> validator=are_valid_tags)' in our table with queries like:
>>
>>>>> reports = db.Query(Report).filter('tags =', subdomain)
>>>>> for tag in tags:
>>>>> reports = reports.filter('tags =', tag)
>>>>> reports = reports.order('-approval_time').fetch(int(25))
>>
>>>>> Advice and/or insight would be much appreciated!
>>
>>>>> On Sep 20, 7:59 am, "Ikai Lan (Google)"
>>>>> <[email protected]<ikai.l%[email protected]>
>>> <ikai.l%[email protected] <ikai.l%[email protected]>>
>>
>>>>> wrote:
>>>>>> In general, we do not release ETAs for features. Alfred's
>>> enhancements
>>>>> are
>>>>>> coming in multiple small feature releases over the coming months.
>>
>>>>>> It'd be more practical to post what's causing an exploding index, as
>>>>> there
>>>>>> are probably ways to design around it and still meet your needs.
>>
>>>>>> --
>>>>>> Ikai Lan
>>>>>> Developer Programs Engineer, Google App Engine
>>>>>> Blogger:http://googleappengine.blogspot.com
>>>>>> Reddit:http://www.reddit.com/r/appengine
>>>>>> Twitter:http://twitter.com/app_engine
>>
>>>>>> On Mon, Sep 20, 2010 at 10:49 AM, DevShop <[email protected]>
>>> wrote:
>>>>>>> At the Google IO Conference in May, Alfred Fuller talked about his
>>>>>>> solution for the exploding indexes problem.
>>
>>>>>>> I cannot deploy my app due to an exploding index, just seeding the
>>>>>>> database with initial data triggers the problem.
>>
>>>>>>> Any updates as to when the solution will be deployed by the App
>>> Engine
>>>>>>> team?
>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>> Groups
>>>>>>> "Google App Engine" group.
>>>>>>> To post to this group, send email to
>>> [email protected]
>>>>> .
>>>>>>> To unsubscribe from this group, send email to
>>>>>>> [email protected]<google-appengine%[email protected]>
>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>
>>>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>
>>>>>>> .
>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>> Groups
>>>>> "Google App Engine" group.
>>>>> To post to this group, send email to [email protected]
>>> .
>>>>> To unsubscribe from this group, send email to
>>>>> [email protected]<google-appengine%[email protected]>
>>> <google-appengine%[email protected]<google-appengine%[email protected]>
>>
>>>>> .
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to [email protected].
>>> To unsubscribe from this group, send email to
>>> [email protected]<google-appengine%[email protected]>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
