Hi Stephen,
Yeah, I didn't explain very that clearly at all; I think you did a
significantly better job: "you have too few overlapping entities in
the two sets." That is exactly what I was trying to say. ;)
For those interested, the merge-join algorithm is explained here:
http://sites.google.com/site/io/under-the-covers-of-the-google-app-engine-datastore
Robert
2011/4/15 Stephen Johnson <[email protected]>:
> Hi Alexander,
> I'm sorry to disagree with Robert, but the zig zag merge-join doesn't return
> the various rows for each index back to the application code to combine them
> in the application. It's a difficult algorithm to explain in an email but
> basically the problem looks to be that with the 90000 entities and < 500
> entites case you have too few overlapping entites in the two sets.
> Basically, think of it as you start off with 90000 possible candidates that
> could be part of the result set. Now, the zig zag merge join goes and checks
> the other list (the less than 500 list), but it basically does this one at a
> time. Every time it doesn't find a match it throws that candidate away and
> goes on to the next. It has to do this for each equality filter so if you
> have 4 then it has to check for indexes for each candidate. You can see that
> this can take some time. Now if it finds enough matches fast enough to
> satisfy the query limit that is requested to be returned then all is fine,
> but if it has to keep throwing candidates away and trying new ones than it
> times out and reports that it needs an index. So with your queries that have
> large individual results (90000, 50000 and 50000) you obviously have a large
> overlapping set and it finds enough matches very quickly, but when you have
> large sets combined with small sets (90000 and <500) then you have few
> overlaps but lots of checking to do.
> I hope this helps and if I misinterpreted Robert's post then I apologize to
> him.
> Stephen
>
> On Thu, Apr 14, 2011 at 8:36 AM, Alexandru Farcaş
> <[email protected]> wrote:
>>
>> Hi Robert,
>> Thanks for your answer.
>> If the number of results returned by each filter individually is
>> smaller than, let's say 1000, the "final" number of results will be
>> returned correctly?
>> I want to know if there are some known limits for this type of query
>> on a single list property (maybe for the size of the list, for the
>> number of the results returned by each filter, for the number of
>> filters, etc...). I didn't find the answer for this question.
>> In all queries that returns datastore need index exception I have one
>> filter that returns almost 90000 entities and one that returns < 500
>> results.
>> But I have also many queries with 3 filters (90000 results, 50000
>> results, 50000 results) that returns the correct results.
>>
>>
>> Hi Andrei,
>> Yes, almost the same queries: the difference is that in the second
>> table I have a single list property. This list contains the rest of
>> fields from the first table.
>> I tried to limit the number of the results from datastore viewer but
>> with the same result :(.
>>
>>
>> On Apr 14, 5:03 pm, Andrei Cosmin Fifiiţă <[email protected]>
>> wrote:
>> > And also, i limit the number of results (for ex when implementing
>> > pagination)... Does that affect the success of the query ?
>> >
>> > 2011/4/14 Andrei Cosmin Fifiiţă <[email protected]>:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > > I don't want to create custom indices cause they will fail... but does
>> > > the
>> > > order of the equality filters matter?
>> >
>> > > On Apr 14, 2011 3:38 AM, "Robert Kluin" <[email protected]>
>> > > wrote:
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
