Hi Andrei, Well, the datastore viewer defaults to a limit of 20 (unless you change it via url). Did you have any limit set on your low-level api call?? I don't know when the zig zag merge join decides it has enough and gives up. Either it is based on some time threshold or it is based on number of candidate keys it has thrown away or perhaps some other metric. So it's definitely possible that the datastore viewer could show 20 records while your code has some higher limit (or none) and thus it surpasses the threshold and aborts. Stephen
2011/4/15 Andrei Cosmin Fifiiţă <[email protected]> > Well the Stephen's explanation seems more logical... > So that means that the limit count influences the success of the query ? > I had a situation where my query (with 3 params) failed in my app > (using low-level api) but succeeded in the App console (datastore > viewer), using GQL... How can that be ? > > On 14 April 2011 21:59, Alexandru Farcaş <[email protected]> > wrote: > > In this presentation, > http://dl.google.com/io/2009/pres/W_0415_Building_Scalable_Complex_App_Engines.pdf > , > > at slide 53, google say that "to many overlapping values = lots of zig > > - zagging" > > > > > > I test my app with queries with individual results (90000, <500) and I > > didn't receive "Datastore need index exception...". > > When I get this exception the query time is ~80ms. It looks like it > > didn't try to zig zag anything... > > > > > > On Apr 14, 8:34 pm, Robert Kluin <[email protected]> wrote: > >> Hi Stephen, > >> Yeah, I didn't explain very that clearly at all; I think you did a > >> significantly better job: "you have too few overlapping entities in > >> the two sets." That is exactly what I was trying to say. ;) > >> > >> For those interested, the merge-join algorithm is explained here: > >> > http://sites.google.com/site/io/under-the-covers-of-the-google-app-en... > >> > >> Robert > >> > >> 2011/4/15 Stephen Johnson <[email protected]>: > >> > >> > >> > >> > >> > >> > >> > >> > Hi Alexander, > >> > I'm sorry to disagree with Robert, but the zig zag merge-join doesn't > return > >> > the various rows for each index back to the application code to > combine them > >> > in the application. It's a difficult algorithm to explain in an email > but > >> > basically the problem looks to be that with the 90000 entities and < > 500 > >> > entites case you have too few overlapping entites in the two sets. > >> > Basically, think of it as you start off with 90000 possible candidates > that > >> > could be part of the result set. Now, the zig zag merge join goes and > checks > >> > the other list (the less than 500 list), but it basically does this > one at a > >> > time. Every time it doesn't find a match it throws that candidate away > and > >> > goes on to the next. It has to do this for each equality filter so if > you > >> > have 4 then it has to check for indexes for each candidate. You can > see that > >> > this can take some time. Now if it finds enough matches fast enough to > >> > satisfy the query limit that is requested to be returned then all is > fine, > >> > but if it has to keep throwing candidates away and trying new ones > than it > >> > times out and reports that it needs an index. So with your queries > that have > >> > large individual results (90000, 50000 and 50000) you obviously have a > large > >> > overlapping set and it finds enough matches very quickly, but when > you have > >> > large sets combined with small sets (90000 and <500) then you have few > >> > overlaps but lots of checking to do. > >> > I hope this helps and if I misinterpreted Robert's post then I > apologize to > >> > him. > >> > Stephen > >> > >> > On Thu, Apr 14, 2011 at 8:36 AM, Alexandru Farcaş > >> > <[email protected]> wrote: > >> > >> >> Hi Robert, > >> >> Thanks for your answer. > >> >> If the number of results returned by each filter individually is > >> >> smaller than, let's say 1000, the "final" number of results will be > >> >> returned correctly? > >> >> I want to know if there are some known limits for this type of query > >> >> on a single list property (maybe for the size of the list, for the > >> >> number of the results returned by each filter, for the number of > >> >> filters, etc...). I didn't find the answer for this question. > >> >> In all queries that returns datastore need index exception I have one > >> >> filter that returns almost 90000 entities and one that returns < 500 > >> >> results. > >> >> But I have also many queries with 3 filters (90000 results, 50000 > >> >> results, 50000 results) that returns the correct results. > >> > >> >> Hi Andrei, > >> >> Yes, almost the same queries: the difference is that in the second > >> >> table I have a single list property. This list contains the rest of > >> >> fields from the first table. > >> >> I tried to limit the number of the results from datastore viewer but > >> >> with the same result :(. > >> > >> >> On Apr 14, 5:03 pm, Andrei Cosmin Fifiiţă <[email protected]> > >> >> wrote: > >> >> > And also, i limit the number of results (for ex when implementing > >> >> > pagination)... Does that affect the success of the query ? > >> > >> >> > 2011/4/14 Andrei Cosmin Fifiiţă <[email protected]>: > >> > >> >> > > I don't want to create custom indices cause they will fail... but > does > >> >> > > the > >> >> > > order of the equality filters matter? > >> > >> >> > > On Apr 14, 2011 3:38 AM, "Robert Kluin" <[email protected]> > >> >> > > wrote: > >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > Groups > >> >> "Google App Engine" group. > >> >> To post to this group, send email to > [email protected]. > >> >> To unsubscribe from this group, send email to > >> >> [email protected]. > >> >> For more options, visit this group at > >> >>http://groups.google.com/group/google-appengine?hl=en. > >> > >> > -- > >> > You received this message because you are subscribed to the Google > Groups > >> > "Google App Engine" group. > >> > To post to this group, send email to > [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]. > >> > For more options, visit this group at > >> >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > [email protected]. > > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
