The zig-zag doesnt try and then gives up... it just doesnt try anything... it knows exactly when the query is executed... The fail happens very quickly... 100ms aprox... Second.. i tryied with 200 limit (which is max) and works
On 15 April 2011 22:04, Stephen Johnson <[email protected]> wrote: > Hi Andrei, > Well, the datastore viewer defaults to a limit of 20 (unless you change it > via url). Did you have any limit set on your low-level api call?? I don't > know when the zig zag merge join decides it has enough and gives up. Either > it is based on some time threshold or it is based on number of candidate > keys it has thrown away or perhaps some other metric. So it's definitely > possible that the datastore viewer could show 20 records while your code has > some higher limit (or none) and thus it surpasses the threshold and aborts. > Stephen > > 2011/4/15 Andrei Cosmin Fifiiţă <[email protected]> >> >> Well the Stephen's explanation seems more logical... >> So that means that the limit count influences the success of the query ? >> I had a situation where my query (with 3 params) failed in my app >> (using low-level api) but succeeded in the App console (datastore >> viewer), using GQL... How can that be ? >> >> On 14 April 2011 21:59, Alexandru Farcaş <[email protected]> >> wrote: >> > In this presentation, >> > http://dl.google.com/io/2009/pres/W_0415_Building_Scalable_Complex_App_Engines.pdf, >> > at slide 53, google say that "to many overlapping values = lots of zig >> > - zagging" >> > >> > >> > I test my app with queries with individual results (90000, <500) and I >> > didn't receive "Datastore need index exception...". >> > When I get this exception the query time is ~80ms. It looks like it >> > didn't try to zig zag anything... >> > >> > >> > On Apr 14, 8:34 pm, Robert Kluin <[email protected]> wrote: >> >> Hi Stephen, >> >> Yeah, I didn't explain very that clearly at all; I think you did a >> >> significantly better job: "you have too few overlapping entities in >> >> the two sets." That is exactly what I was trying to say. ;) >> >> >> >> For those interested, the merge-join algorithm is explained here: >> >> >> >> http://sites.google.com/site/io/under-the-covers-of-the-google-app-en... >> >> >> >> Robert >> >> >> >> 2011/4/15 Stephen Johnson <[email protected]>: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > Hi Alexander, >> >> > I'm sorry to disagree with Robert, but the zig zag merge-join doesn't >> >> > return >> >> > the various rows for each index back to the application code to >> >> > combine them >> >> > in the application. It's a difficult algorithm to explain in an email >> >> > but >> >> > basically the problem looks to be that with the 90000 entities and < >> >> > 500 >> >> > entites case you have too few overlapping entites in the two sets. >> >> > Basically, think of it as you start off with 90000 possible >> >> > candidates that >> >> > could be part of the result set. Now, the zig zag merge join goes and >> >> > checks >> >> > the other list (the less than 500 list), but it basically does this >> >> > one at a >> >> > time. Every time it doesn't find a match it throws that candidate >> >> > away and >> >> > goes on to the next. It has to do this for each equality filter so if >> >> > you >> >> > have 4 then it has to check for indexes for each candidate. You can >> >> > see that >> >> > this can take some time. Now if it finds enough matches fast enough >> >> > to >> >> > satisfy the query limit that is requested to be returned then all is >> >> > fine, >> >> > but if it has to keep throwing candidates away and trying new ones >> >> > than it >> >> > times out and reports that it needs an index. So with your queries >> >> > that have >> >> > large individual results (90000, 50000 and 50000) you obviously have >> >> > a large >> >> > overlapping set and it finds enough matches very quickly, but when >> >> > you have >> >> > large sets combined with small sets (90000 and <500) then you have >> >> > few >> >> > overlaps but lots of checking to do. >> >> > I hope this helps and if I misinterpreted Robert's post then I >> >> > apologize to >> >> > him. >> >> > Stephen >> >> >> >> > On Thu, Apr 14, 2011 at 8:36 AM, Alexandru Farcaş >> >> > <[email protected]> wrote: >> >> >> >> >> Hi Robert, >> >> >> Thanks for your answer. >> >> >> If the number of results returned by each filter individually is >> >> >> smaller than, let's say 1000, the "final" number of results will be >> >> >> returned correctly? >> >> >> I want to know if there are some known limits for this type of query >> >> >> on a single list property (maybe for the size of the list, for the >> >> >> number of the results returned by each filter, for the number of >> >> >> filters, etc...). I didn't find the answer for this question. >> >> >> In all queries that returns datastore need index exception I have >> >> >> one >> >> >> filter that returns almost 90000 entities and one that returns < >> >> >> 500 >> >> >> results. >> >> >> But I have also many queries with 3 filters (90000 results, 50000 >> >> >> results, 50000 results) that returns the correct results. >> >> >> >> >> Hi Andrei, >> >> >> Yes, almost the same queries: the difference is that in the second >> >> >> table I have a single list property. This list contains the rest of >> >> >> fields from the first table. >> >> >> I tried to limit the number of the results from datastore viewer but >> >> >> with the same result :(. >> >> >> >> >> On Apr 14, 5:03 pm, Andrei Cosmin Fifiiţă <[email protected]> >> >> >> wrote: >> >> >> > And also, i limit the number of results (for ex when implementing >> >> >> > pagination)... Does that affect the success of the query ? >> >> >> >> >> > 2011/4/14 Andrei Cosmin Fifiiţă <[email protected]>: >> >> >> >> >> > > I don't want to create custom indices cause they will fail... >> >> >> > > but does >> >> >> > > the >> >> >> > > order of the equality filters matter? >> >> >> >> >> > > On Apr 14, 2011 3:38 AM, "Robert Kluin" <[email protected]> >> >> >> > > wrote: >> >> >> >> >> -- >> >> >> You received this message because you are subscribed to the Google >> >> >> Groups >> >> >> "Google App Engine" group. >> >> >> To post to this group, send email to >> >> >> [email protected]. >> >> >> To unsubscribe from this group, send email to >> >> >> [email protected]. >> >> >> For more options, visit this group at >> >> >>http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "Google App Engine" group. >> >> > To post to this group, send email to >> >> > [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]. >> >> > For more options, visit this group at >> >> >http://groups.google.com/group/google-appengine?hl=en. >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine?hl=en. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
