Actually i think here the main problem is that those indexes that should serve a query without sorting are not completely constructed (or they are built with errors). I updated thousands of entities to create a List property with about 15 words in it and the internal app engine zig-zag method doesn't work either. (exception: The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query. etc) I get the same error in Datastore viewer.
I have searched the web and it seems there are not a lot of developers receiving this error. And i don't think that re-updating or re-inserting them will resolve this problem (that is, instructing the data to be re-indexed), but i'm not sure. On Sep 29, 6:31 pm, Raphael André Bauer <[email protected]> wrote: > On Fri, Sep 24, 2010 at 5:53 PM, Ice13ill <[email protected]> wrote: > > Well, that is what i'm using :) > > FTS with self merged join (if that is what you're refering to) > > > The problem here is with the build-inindexes(it should be able to > > serve those queries that matches a text on a singelistproperty, for > > a large number of entities right ?). As i understand, the FTS in GAE > > is based on the ability of matching alistof keywords against the > >listfield of the entities you want to search (equality filter) right? > > Hi Andrei, > > I am the author of that FTS blogpost mentioned before [1]. What you do > can be done on GAE / FTS. The main obstacle you are currently facing > is the explodingindexesproblem. > > The good thing is: Using self merge joins you do not need explizitindexes. > Everything you need is calculated internally via an algo > called "zig zag". > > First of all I would recommend to watch Brett Slatkin's talk again. I > watched Brett's talk also more than only one time to get the concepts. > Stop thinking in explizitindexesand orderings - start thinking in > Venn Diagramms where you specify the piece of the cake you want > precisely without ordering. > > If you fully understood Brett's talk make also sure to design your > search "bottom up". That means: Make sure you do not get back more > than approx. 200elementsin one FTS search. That also means that you > might include a timestamp or something to search only certainelements > from today or any special day and such. No. No explicit index here. > You simply save anotherStringin your listproperties where you > specify a timestamp such as "2010-09-29" what means today. At the end > you can query allelementsfrom today using that timestamp. No > ordering and no index needed. > > Hope that helps :) > > Cheers, > > Raphael > > [1]http://googleappengine.blogspot.com/2010/04/making-your-app-searchabl... > > > > > > > On Sep 24, 6:29 pm, Cyrille Vincey <[email protected]> wrote: > >> I am afraid you will have to change your data model. > >> By the way, I remember that someone shared so times ago his code for a > >> full text search implementation in GAE. Maybe you should check this out. > > >> On 24/09/10 17:01, "Ice13ill" <[email protected]> wrote: > > >> >Hmm... i understand the problem. But, if i want to query using a text > >> >with multiple words and NO sorting order, i use the build-in index > >> >supplied default right ? > >> >So, what do i do for 90.000 entities ?? because it seams that the > >> >build-in index is not powerful enough ! > > >> >"The built-in indices are not efficient enough for this query and your > >> >data. Please add a composite index for this query.. An index is > >> >missing but we are unable to tell you which one due to a bug in the > >> >App Engine SDK. If your query only contains equality filters you most > >> >likely need a composite index on all the properties referenced in > >> >those filters." > > >> >So the solution iscreatinga index like this: > > >> >keyWordsList ▲ , keyWordsList ▲ , keyWordsList ▲ > > >> >Isn't this really the "build-in" index ? > > >> >On Sep 24, 5:16 pm, Cyrille Vincey <[email protected]> wrote: > >> >> You're facing the explodingindexesproblem. > >> >> You should never put more than 1listproperties in one composite index. > > >> >> This problem is described in the GAE > >> >>doc.http://code.google.com/intl/fr-FR/appengine/docs/python/datastore/que > >> >>... > >> >> dindexes.html#Big_Entities_and_Exploding_Indexes > > >> >> On 24/09/10 15:54, "Ice13ill" <[email protected]> wrote: > > >> >> >I'm trying to implement a search based on alistof keywords (string). > >> >> >Thelistis created based on the content of the objects' fields and i > >> >> >use it when matching against a query text. > >> >> >The problem is that i can't createindexeson App engine (i get Error > >> >> >after about 30 mins, and i have to delete it). There are about 90,000 > >> >> >entities of that kind and the index should looke like this: > > >> >> >keyWordsList ▲ , keyWordsList ▲ , keyWordsList ▲ , key ▼ > > >> >> >and i have about four of them. > > >> >> >My question: do i want to much ? :) I tried uploading about 4indexes > >> >> >and i got Error. I tried uploading 1 and worked (but needed about 2-3 > >> >> >h). Is there a problem if i try to create 4indexesof this magnitude > >> >> >(and why)? Can i be sure that uploading one by one will work? Or maybe > >> >> >this problem is related to AppEngine datastore problems ? > >> >> >Please advice ! > > >> >> >-- > >> >> >You received this message because you are subscribed to the Google > >> >>Groups > >> >> >"Google App Engine for Java" group. > >> >> >To post to this group, send email to > >> >> >[email protected]. > >> >> >To unsubscribe from this group, send email to > >> >> >[email protected]. > >> >> >For more options, visit this group at > >> >> >http://groups.google.com/group/google-appengine-java?hl=en. > > >> >-- > >> >You received this message because you are subscribed to the Google Groups > >> >"Google App Engine for Java" group. > >> >To post to this group, send email to > >> >[email protected]. > >> >To unsubscribe from this group, send email to > >> >[email protected]. > >> >For more options, visit this group at > >> >http://groups.google.com/group/google-appengine-java?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
