obviously, if you have to page a data set more than 50000 items which is not ordered by __key__,
you may find that the __key__ is of no use, because the filtered data is ordered not by key. but by the fields value, and for that reason you need to loop query as you may like to do. but you will encounter a timeout exception before you really finished the action. On Dec 19, 8:26 am, Andy Freeman <[email protected]> wrote: > > > if the type of data is larger than 10000 items, you need reindexing > > for this result. > > and recount each time for getting the proper item. > > What kind of reindexing are you talking about. > > Global reindexing is only required when you change the indices in > app.yaml. It doesn't occur when you add more entities and or have big > entities. > > Of course, when you change an entity, it gets reindexed, but that's a > constant cost. > > Surely you're not planning to change all your entities fairly often, > are you? (You're going to have problems if you try to maintain > sequence numbers and do insertions, but that doesn't scale anyway.) > > > > it seems you have not encountered such a problem. > > on this situation, the indexes on the fields helps nothing for the > > bulk of data you have to be sorted is really big. > > Actually I have. I've even done difference and at-least-# > (intersection and union are special cases - at-least-# also handles > majority), at-most-# (binary xor is the only common case that I came > up with), and combinations thereof on paged queries. > > Yes, I know that offset is limited to 1000 but that's irrelevant > because the paging scheme under discussion doesn't use offset. It > keeps track of where it is using __key__ and indexed data values. > > On Dec 16, 7:56 pm, ajaxer <[email protected]> wrote: > > > > > of course the time is related to the type data you are fetching by one > > query. > > > if the type of data is larger than 10000 items, you need reindexing > > for this result. > > and recount each time for getting the proper item. > > > it seems you have not encountered such a problem. > > on this situation, the indexes on the fields helps nothing for the > > bulk of data you have to be sorted is really big. > > > On Dec 17, 12:20 am, Andy Freeman <[email protected]> wrote: > > > > > it still can result in timout if the data is really big > > > > How so? If you don't request "too many" items with a page query, it > > > won't time out. You will run into runtime.DeadlineExceededErrors if > > > you try to use too many page queries for a given request, but .... > > > > > of no much use to most of us if we really have big data to sort and > > > > page. > > > > You do know that the sorting for the page queries is done with the > > > indexing and not user code, right? Query time is independent of the > > > total amount of data and depends only on the size of the result set. > > > (Indexing time is constant per inserted/updated entity.) > > > > On Dec 16, 12:13 am, ajaxer <[email protected]> wrote: > > > > > it is too complicated for most of us. > > > > and it still can result in timout if the data is really big > > > > > of no much use to most of us if we really have big data to sort and > > > > page. > > > > > On Dec 15, 11:35 pm, Stephen <[email protected]> wrote: > > > > > > On Dec 15, 8:04 am, ajaxer <[email protected]> wrote: > > > > > > > also 1000 index limit makes it not possible to fetcher older data on > > > > > > paging. > > > > > > > for if we need an indexed page more than 10000 items, > > > > > > it would cost us a lot of cpu time to calculate the base for GQL > > > > > > to fetch the data with index less than 1000. > > > > > >http://code.google.com/appengine/articles/paging.html-Hidequotedtext - > > > > > - Show quoted text -- Hide quoted text - > > > - Show quoted text - -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
