What statistics are you talking about? You're claiming that one can't page through an entity type without fetching all instances and sorting them. That claim is wrong because the order by constraint does exactly that.
For example, suppose that you want to page through by a date/time field named "datetime". The query for the first page uses order by datetime while queries for subsequent pages have a "datetime <=" clause for the last datetime value from the previous page and continue to order by datetime. What part of that do you think doesn't work? Do you think that Nick was wrong when he said that time time to execute such query depends on the number of entities? You can even do random access by using markers that are added/ maintained by a sequential process like the above. On Dec 20, 7:34 pm, ajaxer <[email protected]> wrote: > You misunderstand. > if not show me a site with statistics on many fields. > with more than 1000 pages please. > thanks. > > On Dec 21, 9:06 am, Andy Freeman <[email protected]> wrote: > > > > > You misunderstand. > > > If you have an ordering based on one or more indexed properties, you > > can page efficiently wrt that ordering, regardless of the number of > > data items. (For the purposes of this discussion, __key__ is an > > indexed property, but you don't have to use it or can use it just to > > break ties.) > > > If you're fetching a large number of items and sorting so you can find > > a contiguous subset, you're doing it wrong. > > > On Dec 19, 10:26 pm, ajaxer <[email protected]> wrote: > > > > obviously, if you have to page a data set more than 50000 items which > > > is not ordered by __key__, > > > > you may find that the __key__ is of no use, because the filtered data > > > is ordered not by key. > > > but by the fields value, and for that reason you need to loop query as > > > you may like to do. > > > > but you will encounter a timeout exception before you really finished > > > the action. > > > > On Dec 19, 8:26 am, Andy Freeman <[email protected]> wrote: > > > > > > > if the type of data is larger than 10000 items, you need reindexing > > > > > for this result. > > > > > and recount each time for getting the proper item. > > > > > What kind of reindexing are you talking about. > > > > > Global reindexing is only required when you change the indices in > > > > app.yaml. It doesn't occur when you add more entities and or have big > > > > entities. > > > > > Of course, when you change an entity, it gets reindexed, but that's a > > > > constant cost. > > > > > Surely you're not planning to change all your entities fairly often, > > > > are you? (You're going to have problems if you try to maintain > > > > sequence numbers and do insertions, but that doesn't scale anyway.) > > > > > > > it seems you have not encountered such a problem. > > > > > on this situation, the indexes on the fields helps nothing for the > > > > > bulk of data you have to be sorted is really big. > > > > > Actually I have. I've even done difference and at-least-# > > > > (intersection and union are special cases - at-least-# also handles > > > > majority), at-most-# (binary xor is the only common case that I came > > > > up with), and combinations thereof on paged queries. > > > > > Yes, I know that offset is limited to 1000 but that's irrelevant > > > > because the paging scheme under discussion doesn't use offset. It > > > > keeps track of where it is using __key__ and indexed data values. > > > > > On Dec 16, 7:56 pm, ajaxer <[email protected]> wrote: > > > > > > of course the time is related to the type data you are fetching by one > > > > > query. > > > > > > if the type of data is larger than 10000 items, you need reindexing > > > > > for this result. > > > > > and recount each time for getting the proper item. > > > > > > it seems you have not encountered such a problem. > > > > > on this situation, the indexes on the fields helps nothing for the > > > > > bulk of data you have to be sorted is really big. > > > > > > On Dec 17, 12:20 am, Andy Freeman <[email protected]> wrote: > > > > > > > > it still can result in timout if the data is really big > > > > > > > How so? If you don't request "too many" items with a page query, it > > > > > > won't time out. You will run into runtime.DeadlineExceededErrors if > > > > > > you try to use too many page queries for a given request, but .... > > > > > > > > of no much use to most of us if we really have big data to sort > > > > > > > and > > > > > > > page. > > > > > > > You do know that the sorting for the page queries is done with the > > > > > > indexing and not user code, right? Query time is independent of the > > > > > > total amount of data and depends only on the size of the result set. > > > > > > (Indexing time is constant per inserted/updated entity.) > > > > > > > On Dec 16, 12:13 am, ajaxer <[email protected]> wrote: > > > > > > > > it is too complicated for most of us. > > > > > > > and it still can result in timout if the data is really big > > > > > > > > of no much use to most of us if we really have big data to sort > > > > > > > and > > > > > > > page. > > > > > > > > On Dec 15, 11:35 pm, Stephen <[email protected]> wrote: > > > > > > > > > On Dec 15, 8:04 am, ajaxer <[email protected]> wrote: > > > > > > > > > > also 1000 index limit makes it not possible to fetcher older > > > > > > > > > data on > > > > > > > > > paging. > > > > > > > > > > for if we need an indexed page more than 10000 items, > > > > > > > > > it would cost us a lot of cpu time to calculate the base for > > > > > > > > > GQL > > > > > > > > > to fetch the data with index less than 1000. > > > > > > > > >http://code.google.com/appengine/articles/paging.html-Hidequotedtext- > > > > > > > > - Show quoted text -- Hide quoted text - > > > > > > - Show quoted text -- Hide quoted text - > > > > - Show quoted text -- Hide quoted text - > > - Show quoted text - -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
