Filtering in memory works best if you have the all the data aggregated into one or a few entities since there is a very high api-cpu overhead per entity fetch regardless of the amount of data. Either way you go this would be expensive.
That said I'd split into multiple queries so you can sort the current results after each fetch and skip fetching the rest if you find you have enough results to display something useful to the user. Regards, Brandon Thomson On Sat, Jun 19, 2010 at 8:28 AM, coltsith <[email protected]> wrote: > Tim, > > thanks for responding. > > Yes, I know the user won't want to look at all the games. However, I > need to filter and sort the results. For example one way I sort is by > which games have players with the closest rating to you. > > So for that, I need to fetch all games, and custom sort through all > them based on their ratings being closest to the calling player's > rating. > > Also, I need to have an inequality filter on a property that I don't > sort by. I don't want to return games that were created by the calling > player: > > where gameOwnerId != playerId && order by gameCreateDate etc etc. > > The current query limitations restrict me too much. > > On Jun 19, 1:09 pm, Tim Hoffman <[email protected]> wrote: >> I think you should ask yourself, will anyone really page through 20000 >> games >> or even 1000 games? >> >> On Jun 19, 7:39 pm, coltsith <[email protected]> wrote: >> >> >> >> > In my game server, I let the user browse the open games. They can >> > choose to filter them, or not put any filters at all. >> >> > My question is, what's the best way to load the games from the >> > datastore. Let's say there's like 20,000. >> >> > Should I fetch them all in 1 query (no more 1000 limit) or fetch in >> > increments of like 2000 each and combine them into a big list? >> >> > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
