countEntities() won't work because there's no way the datastore can compute (iterate through the index) the result in 30s. I can say this about counting results without entity groups: Expect to be able to process 20-30k keys-only values in each 30s window before you restart the task. At that rate, it would take 20-30 minutes to process 1 million records.
It might be faster if all the entities are in a single entity group, I haven't tried that. Seems unlikely to be more than an order of magnitude faster but I'd love to hear some experimental results. 1k entity groups with 10 million records will almost certainly take hours... but I haven't tried it. Sounds like you figured it out, Max: If you want information that is both timely and accurate, maintaining your own counter is the answer. This is the case with any database system, not just BigTable. Jeff On Wed, Feb 3, 2010 at 3:06 PM, David Chandler <[email protected]> wrote: > Or would this would be more efficient if all you need is a count? > > DatastoreService.prepare(query).countEntities(); > > I don't know the answer, but I've been wondering the same thing... > > /dmc > http://turbomanage.wordpress.com > > On Feb 2, 10:05 pm, Max <[email protected]> wrote: >> Hi, >> >> I am designing my data model and planning to use following KeysOnly >> query to perform a count(*) with some filter conditions by a daily >> cron job. >> >> FetchOptions fetchOptions = FetchOptions.Builder.withOffset(0).limit >> (Integer.MAX_VALUE); >> DatastoreService.prepare(query).asList(fetchOptions).size(); >> >> Here are my questions: >> 1, if I use an ancestor query (in a transaction) over an entity group >> with around 1 million records of this kind, how is the performance? >> >> 2, if I use an ancestor query without a transaction (to avoid locking >> whole entity group), how is the performance? >> >> 3, if I use a normal query over around 1k entity groups with around 10 >> million records of this kind, how is the performance? >> >> Many thanks. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
