You might want to look into the mapper api. It is designed to work with large datasets, so it may be a good fit.
Robert On Fri, Sep 10, 2010 at 07:11, Sümer Cip <[email protected]> wrote: > I have 1 million recordes. I am doing statitical work with the data field. > What I want to do is to work in batches(with crons) with my data. That's > all. > > On Fri, Sep 10, 2010 at 6:56 AM, Robert Kluin <[email protected]> > wrote: >> >> What is "many entities," is it 100, 1,000, 1,000,000, more? What are >> you doing with the info field? Are you aggregating something? >> Transforming it? Writing one new model for each entity? >> >> What do you mean continue another day? Do you mean process some >> entities then stop and continue processing on another day? >> >> >> >> Robert >> >> >> >> >> >> >> >> >> On Thu, Sep 9, 2010 at 22:34, Sumer Cip <[email protected]> wrote: >> > Hi, >> > I have a very basic problem. Below is my model: >> > >> > class DummyModel: >> > id:Integer >> > info:Integer >> > added:DateTime >> > >> > What I need to do is to have a cron job running per day to filter on >> > added field and do something with the info field, but I have many >> > entities and I want to make this process in batches maybe continue >> > another day. How to do that efficiently? Cursors? Can we use them with >> > cron jobs? >> > >> > Any example suggestion would be appreciated. >> > >> > Thanks, >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine?hl=en. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > Sumer Cip > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
