You should be querying for keys only, not the whole entities. This should speed things up considerably.
On Jun 27, 2:41 pm, gae123 <[email protected]> wrote: > I am sharing my experience of deleting all the data of one of my apps. > We are talking about 500MB spread over about a few thousands records > and about 15 Models. The summary is that it worked but caused many > errors and depleted my CPU time, read on for the details... > > The approach was pretty simple. Iterate though all of a the 15 kinds > and create one task for each kind. Then each of these tasks, queries > the datastore for 10 entities of this kind, db.delete() them and then > queues one more similar task. This continues until the datastore > reports no more records of this kind. I use the default queue. I would > guess that 60-70% records are in two Models so I did notice as I > expected that pretty quickly only two tasks reamained in the queue. > One more thing, all the data have the same ancestor. > > At the end of the process the good news is tha the algorithm worked, > no data in the app and the Dashboard reports: > > 96% of my CPU time depleted > 10% of my datastore time depleted > 748 tasks queued > other indicators healthy... > > Now I do see many many logs with the following two traces which I > think is what caused a lot of tasks to fail and be requeued and > eventually depleted almost all my CPU time. So what could I be doing > wrong? What could I do to avoid these errors? > > -------------------------------------------------------------------------------------------------------------------- > > File "/base/data/home/apps/neatschool-test/ > 0-7-22-81dafb9.334504032648079489/swplatform/controllers/admin.py", > line 89, in post > db.delete(res) > File "/base/python_lib/versions/1/google/appengine/ext/db/ > __init__.py", line 1127, in delete > datastore.Delete(keys) > File "/base/python_lib/versions/1/google/appengine/api/ > datastore.py", line 269, in Delete > raise _ToDatastoreError(err) > File "/base/python_lib/versions/1/google/appengine/api/ > datastore.py", line 2055, in _ToDatastoreError > raise errors[err.application_error](err.error_detail) > TransactionFailedError: too much contention on these datastore > entities. please try again. > > ------------------------------------------------------------------------------------------------------------------------ > > File "/base/data/home/apps/neatschool-test/ > 0-7-22-81dafb9.334504032648079489/swplatform/controllers/admin.py", > line 86, in post > results = qq.fetch(10) > File "/base/data/home/apps/neatschool-test/ > 0-7-22-81dafb9.334504032648079489/swplatform/util/db/query.py", line > 118, in fetch > ents = self.query.fetch(limit, offset) > File "/base/python_lib/versions/1/google/appengine/ext/db/ > __init__.py", line 1426, in fetch > raw = self._get_query().Get(limit, offset) > File "/base/python_lib/versions/1/google/appengine/api/ > datastore.py", line 959, in Get > return self._Run(limit, offset)._Get(limit) > File "/base/python_lib/versions/1/google/appengine/api/ > datastore.py", line 903, in _Run > _ToDatastoreError(err) > File "/base/python_lib/versions/1/google/appengine/api/ > datastore.py", line 2055, in _ToDatastoreError > raise errors[err.application_error](err.error_detail) > Timeout --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
