Hi Nick,
Thanks, that's more straightforward. Somewhat related, I'm creating an
auto-generated 'bulk-delete' script that uses the 'remote_api'
handler. Since i know the key_name for each entity that I'm uploading,
I create a script which looks something like (ignoring imports):
---- deleteall_mymodel.py ------
def auth_func():
return raw_input('Username:'), getpass.getpass('Password:')
app_id = 'myapp'
host = app_id + '.appspot.com'
remote_api_stub.ConfigureRemoteDatastore(app_id, '/remote_api',
auth_func, host)
print("Deleting entity: a1")
obj = MyModel.get_by_key_name("a1")
if obj is not None:
obj.delete()
print("Deleting entity: a2")
obj = MyModel.get_by_key_name("a2")
if obj is not None:
obj.delete()
print("Deleting entity: a3")
obj = MyModel.get_by_key_name("a3")
if obj is not None:
obj.delete()
......
-----------------------------------------------------
This creates a huge file but it does work, and at a not unreasonable
speed. What do think of this approach?
Thanks
G.
On Sep 26, 8:17 pm, "Nick Johnson (Google)" <[email protected]>
wrote:
> Hi djerdo,
>
> You can specify key names by overriding the generate_key method of your
> loader. See here for more
> details:http://blog.notdot.net/2009/9/Advanced-Bulk-Loading-part-2-Customization
>
> -Nick Johnson
>
>
>
> On Sat, Sep 26, 2009 at 1:54 PM, djerdo <[email protected]> wrote:
>
> > On 17 Sep, 23:25, djerdo <[email protected]> wrote:
> > > Using the bulkloader with the dev appserver, the script adds rows
> > > (Entities) at a progressively slower rate, to the point where it
> > > becomes unusable when the csv file is large (20,000 rows). Why? Is
> > > this a known issue? Are there any workarounds?
>
> > > Thanks
>
> > Ok, I wasn't doing it right. FWIW - to resolve reference property
> > fields, I was using code such as:
>
> > select __key__ from MyModel where id = :1
>
> > id being the unique field in the incoming csv file. This seemed to
> > work but when i figured out how to specify a key_name when adding an
> > entity, I could replace this with
>
> > MyModel.get_by_key_name(id).key()
>
> > which sped things up a lot.
>
> > How to specify a key name when bulkloading entities? I couldn't find
> > any examples of this, so (again FWIW), here's a 'Baseloader' class
> > that i used:
>
> > from google.appengine.api import datastore
> > from google.appengine.ext import bulkload
>
> > class BaseLoader(bulkload.Loader):
> > #abstract class
>
> > def __init__(self):
> > bulkload.Loader.__init__(self, self.model, self.mapping)
>
> > def HandleEntity(self, entity):
> > if not hasattr(self, 'unique_field'):
> > return entity
> > #entity is a dict subclass
> > keyname = entity.pop(self.unique_field)
> > named_entity = datastore.Entity(self.model, name=keyname)
> > named_entity.update(entity)
> > return named_entity
>
> > Example:
>
> > class ContactLoader(Loader):
> > model = 'Contact'
> > mapping = [
> > ('id', str),
> > ('name', str),
> > ('phone', str),
> > ]
> > unique_field = 'id'
>
> --
> Nick Johnson, Developer Programs Engineer, App Engine
> Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
> 368047
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
