I am encountering multiple issues when trying to use PolyModels and
bulkload data. At a minimum, it appears the documentation is
incomplete on how to bulkload data for PolyModels. For instance, this
issue http://code.google.com/p/googleappengine/issues/detail?id=1133
has been fixed, but I cannot find the documentation for why one must
use 'model:' in the bulkloader config instead of 'kind:'.
Here is an example set of models, loaders and the issues I encounter:
class Record(db.Model);
...
class Parent(polymodel.PolyModel):
ival = db.IntegerValue()
class Child(Parent):
sval = db.StringValue(required=True)
record = db.ReferenceProperty(Record, required=True)
bulkloader.yaml:
- model:
models.Child
name:
Child
connector:
simplexml
connector_options:
xpath_to_nodes: /Children/
Child
style:
element_centric
property_map:
- property:
__key__
external_name:
key
export_transform:
transform.key_id_or_name_as_string
import_transform: transform.create_foreign_key('Parent', True)
- property:
ival
external_name: ival
import_transform: transform.none_if_empty(long)
- property:
sval
external_name: sval
- property:
record
external_name:
record
import_transform: transform.create_foreign_key('Record',
True)
export_transform: transform.key_id_or_name_as_string
Here is my XML data file which I am trying to import into the
datastore:
<Children>
<Child>
<key>11</
key>
<ival>10</
ival>
<sval>someval</
sval>
<record>1</
record>
</
Child>
<Child>
<key>2</
key>
<ival>20</
ival>
<sval>20val</
sval>
<record>2</
record>
</
Child>
<Child>
<key>3</
key>
<ival>30</
ival>
<sval>30val</
sval>
<record>1</
record>
</
Child>
</Children>
There are two issues that I hit.
a) If the bulkloader.yaml specifies a property transform for __key__
then, no instances are uploaded (and bulkloader says N records
successfully transferred, N being correct).
b) If I implement workarounds for a) and proceed, then I hit the
second issue. The second issue is that if a property is marked with
required=True, then a BadValueError is thrown since the created model
does not have any properties set.
Both issues center around DictConverter.__create_instance in
bulkloader_config.py.
def __create_instance(self, input_dict, bulkload_state):
key =
None
if
self._create_key:
key = self.__dict_to_prop(self._create_key, input_dict,
bulkload_state)
if isinstance(key, (int,
long)):
key = datastore.Key.from_path(self._transformer_spec.kind,
key)
if isinstance(key,
datastore.Key):
parent =
key.parent()
if key.name() ==
None:
return
datastore.Entity(self._transformer_spec.kind,
parent=parent,
id=key.id())
else:
return
datastore.Entity(self._transformer_spec.kind,
parent=parent,
name=key.name())
if
self._transformer_spec.model:
return
self._transformer_spec.model(key=key)
return datastore.Entity(self._transformer_spec.kind, name=key)
For issue a), the code ends up creating an entity with
self._transformer_spec.kind (and the specified id) which causes
problems later in __run_import_transforms() which are unable to set
attributes on the instance. So the instance ends up being empty.
The workaround I tried was to make the code so:
if
self._create_key:
key = self.__dict_to_prop(self._create_key, input_dict,
bulkload_state)
if
self._transformer_spec.model:
return self._transformer_spec.model(key=key)
This works great until I hit issue b). It turns out that creating
models this way causes BadValueErrors during validation of properties
with required=True.
I really do not want to go out and set required=False on my deployed
app. I also really want to use PolyModels for all the usual reasons.
But it seems like bulkloading is a pain and undocumented clearly for
PolyModels.
It would be great if someone can suggest some ways to proceed.
I am using Appengine SDK 1.4.2 for MacOSX.
thanks,
Vasanth
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
