> Den 12/08/2015 kl. 04.47 skrev [email protected]:
>
> for row in rows:
> dp = DataPoint.objects.get(Taken_datetime=row['date'])
>
> sensorToAdd = []
> for sensor in sensors:
> s = Sensor.objects.get(Name=sensor.name, Value=sensor.value )
> sensorToAdd.append( s )
>
> dp.Sensors.add( sensorToAdd )
>
> In the actually app I bulk create all the DataPoint and Sensor instances.
> The problem is that |dp.Sensors.add( sensorToAdd )| does a lot of hits on the
> db. I want a way bulk add all the sensors.
Try fetching all the data you need from the database up-front and place objects
in a dict. Depending on the volume of your data, you may want to split this
into reasonable batches:
from django.db.models import Q
datapoints = DataPoint.objects.filter(Taken_datetime__in={r['date'] for r in
rows})
datapoints_map = {(d.Taken_datetime, d) for d in datapoints}
# To generate efficient SQL, make sure (name, value) pairs are unique
unique_sensor_values = {(s.name, s.value) for s in my_list_of_sensors}
sensors_q = Q()
for name, value in my_unique_sensor_values:
sensors |= Q(Name=name, Value=value)
sensors = Sensor.objects.filter(sensors_q)
sensors_map = {((s.name, s.value), s) for s in sensors}
This reduces your queries to only two. You can then bulk-insert m2m relations
per-object like this:
for some_date, some_sensors in my_data:
dp = datapoints_map[some_date]
dp.Sensors.add(*[sensors_map[(s.name, s.value)] for s in some_sensors])
If you need better bulk-insert performance than this, you can convert the m2m
relation between Sensor and DataPoint to an explicit m2m model. You can then
bulk-insert all m2m relations in one go instead of per-object.
You should probably add indexes on DataPoint.Taken_datetime and Sensors.[Name,
Value] to increase query performance.
Erik
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/7FCE3968-7C15-4D92-8DB3-E4A6EE56442B%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.
