Thanks, this is in fact what I implemented now as a method in the Sensor
class, exploiting also the bulk_insert_mappings since the number of
readings are quite a lot (400k each time):
def store_readings(self, session):
if not(self.id):
session.add(self)
session.flush()
sensor_id = self.id
times, voltages = self.get_values_from_somewhere()
n = len(times)
chunk_size = 100000
for i in six.moves.range(0, n, chunk_size):
begin = i
end = i+chunk_size
time_chunk = times[begin:end]
voltage_chunk = voltages[begin:end]
session.bulk_insert_mappings(PassageData, [
{
'time': t,
'voltage': v,
'sensor_id': sensor_id
} for t, v in
zip(time_chunk, voltave_chunk)]
)
session.commit()
The problem with the custom collection is that maybe we lose performances
that are gained while using the numpy powerful indexing. I need, still, to
study a bit the suggestion given by Mike.
Thanks,
On Friday, July 28, 2017 at 9:28:19 PM UTC+2, Jonathan Vanasco wrote:
>
> Unless you need to use all the readings immediately, have you considered
> just making a custom def under the Sensor model, and then inserting all the
> readings via sqlalchemy core? That would allow you to insert them without
> creating ORM objects, which people using numpy and a lot of data often like
> to avoid.
>
> Then you could do...
>
> s = Sensor()
> session.add(s)
> session.add_readings(dates, voltages, values)
>
> it would look something like this...
>
> class Sensor(Base):
> def add_readings(self, dates, voltages, values):
> if not self.id:
> # flush this to the session
> session = object_session(self)
> session.flush(objects=[self])
> # insert via core
>
>
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable
Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
