Hi,
I have some very big datasets that we want to process in batches of
reasonable size.
In order to do that, we enable the 'stream_results' execution option
(available in Postgresql) and we use the fetchmany method to retrieve the
records in batches.
With those records we must build a numpy array, but its constructor
complains about the RowProxy wrapper.
So I tried using the cursor directly to retrieve the selected rows, but the
cursor skips the first one if 'stream_results' is enabled.
Let me explain with an example:
engine = create_engine(db_url, execution_options = {'stream_results' :
True})
rs = engine.execute("SELECT * FROM table")
print len(rs.fetchall())
rs = engine.execute("SELECT * FROM table")
print len(rs.cursor.fetchall())
The first one displays *N*, while the second one displays *N-1*.
Any clues? :/
Thank you in advance!
Pau.
--
----------------------------------
Pau Tallada Crespí
Dep. d'Astrofísica i Cosmologia
Port d'Informació Científica (PIC)
Tel: +34 93 586 8233
----------------------------------
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
