On 06/16/2016 12:32 PM, Andy Crain wrote:
Hi,
I'm attempting to do a bulk insert from a large .csv file. I've read
through the various options
at
http://docs.sqlalchemy.org/en/latest/faq/performance.html#i-m-inserting-400-000-rows-with-the-orm-and-it-s-really-slow,
and I would like to perform a bulk insert using Core, along the lines of:
|
engine.execute(MyTable.__table__.insert(),[...])
|
Instead of a list, though, I'd like to use a generator, as I'm very
memory constrained. When I try something like this:
|
engine.execute(MyTable.__table__.insert(),(d ford inmy_dict_generator()))
|
...I get "AttributeError: 'list' object has no attribute 'keys'".
Is it not possible to use a generator in lieu of a list here?
not directly. You should break your generator into batches:
import itertools
with engine.begin() as conn:
while True:
chunk = [elem for elem in itertools.islice(generator, 0, 10000)]
if not chunk:
break
conn.execute(table.__insert__, chunk)
Thanks,
Andy
--
You received this message because you are subscribed to the Google
Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
