On 01/11/2011 07:06 AM, David Blewett wrote: > On Thu, Jan 6, 2011 at 11:51 PM, Shane Hathaway<sh...@hathawaymix.org> wrote: >>> I am no zodb or relstorage expert, but the changes for the >>> --single-transaction code look good to me. >> >> I looked at that branch before, but I felt like the changes were >> complicated enough to require a comparison with simpler solutions first. >> In particular, Postgres has an option to disable fsync. Set it in >> postgresql.conf. Disabling fsync is not normally recommended, but for a >> large import it's obviously a good idea. Would you compare the speed of >> --single-transaction with disabled fsync on vanilla RelStorage? > > Sorry, it's been awhile since I started work on that branch. Disabling > fsync is an option for the load, and it should help. However, using a > single transaction is much faster in PostgreSQL. Off the top of my > head, I can't recall if the branch also uses COPY mode. I recall > encountering some problems trying to get COPY to work correctly. > Single-transaction mode + COPY, however, is the fastest way to get > data into PostgreSQL.
Right, but I want zodbconvert to remain a very generic tool. I think of zodbconvert as a reference implementation. A tool that uses PostgreSQL COPY is a good idea but it ought to be a different tool. IMHO, it's actually pretty easy to write optimized ZODB conversion tools. I would suggest writing conversion tools as independent packages, meaning they probably should not import RelStorage, but instead talk to the database directly. Shane _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev