On 01/11/2011 07:06 AM, David Blewett wrote:
> On Thu, Jan 6, 2011 at 11:51 PM, Shane Hathaway<sh...@hathawaymix.org> wrote:
>>> I am no zodb or relstorage expert, but the changes for the
>>> --single-transaction code look good to me.
>> I looked at that branch before, but I felt like the changes were
>> complicated enough to require a comparison with simpler solutions first.
>> In particular, Postgres has an option to disable fsync. Set it in
>> postgresql.conf. Disabling fsync is not normally recommended, but for a
>> large import it's obviously a good idea. Would you compare the speed of
>> --single-transaction with disabled fsync on vanilla RelStorage?
> Sorry, it's been awhile since I started work on that branch. Disabling
> fsync is an option for the load, and it should help. However, using a
> single transaction is much faster in PostgreSQL. Off the top of my
> head, I can't recall if the branch also uses COPY mode. I recall
> encountering some problems trying to get COPY to work correctly.
> Single-transaction mode + COPY, however, is the fastest way to get
> data into PostgreSQL.
Right, but I want zodbconvert to remain a very generic tool. I think of
zodbconvert as a reference implementation. A tool that uses PostgreSQL
COPY is a good idea but it ought to be a different tool.
IMHO, it's actually pretty easy to write optimized ZODB conversion
tools. I would suggest writing conversion tools as independent
packages, meaning they probably should not import RelStorage, but
instead talk to the database directly.
For more information about ZODB, see the ZODB Wiki:
ZODB-Dev mailing list - ZODB-Dev@zope.org