On 01/06/2011 03:35 PM, Maurits van Rees wrote: > davidb started a branch of RelStorage in July 2010 and I wonder if that > can be merged. It does two interesting things in the zodbconvert code: > > - It adds a --single-transaction command line option. From the readme: > "Import into the destination in a single transaction, instead of one > transaction per transaction in the source. This option can > significantly speed up conversion times on PostgreSQL." > > - I adds more logging so you have an indication that yes, something is > in fact happening. This is good for your heart when you want to migrate > a 16 GB Data.fs. :-) Otherwise you just see 'Storages opened > successfully' and then nothing at all for several hours. > > I am no zodb or relstorage expert, but the changes for the > --single-transaction code look good to me.
I looked at that branch before, but I felt like the changes were complicated enough to require a comparison with simpler solutions first. In particular, Postgres has an option to disable fsync. Set it in postgresql.conf. Disabling fsync is not normally recommended, but for a large import it's obviously a good idea. Would you compare the speed of --single-transaction with disabled fsync on vanilla RelStorage? I assume the logging additions are less invasive, and if that assumption is correct, there's no problem with merging those. > I added some more logging on that branch, mostly because the conversion > appeared to be hanging at some unknown point. This was also with the > official 1.4.1 release, which was the reason I started experimenting > with the fastimport branch to see if that would help. It did not. At > least in both test runs mentioned above, the actual time it took was > about twenty minutes longer, possibly because the conversion temporarily > lost the connection with the postgres server. With the logging I could > at least see that it was throwing the old transaction table away; I have > seen the same with other tables. Definitely no one else is accessing > this database at the same time. So if someone has an idea what could be > going on here, that is welcome. Are you sure you were not accidentally running multiple imports in parallel? RelStorage does not throw away tables unless you're rather explicit about it. > Anyway, the --single-transaction seems to work and I would say the > logging is helpful. So: is there any chance this can be merged to > trunk? Not yet. We really need someone to do the fsync test, and if that doesn't do the trick, there has to be a way to accomplish the same thing in a clearer way. > I am also interested in the blob support that has been added > there. :-) The blob changes are also in 1.5.0a1. That release seems to be a lot more stable than I expected and may soon become 1.5.0 final if no one finds bugs in the new blob option. Shane _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
