@jim, resent to the list, sorry.
Am 07.02.2013 17:11, schrieb Jim Fulton:
On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann
<juergen.herrm...@xlhost.de> wrote:
Am 06.02.2013 15:05, schrieb Jürgen Herrmann:
Hi there!
I hav a relstorage with mysql backend that grew out of bounds
and we're looking into different backend solutions now. Possibly
also going back to FileStorage and using zeo...
Anyway we'll have to convert the databases at some point. As these
are live DBs we cannot shut them down for longer than the
ususal maintenance interval during the night, so for maybe 2-3h.
a full conversion process will never complete in this time so
we're looking for a process that can split the conversion into
two phases:
1. copy transactions from backup of the source db to the
destination
db. this can take a long time, we don't care. note the last
timestamp/transaction_id converted.
2. shut down the source db
3. copy transactions from the source db to the destination db,
starting
at the last converted transaction_id. this should be fast, as
only
a few transactions need to be converted, say < 1% .
if i would reimplement copyTransactionsFrom() to accept a start
transaction_id/timestamp, would this result in dest being an exact
copy of source?
source = open_my_source_storage()
dest = open_my_destination_storage()
dest.copyTransactionsFrom(source)
last_txn_id = source.lastTransaction()
source.close()
dest.close()
source = open_my_source_storage()
# add some transactions
source.close()
source = open_my_source_storage()
dest = open_my_destination_storage()
dest.copyTransactionsFrom(source, last_txn_id=last_txn_id)
source.close()
dest.close()
I will reply to myself here :) This actually works, tested with a
modified version of FileStorage for now. I modified the signature
of copyTransactionsFrom to look like this:
def copyTransactionsFrom(self, source, verbose=0,
not_before_tid=None):
``start`` would be better to be consistent with the iterator API.
this was my first approach, though for my usecase it would be
misleading
as the code roughly looks like this:
if tid < not_before_tid:
continue
and it excludes the given tid from the transactions re-stored. maybe
we can come up with a better name but start doesn't nail it :)
not_before_tid is a packed tid or None, None meaning "copy all"
(the default, so no existing API usage would break).
Is there public interest in modifying this API permamently?
+.1
This "API" is a bit of an attractive nuisance. I'd rather people
learn how to use iterators in their own scripts, as they are very
useful and powerful. This API just hides that.
not sure i understand this correctly, maybe you could elaborate a
bit more? for my usecase you'd suggest i just use the storage iterator
and walk/re-store the transactions in my own code?
there's a lot of checking and branching going on inside
copyTransactionsFrom(), that's why i asked if this would work in the
first place.
The second part, replaying old transactions is a bit more subtle,
but it's still worth it for people to be aware of it.
If I were doing this today, I'd make this documentation
rather than API. But then, documentation ... whimper.
Anybody want to look at the actual code changes?
Sure, if they have tests. Unfortunately, we can only accept
pull requests from zope contributors. Are you one?
Wanna be one? :)
i'll look at the supplied test and see if i can make my test script
a proper test case for the test suite. shouldn't be too hard.
we'll decide about the contributor stuff after that :)
btw i need this to be in the ZODB version current Zope2 uses,
is this one on github already? if so, where can i find it? even
if i don't become a contributor this would make generating patches
much easier.
Jim
thanks for your help!
Jürgen
--
XLhost.de ® - Webhosting von supersmall bis eXtra Large <<
XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany
Geschäftsführer: Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218
Fon: +49 (0)800 XLHOSTDE [0800 95467833]
Fax: +49 (0)800 95467830
Web: http://www.XLhost.de
_______________________________________________
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
