Re: [ZODB-Dev] zodb conversion questions
Am 07.02.2013 17:11, schrieb Jim Fulton: On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: Am 06.02.2013 15:05, schrieb Jürgen Herrmann: Hi there! I hav a relstorage with mysql backend that grew out of bounds and we're looking into different backend solutions now. Possibly also going back to FileStorage and using zeo... Anyway we'll have to convert the databases at some point. As these are live DBs we cannot shut them down for longer than the ususal maintenance interval during the night, so for maybe 2-3h. a full conversion process will never complete in this time so we're looking for a process that can split the conversion into two phases: 1. copy transactions from backup of the source db to the destination db. this can take a long time, we don't care. note the last timestamp/transaction_id converted. 2. shut down the source db 3. copy transactions from the source db to the destination db, starting at the last converted transaction_id. this should be fast, as only a few transactions need to be converted, say 1% . if i would reimplement copyTransactionsFrom() to accept a start transaction_id/timestamp, would this result in dest being an exact copy of source? source = open_my_source_storage() dest = open_my_destination_storage() dest.copyTransactionsFrom(source) last_txn_id = source.lastTransaction() source.close() dest.close() source = open_my_source_storage() # add some transactions source.close() source = open_my_source_storage() dest = open_my_destination_storage() dest.copyTransactionsFrom(source, last_txn_id=last_txn_id) source.close() dest.close() I will reply to myself here :) This actually works, tested with a modified version of FileStorage for now. I modified the signature of copyTransactionsFrom to look like this: def copyTransactionsFrom(self, source, verbose=0, not_before_tid=None): ``start`` would be better to be consistent with the iterator API. not_before_tid is a packed tid or None, None meaning copy all (the default, so no existing API usage would break). Is there public interest in modifying this API permamently? +.1 This API is a bit of an attractive nuisance. I'd rather people learn how to use iterators in their own scripts, as they are very useful and powerful. This API just hides that. The second part, replaying old transactions is a bit more subtle, but it's still worth it for people to be aware of it. If I were doing this today, I'd make this documentation rather than API. But then, documentation ... whimper. Anybody want to look at the actual code changes? Sure, if they have tests. Unfortunately, we can only accept pull requests from zope contributors. Are you one? Wanna be one? :) Jim 1. SUCCESS. I migrated to FileStorage/ZEO successfully with the modified copyTransactionsFrom() with downtime of only 10 minutes. Very cool :) DB sizes are reasonable again and repozo backups are set up, just like before the migration to RelStorage. Feels robust again, a good feeling. I'd advise anybody thinking about migrating to RelStorage to think about a proper (incremental) backup strategy for the underlying sql db first, this might become crucial! repozo makes your life easy there when using FileStorage. 2. Regarding the tests: I did not find any test for copyTransactionsFrom() in the current test suite, am I blind? If there isn't any test yet, maybe you could suggest a proper file for the test, maybe even a similar one to start from? 3. Regarding my request to have this in ZODB 3.10.5: After reading my own statement I quickly realized that changing an API can only happen in a major version. Anyway, this was a one-shot action so I'll probably never need my own code again :) best regards, Jürgen -- XLhost.de ® - Webhosting von supersmall bis eXtra Large XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 Web: http://www.XLhost.de ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb conversion questions
Hi, Will be cool if you create a package with this script and with instructions to patch ZODB and maybe new major version could have the new API. If you need help I'm interested to do it. :-) Cheers, On Thu, Feb 14, 2013 at 6:29 PM, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: Am 07.02.2013 17:11, schrieb Jim Fulton: On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: Am 06.02.2013 15:05, schrieb Jürgen Herrmann: Hi there! I hav a relstorage with mysql backend that grew out of bounds and we're looking into different backend solutions now. Possibly also going back to FileStorage and using zeo... Anyway we'll have to convert the databases at some point. As these are live DBs we cannot shut them down for longer than the ususal maintenance interval during the night, so for maybe 2-3h. a full conversion process will never complete in this time so we're looking for a process that can split the conversion into two phases: 1. copy transactions from backup of the source db to the destination db. this can take a long time, we don't care. note the last timestamp/transaction_id converted. 2. shut down the source db 3. copy transactions from the source db to the destination db, starting at the last converted transaction_id. this should be fast, as only a few transactions need to be converted, say 1% . if i would reimplement copyTransactionsFrom() to accept a start transaction_id/timestamp, would this result in dest being an exact copy of source? source = open_my_source_storage() dest = open_my_destination_storage() dest.copyTransactionsFrom(**source) last_txn_id = source.lastTransaction() source.close() dest.close() source = open_my_source_storage() # add some transactions source.close() source = open_my_source_storage() dest = open_my_destination_storage() dest.copyTransactionsFrom(**source, last_txn_id=last_txn_id) source.close() dest.close() I will reply to myself here :) This actually works, tested with a modified version of FileStorage for now. I modified the signature of copyTransactionsFrom to look like this: def copyTransactionsFrom(self, source, verbose=0, not_before_tid=None): ``start`` would be better to be consistent with the iterator API. not_before_tid is a packed tid or None, None meaning copy all (the default, so no existing API usage would break). Is there public interest in modifying this API permamently? +.1 This API is a bit of an attractive nuisance. I'd rather people learn how to use iterators in their own scripts, as they are very useful and powerful. This API just hides that. The second part, replaying old transactions is a bit more subtle, but it's still worth it for people to be aware of it. If I were doing this today, I'd make this documentation rather than API. But then, documentation ... whimper. Anybody want to look at the actual code changes? Sure, if they have tests. Unfortunately, we can only accept pull requests from zope contributors. Are you one? Wanna be one? :) Jim 1. SUCCESS. I migrated to FileStorage/ZEO successfully with the modified copyTransactionsFrom() with downtime of only 10 minutes. Very cool :) DB sizes are reasonable again and repozo backups are set up, just like before the migration to RelStorage. Feels robust again, a good feeling. I'd advise anybody thinking about migrating to RelStorage to think about a proper (incremental) backup strategy for the underlying sql db first, this might become crucial! repozo makes your life easy there when using FileStorage. 2. Regarding the tests: I did not find any test for copyTransactionsFrom() in the current test suite, am I blind? If there isn't any test yet, maybe you could suggest a proper file for the test, maybe even a similar one to start from? 3. Regarding my request to have this in ZODB 3.10.5: After reading my own statement I quickly realized that changing an API can only happen in a major version. Anyway, this was a one-shot action so I'll probably never need my own code again :) best regards, Jürgen -- XLhost.de ® - Webhosting von supersmall bis eXtra Large XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-**Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 Web: http://www.XLhost.de __**_ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/**listinfo/zodb-devhttps://mail.zope.org/mailman/listinfo/zodb-dev -- Rudá Porto Filgueiras http://python-blog.blogspot.com http://twitter.com/rudaporto ___ For more information about ZODB, see http://zodb.org/
[ZODB-Dev] A certain code path seems to be blocking everything
I've got a weird bug with my server and I'm wondering if anyone could provide some insight into it. The general idea of the server is that I have 8 or so clients constantly pinging it for information. Each ping usually only takes up to 2 seconds to process. Essentially, everything runs fine, until a certain code path is executed - basically, one of the clients causes the server to try to give it some other information than usual. When this client is turned on and starts pinging the server, suddenly all those requests that only took 0.5-2 seconds now take 10, 20, or even 30 seconds. I'm using paster and using its thread-tracker tool I can see what all the threads are doing at any given point. Usually there's anywhere from 0 to 4 requests going on, each having taken less than 1 second up to that point. When the 'special client' is turned on, there's more like 10 requests each having taken 6 seconds or more up to that point. There aren't more requests, it's just that they don't clear out quickly enough. The tail end of the tracebacks are one of two things: File /home/tsa/env/lib/python2.6/site-packages/transaction/_manager.py, line 89, in commit return self.get().commit() File /home/tsa/env/lib/python2.6/site-packages/transaction/_transaction.py, line 329, in commit self._commitResources() File /home/tsa/env/lib/python2.6/site-packages/transaction/_transaction.py, line 441, in _commitResources rm.tpc_begin(self) File /home/tsa/env/lib/python2.6/site-packages/ZODB/Connection.py, line 547, in tpc_begin self._normal_storage.tpc_begin(transaction) File /home/tsa/env/lib/python2.6/site-packages/ZEO/ClientStorage.py, line 1118, in tpc_begin self._tpc_cond.wait(30) File /usr/lib/python2.6/threading.py, line 258, in wait _sleep(delay) Or: File /home/tsa/env/lib/python2.6/site-packages/ZODB/Connection.py, line 856, in setstate self._setstate(obj) File /home/tsa/env/lib/python2.6/site-packages/ZODB/Connection.py, line 897, in _setstate p, serial = self._storage.load(obj._p_oid, '') File /home/tsa/env/lib/python2.6/site-packages/ZEO/ClientStorage.py, line 824, in load self._load_lock.acquire() I read through the code a bit and it sees that only one thread can commit at a time. So, my first guess is that the 'special' request is taking particularly long to commit, which ends up slowing everything else down. Would that be a fair estimation? What might cause a commit to take abnormally long? Just looking at the code I can't see anything that particularly stands out, especially since this bug only started happening recently and I haven't really changed that part of the code on the server-side recently. I did refactor the client code but I don't yet see how that might be causing this issue. One last thing. I have certain views which touch a lot of different parts of the database and do a lot of processing. I had many conflict errors in the past, and retrying the entire view all the time proved to be too inefficient, so I separated the code out into blocks, where I retry each block separately. This function describes the pattern well: def commit_and_inner_retry(transaction_manager, func, num_retries, ingdescription)): Using the `transaction_manager`: 1) commit the transaction thus far 2) try `num_retries` times to begin a new transaction, execute `func`, which is passed the current transaction as its only argument, and then commit the new transaction. if ConflictError is raised, abort try again 3) if a failure happens, then an error is printed involving `ingdescription` 4) raise ConflictError if retrying `num_retries` times does not work. tm = transaction_manager tm.commit() for retry in range(num_retries): tm.begin() return_value = func(tm.get()) try: tm.commit() except ConflictError: print Conflict error attempt #%d %s, trying again % (retry+1, ingdescription) tm.abort() continue break else: raise ConflictError(Was never able to commit %s % ingdescription) return return_value With the view code looking something like this: def long_view(context, request): db_objs = prepare_stuff() def do_stuff_1(): return stuff_1_with_db_objs(db_objs) stuff1 = commit_and_inner_retry( transaction.manager, block, 10, 'doing stuff 1') def do_stuff_2(): return stuff_2_with_db_objs(stuff1) stuff2 = commit_and_inner_retry( transaction.manager, block, 10, 'doing stuff 2') def do_stuff_3(): stuff_3_with_db_objs(stuff2) try: commit_and_inner_retry( transaction.manager, block, 10, 'doing stuff 3') except ConflictError: #this stuff not important
