Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-28 Thread Martijn Pieters
On Fri, Feb 25, 2011 at 20:16, Shane Hathaway sh...@hathawaymix.org wrote: My Buildbot is reporting success with all 3 databases on several platforms, so it's looking good.  It's having trouble on Windows, but I suspect that's a build bug, not a software bug.  RelStorage supports MySQL 5.1,

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-25 Thread Shane Hathaway
On 02/22/2011 01:41 PM, Martijn Pieters wrote: On Tue, Feb 22, 2011 at 19:12, Shane Hathawaysh...@hathawaymix.org wrote: On 02/22/2011 09:25 AM, Martijn Pieters wrote: I haven't yet actually run this code, but the change isn't big. I didn't find any relevant tests to update. Anyone want to

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-25 Thread Martijn Pieters
On Fri, Feb 25, 2011 at 12:48, Shane Hathaway sh...@hathawaymix.org wrote: Do your best and I'll take a look after you've committed the patches to the trunk. :-) The pack progress and two-phase pack patches have been committed. -- Martijn Pieters

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-25 Thread Martijn Pieters
On Thu, Feb 24, 2011 at 16:56, Martijn Pieters m...@zopatista.com wrote: I see a lot of transaction aborted errors on the ZODB multi-thread tests with this patch in place, so I'll have to investigate more. Thread debugging joy! In the end it was a simple mistake in the PostgreSQL version of

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 15:08, Martijn Pieters m...@zopatista.com wrote: I've started a optimistic locking strategy patch in my patch queue, contains this locking strategy change only for now:  https://bitbucket.org/mjpieters/relstorage-mq/src/tip/optimistic_commitlock_pack.patch I've made

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 14:41, Martijn Pieters m...@zopatista.com wrote: I've moved this patch to bitbucket at https://bitbucket.org/mjpieters/relstorage-mq/src/tip/twophasepack.patch and updated the README a little more to document the options to zodbpack. The two-phase pack patch has been

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Thu, Feb 24, 2011 at 14:26, Martijn Pieters m...@zopatista.com wrote: I've made progress on the patch this afternoon. Next up are tests for both patches. The above patch now uses the nowait locking strategy to run pack batches. It has been renamed though, and now lives at:  

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 22:51, Maurits van Rees m.van.r...@zestsoftware.nl wrote: I wonder it it may help to set pack-gc to false during the first pack. According to the docs this is faster, though it of course leaves more unused objects behind.  Set pack-gc to the default true value for

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 21:41, Martijn Pieters m...@zopatista.com wrote: BTW, should I just commit the patch, or do you want to integrate it yourself? Updated patch attached; added the options changes to component.xml and README.txt. -- Martijn Pieters twophasepack.patch Description: Binary

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 11:55, Martijn Pieters m...@zopatista.com wrote: Updated patch attached; added the options changes to component.xml and README.txt. I've moved this patch to bitbucket at https://bitbucket.org/mjpieters/relstorage-mq/src/tip/twophasepack.patch and updated the README a

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 21:41, Martijn Pieters m...@zopatista.com wrote: I'll look into working the locking idea into a patch too, but I'll need help with supporting Postgres and MySQL as I don't know their locking semantics. Both MySQL and Oracle support lock timeouts and already use a

[ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-22 Thread Martijn Pieters
Hi, I was already investigating the possibility to split the RelStorage packing process up into smaller chunks. Due to the expected load on the Oracle cluster during a pack, we'll have to run the pack at night and want to be absolutely certain that database is ready for normal site operations

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-22 Thread Shane Hathaway
On 02/22/2011 09:25 AM, Martijn Pieters wrote: I haven't yet actually run this code, but the change isn't big. I didn't find any relevant tests to update. Anyone want to venture some feedback? Both ideas are excellent. The new options even open the possibility of running the pre-pack on a

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-22 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 19:12, Shane Hathaway sh...@hathawaymix.org wrote: Both ideas are excellent.  The new options even open the possibility of running the pre-pack on a copy of the database, then copying the pack tables to the main database for a final run. For this project, we have a