Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-28 Thread Martijn Pieters
On Fri, Feb 25, 2011 at 20:16, Shane Hathaway wrote: > My Buildbot is reporting success with all 3 databases on several > platforms, so it's looking good.  It's having trouble on Windows, but I > suspect that's a build bug, not a software bug.  RelStorage supports > MySQL 5.1, but not 5.5 yet. Th

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-25 Thread Shane Hathaway
On 02/25/2011 08:31 AM, Martijn Pieters wrote: > On Thu, Feb 24, 2011 at 16:56, Martijn Pieters wrote: >> I see a lot of transaction aborted errors on the ZODB multi-thread >> tests with this patch in place, so I'll have to investigate more. >> Thread debugging joy! > > In the end it was a simple

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-25 Thread Martijn Pieters
On Thu, Feb 24, 2011 at 16:56, Martijn Pieters wrote: > I see a lot of transaction aborted errors on the ZODB multi-thread > tests with this patch in place, so I'll have to investigate more. > Thread debugging joy! In the end it was a simple mistake in the PostgreSQL version of the commit lock me

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-25 Thread Martijn Pieters
On Fri, Feb 25, 2011 at 12:48, Shane Hathaway wrote: > Do your best and I'll take a look after you've committed the patches to > the trunk. :-) The pack progress and two-phase pack patches have been committed. -- Martijn Pieters ___ For more informati

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-25 Thread Shane Hathaway
On 02/22/2011 01:41 PM, Martijn Pieters wrote: > On Tue, Feb 22, 2011 at 19:12, Shane Hathaway wrote: >> On 02/22/2011 09:25 AM, Martijn Pieters wrote: >>> I haven't yet actually run this code, but the change isn't big. I >>> didn't find any relevant tests to update. Anyone want to venture some >>

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Thu, Feb 24, 2011 at 14:26, Martijn Pieters wrote: > I've made progress on the patch this afternoon. Next up are tests for > both patches. The above patch now uses the nowait locking strategy to > run pack batches. It has been renamed though, and now lives at: > >  https://bitbucket.org/mjpiete

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 14:41, Martijn Pieters wrote: > I've moved this patch to bitbucket at > https://bitbucket.org/mjpieters/relstorage-mq/src/tip/twophasepack.patch > and updated the README a little more to document the options to > zodbpack. The two-phase pack patch has been updated again:

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 15:08, Martijn Pieters wrote: > I've started a optimistic locking strategy patch in my patch queue, > contains this locking strategy change only for now: > >  https://bitbucket.org/mjpieters/relstorage-mq/src/tip/optimistic_commitlock_pack.patch> I've made progress on the

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 21:41, Martijn Pieters wrote: > I'll look into working the locking idea into a patch too, > but I'll need help with supporting Postgres and MySQL as I don't know > their locking semantics. Both MySQL and Oracle support lock timeouts and already use a timeout for the commit

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 11:55, Martijn Pieters wrote: > Updated patch attached; added the options changes to component.xml and > README.txt. I've moved this patch to bitbucket at https://bitbucket.org/mjpieters/relstorage-mq/src/tip/twophasepack.patch and updated the README a little more to docum

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 21:41, Martijn Pieters wrote: > BTW, should I just commit the patch, or do you want to integrate it > yourself? Updated patch attached; added the options changes to component.xml and README.txt. -- Martijn Pieters twophasepack.patch Description: Binary data ___

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 22:51, Maurits van Rees wrote: > I wonder it it may help to set pack-gc to false during the first pack. > According to the docs this is faster, though it of course leaves more > unused objects behind.  Set pack-gc to the default true value for > subsequent packs.  Theoretic

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-22 Thread Maurits van Rees
Op 22-02-11 17:25, Martijn Pieters schreef: > Hi, > > I was already investigating the possibility to split the RelStorage > packing process up into smaller chunks. > > Due to the expected load on the Oracle cluster during a pack, we'll > have to run the pack at night and want to be absolutely certa

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-22 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 19:12, Shane Hathaway wrote: > On 02/22/2011 09:25 AM, Martijn Pieters wrote: >> I haven't yet actually run this code, but the change isn't big. I >> didn't find any relevant tests to update. Anyone want to venture some >> feedback? > > Both ideas are excellent.  The new op

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-22 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 19:12, Shane Hathaway wrote: > Both ideas are excellent.  The new options even open the possibility of > running the pre-pack on a copy of the database, then copying the pack > tables to the main database for a final run. For this project, we have a mirrorred test cluster

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-22 Thread Shane Hathaway
On 02/22/2011 09:25 AM, Martijn Pieters wrote: > I haven't yet actually run this code, but the change isn't big. I > didn't find any relevant tests to update. Anyone want to venture some > feedback? Both ideas are excellent. The new options even open the possibility of running the pre-pack on a

[ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

2011-02-22 Thread Martijn Pieters
Hi, I was already investigating the possibility to split the RelStorage packing process up into smaller chunks. Due to the expected load on the Oracle cluster during a pack, we'll have to run the pack at night and want to be absolutely certain that database is ready for normal site operations aga