[ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-22 Thread Martijn Pieters
-intrusive background operation. Is this a viable scenario? -- Martijn Pieters twophasepack.patch Description: Binary data ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-22 Thread Martijn Pieters
a mirrorred test cluster high on our wishlist, but having the database packed first would make that easier given the size of the database now. In the future however, copying back the prepared pack tables is an excellent prospect. -- Martijn Pieters

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-22 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 19:12, Shane Hathaway sh...@hathawaymix.org wrote: On 02/22/2011 09:25 AM, Martijn Pieters wrote: I haven't yet actually run this code, but the change isn't big. I didn't find any relevant tests to update. Anyone want to venture some feedback? Both ideas are excellent

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
, presumably queries during a GC run should be easier on the database as there is less data to scan through. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 21:41, Martijn Pieters m...@zopatista.com wrote: BTW, should I just commit the patch, or do you want to integrate it yourself? Updated patch attached; added the options changes to component.xml and README.txt. -- Martijn Pieters twophasepack.patch Description: Binary

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 11:55, Martijn Pieters m...@zopatista.com wrote: Updated patch attached; added the options changes to component.xml and README.txt. I've moved this patch to bitbucket at https://bitbucket.org/mjpieters/relstorage-mq/src/tip/twophasepack.patch and updated the README

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-23 Thread Martijn Pieters
On Tue, Feb 22, 2011 at 21:41, Martijn Pieters m...@zopatista.com wrote: I'll look into working the locking idea into a patch too, but I'll need help with supporting Postgres and MySQL as I don't know their locking semantics. Both MySQL and Oracle support lock timeouts and already use

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 15:08, Martijn Pieters m...@zopatista.com wrote: I've started a optimistic locking strategy patch in my patch queue, contains this locking strategy change only for now:  https://bitbucket.org/mjpieters/relstorage-mq/src/tip/optimistic_commitlock_pack.patch I've made

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Wed, Feb 23, 2011 at 14:41, Martijn Pieters m...@zopatista.com wrote: I've moved this patch to bitbucket at https://bitbucket.org/mjpieters/relstorage-mq/src/tip/twophasepack.patch and updated the README a little more to document the options to zodbpack. The two-phase pack patch has been

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-24 Thread Martijn Pieters
On Thu, Feb 24, 2011 at 14:26, Martijn Pieters m...@zopatista.com wrote: I've made progress on the patch this afternoon. Next up are tests for both patches. The above patch now uses the nowait locking strategy to run pack batches. It has been renamed though, and now lives at:  https

[ZODB-Dev] Large ZODB packing and progress

2011-02-25 Thread Martijn Pieters
to RelStorage SVN after we tried it on for size, in the next few days. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman

Re: [ZODB-Dev] Large ZODB packing and progress

2011-02-25 Thread Martijn Pieters
23:04:01,513 [relstorage.adapters.packundo] INFO pre_pack: finished successfully 2011-02-24 23:04:01,531 [relstorage] INFO pack: pre-pack complete 2011-02-24 23:04:01,539 [zodbpack] INFO Packed storage (RelStorageFactory). so it took under 2 hours. -- Martijn Pieters

Re: [ZODB-Dev] Large ZODB packing and progress

2011-02-25 Thread Martijn Pieters
to lock. It'll only pause with packing if the lock is already taken. That way packing will go faster when there are no other transactions occurring anyway. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-25 Thread Martijn Pieters
On Fri, Feb 25, 2011 at 12:48, Shane Hathaway sh...@hathawaymix.org wrote: Do your best and I'll take a look after you've committed the patches to the trunk. :-) The pack progress and two-phase pack patches have been committed. -- Martijn Pieters

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-25 Thread Martijn Pieters
On Thu, Feb 24, 2011 at 16:56, Martijn Pieters m...@zopatista.com wrote: I see a lot of transaction aborted errors on the ZODB multi-thread tests with this patch in place, so I'll have to investigate more. Thread debugging joy! In the end it was a simple mistake in the PostgreSQL version

Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour

2011-02-28 Thread Martijn Pieters
, but not 5.5 yet. The nowait locking strategy worked beautifully on our mammoth pack, so I've committed it to RelStorage trunk. I didn't get a chance to test it against MySQL 5.1 though, but the MySQL changes were minimal and mirrored those for the Oracle version. -- Martijn Pieters

[ZODB-Dev] Pack complete: 80% of a 425GB database packed

2011-02-28 Thread Martijn Pieters
deletion SQL statements have been optimized for MySQL, but not for the history-free version. Wouldn't those statements not also benefit from using a JOIN? -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Martijn Pieters
BlobCacheChecker.loaded with check=True (the default). Try calling it with check=False instead. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Martijn Pieters
established that the cache cleanup in RelStorage is to blame here, so we can focus on _check_blob_cache_size, specifically on line 490 and further. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB

Re: [ZODB-Dev] Pack complete: 80% of a 425GB database packed

2011-03-01 Thread Martijn Pieters
package bit me at some point, disabling our cluster. Luckily a swift restart of one of the clients reinstated the 1.4 version quickly. :-) We are not quite ready for a full 1.5.0 upgrade just yet.. ;-) -- Martijn Pieters ___ For more information about

Re: [ZODB-Dev] python path problem: ImportError when calling zodbconvert (FreeBSD 8.1)

2011-04-26 Thread Martijn Pieters
://stackoverflow.com/q/5769777/100297 -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Relstorage Blob support and Oracle

2011-06-09 Thread Martijn Pieters
after my refactor. Any reason to keep it, other than that others whom already are using 1.5b2 on an Oracle database will now have to drop that column again (or set it to a default value of 0 on insert)? Should the code support reading blob chunks still? -- Martijn Pieters

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-09 Thread Martijn Pieters
overhead compared with streaming. I'm retaining the schema; there is a chance people have updated to 1.5b2 already and are using blobs in production. My refactor maintains compatibility with the chunked blob storage. -- Martijn Pieters ___ For more

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-09 Thread Martijn Pieters
On Thu, Jun 9, 2011 at 22:03, Martijn Pieters m...@zopatista.com wrote: I'm retaining the schema; there is a chance people have updated to 1.5b2 already and are using blobs in production. My refactor maintains compatibility with the chunked blob storage. I've attached my patch for review

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-10 Thread Martijn Pieters
in that it let's you specify files directly to upload from or download to. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-12 Thread Martijn Pieters
/usage.html#index-7 Not encoding ZODB pickles for the object_state table from and to base64 would save a lot of space (base64 encoding expands data to 133% it's original size, leading to up to 1/3rd more space required to store a Zope ZODB in PostgreSQL than in other databases. -- Martijn Pieters

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-13 Thread Martijn Pieters
, but RelStorage never does that; it's either INSERT or DELETE. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-15 Thread Martijn Pieters
On Tue, Jun 14, 2011 at 22:22, Martijn Pieters m...@zopatista.com wrote: I am considering altering the ON COMMIT behaviour of the temp_blob_chunk table to TRUNCATE to extend this behaviour to all remaining blob rows on commit. However, such events should be rare enough

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-15 Thread Martijn Pieters
On Wed, Jun 15, 2011 at 16:23, Martijn Pieters m...@zopatista.com wrote: Last but not least I'll need to write a migration script for those users of RelStorage 1.5.0b2 already in production. I have figured out how to do this (see http://archives.postgresql.org/pgsql-general/2009-01/msg00771

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-15 Thread Martijn Pieters
without issue. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Bug during pack with Relstorage 1.5.0

2011-08-31 Thread Martijn Pieters
particular problem though. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] RelStorage breaks History tab

2011-09-02 Thread Martijn Pieters
of the stock FileStorage implementation, not on public ZODB APIs. I haven't touched the archetypes History code in 5 years at least though, so I am not sure about this. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http

[ZODB-Dev] Debugging RelStorage hang-ups

2011-10-20 Thread Martijn Pieters
at where to start debugging this. Any hints from anyone? -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Debugging RelStorage hang-ups

2011-10-20 Thread Martijn Pieters
has weird edge cases.) The test rig is not using RAC. In fact, the production environment no longer uses RAC either. -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https

Re: [ZODB-Dev] zeopack error

2012-01-30 Thread Martijn Pieters
there. -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] zeopack error

2012-01-31 Thread Martijn Pieters
ZEO server and attempt to pack again. Now it should log the exception instead of None. -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb

Re: [ZODB-Dev] zeopack error

2012-01-31 Thread Martijn Pieters
for people on this list to analyze. I cannot promise anyone will, of course, but if someone could and the pickle is shown to only contain primitive data types (no references at all) then the pass is certainly going to solve your problem. -- Martijn Pieters

Re: [ZODB-Dev] Migrating from Plone 4.1.4 ZEO data.fs and blobs to Relstorage

2012-04-16 Thread Martijn Pieters
all ZODB blobs also live in the same database. Performance is quite decent but the customer doesn't try to store more than a few MBs per file. Better keep them in a NFS setup for most cases. -- Martijn Pieters ___ For more information about ZODB, see

Re: [ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

2012-04-19 Thread Martijn Pieters
on SIGUSR1, like the Zope signalstack product does (see http://svn.plone.org/svn/collective/Products.signalstack/trunk/Products/signalstack/__init__.py for the exact code to bind the signal handler). That'll tell you exactly what each thread is busy with when you send the signal. -- Martijn

Re: [ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

2012-04-19 Thread Martijn Pieters
locked up, then it could be that the server is not answering your is_alive request at all because it is waiting for a thread to free up first. -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev

Re: [ZODB-Dev] Solved!!! Re: unable to import ZODB: class ConflictError, AttributeError

2012-06-14 Thread Martijn Pieters
switch; this makes sure your environment does not try to import globally-installed packages. -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo

Re: [ZODB-Dev] Solved!!! Re: unable to import ZODB: class ConflictError, AttributeError

2012-06-14 Thread Martijn Pieters
On Thu, Jun 14, 2012 at 5:14 PM, Paul Winkler sli...@gmail.com wrote: That's backwards, I'm sure you just made a typo but I don't want Ralf to get confused :) Yeah, my mistake. *with*, not *without*. Sorry! -- Martijn Pieters ___ For more

Re: [ZODB-Dev] Solved!!! Re: unable to import ZODB: class ConflictError, AttributeError

2012-06-14 Thread Martijn Pieters
of persistent objects if parts need changing over time. Future commits would be a lot smaller that way. I'd parse out the original large dictionary in a separate process, perhaps chunk the commits (so build up the structure over several commits). -- Martijn Pieters

Re: [ZODB-Dev] PosKeyError in DateRangeIndex with RelStorage

2012-08-03 Thread Martijn Pieters
(upgraded from 3.x last year). -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB

2012-08-28 Thread Martijn Pieters
act as a BlobStorage, NEO can not. Anything else different? Did you make any performance comparisons between RelStorage and NEO? -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org

Re: [ZODB-Dev] Relstorage and over growing database.

2013-11-12 Thread Martijn Pieters
garbage collection. -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Relstorage and over growing database.

2013-11-14 Thread Martijn Pieters
* not be reading this list actively. -- Martijn Pieters ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev