Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
On 28/01/2011 21:58, Jürgen Herrmann wrote: Afaics you use zodbpack's default of days=0. This is known to produce POSKeyErrors if the database is written to while packing. Where is this documented? Does this apply to history-keeping or history-free storages? Try with something like days=0.1 . I'll not pack a history-free storage until Shane gives it the okay... However, I find it a bit odd that RelStorage ships with a default that could cause these kind of problems and no warning in the readme that this is the case... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
On 01/31/2011 06:02 PM, Chris Withers wrote: On 28/01/2011 21:58, Jürgen Herrmann wrote: Afaics you use zodbpack's default of days=0. This is known to produce POSKeyErrors if the database is written to while packing. Where is this documented? It is not known to me. :-) I'll not pack a history-free storage until Shane gives it the okay... Good, because although I fixed a bug your test revealed, your test still fails, and I don't yet know why. :-( Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
Bingo:
svn co https://secure.simplistix.co.uk/svn/relstorage_pack -r 4595
cd relstorage_pack
python2.6 bootstrap.py
bin/buildout
Then in one terminal:
bin/generate --zap --folder_depth 3
Leave it running for a minute or two, and then in another do:
bin/zodbpack pack.conf
After a bit I get:
Traceback (most recent call last):
File bin/generate, line 100, in module
generate.generate('/home/zope/relstorage_pack/zodb.conf')
File /home/zope/relstorage_pack/generate.py, line 64, in generate
container.manage_delObjects([path[-1]])
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line
847, in setstate
self._setstate(obj)
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line
897, in _setstate
p, serial = self._storage.load(obj._p_oid, '')
File
/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py,
line 462, in load
raise POSKeyError(oid)
ZODB.POSException.POSKeyError: 0x1675
cheers,
Chris
--
Simplistix - Content Management, Batch Processing Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
On 28 Jan 2011, at 21:58, Jürgen Herrmann juergen.herrm...@xlhost.de wrote:
On Fri, 28 Jan 2011 12:34:35 +, Chris Withers wrote:
Bingo:
svn co https://secure.simplistix.co.uk/svn/relstorage_pack -r 4595
cd relstorage_pack
python2.6 bootstrap.py
bin/buildout
Then in one terminal:
bin/generate --zap --folder_depth 3
Leave it running for a minute or two, and then in another do:
bin/zodbpack pack.conf
After a bit I get:
Traceback (most recent call last):
File bin/generate, line 100, in module
generate.generate('/home/zope/relstorage_pack/zodb.conf')
File /home/zope/relstorage_pack/generate.py, line 64, in
generate
container.manage_delObjects([path[-1]])
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py,
line
847, in setstate
self._setstate(obj)
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py,
line
897, in _setstate
p, serial = self._storage.load(obj._p_oid, '')
File
/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py,
line 462, in load
raise POSKeyError(oid)
ZODB.POSException.POSKeyError: 0x1675
cheers,
Chris
Afaics you use zodbpack's default of days=0. This is known to produce
POSKeyErrors if the database is written to while packing. Try with
something
like days=0.1 .
Regards,
Jürgen
--
XLhost.de ® - Webhosting von supersmall bis eXtra Large
XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany
Geschäftsführer: Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218
Fon: +49 (0)800 XLHOSTDE [0800 95467833]
Fax: +49 (0)800 95467830
Web: http://www.XLhost.de
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
__
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
On 01/28/2011 05:34 AM, Chris Withers wrote: bin/generate --zap --folder_depth 3 Leave it running for a minute or two, and then in another do: bin/zodbpack pack.conf Thanks to this test, I found a serious flaw in the history-free variant of packing. Currently, the history-free and history-preserving variants share the fill_object_refs() method, which assumes that it's possible to enumerate the references from all objects in a specific transaction. However, that assumption is only correct in the history-preserving variant. In history-free storages, object states disappear every time they are replaced by a newer version, causing fill_object_refs() to think the replaced objects have no references. That mistake leads to garbage collection of too many objects. I'm surprised that existing tests did not reveal this. Until I release a fix, I recommend not packing history-free databases. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
