Re: [ZODB-Dev] Changing namespace - best strategy
On 08/23/2013 09:07 AM, Marius Gedminas wrote: And then there's https://pypi.python.org/pypi/zodbupdate which sounds exactly like the tool you want for this. Yeah, that definitely works. You just got to get the moving classes/definitions right. -- Best regards, Adam GROSZER -- Quote of the day: Knowledge by suffering entereth, And life is perfected by death. - Elizabeth Barrett Browning ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zc.zlibstorage missing from zodb package
On 07/21/2013 05:09 AM, Christian Tismer wrote: - discussion zc.zlibstorage requites a wrapper to add it to filestorage. I consider this an option, instead, and a simple boolean flag to switch it on and off. The module is way too simple to add all this config extra complication to even think of it. IMHO the wrapper architecture is a good thing, you can do some handy things as: https://pypi.python.org/pypi/cipher.encryptingstorage -- Best regards, Adam GROSZER -- Quote of the day: It's not whether you win or lose, it's how good you look playing! - David Lee Roth ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction: synchronizer newTransaction() behavior
Hello, Slightly offtopic: https://pypi.python.org/pypi/mongopersist On 03/09/2013 03:54 AM, Siddhartha Kasivajhula wrote: ... -- Best regards, Adam GROSZER -- Quote of the day: The only thing worse than an alarm going off is one that doesn't. - Unknown ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Zope3-Users] MemoryError Evolving a ZODB
Hello, That approach works for us, on DBs over 100GB. Let's CC zodb-dev, which seems to be the better place to discuss this. On 12/12/2012 09:39 AM, Jeroen Michiel wrote: Thanks for the reply! I already tried transaction.savepoint() every minute, but that didn't help: I only saw the memory usage dropping the first time, but never after. I changed the code to what you suggested, but it still doesn't seem to help. Something must be wrong somewhere along the line, but I don't have a clue where to begin looking. Would using something like guppy (or heapy, or what it's called) reveal something? Could it be something about objects with circular references not being able to be garbage-collected? The objects in my DB are quite complex, so something like that might actually be happening. Adam Groszer-3 wrote: Well it loads too many objects in a single transaction. Doing this after some iterations (10k?, depends on your object sizes) helps usually: def forceSavepoint(anyPersistentObject=None): transaction.savepoint(optimistic=True) if anyPersistentObject is not None: #and clear picklecache conn = anyPersistentObject._p_jar conn.cacheGC() -- Best regards, Adam GROSZER -- Quote of the day: A liberal is someone too poor to be a capitalist and too rich to be a communist. ___ Zope3-users mailing list zope3-us...@zope.org https://mail.zope.org/mailman/listinfo/zope3-users -- Best regards, Adam GROSZER -- Quote of the day: The Atomic Age is here to stay - but are we? - Bennett Cerf ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
Hello, Now we have some problems making the binary eggs. See the attached txt for the full output. On 08/26/2012 09:07 PM, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/26/2012 07:05 AM, Hanno Schlichting wrote: On Sat, Aug 25, 2012 at 4:11 PM, Tres Seaver tsea...@palladion.com wrote: This failure, and the others like it, indicate that the buildout has failed to install persistent correctly (there is nothing Windows-specific about the failure):: I actually got the same error locally. The trouble is the timestamp.py module and the TimeStamp.c extension. During installation setuptools creates a wrapper for the extension file. So you end up with TimeStamp.so but also a timestamp.py wrapper. This wrapper overwrites the original timestamp.py module. You likely have a case-sensitive file system and therefor didn't get this problem. Whoever came up with the idea of a case-insensitive filesystem should be shot (or at least, whoever decided to keep doing that after moving off of DOS). I have worked around this problem (I think) by renaming the extension module to '_timestamp.so'. I released a 4.0.1 to PyPI with this fix, along with a fix for the int - long overflow on 32-bit systems. I think TimeStamp.c has been there before and is imported in other modules as from persistent.TimeStamp import TimeStamp. So we should keep that API and use a different name for the new timestamp.py module. I don't see the need for the BBB-preservation here: I documented the backward-incompatibility in CHANGES.txt instead. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA6c9IACgkQ+gerLs4ltQ6ocACeIqlGWazwos+yWGaMvgMCOkQD ZQoAn3oR2rY70a1oT6aTjLpofIVtVBNd =GzPs -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Best regards, Adam GROSZER -- Quote of the day: Never let a man imagine that he can pursue a good end by evil means, without sinning against his own soul. The evil effect on himself is certain. - Robert Southey running build_ext building 'persistent.cPersistence' extension creating build creating build\temp.win32-2.6 creating build\temp.win32-2.6\Release creating build\temp.win32-2.6\Release\persistent C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python26_32\include -Ic:\Python26_32\PC /Tcpersistent/cPersistence.c /Fobuild\temp.win32-2.6\Release\persistent/cPersistence.obj cPersistence.c persistent/cPersistence.c(1124) : warning C4047: '' : '__int64' differs in levels of indirection from 'PyObject *' C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python26_32\include -Ic:\Python26_32\PC /Tcpersistent/ring.c /Fobuild\temp.win32-2.6\Release\persistent/ring.obj ring.c creating build\lib.win32-2.6 creating build\lib.win32-2.6\persistent C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\Python26_32\libs /LIBPATH:c:\Python26_32\PCbuild /EXPORT:initcPersistence build\temp.win32-2.6\Release\persistent/cPersistence.obj build\temp.win32-2.6\Release\persistent/ring.obj /OUT:build\lib.win32-2.6\persistent\cPersistence.pyd /IMPLIB:build\temp.win32-2.6\Release\persistent\cPersistence.lib /MANIFESTFILE:build\temp.win32-2.6\Release\persistent\cPersistence.pyd.manifest Creating library build\temp.win32-2.6\Release\persistent\cPersistence.lib and object build\temp.win32-2.6\Release\persistent\cPersistence.exp C:\Program Files\Microsoft SDKs\Windows\v6.1\bin\mt.exe -nologo -manifest build\temp.win32-2.6\Release\persistent\cPersistence.pyd.manifest -outputresource:build\lib.win32-2.6\persistent\cPersistence.pyd;2 building 'persistent.cPickleCache' extension C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python26_32\include -Ic:\Python26_32\PC /Tcpersistent/cPickleCache.c /Fobuild\temp.win32-2.6\Release\persistent/cPickleCache.obj cPickleCache.c C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic:\Python26_32\include -Ic:\Python26_32\PC /Tcpersistent/ring.c /Fobuild\temp.win32-2.6\Release\persistent/ring.obj ring.c C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\Python26_32\libs /LIBPATH:c:\Python26_32\PCbuild /EXPORT:initcPickleCache build\temp.win32-2.6\Release\persistent/cPickleCache.obj build\temp.win32-2.6\Release\persistent/ring.obj
Re: [ZODB-Dev] Build compression into ZODB 3.11?
Hello, What about LZ4HC? http://code.google.com/p/lz4hc/ On Tue, 20 Mar 2012 10:27:26 -0400 you wrote: On Thu, Mar 15, 2012 at 11:09 AM, Jim Fultonj...@zope.com wrote: On Wed, Mar 14, 2012 at 1:47 PM, Jim Fultonj...@zope.com wrote: ... At some point soonish, I'll do some tests against a large database. On a database with 180 million records, 150 million of which are compressable: CompressedCompress Uncompress Size % of time time uncompressed microseconds microseconds --- zlib3896 12.5 lz4 527.4 1.6 --- lz4 is an order of magnitude faster than zlib, however, lz4-compressed records were 36% larger. For me, I don't think the speedup is worth the loss of compression. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev . -- Best regards, Adam GROSZER -- Quote of the day: Facts do not cease to exist because they are ignored. ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Build compression into ZODB 3.11?
Hello, On Wed, 14 Mar 2012 13:47:54 -0400 you wrote: I'm pretty happy with how zc.zlibstorage has worked out. Should I build this into ZODB 3.11? +1 BTW, lz4 compression looks interesting. indeed, looks bloody fast The Python binding (at least from PyPI) is broken. I submitted an issue. Hopefully it will be fixed. emmm, windows pains in sight? -- Best regards, Adam GROSZER -- Quote of the day: The truest test of independent judgment is being able to dislike someone who admires us, and to admire someone who dislikes us. - Sydney J. Harris ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] the difference between normal class object and the ZODB Persistent subclass object
Hello, Some links for you: http://www.ibm.com/developerworks/aix/library/au-zodb/ http://www.zodb.org/zodbbook/ http://pypi.python.org/pypi/zodbbrowser On Tue, 27 Sep 2011 18:16:08 +0800 you wrote: Hi. I Use eye to browse the ZODB database. There, the methods of the float object or any normal class objects are also shown. Are these methods stored in the database? Besides doing the _p_change magic, is there any extra advantage to write class derived from persistent.Persistent? Does ZODB store the persistent.Persistent subclass object more efficiently (much faster? smaller size?) Thanks -- /Dr. Yi-Xin Liu/ /Department of Macromolecular Science/ /Fudan University/ /Room 415, Yuejing Building / /Handan Rd. 220, //Shanghai, China/ /Tel +86-021-65642863/ /Mobile +86-13916819745/ http://www.mendeley.com/profiles/yi-xin-liu/ ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Best regards, Adam GROSZER -- Quote of the day: A city is a large community where people are lonesome together ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Delete an object in a IOBTree
Hello, On Fri, 05 Aug 2011 11:56:40 +0200 you wrote: Hello! I'm having a KeyReferenceToPersistent object in IOBTree.__getstate__() that I don't see in IOBTree.items(). Is there a way to remove it? In that KeyReferenceToPersistent I think there is an instance of a no longer existing class (broken). BTrees had a bug earlier that kept some already deleted key-values referenced. Worst case I'd build a new BTree: old = some.object.attribute new = old.__class__() for k,v in old.items(): new[k] = v some.object.attribute = new -- Best regards, Adam GROSZER -- Quote of the day: A ship on the beach is a lighthouse to the sea. - Dutch proverb ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] packing ZODB
Hello, After investigating FileStorage a bit, I found that GC runs on objects, but pack later by transactions. That means that if there's a bigger-ish transaction, we can't get rid of it until all of it's objects are GCed (or superseeded by newer states). Is that correct? I think yes, so an idea might be to split up transactions to one transaction per object state somehow and pack again. This would definitely work only offline of course. How is this handled by relstorage? -- Best regards, Adam GROSZER -- Quote of the day: The most beautiful thing we can experience is the mysterious. It is the source of all true art and science. - Albert Einstein ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packing ZODB
Hello, On Thu, 31 Mar 2011 06:33:30 -0600 you wrote: On 03/31/2011 05:25 AM, Hanno Schlichting wrote: Hi. On Thu, Mar 31, 2011 at 12:46 PM, Adam GROSZERagros...@gmail.com wrote: After investigating FileStorage a bit, I found that GC runs on objects, but pack later by transactions. That means that if there's a bigger-ish transaction, we can't get rid of it until all of it's objects are GCed (or superseeded by newer states). Is that correct? I think so. Actually we can remove pieces of the old transaction and that's the way both FileStorage and RelStorage already work. RelStorage owes much of its success to simply duplicating the functionality of FileStorage and ZEO. ;-) Yup, checked that and it works... You just got to get rid of all references ;-) -- Best regards, Adam GROSZER -- Quote of the day: The proud man counts his newspaper clippings- the humble man his blessings. - Bishop Fulton J. Sheen ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] savepoint not doing cacheGC?
Hello, I'm trying to iterate over the IntIds utility's object in a huge FileStorage DB, calling transaction.savepoint(optimistic=True) every 1000 iterations or so, but memory usage simply skyrockets. Weird is that almost the same code was working with python 2.5 and ZODB 3.9.6, now it does not with python 2.6 and ZODB 3.9.6. config is: cache-size 1 cache-size-bytes 50MB -- Best regards, Adam GROSZER -- Quote of the day: Patience is the companion of wisdom. - St. Augustine ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate install problem
Hello,
On Tue, 15 Feb 2011 18:27:46 +0100 you wrote:
On Tue, Feb 15, 2011 at 6:11 PM, Tres Seaver tsea...@palladion.com
mailto:tsea...@palladion.com wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 02/15/2011 12:06 PM, Tres Seaver wrote:
On 02/15/2011 11:37 AM, Chris Fanning wrote:
On Tue, Feb 15, 2011 at 5:16 PM, Adam GROSZER
agros...@gmail.com mailto:agros...@gmail.com wrote:
Hello,
You got to use python 2.6, at least for zodbupdate.
Thanks. I'd managed to get it going with 2.5 but I was getting this
warning..
Warning: Missing factory for Persistence PersistentMapping
Now, using python 2.6, I'm still getting the same warning.
Looks like I need to get this missing factory because the
resulting zodb.fs
still gives the same error
AttributeError: 'PersistentMapping' object has no attribute
'has_key'
How do I install/configure this?
You need to make the class importable at the old location in
order to
use instances of that class to convert to the new location.
Something
like::
import persistent.mapping
import sys
sys.modules['Persistence.PersistentMapping'] = \
sys.modules['persistent.mapping']
Then you should be able toload you old 'pages' object and create
a new
one (using the new class) and copy the contents into it from the
old one.
Actually, you can install the 'Persistence' package directly (it is
distributed separately from Zope):
http://pypi.python.org/pypi/Persistence
Thankyou both,
I've can load the database now.
Or use the rename rules feature of zodbupdate something like this:
renames = {'persistent.mapping PersistentMapping':
'Persistence PersistentMapping'}
--
Best regards,
Adam GROSZER
--
Quote of the day:
I intend to live forever. So far, so good
- Stephen Wright
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate install problem
Hello,
You got to use python 2.6, at least for zodbupdate.
On Tue, 15 Feb 2011 17:10:23 +0100 you wrote:
On Tue, Feb 15, 2011 at 5:03 PM, Chris Fanning
christopher.fann...@gmail.com mailto:christopher.fann...@gmail.com
wrote:
Hi,
I've been using zodb as the database on a homebrew cms since 2004
without one single problem. :)
Now we're changing hardware so we're upgrading software too.
I'm running into problems.
The client starts like this.
from Persistence import PersistentMapping
def __init__(self):
.
if not dbroot.has_key('pages'):
dbroot['pages'] = PersistentMapping()
So I changed the Persistence to
from persistent.mapping import PersistentMapping
And now I get the error
'PersistentMapping' object has no attribute 'has_key'
From what I understand the objects are Broken and I need to upgrade
the zodb.
So I did
#easy_install zodbupdate
However I get this error.
Running zodbupdate-0.5/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-iNI2JT/zodbupdate-0.5/egg-dist-tmp-AstNfF
File build/bdist.linux-x86_64/egg/zodbupdate/serialize.py, line 165
symb = find_global(*symb_info, Broken=ZODBBroken)
^
SyntaxError: invalid syntax
I'm at a real loss here.
Any help please?
Chris.
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Emulating site import/export or walking through all site objects?
Hello Mikko, Friday, November 5, 2010, 10:11:43 AM, you wrote: MO Hi, MO I would like to create an automatic test to check that Plone add-ons MO install and uninstall cleanly. There has been problems that after the MO add-on has been uninstalled, there are persistent objects based on the MO add-on packages left around. If this happens you cannot import or MO export the site anymore. MO I would like to walk through all persistent objects contained inside a MO site, or emulate import/export functionality to see if this happens or MO not. Since I cannot actually unload Python eggs from the memory easily MO (is this assumption correct?) I can just check pickled objects by MO their dotted name and see if they are in blacklisted packages. MO Are there any ready Python recipes or example code where I should start? MO The question is also open here: MO http://plone.293351.n2.nabble.com/Automatizing-uninstalls-cleanly-test-tp5705298p5705298.html zodbupdate comes to my mind. http://pypi.python.org/pypi/zodbupdate/0.5 It will find ALL objects which can't be loaded. -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: Art is anything you can get away with. -- Marshall McLuhan. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] weird error while packing
Hello, This exception came up while packing a filestorage: 2010-08-29T05:05:23 ERROR ZODB.DB packing Traceback (most recent call last): File D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\DB.py, line 826, in pack self.storage.pack(t, self.references) File D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\FileStorage.py, line 1126, in pack pack_result = self.packer(self, referencesf, stop, gc) File D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\FileStorage.py, line 1079, in packer opos = p.pack() File D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\fspack.py, line 398, in pack self.gc.findReachable() File D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\fspack.py, line 191, in findReachable self.findReachableAtPacktime([z64]) File D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\fspack.py, line 276, in findReachableAtPacktime for oid in self.findrefs(pos): SystemExit Any pointers appreciated. -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: When we walk to the edge of all the light we have and take the step into the darkness of the unknown, we must believe that one of two things will happen. There will be something solid for us to stand on or we will be taught to fly. - Patrick Overton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] buildbot for ZODB on windows
Hello, There's a buildbot for ZODB for various windows platforms at http://winbot.zope.org -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: Pain can be a positive thing ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Book
Hello Alan, How to sign up as a volunteer? Tuesday, June 15, 2010, 10:23:35 PM, you wrote: AR Hello ZODB Community, AR I wanted to announce a community-led initiative to create a ZODB book. AR The goal is to solicit contributions from the community and when we AR reach a threshold, Carlos de la Guardia will begin work on writing the AR book. AR I have started a blog, http://zodbdocs.blogspot.com/ AR The blog contains the contribution levels and the proposed outline. AR Any feedback is appreciated. AR If you are interested in contributing financially please subscribe to AR the email list on the blog. AR I am working on getting a paypal account. I will keep the blog up to AR date and will ping the mailing list when we start accepting contributions. AR cheers AR alan runyan AR ___ AR For more information about ZODB, see the ZODB Wiki: AR http://www.zope.org/Wikis/ZODB/ AR ZODB-Dev mailing list - ZODB-Dev@zope.org AR https://mail.zope.org/mailman/listinfo/zodb-dev -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: People will buy anything that's one to a customer. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] My bugday
Hello, My only task was to setup the zope.wineggbuilder on the server with a buildbot. It is here: http://173.203.65.130/ will be http://winbot.zope.org once DNS entries propagate For this night it will do a testing round with zope.i18nmessageid, for tomorrow night I'm going to switch it to full mode if nothing big happens. -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: Desire joy and thank God for it. Renounce it, if need be, for other's sake. That's joy beyond joy. - Robert Browning ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Some interesting (to some:) numbers
Hello Jim, Tuesday, May 11, 2010, 8:36:46 PM, you wrote: JF On Sun, May 9, 2010 at 4:59 PM, Roel Bruggink r...@fourdigits.nl wrote: On Sun, May 9, 2010 at 8:33 PM, Jim Fulton j...@zope.com wrote: Our recent discussion of compression made me curious so I did some analysis of pickle sizes in one of our large databases. This is for a content management system. The database is packed weekly. It doesn't include media, which are in blobs. There were ~19 million transaction in the database and around 130 million data records. About 60% of the size was taken up by BTrees. Compressing pickles using zlib with default compression reduced the pickle sizes by ~58%. The average uncompressed record size was 1163 bytes. The average compressed size was ~493 bytes. This is probably enough of a savings to make compression interesting. JF ... That's really interesting! Did you notice any issues performance wise, or didn't you check that yet? JF OK, I did some crude tests. It looks like compressing is a little JF less expensive than pickling and decompressing is a little more JF expensive than unpickling, which is to say this is pretty cheap. For JF example, decompressing a data record took around 20 microseconds on my JF machine. A typical ZEO load takes 10s of milliseconds. Even in Shane's JF zodb shootout benchmark which loads data from ram, load times are JF several hundred microseconds or more. JF I don't think compression will hurt performance. It is likeley to JF help it in practice because: JF - There will be less data to send back and forth to remote servers. JF - Smaller databases will get more benefit from disk caches. JF (Databases will be more likely to fit on ssds.) JF - ZEO caches (and relstorage memcached caches) will be able to hold JF more object records. I was thinking about using other compressors. I found this: http://tukaani.org/lzma/benchmarks.html Seems like gzip/zlib is the fastest with some expense of efficiency. -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: What this country needs is a good five-cent microcomputer. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Some interesting (to some:) numbers
Hello Jim, Monday, May 10, 2010, 1:27:00 PM, you wrote: JF On Sun, May 9, 2010 at 4:59 PM, Roel Bruggink r...@fourdigits.nl wrote: That's really interesting! Did you notice any issues performance wise, or didn't you check that yet? JF I didn't check performance. I just iterated over a file storage file, JF checking compressed and uncompressed pickle sizes. I'd say some checksum is then also needed to detect bit failures that mess up the compressed data. -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: A true friend is someone who is there for you when he'd rather be anywhere else. - Len Wein ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Some interesting (to some:) numbers
Hello Jim, Tuesday, May 11, 2010, 12:33:04 PM, you wrote: JF On Tue, May 11, 2010 at 3:16 AM, Adam GROSZER agros...@gmail.com wrote: Hello Jim, Monday, May 10, 2010, 1:27:00 PM, you wrote: JF On Sun, May 9, 2010 at 4:59 PM, Roel Bruggink r...@fourdigits.nl wrote: That's really interesting! Did you notice any issues performance wise, or didn't you check that yet? JF I didn't check performance. I just iterated over a file storage file, JF checking compressed and uncompressed pickle sizes. I'd say some checksum is then also needed to detect bit failures that mess up the compressed data. JF Why? I think the gzip algo compresses to a bit-stream, where even one bit has an error the rest of the uncompressed data might be a total mess. If that one bit is relatively early in the stream it's fatal. Salvaging the data is not a joy either. I know at this level we should expect that the OS and any underlying infrastructure should provide error-free data or fail. Tho I've seen some magic situations where the file copied without error through a network, but at the end CRC check failed on it :-O -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: You may have to fight a battle more than once to win it. - Margaret Thatcher ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Some interesting (to some:) numbers
Hello, Tuesday, May 11, 2010, 1:59:17 PM, you wrote: N Am 11.05.2010, 13:47 Uhr, schrieb Adam GROSZER agros...@gmail.com: Hello Jim, Tuesday, May 11, 2010, 1:37:19 PM, you wrote: JF On Tue, May 11, 2010 at 7:13 AM, Adam GROSZER agros...@gmail.com wrote: Hello Jim, Tuesday, May 11, 2010, 12:33:04 PM, you wrote: JF On Tue, May 11, 2010 at 3:16 AM, Adam GROSZER agros...@gmail.com wrote: Hello Jim, Monday, May 10, 2010, 1:27:00 PM, you wrote: JF On Sun, May 9, 2010 at 4:59 PM, Roel Bruggink r...@fourdigits.nl wrote: That's really interesting! Did you notice any issues performance wise, or didn't you check that yet? JF I didn't check performance. I just iterated over a file storage file, JF checking compressed and uncompressed pickle sizes. I'd say some checksum is then also needed to detect bit failures that mess up the compressed data. JF Why? I think the gzip algo compresses to a bit-stream, where even one bit has an error the rest of the uncompressed data might be a total mess. If that one bit is relatively early in the stream it's fatal. Salvaging the data is not a joy either. I know at this level we should expect that the OS and any underlying infrastructure should provide error-free data or fail. Tho I've seen some magic situations where the file copied without error through a network, but at the end CRC check failed on it :-O JF How would a checksum help? All it would do is tell you your hosed. JF It wouldn't make you any less hosed. Yes, but I would know why it's hosed. Not like I'm expecting 2+2=4 and get 5 somewhere deep in the custom app that does some calculation. N You could have bitflips anywhere in the database, not just the payload N parts. You'd have to checksum and test everything all the time. Imo it's N not worth the complexity and performance penalty given today's redundant N storages like RAID, ZRS or zeoraid. N Btw, the current pickle payload format is not secured against any bitflips N either I think. The difference between the uncompressed and compressed is that if you have bitflips in an uncompressed data stream then you get let's say a B instead of A, or 3 instead of 1. That hits hard in numbers/IDs, but keeps string still human readable. Because the rest is still there. In a compressed stream the rest of the pickle/payload would be probably crap. Probably that crappy data would make the unpickler fail... or wait a second... the unpickler is a **SECURITY HOLE** in python, isn't it? That means feed it some random data... and stay tuned for the unexpected. The thing is that a single bitflip could cause a LOT of crap. You're right that currently there's no protection against such bitflips, but I'd rather present the user a nice error than some crappy data. -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: If necessity is the mother of invention, discontent is the father of progress. - David Rockefeller ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Some interesting (to some:) numbers
Hello Jim, Tuesday, May 11, 2010, 4:46:46 PM, you wrote: JF On Tue, May 11, 2010 at 7:47 AM, Adam GROSZER agros...@gmail.com wrote: Hello Jim, Tuesday, May 11, 2010, 1:37:19 PM, you wrote: JF On Tue, May 11, 2010 at 7:13 AM, Adam GROSZER agros...@gmail.com wrote: Hello Jim, Tuesday, May 11, 2010, 12:33:04 PM, you wrote: JF On Tue, May 11, 2010 at 3:16 AM, Adam GROSZER agros...@gmail.com wrote: Hello Jim, Monday, May 10, 2010, 1:27:00 PM, you wrote: JF On Sun, May 9, 2010 at 4:59 PM, Roel Bruggink r...@fourdigits.nl wrote: That's really interesting! Did you notice any issues performance wise, or didn't you check that yet? JF I didn't check performance. I just iterated over a file storage file, JF checking compressed and uncompressed pickle sizes. I'd say some checksum is then also needed to detect bit failures that mess up the compressed data. JF Why? I think the gzip algo compresses to a bit-stream, where even one bit has an error the rest of the uncompressed data might be a total mess. If that one bit is relatively early in the stream it's fatal. Salvaging the data is not a joy either. I know at this level we should expect that the OS and any underlying infrastructure should provide error-free data or fail. Tho I've seen some magic situations where the file copied without error through a network, but at the end CRC check failed on it :-O JF How would a checksum help? All it would do is tell you your hosed. JF It wouldn't make you any less hosed. Yes, but I would know why it's hosed. JF How so? How would you know why it is hosed. Because of data corruption in the compressed stream. JF Note BTW that the zlib format already includes a checksum. JF http://www.faqs.org/rfcs/rfc1950.html I missed that. Case closed then ;-) Sorry for the noise. -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: Some are atheists by neglect; others are so by affectation; they that think there is no God at some times do not think so at all times. - Benjamin Whichcote ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
Hello Christian, Wednesday, April 14, 2010, 8:30:50 AM, you wrote: CT I don't think the transfer rate is actually that interesting. For small CT but many transactions the seek time/spinning speed should have the CT limiting influence. CT I've run the attached script a couple of times on my notebook, here's CT the results: CT 0.11 909.090909091 CT 0.15 666.7 CT 0.2 500.0 CT 0.07 1428.57142857 CT 0.07 1428.57142857 CT 0.14 714.285714286 CT The initial runs are a bit lower as they were interfered with by other CT applications writing to the disk. CT It's a notebook w/ Intel P9600, Seagate 7.2k SATA drive, 4GB RAM, Ubuntu CT 10.04, linux 2.6.32, ext4 Something is wrong with the seek time. Trying it on an Intel G1 80G SSD goes hardly over a 1000. NB, P7400 3GB RAM, ubuntu 9.10, ext4 0.14 714.285714286 0.14 714.285714286 0.1 1000.0 0.14 714.285714286 Just for fun, it's flying on a tmpfs 0.36 2777.7778 0.37 2702.7027027 0.37 2702.7027027 Vmware makes it fly also: with 1000 runs on a 4GHz E8400, ubuntu 9.10, ext4 within VMware 0.24 4166.6667 0.33 3030.3030303 0.26 3846.15384615 0.31 3225.80645161 0.29 3448.27586207 in the VM tmpfs does not boost a lot: 0.2 5000.0 0.2 5000.0 0.2 5000.0 0.19 5263.15789474 0.19 5263.15789474 -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: People seem to enjoy things more when they know a lot of other people have been left out of the pleasure. - Russell Baker ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Checking the length of OOBTree
Hello, And have a look at http://svn.zope.org/zope.container/trunk/src/zope/container/btree.py?rev=107468view=markup for usage. self.__len = Length() Thursday, April 8, 2010, 5:41:04 PM, you wrote: LR A BTree does not keep track of it's length. See BTrees.Length.Length: LR http://apidoc.zope.org/++apidoc++/Code/BTrees/Length/Length/index.html LR Laurence LR On 8 April 2010 16:36, Leszek Syroka LR leszek.marek.syr...@cern.ch wrote: Hi, what is the fastest way of checking the number of elements in OOBtree. Execution time of len( OOBtree.keys() ) and len(OOBtree) is exactly the same. For big data sets execution time is unacceptable. I found out that in the implementation of OOBtree (written in C) there is a variable called 'len', which seems to contain the length of the tree. Is it possible to access that variable from the python code without modifying the source? Best regards Leszek ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - zodb-...@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev LR ___ LR For more information about ZODB, see the ZODB Wiki: LR http://www.zope.org/Wikis/ZODB/ LR ZODB-Dev mailing list - ZODB-Dev@zope.org LR https://mail.zope.org/mailman/listinfo/zodb-dev -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: The journey of a thousand miles begins with one step. - Lao-Tsze ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] cache-size-bytes problem
Hello, So far I know Hanno fixed that. Any chance to get it released as 3.9.5? -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: If you don't care where you are, then you ain't lost. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Understanding the ZODB cache-size option
Hello, On top of that, be aware that those numbers 'cache-size' (or cache-size-bytes) are for each connection. By default a Z3 ZODB can have up to 7 connections in the pool. That means you might easily end with 7 times the cache size set. Most people tend to limit the number of connections too. Monday, March 22, 2010, 8:01:30 PM, you wrote: HS On Mon, Mar 22, 2010 at 7:05 PM, Jeff Shell j...@bottlerocket.net wrote: Are there any metrics about how to set the ZODB 'cache-size' (or cache-size-bytes) option? We've been using '5000' (arbitrarily chosen) for our Zope 3.4-ish app servers. We have access to zc.z3monitor which can output the number of objects in the object caches (combined) and number of non-ghost objects in the object caches (combined). But I don't know understand how to interpret those numbers and use them to make better settings. HS ZODB cache size optimization is a typical case of black art. There's HS no real good way to find the perfect number. I would advise against HS using the cache-size-bytes option. There's a known critical problem HS with it, that is currently only fixed on the 3.9 SVN branch. So stick HS to the old object count cache-size for now. HS But generally database caching is a trade-off between performance and HS available RAM. As the upper limit, you could have your entire data set HS in each ZODB cache. So you could look at the number of persistent HS objects in the database and match your cache-size to that number. HS That's usually not want you want. HS As a real strategy, you should set up detailed monitoring of the HS server. Monitor and graph overall RAM usage, RAM usage per Zope HS process, number of DB loads and writes over time. Preferably include HS some way of measuring application request performance and track CPU HS and I/O usage on the server hosting the database. HS If you have those numbers, you can play around with the cache setting HS and increase it. See what impact it has on your application and data HS set. At some point you run out of memory and need to decrease the HS number or the increased cache size doesn't actually buy you any HS application performance anymore. HS For general Zope 3 application there are no rules of thumb that I HS know. The dataset and load patterns of the applications are too HS different to have any of those. It's affected a lot by the dataset and HS if you use ZODB blobs or another mechanism to store large binary HS content outside the DB. 5000 persistent objects including 1000 images HS of 5mb each are obviously very different, than 1000 BTree buckets HS containing only integers. One main advantage of blobs is that they HS aren't loaded into the ZODB connection caches, so they lower the HS memory requirements for applications with binary content a lot. HS In the Plone context I generally use something like number of content HS objects in the catalog + 5000 objects for the general application as HS a starting point. But that has a lot of assumptions on the type of HS content and the application in it. HS Hanno HS ___ HS For more information about ZODB, see the ZODB Wiki: HS http://www.zope.org/Wikis/ZODB/ HS ZODB-Dev mailing list - ZODB-Dev@zope.org HS https://mail.zope.org/mailman/listinfo/zodb-dev -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: You can't cross a chasm in two small jumps. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] redirect burps on unicode URLs
Hello,
Looks like zope.publisher burps on unicode URL which contain non-ascii
chars. This is from a KGS 3.4 application, but looking at the source
it still seems to have the same problems.
opinions?
...
self.request.response.redirect(url)
File
d:\home\.buildout\eggs\zope.publisher-3.4.6-py2.5.egg\zope\publisher\browser.py,
line
729, in redirect
return super(BrowserResponse, self).redirect(location, status)
File
d:\home\.buildout\eggs\zope.publisher-3.4.6-py2.5.egg\zope\publisher\http.py,
line 882,
in redirect
self.setHeader('Location', location)
File
d:\home\.buildout\eggs\zope.publisher-3.4.6-py2.5.egg\zope\publisher\http.py,
line 676,
in setHeader
value = str(value)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xd6' in position
71: ordinal not in
range(128)
--
Best regards,
Adam GROSZER mailto:agros...@gmail.com
--
Quote of the day:
Exasperation is the mind's way of spinning its wheels until patience restores
traction.
- George L. Griggs
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConnectionStateError from ZODB.Connection
Hello, Yah, eg. hooks.setSite(my-persistent-site) and no setSite(None). Tuesday, February 2, 2010, 2:52:31 PM, you wrote: JF On Tue, Feb 2, 2010 at 2:23 AM, Baiju M mba...@zeomega.com wrote: Hi, I am getting an error like this: Module ZODB.Connection, line 808, in setstate ConnectionStateError: Shouldn't load state for 0x0ba91e when the connection is closed Few details from ZMI: Zope Version (Zope 2.11.3-final, python 2.4.6, linux2) Python Version 2.4.6 (#1, Jul 8 2009, 10:09:58) [GCC 3.4.6 20060404 (Red Hat 3.4.6-10)] System Platform linux2 Network Services ZServer.HTTPServer.zhttp_server (Port: 6002) I can see recent commit (r106543) by Jim, is this related ? - A ZEO threading bug could cause transactions to read inconsistent data. (This sometimes caused an AssertionError in Connection._setstate_noncurrent.) JF No. Most likely, You're storing a persistent object in a global JF variable and accessing it after JF its connection has been closed. JF Jim -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: There is often less danger in the things we fear than in the things we desire. - John C. Collins ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] how to estimate cache size
Hello, There are some methods in PickleCache That might be good for object count: * __len__ * ringlen which might be better to get the number of objects in the cache? Is there any chance to estimate the current cache size in bytes? Also I guess I need to add all connection's picklecache's object count to get the total of objects in the cache, right? -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: Instead of crying over spilt milk, go milk another cow. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] how to estimate cache size
Yay! The answer to my object count question lies in zc.z3monitor. But what about cache size in bytes? Friday, January 15, 2010, 1:39:02 PM, you wrote: AG Hello, AG There are some methods in PickleCache AG That might be good for object count: AG * __len__ AG * ringlen AG which might be better to get the number of objects in the cache? AG Is there any chance to estimate the current cache size in bytes? AG Also I guess I need to add all connection's picklecache's object count AG to get the total of objects in the cache, right? -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: Swipple's Rule of Order: He who shouts the loudest has the floor. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] relstorage - memcached - noSQL(?)
Hello Shane, This is some tinkering about whether/how it is possible to make a ZODB storage that uses a simple kev-value database (KVDB) (e.g. tokyo tyrant). A KVDB that is persistent on disk or even replicated should not be less fault tolerant as MySQL. As I see you got the caching right with memcached in relstorage. That is also a step further on this direction. Problems that I see: - read/write separation: as the KVDB does not have transactions, reads should not overlap with writes. That should easily be possible with locking. (Yah, that does not make it so shiny anymore.) Either DB global or fine-grained on the oid level. That would maybe require a separate service that does the locking? - loadBefore: it's a good question how to solve that easily/speedy without lots of requests to the KVDB with the KVDB's reduced set of features. - performance: my fear is that getting this right could be slower as relstorage/ZEO. Because of the mismatches between the requirements of a ZODB storage and the features of the simple KVDB. There is also an S3 storage out there, by Laurence: http://code.google.com/p/s3storage/ Jim is working on zc.bsddbstorage, where BerkeleyDB should be (sofar I see -- but never used it) also a (not that minimal) KVDB. I'd like to personally chase these ideas, but I don't think I'll get some time soon. Also I guess you have a deeper knowledge about storages. -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: Baseball is a skilled game. It's America's game -- it, and high taxes. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] relstorage - memcached - noSQL(?)
Hey Shane, Well, first feasibility. But I think it's doable. The question is how much cruft it will require. (I think on the high level a KVDB is a perfect match for a ZODB backend. oid - pickle is the mapping. If it's oid_tid - pickle it's still not bad.) So if the story about a KVDB being superfast is true, that should bring some speed. I'd start with a single process multithreaded scaffolded something with locally implemented locking, that should be rather easy to build together from the various storages. But still make sure with some dumb requests that the locking and other left out features take their own time and benchmark this against the other storages out there. Then we'll see whether it's still worth chasing this. - Reliability might be the next thing as (most) KVDBs seem to support (some sort of) replication. - Scalability is a tough question. First problem is the commit lock. And the CAP theorem - I think a ZODB backend cannot get by without all 3 properties. But about the latter two I did not think yet in depth and there are just too many KVDBs out there to explore. On Wed, Nov 25, 2009 at 2:29 PM, Shane Hathaway sh...@hathawaymix.org wrote: Adam GROSZER wrote: This is some tinkering about whether/how it is possible to make a ZODB storage that uses a simple kev-value database (KVDB) (e.g. tokyo tyrant). A KVDB that is persistent on disk or even replicated should not be less fault tolerant as MySQL. I don't quite understand. What do you hope to accomplish with Tokyo Tyrant or some other key/value store as a ZODB backend? Speed? Reliability? Scalability? I'd like to personally chase these ideas, but I don't think I'll get some time soon. Also I guess you have a deeper knowledge about storages. Well, I like to hear ideas, but I can't chase them all either. Shane -- Best regards, Adam ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [OT] NoSQL
Hello, I think we can look at this at 2 levels. 1.: As your app uses ZODB. Then this is your app's problem/reponsibility. You use a nosql contender directly from your app and it's your responsibility to deal with it. 2.: On the ZODB Storage level. So far I can see that level needs consistency, transactions and locking support. Those are usually missing from nosql implementations (unless I miss some). OTOH a key-value store would fit the ZODB storage. If someone finds/writes a key-value storage that has the above properties we could give it a try. Thursday, November 12, 2009, 8:24:43 PM, you wrote: SH Encolpe Degoute wrote: Is there someone in the ZODB development team following this: http://www.rackspacecloud.com/blog/2009/11/09/nosql-ecosystem/ SH It is possible that ZODB unfortunately occupies the same space as SQL in SH the CAP triangle: SH http://camelcase.blogspot.com/2007/08/cap-theorem.html SH That is to say, ZODB applications require consistency and availability, SH so if the CAP theorem is true, then ZODB applications can not be very SH partition-tolerant. SH The NoSQL databases provide availability and partition tolerance while SH foregoing absolute consistency. SH Shane SH ___ SH For more information about ZODB, see the ZODB Wiki: SH http://www.zope.org/Wikis/ZODB/ SH ZODB-Dev mailing list - ZODB-Dev@zope.org SH https://mail.zope.org/mailman/listinfo/zodb-dev -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: For a good time, call 836-3100. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] copy on write
Hi Jim, Have you seen Malthe's latest package that tries to implement a persistency with copy on write? ( http://svn.repoze.org/dobbin/trunk ) Do you see any chance to do the same with the good old ZODB? -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: It is a great mistake to suppose that God is only, or even chiefly, concerned with religion. - William Temple, Archbishop of Canterbury ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] CRC check failed when running repozo
Hello, +1 on that. Sometimes copy operations (over the network) screw up some bits. Happens rarely but then it hits hard. Monday, September 14, 2009, 10:13:39 AM, you wrote: CT Hmm. Haven't seen that. My guess would be some low-level data corruption. CT Christian -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: People are boring unless they're extremists ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Hello Jim, Where's that certain size on the scale? Tuesday, May 26, 2009, 3:35:56 PM, you wrote: JF FileStorage indexes can't be saved after they reach a certain size, JF where size roughly based on the number of objects. -- Best regards, Adam GROSZERmailto:agros...@gmail.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB browse
Hello Anurag, This one might be some help: svn://svn.zope.org/repos/main/z3c.zodbbrowser/trunk Saturday, April 25, 2009, 12:14:17 PM, you wrote: UAGH We are using ZODB [in Zenoss] and we would like to browse UAGH Zenoss' ZODB schema. How can we do this? Also, how can we create UAGH our own tables, etc? Any links, documentation? Thanks! UAGH ___ UAGH For more information about ZODB, see the ZODB Wiki: UAGH http://www.zope.org/Wikis/ZODB/ UAGH ZODB-Dev mailing list - ZODB-Dev@zope.org UAGH http://mail.zope.org/mailman/listinfo/zodb-dev -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: Every time I think I know where it's at, they move it. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] error installing ZODB 3.9.0a12 with Python 2.6 on Windows
Hello Chris, You have a c compiler working on the machine? Friday, April 3, 2009, 1:48:39 PM, you wrote: CW Wichert Akkerman wrote: Any ideas? buildout hides all compile errors unless you run it with -v (or -vv). CW buildout -vv didn't exactly provide much more help ;-) CW C:\Python26\python.exe -c from setuptools.command.easy_install CW import main; CW main() -mUNxd C:\buildout-eggs\tmpctofbd CW c:\docume~1\chris\locals~1\te CW mp\tmpiks1grget_dist\ZODB3-3.9.0a11.tar.gz CW path=c:\buildout-eggs\setuptools-0.6c9-py2.6.egg CW Processing ZODB3-3.9.0a11.tar.gz CW Running ZODB3-3.9.0a11\setup.py -q bdist_egg --dist-dir CW c:\docume~1\chris\locals CW ~1\temp\easy_install-7amb3r\ZODB3-3.9.0a11\egg-dist-tmp-4eufhq CW error: Setup script exited with error: None CW An error occured when trying to install ZODB3 3.9.0a11.Look above this CW message for any errors thatwere output by easy_install. CW While: CWInstalling zeoinstance. CWGetting distribution for 'ZODB3==3.9.0a11'. CW Error: Couldn't install: ZODB3 3.9.0a11 CW Is returning None from bdist_egg an error? If so, what does it mean? CW Chris -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: They that know God will be humble; they that know themselves cannot be proud. - John Flavel ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] create unique container keys
Hello, What's a good practice to create unique container keys for a heavily loaded application? (I mean lots of writes to the same container) Obviously having a counter on the container and incrementing and using that for key gives write conflicts. What do the experts use? thanks -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: And the number one thing you'll never hear in an internet chat room: 1. Where have I been lately? Hey, some of us have a life, ya know! ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] _p_oid
Hello, Does anyone know how _p_oid is generated? I'm asking because we would like to use _p_oid as a unique key to store BLOB-like files on the filesystem. Of course more threads/processes/machines are in the picture. So the question is once an object gets a _p_oid assigned, is it guaranteed to be unique over all instances accessing the same DB? thanks -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: It is the province of knowledge to speak, and its a privilege of wisdom to listen. - Oliver Wendell Homes ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] example for the following?
Hello Jamie, I would say to persist that object either you rewrite it in python, (and use the persistent module) or fiddle around the __getstate__ and __setstate__ methods. Though no idea how you can get those working with your swig'ed objects. See: http://docs.python.org/library/pickle.html Thursday, January 22, 2009, 5:46:08 PM, you wrote: JM Hi, JM I'm using ZODB in a desktop app (i.e. not using zope). JM I have class A which contains a list of instances for class B. JM I can persist class A when the list of class B is empty. When i JM add an instance of class B to the list (and set _p_changed = 1) i get the error: JM TypeError: can't pickle PySwigObject objects JM Can someone please give me a link of some advice in how i go JM about persisting lists of class instances? JM Thanks, JM Jamie JM JM ~~~ JM Jamie McQuay JM Scimatic Software Inc. JM www.scimatic.com JM JM We build software for scientists. JM -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: You will feel hungry again in another hour. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with broken
Hello, Thanks for the advices on the generation script, but I think the problem lies in that the broken object is loaded fine but cannot be written back again to ZODB. Or do I misunderstand that broken is a sort of proxy for the missing class that allows it to be around? Wednesday, November 5, 2008, 11:31:21 AM, you wrote: LR Broken objects occur when the class for a pickled object cannot be LR imported. To change the location of a class, you need to provide an LR alias at the old location so that the object can be unpickled, i.e. LR MyOldClassName = MyNewClassName. You can only remove MyOldClassName LR after you have updated all of the pickles (with your code below). LR Laurence LR Adam GROSZER wrote: Hello, I'm having a problem with broken objects here. It's coming when I'm trying to evolve generations and the generation just touches all objects in the ZODB to store them again with the non-deprecated classes. The code is like this: storage = context.connection._storage next_oid = None n = 0 while True: oid, tid, data, next_oid = storage.record_iternext(next_oid) obj = context.connection.get(oid) # Make sure that we tell all objects that they have been changed. Who # cares whether it is true! :-) obj._p_activate() obj._p_changed = True if next_oid is None: break 2008-11-04T19:40:16 ERROR SiteError http://localhost:8080/++etc++process/@@generations.html Traceback (most recent call last): File F:\W\Zope3\src\zope\publisher\publish.py, line 133, in publish result = publication.callObject(request, obj) ... File F:\W\Zope3\src\zope\tal\talinterpreter.py, line 343, in interpret handlers[opcode](self, args) File F:\W\Zope3\src\zope\tal\talinterpreter.py, line 583, in do_setLocal_tal self.engine.setLocal(name, self.engine.evaluateValue(expr)) File F:\W\Zope3\src\zope\tales\tales.py, line 696, in evaluate return expression(self) File F:\W\Zope3\src\zope\tales\expressions.py, line 217, in __call__ return self._eval(econtext) File F:\W\Zope3\src\zope\tales\expressions.py, line 211, in _eval return ob() File F:\W\Zope3\src\zope\app\generations\browser\managers.py, line 182, in evolve transaction.commit() File F:\W\Zope3\src\transaction\_manager.py, line 93, in commit return self.get().commit() File F:\W\Zope3\src\transaction\_transaction.py, line 322, in commit self._commitResources() File F:\W\Zope3\src\transaction\_transaction.py, line 416, in _commitResources rm.commit(self) File F:\W\Zope3\src\ZODB\Connection.py, line 541, in commit self._commit(transaction) File F:\W\Zope3\src\ZODB\Connection.py, line 586, in _commit self._store_objects(ObjectWriter(obj), transaction) File F:\W\Zope3\src\ZODB\Connection.py, line 620, in _store_objects p = writer.serialize(obj) # This calls __getstate__ of obj File F:\W\Zope3\src\ZODB\serialize.py, line 405, in serialize meta = klass, newargs() File F:\W\Zope3\src\ZODB\broken.py, line 325, in __getnewargs__ return self.__Broken_newargs__ AttributeError: 'VocabularyManager' object has no attribute '__Broken_newargs__' LR ___ LR For more information about ZODB, see the ZODB Wiki: LR http://www.zope.org/Wikis/ZODB/ LR ZODB-Dev mailing list - ZODB-Dev@zope.org LR http://mail.zope.org/mailman/listinfo/zodb-dev -- Best regards, Adam GROSZERmailto:[EMAIL PROTECTED] -- Quote of the day: Every absurdity has a champion who will defend it. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: Hooking persistent.Persistent.__setstate__ was Re: [ZODB-Dev] Analyzing a ZODB.
Hello Alan, Sunday, April 6, 2008, 12:30:21 AM, you wrote: AR Question: Is it possible for ZODB 3.9 to have a pure python AR implementation of persistent.Persistent? Maybe this would be a good AR ZODB GSOC project? That would give some help to the Zope 3 components on Jython project too ;-) Maybe you should post that on [EMAIL PROTECTED] btw, there is also a proposal on the GSoC list to further elaborate the zodbbrowser. svn://svn.zope.org/repos/main/z3c.zodbbrowser/sandbox/src/z3c/zodbbrowser Any ideas welcome. -- Best regards, Adam GROSZERmailto:[EMAIL PROTECTED] -- Quote of the day: The price of success is perseverance. The price of failure comes cheaper. (Anonymous) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
Hello, Somehow relevant to the subject I just found an article on Wickert's site: http://www.wiggy.net/ , Using a seperate Data.fs for the catalog -- Best regards, Groszer Adam -- Quote of the day: Keep your fears for yourself, but share your courage with others. - Robert Louis Stevenson ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Community opinion about search+filter
Hello, I'd like to ask your opinion, your experiences about searching and filtering in quite large object DBs. We need to add search and filter functions to our current app, where the user might be able to create quite _sophisticated_ filter criterias. (The app is a pure Z3 app, subject is document management) Currently we're looking at something based on catalog/indexes. As I checked the most comfortable solution would be based on hurry.query. Some questions arose: - Is it necessary/worth adding indexes on all attributes? - How does the index perform on modification and retrieval? The biggest problem is that this will be our first try, so we're missing experiences and are a bit puzzled about the right solution. Certain is that moving to RDB is not an option. Thanks, -- Best regards, Adam mailto:[EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] UnpickleableError
Hello, Just run into a usual Cannot pickle type 'zope.security._proxy._Proxy' objects exception. What about doing the following patch, that might help a lot determining what cannot be pickled? Index: serialize.py === --- serialize.py(revision 71248) +++ serialize.py(working copy) @@ -413,7 +413,13 @@ self._file.seek(0) self._p.clear_memo() self._p.dump(classmeta) -self._p.dump(state) +try: +self._p.dump(state) +except Exception, msg: +log = logging.getLogger(ZODB.serialize) +log.exception(Pickling error: %s, classmeta: %s, str(msg), str(classmeta)) +raise + self._file.truncate() return self._file.getvalue() -- Best regards, Groszer Adam -- Quote of the day: Never close your lips to those to whom you have opened your heart. - Charles Dickens ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[2]: [ZODB-Dev] UnpickleableError
Hello Chris, Actually that message is not quite helpful when trying to locate the code which put that property there. There is almost zero context that helps debugging. The traceback is worthless, because this happens in the commit. What helps at least for me a _lot_ is the parent object's class and the parent object state. This data is present in the method just needs to be printed. The printing/logging method can be negotiated I found this way of logging some lines below. Thursday, March 8, 2007, 5:09:29 PM, you wrote: CW Adam Groszer wrote: Hello, Just run into a usual Cannot pickle type 'zope.security._proxy._Proxy' objects exception. CW What does your patch give you that this error message doesn't? +try: +self._p.dump(state) +except Exception, msg: CW it's logger msg, it's the exception object being caught. +log = logging.getLogger(ZODB.serialize) CW Why get the logger here? get it in some global context. CW Chris -- Best regards, Groszer Adam -- Quote of the day: You auto buy now. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[4]: [ZODB-Dev] ZODB load/save tracing
Hello Win, I think you'll have to write a script. Have a look at http://svn.zope.org/z3c.zodbbrowser/sandbox/src/z3c/zodbbrowser/ From plugin_fs.py you can extract how to open the ZODB without zope. You can then traverse (access the properties) of the root and object thereunder. Matching an OODB with an RDB might not be a simple task as the OODB is a tree or a graph of objects. As Lennart already said on the list: But the best way to export data from ZODB is to write a script which outputs CSV values, which you then can import to MySQL. (Note that ZODB is not a type of SQL database but an object oriented database, and that a generic export therefore is not possible, you must write a dedicated script for your specific data set. This however is usually quite easy). Tuesday, March 6, 2007, 4:08:45 AM, you wrote: WMA Hello Adam, WMA I would like to request you to help me if possible. I am tryiny to WMA export contents (text data) of objects in ZODB. But I am a newbie and WMA no idea to do that. Could you please show some ways with exact WMA instruction? I am a bit hurry to do that. WMA Thank you for your attention and time. WMA Kind regards, WMA Win Myint Aung WMA Quoting Adam Groszer [EMAIL PROTECTED] on 03/06/07: Hello Christian, It's mostly done. Available at z3c.zodbtracing as inbetween I figured out that tracing the connection should be also possible. Saturday, February 24, 2007, 8:00:40 PM, you wrote: Hi, I'm pretty sure there are no hooks around that do what you need. The methods you want to tap into would be store() load*() on the storages. You might want to look into how the BlobStorage was created to make yourself a tracing storage that can be wrapped around an existing storage to allow usage with multiple other storages. This could be a useful extensions to ZODB. If you go this way, I'd be happy to help out with some of the packaging mechanics and getting the ZConfig parts fletched out. Christian Am Samstag, den 24.02.2007, 14:33 +0100 schrieb Adam Groszer: Hello, I'm using ZODB in a GUI application, so outside of Zope. I'm having performance problems. I'm already on the way of figuring out what causes a lot of object load. getTransferCounts helped in that. But now I would need something more detailed. Are there any hooks or something to get detailed object load statistics? Worst case I could hook into some methods of ZODB temporarly to collect the data, but into which methods? Any help or pointers are welcome. -- Best regards, Adammailto:[EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev WMA Master Student-Information Management WMA School of Engineering and Technology, AIT WMA Klong Luang, Pathumthani 12120, WMA Thailand WMA Mail Box 1272 -- Best regards, Groszer Adammailto:[EMAIL PROTECTED] -- Quote of the day: Love is a matter of chemistry, but Sex is a matter of physics. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[2]: [ZODB-Dev] ZODB load/save tracing
Hello Christian, It's mostly done. Available at z3c.zodbtracing as inbetween I figured out that tracing the connection should be also possible. Saturday, February 24, 2007, 8:00:40 PM, you wrote: Hi, I'm pretty sure there are no hooks around that do what you need. The methods you want to tap into would be store() load*() on the storages. You might want to look into how the BlobStorage was created to make yourself a tracing storage that can be wrapped around an existing storage to allow usage with multiple other storages. This could be a useful extensions to ZODB. If you go this way, I'd be happy to help out with some of the packaging mechanics and getting the ZConfig parts fletched out. Christian Am Samstag, den 24.02.2007, 14:33 +0100 schrieb Adam Groszer: Hello, I'm using ZODB in a GUI application, so outside of Zope. I'm having performance problems. I'm already on the way of figuring out what causes a lot of object load. getTransferCounts helped in that. But now I would need something more detailed. Are there any hooks or something to get detailed object load statistics? Worst case I could hook into some methods of ZODB temporarly to collect the data, but into which methods? Any help or pointers are welcome. -- Best regards, Adammailto:[EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[4]: [ZODB-Dev] ZODB load/save tracing
Hello Jim, What do you say, One event type with parameters: - method name - method parameters OR Lots of event types with parameters: - method parameters Sunday, February 25, 2007, 3:21:52 PM, you wrote: It might also be nice to have this generate events. That is, the tracing storage should call zope.event.notify. I intent in 3.8 or 3.9 to start having ZODB depend on zope.event. We really should have used events rather than adding the callback's we've added recently. Jim On Feb 25, 2007, at 3:19 AM, Adam Groszer wrote: Hello Christian, Gosh, that looks simple. A simple decorator pattern on the storage. So I shall create a branch, something like svn://svn.zope.org/repos/main/ZODB/branches/tracing-storage from svn://svn.zope.org/repos/main/ZODB/trunk and start there? Saturday, February 24, 2007, 8:00:40 PM, you wrote: Hi, I'm pretty sure there are no hooks around that do what you need. The methods you want to tap into would be store() load*() on the storages. You might want to look into how the BlobStorage was created to make yourself a tracing storage that can be wrapped around an existing storage to allow usage with multiple other storages. This could be a useful extensions to ZODB. If you go this way, I'd be happy to help out with some of the packaging mechanics and getting the ZConfig parts fletched out. Christian Am Samstag, den 24.02.2007, 14:33 +0100 schrieb Adam Groszer: Hello, I'm using ZODB in a GUI application, so outside of Zope. I'm having performance problems. I'm already on the way of figuring out what causes a lot of object load. getTransferCounts helped in that. But now I would need something more detailed. Are there any hooks or something to get detailed object load statistics? Worst case I could hook into some methods of ZODB temporarly to collect the data, but into which methods? Any help or pointers are welcome. -- Best regards, Adammailto:[EMAIL PROTECTED] -- Quote of the day: Faith is believing in things when common sense tells you not to. - George Seaton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org -- Best regards, Adammailto:[EMAIL PROTECTED] -- Quote of the day: When you close your eyes to the devil, make sure that it is not a wink. - John C. Kulp ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[2]: [ZODB-Dev] ZODB load/save tracing
Hello Christian, Gosh, that looks simple. A simple decorator pattern on the storage. So I shall create a branch, something like svn://svn.zope.org/repos/main/ZODB/branches/tracing-storage from svn://svn.zope.org/repos/main/ZODB/trunk and start there? Saturday, February 24, 2007, 8:00:40 PM, you wrote: Hi, I'm pretty sure there are no hooks around that do what you need. The methods you want to tap into would be store() load*() on the storages. You might want to look into how the BlobStorage was created to make yourself a tracing storage that can be wrapped around an existing storage to allow usage with multiple other storages. This could be a useful extensions to ZODB. If you go this way, I'd be happy to help out with some of the packaging mechanics and getting the ZConfig parts fletched out. Christian Am Samstag, den 24.02.2007, 14:33 +0100 schrieb Adam Groszer: Hello, I'm using ZODB in a GUI application, so outside of Zope. I'm having performance problems. I'm already on the way of figuring out what causes a lot of object load. getTransferCounts helped in that. But now I would need something more detailed. Are there any hooks or something to get detailed object load statistics? Worst case I could hook into some methods of ZODB temporarly to collect the data, but into which methods? Any help or pointers are welcome. -- Best regards, Adammailto:[EMAIL PROTECTED] -- Quote of the day: Faith is believing in things when common sense tells you not to. - George Seaton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB load/save tracing
Hello, I'm using ZODB in a GUI application, so outside of Zope. I'm having performance problems. I'm already on the way of figuring out what causes a lot of object load. getTransferCounts helped in that. But now I would need something more detailed. Are there any hooks or something to get detailed object load statistics? Worst case I could hook into some methods of ZODB temporarly to collect the data, but into which methods? Any help or pointers are welcome. -- Best regards, Groszer Adam -- Quote of the day: You cannot kill time without injuring eternity. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[2]: [ZODB-Dev] Commit Progress Indictor
Hello Tino, Just trying to take the road with the least number of stones. Obviously it is not worth to touch the ZODB internals. Thursday, August 17, 2006, 1:59:19 PM, you wrote: TW Jim Fulton schrieb: TW ... I can put a fake progress indicator anytime that advances every second a bit and never reaches 100%. Good idea. :) TW I fear this idea is already patented by Microso~1 ;) TW Regards TW Tino -- Best regards, Groszer Adam -- Quote of the day: The probability that we may fail in the struggle ought not to deter us from the support of a cause we believe to be just. - Abraham Lincoln ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[2]: [ZODB-Dev] Commit Progress Indictor
Hello Dieter, Do you think that's possible? Any ideas or tips? Monday, August 14, 2006, 7:09:51 PM, you wrote: Chris S wrote at 2006-8-14 11:06 -0400: When commiting a transaction, is there any way to track the progress of data commited? No, nobody has yet envisaged this use case... I have a small gui app, and I'd like to display a progress bar indicating the status of the commit. -- Best regards, Adammailto:[EMAIL PROTECTED] -- Quote of the day: This fortune is inoperative. Please try another. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[4]: [ZODB-Dev] Commit Progress Indictor
Hello Andreas, Yep, you're right :-) Are there any hooks in ZODB to support that or that should be a major overhaul? Wednesday, August 16, 2006, 11:03:54 AM, you wrote: --On 16. August 2006 10:52:51 +0200 Adam Groszer [EMAIL PROTECTED] wrote: Hello Dieter, Do you think that's possible? This is not the question. With some effort you can develop almost everything...you just have to find a volunteer and a budget :-) -aj -- Best regards, Adammailto:[EMAIL PROTECTED] -- Quote of the day: It is astonishing how little one feels poverty when one loves. - John Bulwer ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[2]: [ZODB-Dev] What makes the ZODB slow?
Hello Tim, Monday, June 26, 2006, 11:28:51 PM, you wrote: [snip] AFAIK, nobody anywhere has used this yet, outside of Python's test suite. It was intended to be a simple, cheap approach to cutting pickle bloat for apps motivated enough to set up the registry. You'll note that half the one-byte codes are reserved for Zope :-) [snip] What about making this an option in my (z3) instance? If I want to be compatible I leave it alone, if I want to be hell-fast I turn it on? -- Best regards, Adammailto:[EMAIL PROTECTED] -- Quote of the day: An ounce of mother is worth a ton of priest. - Proverb ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re[2]: [ZODB-Dev] Advice needed
Hi Andreas, Thank you for your quick answer. Friday, June 23, 2006, 2:36:56 PM, you wrote: AJ --On 23. Juni 2006 14:27:52 +0200 Adam Groszer [EMAIL PROTECTED] wrote: Hi there, I know it is not easy, but I need some advices to keep our bosses happy at the planning stage of our application. The application we are planning is a document management application which follows documents in a company. Document and version count could be around 1s, AJ 1s per document? Sorry, around 1s of document, each having 10s of versions, so around 100k versions/files. That grows to 300k objects or more with all bells and whistles. (Version history, Workflow history, ...) quite frequent writes are possible so concurrency might be an issue. Document files might be stored as BLOBs or separately on a files system to relieve ZODB. Some fears they are having and I can't find unambiguous information: - Is ZODB a good choice for this app? Which storage to consider? Filestorage? AJ We run a CMS with several ZEO storages using FileStorage with up 300k AJ different objects per storage. May you mention some or some sites on the internet that run Zope/ZODB to have some examples? maybe PGStorage? - ACID properties. Is it really ACID, I mean data consistency level could be compared to a RDB? AJ What do you mean? The ZODB is transactional and other systems (RBDMS) can AJ be hooked with the transaction system of Zope...this is daily practice. My problem is that I'm quite confident that ZODB may be able to handle the application, but I have to convince some other people too. They are coming with a strong RDB background. I would appreciate some facts to hit them hard. Here we do not have any experience and the application should be a real good one. - Coming from the RDB world, what are our current choices to provide a referential integrity like service on top of ZODB? AJ See above AJ -aj -- Best regards, Groszer Adam -- Quote of the day: Things turn out best for the people who make the best of the way things turn out. - Art Linkletter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
