[ZODB-Dev] Pickle protocol version 2

2007-12-22 Thread Hanno Schlichting
Hi, for those of you watching the commit lists closely I made a branch at: svn://svn.zope.org/repos/main/ZODB/branches/hannosch-pickle-protocol-version2 which uses a very simple approach to using the pickle protocol version 2 in all parts of the ZODB by externalizing it into a constant in

Re: [ZODB-Dev] 3.8.1b8 released and would like to release 3.8.1 soon

2008-09-30 Thread Hanno Schlichting
Dieter Maurer wrote: Wichert Akkerman wrote at 2008-9-24 09:44 +0200: Jim Fulton wrote: I'd appreciate it if people would try it out soon. I can say that the combination of 3.8.1b8 and Dieter's zodb-cache-size-bytes patch does not seem to work. With zodb-cache-size-bytes set to 1

Re: [ZODB-Dev] 3.8.1b8 released and would like to release 3.8.1 soon

2008-09-30 Thread Hanno Schlichting
Jim Fulton wrote: I'd appreciate it if people would try it out soon. Besides the RelStorage site where we were running into problems as mentioned in the other thread, we also use the ZODB 3.8 branch (beta8 plus the first two beta fixes) in a smaller site with blob storage for about a week in

Re: [ZODB-Dev] Optimize BTree node sizes?

2008-11-04 Thread Hanno Schlichting
Jim Fulton wrote: On Nov 4, 2008, at 12:12 PM, Benji York wrote: On Tue, Nov 4, 2008 at 12:01 PM, Jim Fulton [EMAIL PROTECTED] wrote: A few months back, there was a lot of discussion here about BTree performance. I got a sense that maximum BTree-node and bucket sizes should be increased.

Re: [ZODB-Dev] Minimal changes for Plone 3.1.6 running on ZODB 3.8.1 ?

2008-11-17 Thread Hanno Schlichting
Russ Ferriday wrote: In a warty old application, I'm hitting memory constraints that I think the 3.8.1 changes in ZODB might fix. We are running both Plone 3.0 and 3.1 in production with ZODB 3.8.1 on two quite large sites (one has RelStorage, the other one blob storage). I'd consider it

Re: [ZODB-Dev] ZODB alternate serialization format patch

2009-01-15 Thread Hanno Schlichting
Shane Hathaway wrote: I have just created a patch for ZODB that makes the object serialization format pluggable, meaning that it should now be possible to write Python code that causes ZODB to store in formats other than Python pickles. Cool :) In addition to making the data format

Re: [ZODB-Dev] ZODB alternate serialization format patch

2009-01-16 Thread Hanno Schlichting
Shane Hathaway wrote: Hanno Schlichting wrote: Shane Hathaway wrote: In addition to making the data format pluggable, the patch puts most of ZODB's dependencies on the cPickle module in one place, so now if we decide to improve our usage of the cPickle module, we can make that change

Re: [ZODB-Dev] Can I use RelStorage with ZODB3.9.0

2009-03-06 Thread Hanno Schlichting
Wichert Akkerman wrote: On 3/6/09 6:27 AM, eastxing wrote: About one month ago, I asked a question about 'ZODB pack' and got suggestions to update to new ZODB version. It took me one month to update my site from Plone2.5.5(with Zope2.9.6-final,ZODB3.6.2) to Plone3.1.7(with

Re: [ZODB-Dev] ZODB 3.9

2009-04-07 Thread Hanno Schlichting
Alan Runyan wrote: Is a ZODB 3.9 beta around the corner? Just be aware that ZODB 3.9 is not compatible with any stable Zope 2.x release. It only works and is required for Zope 2.12. It can be made to work with prior versions of Zope2 but that is a mild pain. Hanno

Re: [ZODB-Dev] ZODB 3.9

2009-04-11 Thread Hanno Schlichting
Chris Withers wrote: Hanno Schlichting wrote: Just be aware that ZODB 3.9 is not compatible with any stable Zope 2.x release. It only works and is required for Zope 2.12. It can be made to work with prior versions of Zope2 but that is a mild pain. What are the problems with using ZODB 3.9

Re: [ZODB-Dev] ZODB 3.9.0b1 released

2009-05-05 Thread Hanno Schlichting
Jim Fulton wrote: I've just released the first beta of ZODB 3.9. Awesome :) I thought there were some outstanding issues with the cache-size-bytes functionality. Got all of these resolved / reviewed by now? Thanks, Hanno ___ For more information

[ZODB-Dev] Blobs without shared-blob directory

2009-05-15 Thread Hanno Schlichting
Hi. I have a configuration with one ZEO database server (ZODB 3.8.1) and two physical servers housing four single-threaded ZEO clients each. There's some 15gb of blob data in the (Plone) site. Setting up and maintaining NFS to share direct access to the blob data is beyond what I can support in

Re: [ZODB-Dev] Blobs without shared-blob directory

2009-05-15 Thread Hanno Schlichting
Jim Fulton wrote: On May 15, 2009, at 6:22 AM, Hanno Schlichting wrote: Am I correct to assume that sharing the blob-cache between the four processes on the same physical machine is not supported, as these don't have any mechanism to synchronize read/write activity to the blob- cache

Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

2009-05-26 Thread Hanno Schlichting
of a separate database object. +lots Hanno Schlichting recently posted a nice graph showing the persistent structure of a Plone Page object and it's 9 (!) sub-objects. http://blog.hannosch.eu/2009/05/visualizing-persistent-structure-of.html That graph isn't quite correct ;-) workflow_history

Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

2009-05-27 Thread Hanno Schlichting
Chris Withers wrote: Hanno Schlichting wrote: Nope. DateTime objects are plain old-style classes and don't inherit from persistent.*. Hmm, oh well, my bad... In that case it must just be that their pickled form is huge compared to an int ;-) Sure: len(cPickle.dumps(DateTime.DateTime

Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

2009-05-27 Thread Hanno Schlichting
Marius Gedminas wrote: On Wed, May 27, 2009 at 12:17:36PM +0200, Hanno Schlichting wrote: Chris Withers wrote: Hanno Schlichting wrote: They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call

[ZODB-Dev] cache-size-bytes usable?

2009-11-07 Thread Hanno Schlichting
Hi. I've been playing around with the new cache-size-bytes setting to limit the ZODB cache sizes. My ZODB 3.9.3 (Zope 2.12 / Plone) config had a setting of cache-size-bytes to 128mb and a cache-size of one million (to effectively disable the object number based checks). This was without ZEO, a

Re: [ZODB-Dev] cache-size-bytes usable?

2009-11-07 Thread Hanno Schlichting
On Sat, Nov 7, 2009 at 6:34 PM, Andreas Jung li...@zopyx.com wrote: Am 07.11.09 18:25, schrieb Hanno Schlichting: Has anyone used the bytes limited cache in production yet and has any experience with it? I would assume that this feature is working since it comes from the Haufe Zope fork

Re: [ZODB-Dev] ZEO and blobs: missing tmp file breaks transaction on retry

2009-11-13 Thread Hanno Schlichting
On Fri, Nov 13, 2009 at 5:40 PM, Jim Fulton j...@zope.com wrote: On Fri, Nov 13, 2009 at 10:18 AM, Mikko Ohtamaa mi...@redinnovation.com wrote: Unfortunately the application having the issues is Plone 3.3. ZODB 3.9 depends on Zope 2.12 so, right? ZODB does depend on Zope anything. :)

Re: [ZODB-Dev] undo (and storage interface) brokenness

2009-12-23 Thread Hanno Schlichting
On Wed, Dec 23, 2009 at 9:26 PM, Jim Fulton j...@zope.com wrote: Undo is broken in a number of ways. Does anyone care?  Does anyone use undo? Speaking from the Zope2/Plone crowd: I'm using it during development with a local file storage at times. In a real ZEO production setup, undo in Plone

Re: [ZODB-Dev] undo (and storage interface) brokenness

2009-12-23 Thread Hanno Schlichting
On Thu, Dec 24, 2009 at 1:34 AM, Martin Aspeli optilude+li...@gmail.com wrote: Hanno Schlichting wrote: Throwing away all changes done in the meantime is usually not practical. ... although sometimes it is preferable to a full backup restore or living with whatever it is you want to restore

[ZODB-Dev] An interface for broken objects?

2009-12-31 Thread Hanno Schlichting
Hi. We currently have a package called zope.broken whose entire real content is: import zope.interface class IBroken(zope.interface.Interface): Marker interface for broken objects This is used for example by zope.container, which won't try to set __name__ and __parent__ pointers on

Re: [ZODB-Dev] An interface for broken objects?

2009-12-31 Thread Hanno Schlichting
On Thu, Dec 31, 2009 at 6:03 PM, Jim Fulton j...@zope.com wrote: On Thu, Dec 31, 2009 at 11:46 AM, Hanno Schlichting ha...@hannosch.eu wrote: Is this something that could be put into the ZODB itself and have the ZODB.broken.Broken class directly implement this interface? +1. although

Re: [ZODB-Dev] An interface for broken objects?

2010-01-08 Thread Hanno Schlichting
On Sun, Jan 3, 2010 at 5:38 PM, Jim Fulton j...@zope.com wrote: I suggest including these in the interface. Thanks for fixing this yourself and merging. Sorry, I didn't get around to extend things myself. Hanno ___ For more information about ZODB, see

[ZODB-Dev] ZODB on Windows 64bit?

2010-01-15 Thread Hanno Schlichting
Hi. I recently updated my Windows build environment to a 64bit Windows 7 environment with cross compilation support for 32/64 bit. I created and uploaded Windows binary eggs for all Zope 2.12.3 dependencies in both flavors for Python 2.6. The last one missing with a 64bit flavor is the ZODB3

Re: [ZODB-Dev] ZODB on Windows 64bit?

2010-01-16 Thread Hanno Schlichting
On Sat, Jan 16, 2010 at 9:05 AM, Adam GROSZER agros...@gmail.com wrote: Please don't stop with those binaries. There are some win32 users out here ;-) Sure. But Jim is building the 32bit binaries himself. So there's no need for me to do that. As he seems to be building things on WinXP, he

Re: [ZODB-Dev] ZODB on Windows 64bit?

2010-01-17 Thread Hanno Schlichting
On Sun, Jan 17, 2010 at 2:51 PM, Jim Fulton j...@zope.com wrote: On Fri, Jan 15, 2010 at 10:01 PM, Hanno Schlichting ha...@hannosch.eu wrote: P.S. Running the tests on the checkout of the 3.9.4 tag gives me two test failures, of which one is a test cleanup problem caused by the other. Does

Re: [ZODB-Dev] ZODB on Windows 64bit?

2010-01-22 Thread Hanno Schlichting
On Sun, Jan 17, 2010 at 3:42 PM, Hanno Schlichting ha...@hannosch.eu wrote: On Sun, Jan 17, 2010 at 2:51 PM, Jim Fulton j...@zope.com wrote: On Fri, Jan 15, 2010 at 10:01 PM, Hanno Schlichting ha...@hannosch.eu wrote: P.S. Running the tests on the checkout of the 3.9.4 tag gives me two test

Re: [ZODB-Dev] Understanding the ZODB cache-size option

2010-03-22 Thread Hanno Schlichting
On Mon, Mar 22, 2010 at 7:05 PM, Jeff Shell j...@bottlerocket.net wrote: Are there any metrics about how to set the ZODB 'cache-size' (or cache-size-bytes) option? We've been using '5000' (arbitrarily chosen) for our Zope 3.4-ish app servers. We have access to zc.z3monitor which can output

Re: [ZODB-Dev] DemoStorage stacking in zope.testing layers

2010-04-05 Thread Hanno Schlichting
Hi. On Mon, Apr 5, 2010 at 4:36 PM, Martin Aspeli optilude+li...@gmail.com wrote: What is the correct way to use DemoStorage stacking in test layers? Not sure if this helps, but zope.app.testing's functional.py [1] did something similar here (stacking DemoStorages). Maybe the

Re: [ZODB-Dev] Using zodb and blobs

2010-04-14 Thread Hanno Schlichting
On Wed, Apr 14, 2010 at 11:52 AM, Nitro ni...@dr-code.org wrote: Yes, in my case it's nothing critical or related to money. If there's a hardware outage a day of work is lost at worst. In case of corruption (which can happen also without fsync as data within the file can just be garbled) you

Re: [ZODB-Dev] ZODB 3.9.5 has been released

2010-04-23 Thread Hanno Schlichting
On Fri, Apr 23, 2010 at 11:30 PM, Jim Fulton j...@zope.com wrote: See http://pypi.python.org/pypi/ZODB3/3.9.5 Cool! I added the Python 2.6 / Win64 egg. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/

[ZODB-Dev] Problems in ZEO pack in 3.9.x?

2010-04-26 Thread Hanno Schlichting
Hi. I upgraded a ZODB 3.8 database server to 3.9 and suddenly zeopack fails. It looks to me like it is now failing on PosKeyErrors, which had been ignored so far. Is this a known problem and should be fixed? Or should I repair all the PosKeyErrors? Tracebacks below. Hanno Server traceback:

Re: [ZODB-Dev] Problems in ZEO pack in 3.9.x?

2010-04-27 Thread Hanno Schlichting
On Mon, Apr 26, 2010 at 7:44 PM, Jim Fulton j...@zope.com wrote: Hm.  I don't know if it was intentional to ignore POSKeyErrors in the old pack code.  It seems like a bad idea to me. Yep, I was wondering if that was a conscious design choice or just accidental behavior. What do folks think

Re: [ZODB-Dev] Problems in ZEO pack in 3.9.x?

2010-04-27 Thread Hanno Schlichting
On Tue, Apr 27, 2010 at 4:08 PM, Jim Fulton j...@zope.com wrote: On Tue, Apr 27, 2010 at 6:29 AM, Hanno Schlichting ha...@hannosch.eu wrote: But doing GC on an inconsistent state is probably a bad idea. Then I think the current behavior is correct. You can now disable GC using the pack-gc

[ZODB-Dev] Changing the pickle protocol?

2010-04-28 Thread Hanno Schlichting
Hi. The ZODB currently uses a hardcoded pickle protocol one. There's both the more efficient protocol two and in Python 3 protocol 3. Protocol two has seen various improvements in recent Python versions, triggered by its use in memcached. I'd be interested to work on changing the protocol. How

Re: [ZODB-Dev] Changing the pickle protocol?

2010-04-28 Thread Hanno Schlichting
On Wed, Apr 28, 2010 at 5:11 PM, Jim Fulton j...@zope.com wrote: Do you know of specific benefits you expect from protocol 2? Any specific reasons you think it would be better in practice? I have just seen some ongoing work on pickles in recent times, for example from the Python 2.7 what's

Re: [ZODB-Dev] ZEO monitor server

2010-04-30 Thread Hanno Schlichting
On Fri, Apr 30, 2010 at 8:07 PM, Jim Fulton j...@zope.com wrote: Does anyone use the monitor server? Would anyone object if it went away, to be replaced by a server method on the standard ZEO protocol? I don't know anyone using the monitor server. So changing it is not a problem here. We only

[ZODB-Dev] repozo full backup and index files

2010-05-12 Thread Hanno Schlichting
Hi. I was wondering if there's a specific reason why repozo does not backup or create index files, whenever it does a full backup. I understand that creating index files for incremental backups is probably hard, but for a full backup it should be possible. Recreating the index file after

Re: [ZODB-Dev] repozo full backup and index files

2010-05-14 Thread Hanno Schlichting
On Fri, May 14, 2010 at 10:24 AM, Christian Theune c...@gocept.com wrote: Hmm. If the full backup is just a regular FS file then you could start with the naive approach and just open/close it once after performing a backup as that would create the index file. Sure. That would be an easy but

Re: [ZODB-Dev] repozo full backup and index files

2010-05-14 Thread Hanno Schlichting
On Fri, May 14, 2010 at 4:33 PM, Tres Seaver tsea...@palladion.com wrote: I would be willing to make a stab at this, if we can hold off on a 3.10.0 beta until I've had a chance to try it. Oh, awesome! Best possible outcome I could hope for - someone else wants to do the work for me ;-) Hanno

Re: [ZODB-Dev] Storage iterators and IndexError

2010-05-15 Thread Hanno Schlichting
On Fri, May 14, 2010 at 8:43 PM, Jim Fulton j...@zope.com wrote: There's a test for storage iterators that verifies that they raise a special exception that extends StopIteration and IndexError. This makes storage iterators a bit harder to implement than necessary. Does anyone know of a

Re: [ZODB-Dev] ZEO and access permissions

2010-05-22 Thread Hanno Schlichting
On Sat, May 22, 2010 at 2:17 PM, Nitro ni...@dr-code.org wrote: ZEO already supports authenticated logins. Based on the login I'd like people to be able to access some objects and deny access to others. First I thought I'd do the access restrictions on the application level. That's the only

Re: [ZODB-Dev] Changing the pickle protocol?

2010-05-22 Thread Hanno Schlichting
Hi. Following up on my idea of using pickle protocol 2. I implemented this in a fully configurable fashion on a branch, mainly to ease benchmarking and testing of the different variants. My conclusions (maybe for future reference): - There's no significant win of just switching the pickle

Re: [ZODB-Dev] Changing the pickle protocol?

2010-05-23 Thread Hanno Schlichting
On Sun, May 23, 2010 at 5:45 PM, Jim Fulton j...@zope.com wrote: On Sat, May 22, 2010 at 12:30 PM, Hanno Schlichting ha...@hannosch.eu wrote: - The code to make the protocol configurable on all levels (storage, index, persistent cache, ...) is large and ugly, I'm puzzled.  Why were changes so

Re: [ZODB-Dev] Restoring from repozo and reusing an index file?

2010-06-11 Thread Hanno Schlichting
On Fri, Jun 11, 2010 at 3:25 PM, Paul Winkler sli...@gmail.com wrote: I'm preparing to move a zope site from one host to another. We've been planning to use repozo backups to copy the filestorage from the old host to the new one.  I'd like to figure out how to minimize downtime while not

Re: [ZODB-Dev] [Checkins] SVN: ZODB/trunk/src/ Merged zagy-lp509801 branch.

2010-07-11 Thread Hanno Schlichting
Hi Jim. This error is biting us in ZODB 3.8.5 and 3.9.5. Are you intending to backport this and the other bugfixes? Do you want help with backporting and in what way? Create bugfix branches against the older branches for your review or directly apply the patches as you reviewed them already?

Re: [ZODB-Dev] KeyError in BTrees.check

2010-07-13 Thread Hanno Schlichting
On Tue, Jul 13, 2010 at 11:23 AM, Suresh V. suresh...@yahoo.com wrote: While trying to run manage_cleanup on my BTreeFolder2, I get a KeyError from the classify function in Module BTrees.check: def classify(obj):     return _type2kind[type(obj)] I see that obj is None at this point.

Re: [ZODB-Dev] I released 3.10.0b2 this morning

2010-07-14 Thread Hanno Schlichting
On Wed, Jul 14, 2010 at 2:47 PM, Jim Fulton j...@zope.com wrote: It got a lot larger recently when someone moved a bunch of reports from the Zope 2 lists.  A lot of these bugs are obsolete. That one was me. I tried hard to only move those bugs, which were indeed pure ZODB problems. A lot of

Re: [ZODB-Dev] Remember date of last pack?

2010-07-26 Thread Hanno Schlichting
On Mon, Jul 26, 2010 at 5:14 PM, Christian Zagrodnick c...@gocept.com wrote: for monitoring if a storage has been packed it would be handy if it remembered the date of its last pack. An easy solution for file storage would be to create a Data.fs.packed file. Other ways do determine the date

Re: [ZODB-Dev] Remember date of last pack?

2010-07-26 Thread Hanno Schlichting
On Mon, Jul 26, 2010 at 8:12 PM, Christian Zagrodnick c...@gocept.com wrote: On 2010-07-26 17:38:10 +0200, Hanno Schlichting said: - Configure repozo to do incremental backups and keep some old backups. Configure zeopack to run regularly. Check that you have a recent full repozo backup around

Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python

2010-07-29 Thread Hanno Schlichting
On Thu, Jul 29, 2010 at 1:20 PM, Christian Theune c...@gocept.com wrote: ZEO in version 3.10 is supposed to work with older clients. Is it also intended to work with older clients running Python 2.4? The server needs to run on Python 2.5 due to the with statement. The docs explicitly state

Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python

2010-07-29 Thread Hanno Schlichting
On Thu, Jul 29, 2010 at 2:49 PM, Christian Theune c...@gocept.com wrote: On 07/29/2010 02:42 PM, Hanno Schlichting wrote: The docs explicitly state that Python 2.4 is no longer supported at all. Go with the times and use Python 2.6 or 2.7 ;) The docs also say that ZEO supports older clients

Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python

2010-07-29 Thread Hanno Schlichting
On Thu, Jul 29, 2010 at 2:57 PM, Christian Theune c...@gocept.com wrote: Thanks for digging this out. I'll try to find the discussion and refresh my memory. See for example your response here https://mail.zope.org/pipermail/zodb-dev/2010-April/013269.html ;-) And Andreas was the only one

Re: [ZODB-Dev] Weird KeyError with OOBTree

2010-08-16 Thread Hanno Schlichting
On Mon, Aug 16, 2010 at 12:14 PM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Could this be some problem with using persistent objects as keys in a BTree? Some comparison problem? I'm not entirely sure about this, but I think using persistent objects as keys isn't supported. Looking at

Re: [ZODB-Dev] Weird KeyError with OOBTree

2010-08-16 Thread Hanno Schlichting
On Mon, Aug 16, 2010 at 2:04 PM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: I think implementing a stable hash function for your type could make this work though. From what I read, ZODB doesn't use hash functions, relying on __cmp__ instead. So, I guess I should make my class

[ZODB-Dev] 3.10 final?

2010-08-30 Thread Hanno Schlichting
Hi, just wondering what the state of 3.10 final is? We had a couple beta releases already. What needs to be done to move it forward and is there anything the community can help with? Hanno ___ For more information about ZODB, see the ZODB Wiki:

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-20 Thread Hanno Schlichting
On Mon, Sep 20, 2010 at 6:00 PM, Jim Fulton j...@zope.com wrote: Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm curious why? Zope 2.10 includes ZODB 3.7 by default. 3.8 does work fine with it. But in 3.9 versions got removed. There was a good deal of code in Zope 2 that still

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-20 Thread Hanno Schlichting
On Mon, Sep 20, 2010 at 8:08 PM, Jim Fulton j...@zope.com wrote: Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use versions? Or is this a matter of test failures? Last time I checked it did break. There's at least the temporary storage implementation that required changes to

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-21 Thread Hanno Schlichting
On Tue, Sep 21, 2010 at 3:48 PM, Jim Fulton j...@zope.com wrote: 3.8.6 is available now with this bug fix and a few others. Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like CMFEditions). So

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-21 Thread Hanno Schlichting
On Tue, Sep 21, 2010 at 4:30 PM, Jim Fulton j...@zope.com wrote: On Tue, Sep 21, 2010 at 10:00 AM, Hanno Schlichting ha...@hannosch.eu wrote: Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-21 Thread Hanno Schlichting
On Tue, Sep 21, 2010 at 5:20 PM, Jim Fulton j...@zope.com wrote: It was changes like this one http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442r1=115279r2=115442 I don't understand how the BTree changes would run afoul of this. This code should

Re: [ZODB-Dev] RFC: deprecate transaction user and description fields in favor of extened info and simplify extended info API

2010-09-23 Thread Hanno Schlichting
On Thu, Sep 23, 2010 at 4:24 PM, Jim Fulton j...@zope.com wrote: The user and description fields are somewhat archaic. They can only be strings (not unicode) and can as easily be handled as extended info. I propose to deprecate the 'user' and 'description' attributes, and the 'setUser' and

Re: [ZODB-Dev] [Zope-dev] Zope Tests: 31 OK, 19 Failed, 2 Unknown

2010-09-25 Thread Hanno Schlichting
On Fri, Sep 10, 2010 at 10:52 PM, Tres Seaver tsea...@palladion.com wrote: My attempts at diagnosis. Subject: FAILED : Zope Buildbot / zope2 slave-ubuntu64 From: jdriessen at thehealthagency.com Date: Thu Sep  9 09:31:14 EDT 2010 URL:

Re: [ZODB-Dev] Default comparison considered harmful in BTrees.

2010-10-27 Thread Hanno Schlichting
On Mon, Oct 25, 2010 at 11:51 PM, Jim Fulton j...@zope.com wrote: I'm inclined to treat the use of the comparison operator inherited from object in BTrees to be a bug.  I plan to fix this on the trunk. Did you mean to throw warnings for simple built-in types? I'm now getting warnings for

Re: [ZODB-Dev] Default comparison considered harmful in BTrees.

2010-10-27 Thread Hanno Schlichting
On Wed, Oct 27, 2010 at 6:45 PM, Jim Fulton j...@zope.com wrote: If not, I wonder if the existing indexes have some bad values in them that are triggering this somehow.  The relevant check is being done when loading state.  I bet you have some bad keys (e.g. None) in your data structures.

Re: [ZODB-Dev] Default comparison considered harmful in BTrees.

2010-10-28 Thread Hanno Schlichting
On Wed, Oct 27, 2010 at 8:56 PM, Jim Fulton j...@zope.com wrote: On Wed, Oct 27, 2010 at 1:59 PM, Hanno Schlichting ha...@hannosch.eu wrote: I suppose that depends on the application.  Was the use of None intentional, or the result of sloppy coding? I'd have to check the code. I expect

Re: [ZODB-Dev] 32-bit vs 64-bit - RelStorage on MySQL

2010-11-18 Thread Hanno Schlichting
On Thu, Nov 18, 2010 at 5:19 PM, Leonardo Santagada santag...@gmail.com wrote: On Thu, Nov 18, 2010 at 1:47 PM, Chris Withers ch...@simplistix.co.uk wrote: On 18/11/2010 15:39, Marius Gedminas wrote: About the only noticeable difference -- other than the obvious memory growth What obvious

Re: [ZODB-Dev] Sharing (persisted) strings between threads

2010-12-08 Thread Hanno Schlichting
On Wed, Dec 8, 2010 at 11:06 AM, Malthe Borch mbo...@gmail.com wrote: With 20 active threads, each having rendered the Plone 4 front page, this approach reduced the memory usage with 70 MB. Did you measure throughput of the system? In the benchmarks I've seen threads numbers of 3 or above will

Re: [ZODB-Dev] Some tests do not pass load's version parameter

2010-12-13 Thread Hanno Schlichting
Hi. On Mon, Dec 13, 2010 at 10:11 AM, Vincent Pelletier vinc...@nexedi.com wrote: Reading ZODB.interface, I realised the version parameter is defined as mandatory: class IStorage(Interface):   [...]    def load(oid, version): That parameter was deprecated in 3.9. Looks like the interface

Re: [ZODB-Dev] Lazy Instantiation

2011-01-23 Thread Hanno Schlichting
On Mon, Jan 24, 2011 at 1:43 AM, Leonardo Santagada santag...@gmail.com wrote: On Sat, Jan 22, 2011 at 3:23 PM, Jim Fulton j...@zope.com wrote:   a.child == None will because equality comparison will attempt to access methods on the child, which will cause it's activation. Why would it

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Hanno Schlichting
Hi. On Thu, Jan 27, 2011 at 9:00 AM, Matt Hamilton ma...@netsight.co.uk wrote: Alas we are. Or rather, alas, ZCatalog does ;) It would be great if it didn't but it's just the way it is. If I have 300,000 items in my site, and everyone of them visible to someone with the 'Reader' role, then

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Hanno Schlichting
On Thu, Jan 27, 2011 at 11:09 AM, Matt Hamilton ma...@netsight.co.uk wrote: Hanno Schlichting hanno at hannosch.eu writes: There still seem to be instances in which the entire set is loaded.  This could be an artifact of the fact I am clearing the ZODB cache before each ]test, which I think

Re: [ZODB-Dev] Issue with Blobstorage for a third client in Plone

2011-02-11 Thread Hanno Schlichting
On Fri, Feb 11, 2011 at 11:24 PM, Andreas Mantke ma...@gmx.de wrote: I want to create a third instance (client3) in a running zeoserver that uses his own database (data.fs) and blobstorage. This third client should run a repository for extensions. The first client (and the second) is used

Re: [ZODB-Dev] Advice on whether to run relstorage database and Zope on different servers

2011-02-24 Thread Hanno Schlichting
Hi. On Thu, Feb 24, 2011 at 4:20 PM, Anthony Gerrard anthonygerr...@gmail.com wrote: * I'm familiar with enterprise environments where you would have an app server and a database server but are there any advantages to putting Zope and MySQL on different servers? Database servers (be it

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Hanno Schlichting
On Mon, Feb 28, 2011 at 4:19 PM, Maurits van Rees m.van.r...@zestsoftware.nl wrote: This is with RelStorage 1.5.0-b1, with blob-dir enabled, shared-blob-dir false and a low blob-cache-size, say 10 bytes.  Using Plone 3.3.5, plone.app.blob 1.3, plone.app.imaging 1.0.1, but the same is

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Hanno Schlichting
On Mon, Feb 28, 2011 at 5:09 PM, Maurits van Rees m.van.r...@zestsoftware.nl wrote: Op 28-02-11 16:34, Martijn Pieters schreef: On Mon, Feb 28, 2011 at 16:22, Hanno Schlichtingha...@hannosch.eu  wrote: Blobs are considered experimental in ZODB 3.8. Especially the entire blob cache changed

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-03-01 Thread Hanno Schlichting
On Tue, Mar 1, 2011 at 11:50 PM, Shane Hathaway sh...@hathawaymix.org wrote: On 03/01/2011 02:47 PM, Maurits van Rees wrote: - ZODB3 3.8.6-polling Blobs are stored in postgres with RelStorage.  Layout of the blob cache dir is 'zeocache'. I should have spotted this earlier. The zeocache

Re: [ZODB-Dev] packing ZODB

2011-03-31 Thread Hanno Schlichting
Hi. On Thu, Mar 31, 2011 at 12:46 PM, Adam GROSZER agros...@gmail.com wrote: After investigating FileStorage a bit, I found that GC runs on objects, but pack later by transactions. That means that if there's a bigger-ish transaction, we can't get rid of it until all of it's objects are GCed

Re: [ZODB-Dev] [RFC] ZEO: Allow non-packaged products

2011-04-29 Thread Hanno Schlichting
On Fri, Apr 29, 2011 at 3:24 PM, Vincent Pelletier vinc...@nexedi.com wrote: I need ZEO to be able to find non-packaged products for conflict resolution purposes. As ZEO AFAIK doesn't support this I gave it a quick try. I reached the works for me state, that I now would like to get feedback on.

[ZODB-Dev] How to check for setting the same values on persistent objects?

2011-05-04 Thread Hanno Schlichting
Hi. I tried to analyze the overhead of changing content in Plone a bit. It turns out we write back a lot of persistent objects to the database, even tough the actual values of these objects haven't changed. Digging deeper I tried to understand what happens here: 1. persistent.__setattr__ will

Re: [ZODB-Dev] How to check for setting the same values on persistent objects?

2011-05-04 Thread Hanno Schlichting
On Wed, May 4, 2011 at 1:09 PM, Laurence Rowe l...@lrowe.co.uk wrote: Persistent objects are also used as a cache and in that case code relies on an object being invalidated to ensure its _v_ attributes are cleared. Comparing at the pickle level would break these caches. So you would expect

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-05 Thread Hanno Schlichting
On Thu, May 5, 2011 at 10:43 AM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Since we are talking about speed, does anyone have any tips on making ZODB (in general) faster? Query fewer objects from the database. Make sure you don't store lots of tiny persistent objects in the database,

Re: [ZODB-Dev] How to check for setting the same values on persistent objects?

2011-05-06 Thread Hanno Schlichting
On Thu, May 5, 2011 at 6:27 PM, Alexandre Garel alex.ga...@tarentis.com wrote: I'm assuming doing a general check for old == new is not safe, as it might not be implemented correctly for all objects and doing the comparison might be expensive. I know very few of ZODB internals but in Python

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-06 Thread Hanno Schlichting
On Fri, May 6, 2011 at 2:22 PM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: That's hard to do for a project that is already 8 or 9 years old, as you can see in the attached file, we've got have many cases that fall outside your limits. I've noticed, for instance, that pages that involve

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-06 Thread Hanno Schlichting
On Fri, May 6, 2011 at 10:14 PM, Shane Hathaway sh...@hathawaymix.org wrote:  From my experience, most people who want ZODB to be faster want Zope catalogs in particular to be faster.  I don't think prefetching can make catalogs much faster, though. I've spent a lot of time lately on making

Re: [ZODB-Dev] Occasional startup errors in serialize.py/_reconstructor

2011-05-09 Thread Hanno Schlichting
On Mon, May 9, 2011 at 1:14 PM, Andreas Jung li...@zopyx.com wrote: Occasionally I receive the following error after starting my instance after the first request. There is no way to recover out other than restarting the instance...then everything is fine. I don't know why this happens from

Re: [ZODB-Dev] Occasional startup errors in serialize.py/_reconstructor

2011-05-09 Thread Hanno Schlichting
On Mon, May 9, 2011 at 2:08 PM, Andreas Jung li...@zopyx.com wrote: Jup - basically a stripped down version of http://svn.plone.org/svn/collective/collective.examples.userdata/trunk/collective/examples/userdata/ Hhm, that code has

Re: [ZODB-Dev] Immutable blobs?

2011-05-09 Thread Hanno Schlichting
On Mon, May 9, 2011 at 2:26 PM, Laurence Rowe l...@lrowe.co.uk wrote: While looking at the Plone versioning code the other day, it struck me that it would be much more efficient to implement file versioning if we could rely on blobs never changing after their first commit, as a copy of the

Re: [ZODB-Dev] Occasional startup errors in serialize.py/_reconstructor

2011-05-09 Thread Hanno Schlichting
On Mon, May 9, 2011 at 2:34 PM, Andreas Jung li...@zopyx.com wrote: The question is more why this error is happening from time to time after the startup after the first request - this scares me a bit. I also encounter a strange issues with PTS from time to time after startup...but not as

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-09 Thread Hanno Schlichting
On Thu, Jun 9, 2011 at 7:09 PM, Shane Hathaway sh...@hathawaymix.org wrote: On 06/09/2011 06:32 AM, Martijn Pieters wrote: We've looked over the RelsStorage ZODB Blob storage implementation and came to the conclusion that the current use of blob chunks is unnecessary in Oracle when using the

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-10 Thread Hanno Schlichting
On Fri, Jun 10, 2011 at 7:17 AM, Shane Hathaway sh...@hathawaymix.org wrote: BTW, I'd like to release a final of RelStorage 1.5, but the #@$%$ Windows tests are failing on Windows XP (and probably other versions of Windows).  I think the problem is caused by file descriptors being left open

Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?

2011-07-14 Thread Hanno Schlichting
On Thu, Jul 14, 2011 at 4:12 PM, Andreas Jung li...@zopyx.com wrote: I followed the documentation at http://pastebin.com/bL0CbBm2 Wow, looks like I didn't specify an expiration date for my paste ;) If anyone feels like putting that into a more permanent place, feel free to use it. Hanno

Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?

2011-07-14 Thread Hanno Schlichting
On Thu, Jul 14, 2011 at 5:38 PM, Andreas Jung li...@zopyx.com wrote: For the sake of completeness: I found this https://mail.zope.org/pipermail/zodb-dev/2008-February/011606.html and will try it out . There's also instructions in my pastebin dump on how to do this - and quite a bit simpler

Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup

2011-07-14 Thread Hanno Schlichting
On Thu, Jul 14, 2011 at 9:32 PM, Sean Upton sdup...@gmail.com wrote: ZODB.POSException.ConflictError: database conflict error (oid 0x00, class persistent.mapping.PersistentMapping, serial this txn started with 0x038fa89e48c295cc 2011-07-13 14:22:17.053148, serial currently committed

Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?

2011-07-14 Thread Hanno Schlichting
On Thu, Jul 14, 2011 at 9:40 PM, Jim Fulton j...@zope.com wrote: On Thu, Jul 14, 2011 at 3:23 PM, Hanno Schlichting ha...@hannosch.eu wrote: At Jarn we've used that trick many times to repair broken internals in the intid/keyreference data structures. Do you have any theories why objects

Re: [ZODB-Dev] Upgraded from ZODB 3.8 to 3.10, do I need to upgrade my Data.fs?

2012-04-16 Thread Hanno Schlichting
Paul Warner paul.warner at gmail.com writes: We have been running an in house application on ZODB 3.8.2 and recently upgraded to 3.10.5. We run a ZEO server with FileStorage. It seems like everything works great, no actions on our part, but I wondered if it is suggested to some how migrate

Re: [ZODB-Dev] all webserver threads blocking on db.open()

2012-05-07 Thread Hanno Schlichting
I think you might get better help on one of the Pyramid support channels. Your problems all seem to be related to configuring a web server in production mode, rather than database issues. From what I can tell you are dealing with hung requests. I'd look at either the paster configuration options

Re: [ZODB-Dev] Relstorage and MSQL

2012-05-08 Thread Hanno Schlichting
On 09.05.2012, at 02:34, Dylan Jay d...@pretaweb.com wrote: I know it's not all about money but if we were to sponsor development of microsoft sql server support for relstorage, is there someone who knows how and has an estimated cost and available time? It might be helpful if you could add

Re: [ZODB-Dev] RFC: release persistent as a standalone package

2012-06-30 Thread Hanno Schlichting
On Sat, Jun 30, 2012 at 8:02 PM, Tres Seaver tsea...@palladion.com wrote: I would like to release a '4.0.0' version of the package, and switch the ZODB trunk to pull it in as a dependency (deleting the currently included (older) copy of persistent).  One possible issue is that I have not (yet)

  1   2   >