Re: [ZODB-Dev] RFC: ZODB 4.0 (without persistent)

2012-10-21 Thread Hanno Schlichting
On Sat, Oct 20, 2012 at 9:37 PM, Tres Seaver tsea...@palladion.com wrote: I released BTrees 4.0.0, and created a ZODB branch for the (trivial) shift to depending on it: I had to tweak the C header inclusion a bit, so that the winbot could create binary eggs. There's a 4.0.1 release on pypi now,

Re: [ZODB-Dev] RFC: ZODB 4.0 (without persistent)

2012-10-14 Thread Hanno Schlichting
On Sun, Oct 14, 2012 at 10:21 PM, Jim Fulton j...@zope.com wrote: Goal: Give persistent, ZODB and ZEO their own release cycles. In which distribution would BTrees end up in? I think the pure-Python work for BTrees isn't yet finished, but I could be wrong. But if we are extracting packages into

Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.

2012-08-27 Thread Hanno Schlichting
On Mon, Aug 27, 2012 at 8:57 AM, Adam GROSZER agroszer...@gmail.com wrote: Now we have some problems making the binary eggs. See the attached txt for the full output. I think I fixed that on SVN trunk. Looks like on Windows the compiler complains about mismatches between module names and the

Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.

2012-08-27 Thread Hanno Schlichting
On Mon, Aug 27, 2012 at 3:18 PM, Tres Seaver tsea...@palladion.com wrote: I think that was the attempt to convert a too-big number to a C 'long long'. I have adjusted the test to use 2**63 - 1 directly. Yep, tests pass here now. I will make a 4.0.2 release after to buildbots report success

Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.

2012-08-26 Thread Hanno Schlichting
On Sat, Aug 25, 2012 at 4:11 PM, Tres Seaver tsea...@palladion.com wrote: This failure, and the others like it, indicate that the buildout has failed to install persistent correctly (there is nothing Windows-specific about the failure):: I actually got the same error locally. The trouble is

Re: [ZODB-Dev] RFC: release persistent as a standalone package

2012-06-30 Thread Hanno Schlichting
On Sat, Jun 30, 2012 at 8:02 PM, Tres Seaver tsea...@palladion.com wrote: I would like to release a '4.0.0' version of the package, and switch the ZODB trunk to pull it in as a dependency (deleting the currently included (older) copy of persistent).  One possible issue is that I have not (yet)

Re: [ZODB-Dev] Relstorage and MSQL

2012-05-08 Thread Hanno Schlichting
On 09.05.2012, at 02:34, Dylan Jay d...@pretaweb.com wrote: I know it's not all about money but if we were to sponsor development of microsoft sql server support for relstorage, is there someone who knows how and has an estimated cost and available time? It might be helpful if you could add

Re: [ZODB-Dev] all webserver threads blocking on db.open()

2012-05-07 Thread Hanno Schlichting
I think you might get better help on one of the Pyramid support channels. Your problems all seem to be related to configuring a web server in production mode, rather than database issues. From what I can tell you are dealing with hung requests. I'd look at either the paster configuration options

Re: [ZODB-Dev] Upgraded from ZODB 3.8 to 3.10, do I need to upgrade my Data.fs?

2012-04-16 Thread Hanno Schlichting
Paul Warner paul.warner at gmail.com writes: We have been running an in house application on ZODB 3.8.2 and recently upgraded to 3.10.5. We run a ZEO server with FileStorage. It seems like everything works great, no actions on our part, but I wondered if it is suggested to some how migrate

Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?

2011-07-14 Thread Hanno Schlichting
On Thu, Jul 14, 2011 at 4:12 PM, Andreas Jung li...@zopyx.com wrote: I followed the documentation at http://pastebin.com/bL0CbBm2 Wow, looks like I didn't specify an expiration date for my paste ;) If anyone feels like putting that into a more permanent place, feel free to use it. Hanno

Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?

2011-07-14 Thread Hanno Schlichting
On Thu, Jul 14, 2011 at 5:38 PM, Andreas Jung li...@zopyx.com wrote: For the sake of completeness: I found this https://mail.zope.org/pipermail/zodb-dev/2008-February/011606.html and will try it out . There's also instructions in my pastebin dump on how to do this - and quite a bit simpler

Re: [ZODB-Dev] RelStorage zodbconvert, ConflictError on Zope2 startup

2011-07-14 Thread Hanno Schlichting
On Thu, Jul 14, 2011 at 9:32 PM, Sean Upton sdup...@gmail.com wrote: ZODB.POSException.ConflictError: database conflict error (oid 0x00, class persistent.mapping.PersistentMapping, serial this txn started with 0x038fa89e48c295cc 2011-07-13 14:22:17.053148, serial currently committed

Re: [ZODB-Dev] How to fix CorruptedDataError: Error reading unknown oid. Found '' at 81036527?

2011-07-14 Thread Hanno Schlichting
On Thu, Jul 14, 2011 at 9:40 PM, Jim Fulton j...@zope.com wrote: On Thu, Jul 14, 2011 at 3:23 PM, Hanno Schlichting ha...@hannosch.eu wrote: At Jarn we've used that trick many times to repair broken internals in the intid/keyreference data structures. Do you have any theories why objects

Re: [ZODB-Dev] Relstorage Blob support and Oracle

2011-06-10 Thread Hanno Schlichting
On Fri, Jun 10, 2011 at 7:17 AM, Shane Hathaway sh...@hathawaymix.org wrote: BTW, I'd like to release a final of RelStorage 1.5, but the #@$%$ Windows tests are failing on Windows XP (and probably other versions of Windows).  I think the problem is caused by file descriptors being left open

Re: [ZODB-Dev] Occasional startup errors in serialize.py/_reconstructor

2011-05-09 Thread Hanno Schlichting
On Mon, May 9, 2011 at 1:14 PM, Andreas Jung li...@zopyx.com wrote: Occasionally I receive the following error after starting my instance after the first request. There is no way to recover out other than restarting the instance...then everything is fine. I don't know why this happens from

Re: [ZODB-Dev] Occasional startup errors in serialize.py/_reconstructor

2011-05-09 Thread Hanno Schlichting
On Mon, May 9, 2011 at 2:08 PM, Andreas Jung li...@zopyx.com wrote: Jup - basically a stripped down version of http://svn.plone.org/svn/collective/collective.examples.userdata/trunk/collective/examples/userdata/ Hhm, that code has

Re: [ZODB-Dev] Immutable blobs?

2011-05-09 Thread Hanno Schlichting
On Mon, May 9, 2011 at 2:26 PM, Laurence Rowe l...@lrowe.co.uk wrote: While looking at the Plone versioning code the other day, it struck me that it would be much more efficient to implement file versioning if we could rely on blobs never changing after their first commit, as a copy of the

Re: [ZODB-Dev] Occasional startup errors in serialize.py/_reconstructor

2011-05-09 Thread Hanno Schlichting
On Mon, May 9, 2011 at 2:34 PM, Andreas Jung li...@zopyx.com wrote: The question is more why this error is happening from time to time after the startup after the first request - this scares me a bit. I also encounter a strange issues with PTS from time to time after startup...but not as

Re: [ZODB-Dev] How to check for setting the same values on persistent objects?

2011-05-06 Thread Hanno Schlichting
On Thu, May 5, 2011 at 6:27 PM, Alexandre Garel alex.ga...@tarentis.com wrote: I'm assuming doing a general check for old == new is not safe, as it might not be implemented correctly for all objects and doing the comparison might be expensive. I know very few of ZODB internals but in Python

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-06 Thread Hanno Schlichting
On Fri, May 6, 2011 at 2:22 PM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: That's hard to do for a project that is already 8 or 9 years old, as you can see in the attached file, we've got have many cases that fall outside your limits. I've noticed, for instance, that pages that involve

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-06 Thread Hanno Schlichting
On Fri, May 6, 2011 at 10:14 PM, Shane Hathaway sh...@hathawaymix.org wrote:  From my experience, most people who want ZODB to be faster want Zope catalogs in particular to be faster.  I don't think prefetching can make catalogs much faster, though. I've spent a lot of time lately on making

Re: [ZODB-Dev] Speeding up ZODB (was redis cache for RelStorage)

2011-05-05 Thread Hanno Schlichting
On Thu, May 5, 2011 at 10:43 AM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Since we are talking about speed, does anyone have any tips on making ZODB (in general) faster? Query fewer objects from the database. Make sure you don't store lots of tiny persistent objects in the database,

[ZODB-Dev] How to check for setting the same values on persistent objects?

2011-05-04 Thread Hanno Schlichting
Hi. I tried to analyze the overhead of changing content in Plone a bit. It turns out we write back a lot of persistent objects to the database, even tough the actual values of these objects haven't changed. Digging deeper I tried to understand what happens here: 1. persistent.__setattr__ will

Re: [ZODB-Dev] How to check for setting the same values on persistent objects?

2011-05-04 Thread Hanno Schlichting
On Wed, May 4, 2011 at 1:09 PM, Laurence Rowe l...@lrowe.co.uk wrote: Persistent objects are also used as a cache and in that case code relies on an object being invalidated to ensure its _v_ attributes are cleared. Comparing at the pickle level would break these caches. So you would expect

Re: [ZODB-Dev] [RFC] ZEO: Allow non-packaged products

2011-04-29 Thread Hanno Schlichting
On Fri, Apr 29, 2011 at 3:24 PM, Vincent Pelletier vinc...@nexedi.com wrote: I need ZEO to be able to find non-packaged products for conflict resolution purposes. As ZEO AFAIK doesn't support this I gave it a quick try. I reached the works for me state, that I now would like to get feedback on.

Re: [ZODB-Dev] packing ZODB

2011-03-31 Thread Hanno Schlichting
Hi. On Thu, Mar 31, 2011 at 12:46 PM, Adam GROSZER agros...@gmail.com wrote: After investigating FileStorage a bit, I found that GC runs on objects, but pack later by transactions. That means that if there's a bigger-ish transaction, we can't get rid of it until all of it's objects are GCed

Re: [ZODB-Dev] Wrong blob returned in one of the zeo clients

2011-03-01 Thread Hanno Schlichting
On Tue, Mar 1, 2011 at 11:50 PM, Shane Hathaway sh...@hathawaymix.org wrote: On 03/01/2011 02:47 PM, Maurits van Rees wrote: - ZODB3 3.8.6-polling Blobs are stored in postgres with RelStorage.  Layout of the blob cache dir is 'zeocache'. I should have spotted this earlier. The zeocache

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Hanno Schlichting
On Mon, Feb 28, 2011 at 4:19 PM, Maurits van Rees m.van.r...@zestsoftware.nl wrote: This is with RelStorage 1.5.0-b1, with blob-dir enabled, shared-blob-dir false and a low blob-cache-size, say 10 bytes.  Using Plone 3.3.5, plone.app.blob 1.3, plone.app.imaging 1.0.1, but the same is

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Hanno Schlichting
On Mon, Feb 28, 2011 at 5:09 PM, Maurits van Rees m.van.r...@zestsoftware.nl wrote: Op 28-02-11 16:34, Martijn Pieters schreef: On Mon, Feb 28, 2011 at 16:22, Hanno Schlichtingha...@hannosch.eu  wrote: Blobs are considered experimental in ZODB 3.8. Especially the entire blob cache changed

Re: [ZODB-Dev] Advice on whether to run relstorage database and Zope on different servers

2011-02-24 Thread Hanno Schlichting
Hi. On Thu, Feb 24, 2011 at 4:20 PM, Anthony Gerrard anthonygerr...@gmail.com wrote: * I'm familiar with enterprise environments where you would have an app server and a database server but are there any advantages to putting Zope and MySQL on different servers? Database servers (be it

Re: [ZODB-Dev] Issue with Blobstorage for a third client in Plone

2011-02-11 Thread Hanno Schlichting
On Fri, Feb 11, 2011 at 11:24 PM, Andreas Mantke ma...@gmx.de wrote: I want to create a third instance (client3) in a running zeoserver that uses his own database (data.fs) and blobstorage. This third client should run a repository for extensions. The first client (and the second) is used

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Hanno Schlichting
Hi. On Thu, Jan 27, 2011 at 9:00 AM, Matt Hamilton ma...@netsight.co.uk wrote: Alas we are. Or rather, alas, ZCatalog does ;) It would be great if it didn't but it's just the way it is. If I have 300,000 items in my site, and everyone of them visible to someone with the 'Reader' role, then

Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc

2011-01-27 Thread Hanno Schlichting
On Thu, Jan 27, 2011 at 11:09 AM, Matt Hamilton ma...@netsight.co.uk wrote: Hanno Schlichting hanno at hannosch.eu writes: There still seem to be instances in which the entire set is loaded.  This could be an artifact of the fact I am clearing the ZODB cache before each ]test, which I think

Re: [ZODB-Dev] Lazy Instantiation

2011-01-23 Thread Hanno Schlichting
On Mon, Jan 24, 2011 at 1:43 AM, Leonardo Santagada santag...@gmail.com wrote: On Sat, Jan 22, 2011 at 3:23 PM, Jim Fulton j...@zope.com wrote:   a.child == None will because equality comparison will attempt to access methods on the child, which will cause it's activation. Why would it

Re: [ZODB-Dev] Some tests do not pass load's version parameter

2010-12-13 Thread Hanno Schlichting
Hi. On Mon, Dec 13, 2010 at 10:11 AM, Vincent Pelletier vinc...@nexedi.com wrote: Reading ZODB.interface, I realised the version parameter is defined as mandatory: class IStorage(Interface):   [...]    def load(oid, version): That parameter was deprecated in 3.9. Looks like the interface

Re: [ZODB-Dev] Sharing (persisted) strings between threads

2010-12-08 Thread Hanno Schlichting
On Wed, Dec 8, 2010 at 11:06 AM, Malthe Borch mbo...@gmail.com wrote: With 20 active threads, each having rendered the Plone 4 front page, this approach reduced the memory usage with 70 MB. Did you measure throughput of the system? In the benchmarks I've seen threads numbers of 3 or above will

Re: [ZODB-Dev] 32-bit vs 64-bit - RelStorage on MySQL

2010-11-18 Thread Hanno Schlichting
On Thu, Nov 18, 2010 at 5:19 PM, Leonardo Santagada santag...@gmail.com wrote: On Thu, Nov 18, 2010 at 1:47 PM, Chris Withers ch...@simplistix.co.uk wrote: On 18/11/2010 15:39, Marius Gedminas wrote: About the only noticeable difference -- other than the obvious memory growth What obvious

Re: [ZODB-Dev] Default comparison considered harmful in BTrees.

2010-10-28 Thread Hanno Schlichting
On Wed, Oct 27, 2010 at 8:56 PM, Jim Fulton j...@zope.com wrote: On Wed, Oct 27, 2010 at 1:59 PM, Hanno Schlichting ha...@hannosch.eu wrote: I suppose that depends on the application.  Was the use of None intentional, or the result of sloppy coding? I'd have to check the code. I expect

Re: [ZODB-Dev] Default comparison considered harmful in BTrees.

2010-10-27 Thread Hanno Schlichting
On Mon, Oct 25, 2010 at 11:51 PM, Jim Fulton j...@zope.com wrote: I'm inclined to treat the use of the comparison operator inherited from object in BTrees to be a bug.  I plan to fix this on the trunk. Did you mean to throw warnings for simple built-in types? I'm now getting warnings for

Re: [ZODB-Dev] Default comparison considered harmful in BTrees.

2010-10-27 Thread Hanno Schlichting
On Wed, Oct 27, 2010 at 6:45 PM, Jim Fulton j...@zope.com wrote: If not, I wonder if the existing indexes have some bad values in them that are triggering this somehow.  The relevant check is being done when loading state.  I bet you have some bad keys (e.g. None) in your data structures.

Re: [ZODB-Dev] [Zope-dev] Zope Tests: 31 OK, 19 Failed, 2 Unknown

2010-09-25 Thread Hanno Schlichting
On Fri, Sep 10, 2010 at 10:52 PM, Tres Seaver tsea...@palladion.com wrote: My attempts at diagnosis. Subject: FAILED : Zope Buildbot / zope2 slave-ubuntu64 From: jdriessen at thehealthagency.com Date: Thu Sep  9 09:31:14 EDT 2010 URL:

Re: [ZODB-Dev] RFC: deprecate transaction user and description fields in favor of extened info and simplify extended info API

2010-09-23 Thread Hanno Schlichting
On Thu, Sep 23, 2010 at 4:24 PM, Jim Fulton j...@zope.com wrote: The user and description fields are somewhat archaic. They can only be strings (not unicode) and can as easily be handled as extended info. I propose to deprecate the 'user' and 'description' attributes, and the 'setUser' and

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-21 Thread Hanno Schlichting
On Tue, Sep 21, 2010 at 3:48 PM, Jim Fulton j...@zope.com wrote: 3.8.6 is available now with this bug fix and a few others. Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like CMFEditions). So

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-21 Thread Hanno Schlichting
On Tue, Sep 21, 2010 at 4:30 PM, Jim Fulton j...@zope.com wrote: On Tue, Sep 21, 2010 at 10:00 AM, Hanno Schlichting ha...@hannosch.eu wrote: Note that the BTrees 32bit bug fix (OverflowError to TypeError) also needs some adjustments in Zope2 (IIRC in the DateIndex code) and some add-ons (like

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-21 Thread Hanno Schlichting
On Tue, Sep 21, 2010 at 5:20 PM, Jim Fulton j...@zope.com wrote: It was changes like this one http://svn.zope.org/Zope/trunk/src/Products/PluginIndexes/DateIndex/DateIndex.py?rev=115442r1=115279r2=115442 I don't understand how the BTree changes would run afoul of this. This code should

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-20 Thread Hanno Schlichting
On Mon, Sep 20, 2010 at 6:00 PM, Jim Fulton j...@zope.com wrote: Do you know that Zope 2.10 won't work with ZODB 3.9? If so, I'm curious why? Zope 2.10 includes ZODB 3.7 by default. 3.8 does work fine with it. But in 3.9 versions got removed. There was a good deal of code in Zope 2 that still

Re: [ZODB-Dev] ZODB: invalidations out of order

2010-09-20 Thread Hanno Schlichting
On Mon, Sep 20, 2010 at 8:08 PM, Jim Fulton j...@zope.com wrote: Does Zope 2.10 actually break with ZODB 3.9 if a user doesn't use versions? Or is this a matter of test failures? Last time I checked it did break. There's at least the temporary storage implementation that required changes to

[ZODB-Dev] 3.10 final?

2010-08-30 Thread Hanno Schlichting
Hi, just wondering what the state of 3.10 final is? We had a couple beta releases already. What needs to be done to move it forward and is there anything the community can help with? Hanno ___ For more information about ZODB, see the ZODB Wiki:

Re: [ZODB-Dev] Weird KeyError with OOBTree

2010-08-16 Thread Hanno Schlichting
On Mon, Aug 16, 2010 at 12:14 PM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Could this be some problem with using persistent objects as keys in a BTree? Some comparison problem? I'm not entirely sure about this, but I think using persistent objects as keys isn't supported. Looking at

Re: [ZODB-Dev] Weird KeyError with OOBTree

2010-08-16 Thread Hanno Schlichting
On Mon, Aug 16, 2010 at 2:04 PM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: I think implementing a stable hash function for your type could make this work though. From what I read, ZODB doesn't use hash functions, relying on __cmp__ instead. So, I guess I should make my class

Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python

2010-07-29 Thread Hanno Schlichting
On Thu, Jul 29, 2010 at 1:20 PM, Christian Theune c...@gocept.com wrote: ZEO in version 3.10 is supposed to work with older clients. Is it also intended to work with older clients running Python 2.4? The server needs to run on Python 2.5 due to the with statement. The docs explicitly state

Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python

2010-07-29 Thread Hanno Schlichting
On Thu, Jul 29, 2010 at 2:49 PM, Christian Theune c...@gocept.com wrote: On 07/29/2010 02:42 PM, Hanno Schlichting wrote: The docs explicitly state that Python 2.4 is no longer supported at all. Go with the times and use Python 2.6 or 2.7 ;) The docs also say that ZEO supports older clients

Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python

2010-07-29 Thread Hanno Schlichting
On Thu, Jul 29, 2010 at 2:57 PM, Christian Theune c...@gocept.com wrote: Thanks for digging this out. I'll try to find the discussion and refresh my memory. See for example your response here https://mail.zope.org/pipermail/zodb-dev/2010-April/013269.html ;-) And Andreas was the only one

Re: [ZODB-Dev] Remember date of last pack?

2010-07-26 Thread Hanno Schlichting
On Mon, Jul 26, 2010 at 5:14 PM, Christian Zagrodnick c...@gocept.com wrote: for monitoring if a storage has been packed it would be handy if it remembered the date of its last pack. An easy solution for file storage would be to create a Data.fs.packed file. Other ways do determine the date

Re: [ZODB-Dev] Remember date of last pack?

2010-07-26 Thread Hanno Schlichting
On Mon, Jul 26, 2010 at 8:12 PM, Christian Zagrodnick c...@gocept.com wrote: On 2010-07-26 17:38:10 +0200, Hanno Schlichting said: - Configure repozo to do incremental backups and keep some old backups. Configure zeopack to run regularly. Check that you have a recent full repozo backup around

Re: [ZODB-Dev] I released 3.10.0b2 this morning

2010-07-14 Thread Hanno Schlichting
On Wed, Jul 14, 2010 at 2:47 PM, Jim Fulton j...@zope.com wrote: It got a lot larger recently when someone moved a bunch of reports from the Zope 2 lists.  A lot of these bugs are obsolete. That one was me. I tried hard to only move those bugs, which were indeed pure ZODB problems. A lot of

Re: [ZODB-Dev] KeyError in BTrees.check

2010-07-13 Thread Hanno Schlichting
On Tue, Jul 13, 2010 at 11:23 AM, Suresh V. suresh...@yahoo.com wrote: While trying to run manage_cleanup on my BTreeFolder2, I get a KeyError from the classify function in Module BTrees.check: def classify(obj):     return _type2kind[type(obj)] I see that obj is None at this point.

Re: [ZODB-Dev] [Checkins] SVN: ZODB/trunk/src/ Merged zagy-lp509801 branch.

2010-07-11 Thread Hanno Schlichting
Hi Jim. This error is biting us in ZODB 3.8.5 and 3.9.5. Are you intending to backport this and the other bugfixes? Do you want help with backporting and in what way? Create bugfix branches against the older branches for your review or directly apply the patches as you reviewed them already?

Re: [ZODB-Dev] Restoring from repozo and reusing an index file?

2010-06-11 Thread Hanno Schlichting
On Fri, Jun 11, 2010 at 3:25 PM, Paul Winkler sli...@gmail.com wrote: I'm preparing to move a zope site from one host to another. We've been planning to use repozo backups to copy the filestorage from the old host to the new one.  I'd like to figure out how to minimize downtime while not

Re: [ZODB-Dev] Changing the pickle protocol?

2010-05-23 Thread Hanno Schlichting
On Sun, May 23, 2010 at 5:45 PM, Jim Fulton j...@zope.com wrote: On Sat, May 22, 2010 at 12:30 PM, Hanno Schlichting ha...@hannosch.eu wrote: - The code to make the protocol configurable on all levels (storage, index, persistent cache, ...) is large and ugly, I'm puzzled.  Why were changes so

Re: [ZODB-Dev] ZEO and access permissions

2010-05-22 Thread Hanno Schlichting
On Sat, May 22, 2010 at 2:17 PM, Nitro ni...@dr-code.org wrote: ZEO already supports authenticated logins. Based on the login I'd like people to be able to access some objects and deny access to others. First I thought I'd do the access restrictions on the application level. That's the only

Re: [ZODB-Dev] Changing the pickle protocol?

2010-05-22 Thread Hanno Schlichting
Hi. Following up on my idea of using pickle protocol 2. I implemented this in a fully configurable fashion on a branch, mainly to ease benchmarking and testing of the different variants. My conclusions (maybe for future reference): - There's no significant win of just switching the pickle

Re: [ZODB-Dev] Storage iterators and IndexError

2010-05-15 Thread Hanno Schlichting
On Fri, May 14, 2010 at 8:43 PM, Jim Fulton j...@zope.com wrote: There's a test for storage iterators that verifies that they raise a special exception that extends StopIteration and IndexError. This makes storage iterators a bit harder to implement than necessary. Does anyone know of a

Re: [ZODB-Dev] repozo full backup and index files

2010-05-14 Thread Hanno Schlichting
On Fri, May 14, 2010 at 10:24 AM, Christian Theune c...@gocept.com wrote: Hmm. If the full backup is just a regular FS file then you could start with the naive approach and just open/close it once after performing a backup as that would create the index file. Sure. That would be an easy but

Re: [ZODB-Dev] repozo full backup and index files

2010-05-14 Thread Hanno Schlichting
On Fri, May 14, 2010 at 4:33 PM, Tres Seaver tsea...@palladion.com wrote: I would be willing to make a stab at this, if we can hold off on a 3.10.0 beta until I've had a chance to try it. Oh, awesome! Best possible outcome I could hope for - someone else wants to do the work for me ;-) Hanno

[ZODB-Dev] repozo full backup and index files

2010-05-12 Thread Hanno Schlichting
Hi. I was wondering if there's a specific reason why repozo does not backup or create index files, whenever it does a full backup. I understand that creating index files for incremental backups is probably hard, but for a full backup it should be possible. Recreating the index file after

Re: [ZODB-Dev] ZEO monitor server

2010-04-30 Thread Hanno Schlichting
On Fri, Apr 30, 2010 at 8:07 PM, Jim Fulton j...@zope.com wrote: Does anyone use the monitor server? Would anyone object if it went away, to be replaced by a server method on the standard ZEO protocol? I don't know anyone using the monitor server. So changing it is not a problem here. We only

[ZODB-Dev] Changing the pickle protocol?

2010-04-28 Thread Hanno Schlichting
Hi. The ZODB currently uses a hardcoded pickle protocol one. There's both the more efficient protocol two and in Python 3 protocol 3. Protocol two has seen various improvements in recent Python versions, triggered by its use in memcached. I'd be interested to work on changing the protocol. How

Re: [ZODB-Dev] Changing the pickle protocol?

2010-04-28 Thread Hanno Schlichting
On Wed, Apr 28, 2010 at 5:11 PM, Jim Fulton j...@zope.com wrote: Do you know of specific benefits you expect from protocol 2? Any specific reasons you think it would be better in practice? I have just seen some ongoing work on pickles in recent times, for example from the Python 2.7 what's

Re: [ZODB-Dev] Problems in ZEO pack in 3.9.x?

2010-04-27 Thread Hanno Schlichting
On Mon, Apr 26, 2010 at 7:44 PM, Jim Fulton j...@zope.com wrote: Hm.  I don't know if it was intentional to ignore POSKeyErrors in the old pack code.  It seems like a bad idea to me. Yep, I was wondering if that was a conscious design choice or just accidental behavior. What do folks think

Re: [ZODB-Dev] Problems in ZEO pack in 3.9.x?

2010-04-27 Thread Hanno Schlichting
On Tue, Apr 27, 2010 at 4:08 PM, Jim Fulton j...@zope.com wrote: On Tue, Apr 27, 2010 at 6:29 AM, Hanno Schlichting ha...@hannosch.eu wrote: But doing GC on an inconsistent state is probably a bad idea. Then I think the current behavior is correct. You can now disable GC using the pack-gc

Re: [ZODB-Dev] ZODB 3.9.5 has been released

2010-04-23 Thread Hanno Schlichting
On Fri, Apr 23, 2010 at 11:30 PM, Jim Fulton j...@zope.com wrote: See http://pypi.python.org/pypi/ZODB3/3.9.5 Cool! I added the Python 2.6 / Win64 egg. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/

Re: [ZODB-Dev] Using zodb and blobs

2010-04-14 Thread Hanno Schlichting
On Wed, Apr 14, 2010 at 11:52 AM, Nitro ni...@dr-code.org wrote: Yes, in my case it's nothing critical or related to money. If there's a hardware outage a day of work is lost at worst. In case of corruption (which can happen also without fsync as data within the file can just be garbled) you

Re: [ZODB-Dev] DemoStorage stacking in zope.testing layers

2010-04-05 Thread Hanno Schlichting
Hi. On Mon, Apr 5, 2010 at 4:36 PM, Martin Aspeli optilude+li...@gmail.com wrote: What is the correct way to use DemoStorage stacking in test layers? Not sure if this helps, but zope.app.testing's functional.py [1] did something similar here (stacking DemoStorages). Maybe the

Re: [ZODB-Dev] Understanding the ZODB cache-size option

2010-03-22 Thread Hanno Schlichting
On Mon, Mar 22, 2010 at 7:05 PM, Jeff Shell j...@bottlerocket.net wrote: Are there any metrics about how to set the ZODB 'cache-size' (or cache-size-bytes) option? We've been using '5000' (arbitrarily chosen) for our Zope 3.4-ish app servers. We have access to zc.z3monitor which can output

Re: [ZODB-Dev] ZODB on Windows 64bit?

2010-01-17 Thread Hanno Schlichting
On Sun, Jan 17, 2010 at 2:51 PM, Jim Fulton j...@zope.com wrote: On Fri, Jan 15, 2010 at 10:01 PM, Hanno Schlichting ha...@hannosch.eu wrote: P.S. Running the tests on the checkout of the 3.9.4 tag gives me two test failures, of which one is a test cleanup problem caused by the other. Does

Re: [ZODB-Dev] ZODB on Windows 64bit?

2010-01-16 Thread Hanno Schlichting
On Sat, Jan 16, 2010 at 9:05 AM, Adam GROSZER agros...@gmail.com wrote: Please don't stop with those binaries. There are some win32 users out here ;-) Sure. But Jim is building the 32bit binaries himself. So there's no need for me to do that. As he seems to be building things on WinXP, he

[ZODB-Dev] ZODB on Windows 64bit?

2010-01-15 Thread Hanno Schlichting
Hi. I recently updated my Windows build environment to a 64bit Windows 7 environment with cross compilation support for 32/64 bit. I created and uploaded Windows binary eggs for all Zope 2.12.3 dependencies in both flavors for Python 2.6. The last one missing with a 64bit flavor is the ZODB3

Re: [ZODB-Dev] An interface for broken objects?

2010-01-08 Thread Hanno Schlichting
On Sun, Jan 3, 2010 at 5:38 PM, Jim Fulton j...@zope.com wrote: I suggest including these in the interface. Thanks for fixing this yourself and merging. Sorry, I didn't get around to extend things myself. Hanno ___ For more information about ZODB, see

[ZODB-Dev] An interface for broken objects?

2009-12-31 Thread Hanno Schlichting
Hi. We currently have a package called zope.broken whose entire real content is: import zope.interface class IBroken(zope.interface.Interface): Marker interface for broken objects This is used for example by zope.container, which won't try to set __name__ and __parent__ pointers on

Re: [ZODB-Dev] An interface for broken objects?

2009-12-31 Thread Hanno Schlichting
On Thu, Dec 31, 2009 at 6:03 PM, Jim Fulton j...@zope.com wrote: On Thu, Dec 31, 2009 at 11:46 AM, Hanno Schlichting ha...@hannosch.eu wrote: Is this something that could be put into the ZODB itself and have the ZODB.broken.Broken class directly implement this interface? +1. although

Re: [ZODB-Dev] undo (and storage interface) brokenness

2009-12-23 Thread Hanno Schlichting
On Wed, Dec 23, 2009 at 9:26 PM, Jim Fulton j...@zope.com wrote: Undo is broken in a number of ways. Does anyone care?  Does anyone use undo? Speaking from the Zope2/Plone crowd: I'm using it during development with a local file storage at times. In a real ZEO production setup, undo in Plone

Re: [ZODB-Dev] undo (and storage interface) brokenness

2009-12-23 Thread Hanno Schlichting
On Thu, Dec 24, 2009 at 1:34 AM, Martin Aspeli optilude+li...@gmail.com wrote: Hanno Schlichting wrote: Throwing away all changes done in the meantime is usually not practical. ... although sometimes it is preferable to a full backup restore or living with whatever it is you want to restore

Re: [ZODB-Dev] ZEO and blobs: missing tmp file breaks transaction on retry

2009-11-13 Thread Hanno Schlichting
On Fri, Nov 13, 2009 at 5:40 PM, Jim Fulton j...@zope.com wrote: On Fri, Nov 13, 2009 at 10:18 AM, Mikko Ohtamaa mi...@redinnovation.com wrote: Unfortunately the application having the issues is Plone 3.3. ZODB 3.9 depends on Zope 2.12 so, right? ZODB does depend on Zope anything. :)

Re: [ZODB-Dev] cache-size-bytes usable?

2009-11-07 Thread Hanno Schlichting
On Sat, Nov 7, 2009 at 6:34 PM, Andreas Jung li...@zopyx.com wrote: Am 07.11.09 18:25, schrieb Hanno Schlichting: Has anyone used the bytes limited cache in production yet and has any experience with it? I would assume that this feature is working since it comes from the Haufe Zope fork

Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

2009-05-27 Thread Hanno Schlichting
Chris Withers wrote: Hanno Schlichting wrote: Nope. DateTime objects are plain old-style classes and don't inherit from persistent.*. Hmm, oh well, my bad... In that case it must just be that their pickled form is huge compared to an int ;-) Sure: len(cPickle.dumps(DateTime.DateTime

Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

2009-05-27 Thread Hanno Schlichting
Marius Gedminas wrote: On Wed, May 27, 2009 at 12:17:36PM +0200, Hanno Schlichting wrote: Chris Withers wrote: Hanno Schlichting wrote: They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call

Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

2009-05-26 Thread Hanno Schlichting
of a separate database object. +lots Hanno Schlichting recently posted a nice graph showing the persistent structure of a Plone Page object and it's 9 (!) sub-objects. http://blog.hannosch.eu/2009/05/visualizing-persistent-structure-of.html That graph isn't quite correct ;-) workflow_history

[ZODB-Dev] Blobs without shared-blob directory

2009-05-15 Thread Hanno Schlichting
Hi. I have a configuration with one ZEO database server (ZODB 3.8.1) and two physical servers housing four single-threaded ZEO clients each. There's some 15gb of blob data in the (Plone) site. Setting up and maintaining NFS to share direct access to the blob data is beyond what I can support in

Re: [ZODB-Dev] Blobs without shared-blob directory

2009-05-15 Thread Hanno Schlichting
Jim Fulton wrote: On May 15, 2009, at 6:22 AM, Hanno Schlichting wrote: Am I correct to assume that sharing the blob-cache between the four processes on the same physical machine is not supported, as these don't have any mechanism to synchronize read/write activity to the blob- cache

Re: [ZODB-Dev] ZODB 3.9.0b1 released

2009-05-05 Thread Hanno Schlichting
Jim Fulton wrote: I've just released the first beta of ZODB 3.9. Awesome :) I thought there were some outstanding issues with the cache-size-bytes functionality. Got all of these resolved / reviewed by now? Thanks, Hanno ___ For more information

Re: [ZODB-Dev] ZODB 3.9

2009-04-11 Thread Hanno Schlichting
Chris Withers wrote: Hanno Schlichting wrote: Just be aware that ZODB 3.9 is not compatible with any stable Zope 2.x release. It only works and is required for Zope 2.12. It can be made to work with prior versions of Zope2 but that is a mild pain. What are the problems with using ZODB 3.9

Re: [ZODB-Dev] ZODB 3.9

2009-04-07 Thread Hanno Schlichting
Alan Runyan wrote: Is a ZODB 3.9 beta around the corner? Just be aware that ZODB 3.9 is not compatible with any stable Zope 2.x release. It only works and is required for Zope 2.12. It can be made to work with prior versions of Zope2 but that is a mild pain. Hanno

Re: [ZODB-Dev] Can I use RelStorage with ZODB3.9.0

2009-03-06 Thread Hanno Schlichting
Wichert Akkerman wrote: On 3/6/09 6:27 AM, eastxing wrote: About one month ago, I asked a question about 'ZODB pack' and got suggestions to update to new ZODB version. It took me one month to update my site from Plone2.5.5(with Zope2.9.6-final,ZODB3.6.2) to Plone3.1.7(with

Re: [ZODB-Dev] ZODB alternate serialization format patch

2009-01-16 Thread Hanno Schlichting
Shane Hathaway wrote: Hanno Schlichting wrote: Shane Hathaway wrote: In addition to making the data format pluggable, the patch puts most of ZODB's dependencies on the cPickle module in one place, so now if we decide to improve our usage of the cPickle module, we can make that change

Re: [ZODB-Dev] ZODB alternate serialization format patch

2009-01-15 Thread Hanno Schlichting
Shane Hathaway wrote: I have just created a patch for ZODB that makes the object serialization format pluggable, meaning that it should now be possible to write Python code that causes ZODB to store in formats other than Python pickles. Cool :) In addition to making the data format

Re: [ZODB-Dev] Minimal changes for Plone 3.1.6 running on ZODB 3.8.1 ?

2008-11-17 Thread Hanno Schlichting
Russ Ferriday wrote: In a warty old application, I'm hitting memory constraints that I think the 3.8.1 changes in ZODB might fix. We are running both Plone 3.0 and 3.1 in production with ZODB 3.8.1 on two quite large sites (one has RelStorage, the other one blob storage). I'd consider it

Re: [ZODB-Dev] Optimize BTree node sizes?

2008-11-04 Thread Hanno Schlichting
Jim Fulton wrote: On Nov 4, 2008, at 12:12 PM, Benji York wrote: On Tue, Nov 4, 2008 at 12:01 PM, Jim Fulton [EMAIL PROTECTED] wrote: A few months back, there was a lot of discussion here about BTree performance. I got a sense that maximum BTree-node and bucket sizes should be increased.

Re: [ZODB-Dev] 3.8.1b8 released and would like to release 3.8.1 soon

2008-09-30 Thread Hanno Schlichting
Dieter Maurer wrote: Wichert Akkerman wrote at 2008-9-24 09:44 +0200: Jim Fulton wrote: I'd appreciate it if people would try it out soon. I can say that the combination of 3.8.1b8 and Dieter's zodb-cache-size-bytes patch does not seem to work. With zodb-cache-size-bytes set to 1

Re: [ZODB-Dev] 3.8.1b8 released and would like to release 3.8.1 soon

2008-09-30 Thread Hanno Schlichting
Jim Fulton wrote: I'd appreciate it if people would try it out soon. Besides the RelStorage site where we were running into problems as mentioned in the other thread, we also use the ZODB 3.8 branch (beta8 plus the first two beta fixes) in a smaller site with blob storage for about a week in

  1   2   >