Re: [ZODB-Dev] Status of gocept.zeoraid
On 10/20/2010 07:11 PM, Chris Withers wrote: On 20/10/2010 17:03, Sylvain Viollon wrote: One of my customers would like to use gocept.zeoraid in production. So him, and I have some questions: - Is it stable enough to be used in production ? No. Ack. It's called beta. I do need people interested in using it to try it out *seriously*, though. Chris hit quite a wall even when testing it for a while. I've got some data from that but the lesson is: don't think it's ready for (your) production if you haven't run it in (your) production successfully. - Does it bring speed improvements over regular ZEO setup (I don't think so) ? No. It is currently 5 times slower when tested with zodbshootout. And there will likely never will be speed improvements over regular ZEO. I was a bit shocked when I looked at the actual slowness, but then again, I didn't think about performance at all. (In standalone mode there is a slight chance that you can load read-requests off to multiple backend servers. Haven't tried it, though. And it won't be available in co-op mode. - Does it work with blobs ? I will have a blob directory shared between different servers. Can I use it to access and edit directly the blob in that directory ? Don't know. Nope. The blob directory (even with regular ZODB) is a managed area like the data in your Data.fs. (You wouldn't go editing that data with your editor, would you?) Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] tid vs. serial
On 10/18/2010 03:51 PM, Jim Fulton wrote: On Mon, Oct 18, 2010 at 8:53 AM, Vincent Pelletiervinc...@nexedi.com wrote: Hi. During my presentation of NEO at DZUG, Cristian Theune told me that I might be confusing tids and serials. And indeed, I could not define the difference between them. I wonder what he meant. Me too, I still had a distinction in my head back from the 2005 sprint that they are somewhat similar but different. Looks like I never got the signal that we actually consider them to be the same. At least that's clear now. :) -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 10/14/2010 09:58 AM, Chris Withers wrote: On 14/10/2010 02:40, Darryl Dixon - Winterhouse Consulting wrote: ...and fsdump appears to have no way of saying show me the last few transactions in this 35GB file :-( Yeah it's pretty verbose. I usually just pump the output to a text file ...yeah, but a 35GB zodb is going to result in, what, a 120? 200? GB text file. I don't think I actually have that much disk space lying around on these servers, let alone how long fsdump would take to produce it ;-) I wonder how hard it'd be to add some kind of start-at-the-end feature to fstail? With the new iterators that start scanning from the back that shouldn't be too hard. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb 3.9.6 performance tuning?
On 10/12/2010 05:14 PM, Chris Withers wrote: Hi, After recently moving a medium-sized customer project from Zope 2.9.8 to Zope 2.12.7 sitting on ZODB 3.9.6 by way of zeoraid b7, I'm now seeing some fairly weird and bad behaviour that we didn't have before. So, everything was fine after the migration. However, today I turned on a batch job which pumps content into the storage. This appears to be *much* slower than on Zope 2.9.8 (with its included ZODB release) and also appears to cause the web clients to be slow enough when its running that they start dropping out of the load balancer pool. How can I debug what's going on? Is there any way to get diagnostics out of a 3.9.6 zeo server to find out what it's up to? I think the trace log contains timing information. There's also the monitor. Haven't had to do performance tuning within the ZEO server so far. Most times I was basically limited by either network or disk. cheers, Chris PS: If it helps, it looks like the zeoraid process is using an order of magnitude more cpu than the zeo process, according to top, but even that's only 30% of the CPU load... Has ZEORaid been in place before the update, too? Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.10.0 released
On 10/08/2010 08:56 PM, Jim Fulton wrote: ZODB 3.10.0 is available: W00t! -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.10.0b6 traceback
On 09/10/2010 02:25 PM, Jim Fulton wrote: On Fri, Sep 10, 2010 at 2:40 AM, Wichert Akkermanwich...@wiggy.net wrote: I just got the following traceback from a system running 3.10.0b6: As I mentioned before, this looks like a bug. But mainly I want to say: thanks for testing the beta! I hope we're getting close to a final release, and, of course, testing the betas is very helpful. :) Did the tracebacks make it into the bug database? If yes, can someone provide references to them in the mailing list FTR? Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.10 final?
On 08/30/2010 11:22 PM, Jim Fulton wrote: On Mon, Aug 30, 2010 at 3:08 PM, Hanno Schlichtingha...@hannosch.eu wrote: Hi, just wondering what the state of 3.10 final is? I was worried that you were going to ask me about that. :) I haven't done much with it in a while. I need to get back to it. This last weekend I discovered some long-standing (like forever) problems I need to deal with that could explain some of the odd BTree issues we've seen over the years, where certain items become unreachable. We had a couple beta releases already. Yup, I noticed. What needs to be done to move it forward and is there anything the community can help with? Aside from the new issues, the main thing that needs to be done is to review the tracker and decide if there are any issues that need to be dealt with before the final release. The community could help by gardening the tracker -- especially since the tracker recently got a bunch of old issues from the Zope 2 tracker. :) Sounds like a job I'd like for the upcoming sprint after the conference. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python
On 07/29/2010 02:42 PM, Hanno Schlichting wrote: On Thu, Jul 29, 2010 at 1:20 PM, Christian Theunec...@gocept.com wrote: ZEO in version 3.10 is supposed to work with older clients. Is it also intended to work with older clients running Python 2.4? The server needs to run on Python 2.5 due to the with statement. The docs explicitly state that Python 2.4 is no longer supported at all. Go with the times and use Python 2.6 or 2.7 ;) The docs also say that ZEO supports older clients. The application that I currently have in mind has no chance of moving from Python 2.4 but probably would benefit from a threaded ZEO server which I'd like to verify. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python
On 07/29/2010 02:55 PM, Hanno Schlichting wrote: On Thu, Jul 29, 2010 at 2:49 PM, Christian Theunec...@gocept.com wrote: On 07/29/2010 02:42 PM, Hanno Schlichting wrote: The docs explicitly state that Python 2.4 is no longer supported at all. Go with the times and use Python 2.6 or 2.7 ;) The docs also say that ZEO supports older clients. The application that I currently have in mind has no chance of moving from Python 2.4 but probably would benefit from a threaded ZEO server which I'd like to verify. I think we discussed the Python version support policy on this list in regard to the changes to exception classes. It's documented pretty clearly: ZODB 3.10 requires Python 2.5 or later. Note -- When using ZEO and upgrading from Python 2.4, you need to upgrade clients and servers at the same time, or upgrade clients first and then servers. Clients running Python 2.5 or 2.6 will work with servers running Python 2.4. Clients running Python 2.4 won't work properly with servers running Python 2.5 or later due to changes in the way Python implements exceptions. Thanks for digging this out. I'll try to find the discussion and refresh my memory. So the ZODB 3.10 server requires 2.5. You cannot run clients with Python 2.4 with a server running 2.5. Therefor you will have to update the application at some point. Darn. That kinda invalidates the point of supporting older clients, though. :/ -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python
On 07/29/2010 03:03 PM, Hanno Schlichting wrote: On Thu, Jul 29, 2010 at 2:57 PM, Christian Theunec...@gocept.com wrote: Thanks for digging this out. I'll try to find the discussion and refresh my memory. See for example your response here https://mail.zope.org/pipermail/zodb-dev/2010-April/013269.html ;-) And Andreas was the only one responding to the Python 2.5 upgrade proposal at https://mail.zope.org/pipermail/zodb-dev/2009-December/013085.html Yeah, looks like I shot myself in the foot with that response. :) -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO in v3.10 with older clients and older Python
On 07/29/2010 03:02 PM, Jim Fulton wrote: On Thu, Jul 29, 2010 at 7:20 AM, Christian Theunec...@gocept.com wrote: ZEO in version 3.10 is supposed to work with older clients. Is it also intended to work with older clients running Python 2.4? No, see: https://mail.zope.org/pipermail/zodb-dev/2010-April/013268.html and, especially: https://mail.zope.org/pipermail/zodb-dev/2010-April/013269.html Yup, found it. Dug my own hole there ... -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
On 06/18/2010 01:17 PM, Chris Withers wrote: Christian Theune wrote: and a mysterious bug WRT oid generation This one's new to me, where can I find out more? The one you reported quite a while ago that ended in mixed-up OID results. Have you got a url to the tracker issue? I think it was this one: https://bugs.edge.launchpad.net/gocept.zeoraid/+bug/485976 Okay, but again, this one seems to be mitigated by only running one zeoraid server at a time, right? Not completely sure as I didn't find the cause of the OIDs getting out of sync last time. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
On 06/12/2010 09:47 AM, Chris Withers wrote: Christian Theune wrote: On 06/11/2010 05:50 PM, Alan Runyan wrote: I suppose you could look at zeoraid. Hadn't thought of that, will look at it, thanks! Anyone using ZEORaid in production? I hope not. There's still open issues with split brain situations IIUC, that can be mitigated by only having one zeoraid server live at a time? Yes. Our goal is to mitigate even splits by making the system read-only and manually allowing to specify which parts are active in that case. An automatic voting procedure would then require more than 2 participants, but with even splits you at least won't fail horribly. and a mysterious bug WRT oid generation This one's new to me, where can I find out more? The one you reported quite a while ago that ended in mixed-up OID results. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
On 06/11/2010 05:50 PM, Alan Runyan wrote: I suppose you could look at zeoraid. Hadn't thought of that, will look at it, thanks! Anyone using ZEORaid in production? I hope not. There's still open issues with split brain situations and a mysterious bug WRT oid generation that I hope to hunt down in August. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Notes on using wrapper storages for record transformation
On 05/18/2010 10:08 PM, Jim Fulton wrote: The design of ZODB views pickles as opaque to storages. Storages primarily store strings. I have always assumed that writing wrapper storages for compression or encryption would be easy. I was wrong. :) Storages sometimes need to know about pickles: A. When doing garbage collection, a storage (or garbage collector) needs to extract object references. B. When doing conflict resolution, the conflict-resolution code needs to be able to extract state from object records. C. When doing blob-aware iteration, a storage needs to be able to recognize blob records. D. When raising conflict errors, the exception instances *want* to get the object class name from the database record. (A) was dealt with by the existing way the referencesf function was passed into storage pack functions. Wow, good to know about those! Thanks for the info! I dealt (B) and (C) by expanding and renaming the interface of objects passed to the storage registerDB call. These objects now provide an up-call mechanism from a storage to anything that might wrap it, including databases, storage servers, and intermediate wrapping layers. I punted on (D). You can't always get what you want. :) I'm mostly convinced what I've done is reasonable, although I'm just a little uncomfortable. It has always been the theory that storages can't count on the contents of database records. Now that is reality. I've decided I really want to provide a database compression option. I could have built something directly into ZODB or used a wrapper storage. I chose the later because it was more flexible. I think I would have run into the same issues without a wrapper storage. Soon I expect to release a wrapper storage implementation that provides record compression. Cool! :) -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storage iterators and IndexError
On 05/14/2010 08:43 PM, Jim Fulton wrote: The file storage iterator was implemented before Python had iterators. (Actually, Python previously had an iterator based on a corner of the sequence protocol, which FileStorage used.) There's a test for storage iterators that verifies that they raise a special exception that extends StopIteration and IndexError. This makes storage iterators a bit harder to implement than necessary. Does anyone know of a reason why we should have to raise a special error that raises IndexErrors? My first thought was backwards compatibility to something and the docstring of the StorageStopIteration says so but doesn't tall for what. Sounds like support for old clients but I can't say for which use case. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Passing thought: It would be interesting at the application level to have a measure of database record size
On 05/12/2010 01:11 PM, Jim Fulton wrote: It occurs to me that it would be useful, at the application level, to have some measure of an object's record size. This would, for example, be a better basis for OOBucket splits than item count. I thought we gained such an _p_ attribute as an indicator for the size-based cache? -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO monitor server
On 04/30/2010 08:07 PM, Jim Fulton wrote: ZEO servers can optionally listen on a separate monitor port. If you telnet to the monitor port, you get a dump of server statistics. The statistics are useful, but I don't really see why this should be a dedicated server. IMO, it would be just as useful to have a server method to be called from a ZEO client to get server statistics as a Python dictionary. Managing yet another port is a pain. The implementation is pain to maintain. Does anyone use the monitor server? Would anyone object if it went away, to be replaced by a server method on the standard ZEO protocol? I keep meaning to use it and I think I'd do that sooner if it became part of the standard protocol. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
On 04/15/2010 06:00 PM, Adam GROSZER wrote: Hello Christian, Wednesday, April 14, 2010, 8:30:50 AM, you wrote: CT I don't think the transfer rate is actually that interesting. For small CT but many transactions the seek time/spinning speed should have the CT limiting influence. CT I've run the attached script a couple of times on my notebook, here's CT the results: CT 0.11 909.090909091 CT 0.15 666.7 CT 0.2 500.0 CT 0.07 1428.57142857 CT 0.07 1428.57142857 CT 0.14 714.285714286 CT The initial runs are a bit lower as they were interfered with by other CT applications writing to the disk. CT It's a notebook w/ Intel P9600, Seagate 7.2k SATA drive, 4GB RAM, Ubuntu CT 10.04, linux 2.6.32, ext4 Something is wrong with the seek time. Trying it on an Intel G1 80G SSD goes hardly over a 1000. Well, sounds like a broken fsync then. :/ -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development smime.p7s Description: S/MIME Cryptographic Signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
On 04/14/2010 03:30 AM, Nitro wrote: Am 14.04.2010, 04:39 Uhr, schrieb Tim Peterstim.pet...@gmail.com: [Nitro] ... I wonder if _commit is really *that* slow Six years ago I timed factor-of-100 speed differences due to using MS _commit() on WinXP at the time: https://mail.zope.org/pipermail/zodb-dev/2004-July/007720.html or if there's another (faster) function which can be called... No MS function that does the same thing. Seems like ZODB has a long history of discussions on this matter: http://www.mail-archive.com/zodb-dev@zope.org/msg01874.html There's even a proposal for improvement in that thread, also on the wiki: http://wiki.zope.org/ZODB/FsyncBehaviourSetting What I don't really get is why you should never use None on windows. As far as I can judge from the various transaction rates in the thread Tim mentioned, fsync is just a no-op on linux anyways (depending on the specific file system of course). I'm pretty sure it's not. IIRC fsync is defined by POSIX and absolutely requires the implementor to flush data physically to disk ensuring its persistency. If that doesn't hold true then all transactions are borked. I've seen virtualised environments like VMWare ESX lie about fsyncs from a virtual hardware perspective. A similar issue with breaking fsync was the discussion around ext4 on notebooks with a only actually flush discs every 10 seconds. In the end it really depends on what you need your data for. If I'd store the information that a customer paid me 50 EUR for something and I presented a screen to him that told him he'll receive some good for that then I'd rather stick with compliant transactions. I am almost tempted to do os.fsync = lambda fd: 0 and rely on yesterday's backup. 0.49 j/k. j/k? -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Testing gocept.zeoraid
Hi, On 03/22/2010 03:09 PM, Sylvain Viollon wrote: Hello, I am currently testing gocept.zeoraid, and this seems to work fine. However, I have few questions: [...] - I tested like my clumsy customers would use it (that means doing things in the wrong order for instance). I did a similar setup that the one located in the Zope SVN: two ZEO server as backend, and two ZEO Raid server on top of it, connected to two Zope clients. I stop one backend ZEO server. I did raid-manage status, and details and it work fine. After, I executed raid-manage recover 1, 1 being the backend ZEO server I previously stopped, and did details again. I have 'recovering: ???'. Which is ok since I didn't start again my backend ZEO server. But if I start it after, the status doesn't change. If I do again raid-manage recover 1, the status stays the same (and the command display None). I think this is bad, as I can't expect my customer to always do things in the correct step (that would be too easy). Done in the correct order that seems to work fine. But I need to do more testing. Yeah, sounds like a bug. Do you mind reporting that into launchpad? Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeoraid + plone 3.3.4
On 02/18/2010 03:42 PM, Diego Acevedo wrote:
On Thu, Feb 18, 2010 at 11:25 AM, Christian Theune c...@gocept.com
mailto:c...@gocept.com wrote:
On 02/18/2010 03:22 PM, Diego Acevedo wrote:
On Wed, Feb 17, 2010 at 5:20 PM, Christian Theune c...@gocept.com
mailto:c...@gocept.com
mailto:c...@gocept.com mailto:c...@gocept.com wrote:
On 02/17/2010 06:15 PM, Diego Acevedo wrote:
Hello everybody. I have a problem with zeoraid + plone (I
ask for the
problem in the general questions in Plone, and they say me
that i
could
ask here)...
I can do work zeoraid and plone for separate, but if I try to
connect my
zope client instance with the zeoraid, its dont work.
I try different configurations:
zope client - zeoraid - zeoservers;
zope client - zeoserver - zeoraid - zeoservers
etc
I think that the problem is the ZODB, because zeoraid
(gocept.zeoraid)
work with ZODB3 = 3.9, whereas Plone 3.3.4 work with Zope
2.10, and
this with ZODB 3.7.1...
so, the question is, Did someone get that zeoraid and plone work
together??
The correct chain of configuration is:
Zope Server - ZEOServer with a RAIDStorage - other backends
(typically
the second row of ZEO servers)
The correct ZODB versions are:
Zope Server = ZODB 3.6+
ZEO Servers w/ and w/o ZEORAID = ZODB 3.9
If you have those configurations and they don't work, then a
specific
error report would help.
Christian
--
Christian Theune · c...@gocept.com mailto:c...@gocept.com
mailto:c...@gocept.com mailto:c...@gocept.com
gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale)
· germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
mailto:ZODB-Dev@zope.org mailto:ZODB-Dev@zope.org
mailto:ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Oh... nice. thank you, but now I am a little confused...
I have the next configuration right now (briefly):
ZeoRaid:
zeo1 address 8100
zeo2 address 8101
zeoraid address 8200 (and inside raidstorage server 8100; server
8101)
Plone (it is a zeo install, not a stand alone):
zeo-address 8100
That must point to 8200, otherwise Plone will only talk to zeo1 and thus
zeo1 and zeo2 will diverge causing ZEORaid to become inconsistent.
Christian
--
Christian Theune · c...@gocept.com mailto:c...@gocept.com
gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org mailto:ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
I tested it too, but i got this error (for plone client):
2010-02-18 11:33:15 ERROR ZEO.zrpc (3334) CW: error in testConnection
(('127.0.0.1', 8200))
Traceback (most recent call last):
File
/home/gdlk/Plone/Zope-2.10.7-final-py2.4/lib/python/ZEO/zrpc/client.py, line
454, in test_connection
self.preferred = self.client.testConnection(self.conn)
File
/home/gdlk/Plone/Zope-2.10.7-final-py2.4/lib/python/ZEO/ClientStorage.py,
line 442, in testConnection
stub.register(str(self._storage), self._is_read_only)
File
/home/gdlk/Plone/Zope-2.10.7-final-py2.4/lib/python/ZEO/ServerStub.py,
line 74, in register
self.rpc.call('register', storage_name, read_only)
File
/home/gdlk/Plone/Zope-2.10.7-final-py2.4/lib/python/ZEO/zrpc/connection.py,
line 650, in call
raise inst # error raised by server
ValueError: unknown storage: 1
Storage1 is define in zeo1 and zeo2...
You have to use the name you gave to the raidstorage.
Also: after the raid became inconsistent, you better get zeo1 and zeo2
back to the same state or run a recovery.
Christian
--
Christian Theune · c...@gocept.com
gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development
___
For more
Re: [ZODB-Dev] zeoraid + plone 3.3.4
On 02/17/2010 06:15 PM, Diego Acevedo wrote: Hello everybody. I have a problem with zeoraid + plone (I ask for the problem in the general questions in Plone, and they say me that i could ask here)... I can do work zeoraid and plone for separate, but if I try to connect my zope client instance with the zeoraid, its dont work. I try different configurations: zope client - zeoraid - zeoservers; zope client - zeoserver - zeoraid - zeoservers etc I think that the problem is the ZODB, because zeoraid (gocept.zeoraid) work with ZODB3 = 3.9, whereas Plone 3.3.4 work with Zope 2.10, and this with ZODB 3.7.1... so, the question is, Did someone get that zeoraid and plone work together?? The correct chain of configuration is: Zope Server - ZEOServer with a RAIDStorage - other backends (typically the second row of ZEO servers) The correct ZODB versions are: Zope Server = ZODB 3.6+ ZEO Servers w/ and w/o ZEORAID = ZODB 3.9 If you have those configurations and they don't work, then a specific error report would help. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.10.0a1 released
Hi, On 02/08/2010 03:38 PM, Jim Fulton wrote: I've just released 3.10.0a1: http://pypi.python.org/pypi/ZODB3/3.10.0a1#a1-2010-02-08 The main focus of this release is ZEO server performance. For realistic benchmarks involving data stored on magnetic disk (as opposed to SSD or RAM), I've seen speedups of 4x or more, primarily by making the storage server multi- threaded. (BTW, speaking of SSD, we did a test recently using a combination of SSD and magnetic disk. The database was around 1TB in size, with around 900GB being blob data. We stored the non-blob data on an SSD and blob data on a 14-spindle ISCSI magenetic storage systems. The test involved playing back database requests recorded in production. Mean load times with a 3.9 storage server without SSD were 73 milliseconds. Using SSD for the non-blob data reduced this to 20 milliseconds. Using the new ZEO implementation without SSD reduced the mean load times to 16 milliseconds. The use of both SSD and and the new ZEO reduced mean load times to around 4 milliseconds.) This release also includes a change to the way that file-storage indexes are saved leading to make improvements in save and load times and a 30% reduction in index file size. That's good news. Thanks for all the work! -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate branch sylvain-persistent-load
Hi, On 02/01/2010 05:47 PM, Sylvain Viollon wrote: Hello, I did few month ago a branch of zodbupdate, which works completely differently than the trunk. Instead of changing the pickle code of the record, it unpickle it, changing references at that point, and repickle if there is changes. I did this in order to support the ZODB references that are implemented using persistent ID, in which a reference to a class can be expressed as a tuple (module, classname) instead of the class pickle opcode. That tuple (module, classname) can appear in the class meta pickle of a record as well. You can have more details about this by reading the comments of the serialize.py file in the ZODB code. If I understand correctly, those are legacy formats, that the ZODB doesn't use anymore to write data, but is still able to read data stored using those formats for backward compatibility support. The issue is that I want to use zodbupdate on database that have been created a long time ago, and I do need zodbupdate to be able to read those formats and correct class references in those records as well. That's the motivation of my changes. My version of zodbupdate is feature alike with the old one: - it is able to update records which have backward compatibility imports in the code, and report them at the end, even if they are not described in the renaming rules, - it is able to ignore missing modules and classes (it use ZODB.broken), - command line invocation stays the same, - performance-wise, it seems even to be faster (I use cPickle), As well, it's able to work on 'broken data.fs' which trigger POSKey errors. I did my testing using the infrae.com website database, which have been created around 2002 I think, and contain a lot of interesting cases. So I would like to release my work now, since I need to use it in production environment. Since there is lot of changes, I can release it as a fork as well, if it's not possible to merge it with the trunk. I think I'm fine with your approach in general by now. I don't have the time to review anything, though. The old version is happily there in SVN. IIRC you hadn't updated all unit tests last time, did you catch up with that yet? Otherwise, feel free to merge and release. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate branch sylvain-persistent-load
On 02/02/2010 11:17 AM, Sylvain Viollon wrote: On Tuesday 02 February 2010 09:08:42 Christian Theune wrote: Hi, Hello, I think I'm fine with your approach in general by now. I don't have the time to review anything, though. The old version is happily there in SVN. IIRC you hadn't updated all unit tests last time, did you catch up with that yet? I fixed the current tests yesterday. I removed a bunch of them a while ago, all the one which was testing the pickle modification code, since I removed the corresponding code. I plan to add a few more today, Otherwise, feel free to merge and release. Thank you, I think I will have look at the trunk, but if there is no changes in it since my branch, I will move it as a branch called pickle-version, and rename my branch as trunk. I will tag it, but if you want me to do the release, you need to give me rights on pypi. My username is thefunny42. Meh. That looks quirky in the history. If there are no changes, then the merge will be straight-forward. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate branch sylvain-persistent-load
On 02/02/2010 03:58 PM, Sylvain Viollon wrote: On Tue, 02 Feb 2010 11:30:41 +0100 Christian Theune c...@gocept.com wrote: Hello, Meh. That looks quirky in the history. If there are no changes, then the merge will be straight-forward. I merged my branch into the trunk, it's ready for a release. What's your pypi ID? I'll give you release rights then. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate branch sylvain-persistent-load
On 02/02/2010 05:00 PM, Sylvain Viollon wrote: On Tue, 02 Feb 2010 16:44:19 +0100 Christian Theune c...@gocept.com wrote: On 02/02/2010 03:58 PM, Sylvain Viollon wrote: On Tue, 02 Feb 2010 11:30:41 +0100 Christian Theune c...@gocept.com wrote: Hello, Meh. That looks quirky in the history. If there are no changes, then the merge will be straight-forward. I merged my branch into the trunk, it's ready for a release. What's your pypi ID? I'll give you release rights then. thefunny42, I already mentioned it in one of my previous mail. Done. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development smime.p7s Description: S/MIME Cryptographic Signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Data.fs size grows non-stop
On 12/09/2009 04:58 PM, Pedro Ferreira wrote: OOBTrees are complex? No, I meant something list-like, rather than key-value. zc.blist implements a BTree-based list. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On 12/09/2009 12:06 AM, Shane Hathaway wrote: Daniel Kraft wrote: Jim Fulton wrote: On Tue, Dec 8, 2009 at 1:49 PM, Benji Yorkbe...@zope.com wrote: Any incomplete writes that are in progress at the time of the copy will simply be ignored when the resulting backup is opened later. Note that this assumes that you are using a straightforward linear copy. rsync would likely produce unpredictable results. Every modern volume manager or whatever supports snapshotting a filesystem. A snapshot is a good-enough synchonous copy. You simply store its contents to somewhere and you're done, even with blobstorage. Very good point. Tools like LVM and ZFS probably make repozo obsolete. Not exactly: it's still a good idea to store away deltas as single large files are still hard to deal with except if you have really good block-level de-duplication in your filesystem. (Uhmm. I guess ZFS probably has all of that. It also probably serves my dinner.) Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On 12/09/2009 02:25 AM, Shane Hathaway wrote: Christian Theune wrote: Not exactly: it's still a good idea to store away deltas as single large files are still hard to deal with except if you have really good block-level de-duplication in your filesystem. (Uhmm. I guess ZFS probably has all of that. It also probably serves my dinner.) I would run rdiff-backup on the snapshot. Quoting the rdiff-backup feature list: Space efficient: Suppose you have a large database file that changes a little bit every day. A normal incremental backup would keep saving copy after copy of this database, wasting a lot of space. rdiff-backup uses librsync, which implements the same efficient diffing algorithm that rsync uses. It works on binary files as well as text, so only a fraction of the data in your database would be saved in each incremental backup. http://rdiff-backup.nongnu.org/ Interesting. AFAIK this is already in place in our backup scheme. As I haven't put it there I wasn't conciously aware of that detail though. :) -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Data.fs size grows non-stop
On 12/07/2009 05:00 PM, Alan Runyan wrote: I'd just like to add that there's some changes that can be related to this: - we had some classes inheriting from Persistent that now inherit from something else as well (but no extra arguments are being added, AFAIK); - we added some zope.interface definitions to some Persistent classes; maybe this causes some kind of behavior that we were not aware of? I doubt it. Thousnads of people are doing this and do not report the same behavior. What is more likely is that a programmer is changing a persistent object very often. One of the downsides of the ZODB is that it is so transparent it is possible to unwillingly make database changes. A design pattern for RDBMS is to have 2 pools. READ pool and WRITE pool. Often the READ pool comes from some replica and WRITE is to the master. I'm unsure this pattern would work for ZODB. I know Malthe was thinking about this but unsure if he had anything concrete. My guess: have a ZEO fan-out setup and configure the read pool ZEO servers as read-only. Any transactions writing will fail miserably right away. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] newbie ZEO first try. ConflictError.
Hi, On 11/26/2009 10:08 PM, James Bergstra wrote: On Thu, Nov 26, 2009 at 1:51 PM, tsmillertsmil...@gnixterhouse.com wrote: Laurence, Thank you for your very quick reply. I did as you suggested and now the ConflictError is now handled. And that is a most excellent thing because now ConflictError is raised every time. So it still seems to be confused. I am trying to figure out if I will be able to use ZEO in my application. Now my program reads: while True: root[ one ] = time.asctime() while True: try: print Try to commit transaction transaction.commit() print root is, root except POSException.ConflictError: print we have a conflict transaction.abort() time.sleep(.2) else: break time.sleep(10) I have a related question about this code... in the inner loop tom is calling abort() ; sleep(); commit(). Does that make sense? I thought that abort() would revert the root to the database's version of things, and discard any change that the client had tried to make. So what does it mean to call commit() again immediately and why that commit cause a conflict when the client hasn't actually changed anything since abort() ? The abort() actually also marks the point when the next transaction begins implicitly. Your code would work perfectly if whenever you start a transaction, you simply call transaction.begin(): while True: transaction.begin() root[ This will cause the time.sleep(2) to not be included in the transaction and your conflict rate will quickly go down for your example. If you also use a random factor for the sleep (e.g. between 0.2 and 0.3) then you'll have a lesser chance of subsequent conflicts in your example because all your transactions are very similar. In real life transactions would be distinct enough to cause different offsets in runtime. Hope this helps, Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] newbie ZEO first try. ConflictError.
Hi, On 11/26/2009 10:55 PM, tsmiller wrote: Christian, Thanks. How painfully obvious. I have written the words 'transaction.begin()' about a thousand times or so! But I get discombobulated when I start looking at something new ( ZEO ) and forget the obvious. Final code for this little test that works perfectly: while True: transaction.begin() root[ one ] = program 2 - + time.asctime() while True: try: transaction.commit() except POSException.ConflictError: time.sleep(.2) else: break time.sleep(5) Two notes: First, I'd leave the transaction.abort() right before the time.sleep(). Just to be explicit. Second, the second while loop is superfluous - it doesn't do what it looks like and what you might think it does. ;) You can try committing exactly once. If you commit after a conflict error, that will give you a different exception. On the framework level conflict errors are usually handled by re-trying the *whole* transaction once more. E.g. in Zope it means: - abort the transaction - start a new transaction - process the request as it was again If it fails for three times then Zope gives up and hands the conflict error to the user. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Priorities?
Hi there, when jotting down a thought about bug 240381 I got the feedback that caring about the issue isn't among the (current) priorities of ZODB development. Unfortunately, I don't know what those priorities are. In addition, when going through the bug tracker this morning I found that the triage of bugs isn't executed very well currently and I'm a bit wary of trying to do something that will be shot down with a not our priorities quickly. I'm trying to make up time to help where I can and I'm happy to do some leg work. For example I find it extremely important to have a well organised bug database. So one thing I'm trying to do is keep the list of untriaged bugs down so that when someone does have time to work on a fix for a bug he can easily find those that are important/urgent and real issues not duplicates or unreproducable things. This involves moving bugs out of the new status as quickly as possible. For example #183203 has been prioritized to medium but is still sitting there as new. The content doesn't hint to it as confirmed so I'd suggest moving this over to incomplete. I find those thoughts to be relatively obvious, but I'd rather not keep contributing in my spare time with things that aren't of priority to the project. And again: asides from the (to me) obvious cleaning of the bug database. What are the priorities? Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
On 11/20/2009 08:07 PM, Jeff Shell wrote: On Nov 20, 2009, at 11:52 AM, Jim Fulton wrote: If configured, sure. I would not trust repozo + blob backups. I don't trust Blobs. A year ago, I reverted all of the code I had for our CMS that supported Blobs as it was impossible to copy their data in a basic Zope 3 copy/past/move. I think that may have been fixed, but since there's practically zero documentation on how to use blobs, use them wisely, use them well, and use them in fairly plain Zope 3-ish applications, I lost confidence. I think I just found a way to fix this particular issue on the ZODB-level for good. I re-opened https://bugs.edge.launchpad.net/zodb/+bug/240381 and assigned it to myself. I'll see if I can do something over the weekend. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
On 11/20/2009 01:07 AM, Chris Withers wrote: Jim Fulton wrote: On Thu, Nov 19, 2009 at 6:07 PM, Chris Withersch...@simplistix.co.uk wrote: Jim Fulton wrote: So why not add a history method to ZEOStorage that passes size as a keyword argument? I don't know what ZEOStorage is... See StorageServer.py around line 80. As the docstring says it is a: Proxy to underlying storage for a single remote client. Then we shouldn't be having this conversation. I'm working off this traceback: Traceback (most recent call last): File ZODB3-3.9.3-py2.6-linux-i686.egg/ZEO/zrpc/connection.py, line 581, in handle_request ret = meth(*args) File gocept.zeoraid-1.0b6-py2.6.egg/gocept/zeoraid/storage.py, line 219, in history assert version is '' There is no ZEOStorage instance involved in this as far as I can tell. What am I missing? Check the setup_delegation method: the history method is patched through directly so your traceback doesn't see the ZEOStorage anymore. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo uses md5 lib, which is deprecated in python 2.6
On 11/18/2009 06:00 PM, Chris Withers wrote: Hi All, ZODB's repozo script uses the md5 library, which is deprecated in python 2.6, resulting in annoying emails from cron jobs. A few questions: - Where's the tracker for ZODB nowadays to report this issue? https://edge.launchpad.net/zodb - Anyone know what to replace md5 with? The warning suggests hashlib, which landed in 2.5. I assume it's okay to just use hashlib rather than worry about conditionally import md5 or hashlib, given that trunk ZODB and 3.9 branch are only targeted at Python 2.5 and above, right? Not sure about that. Most people I see that need to support 2.4 and 2.6 tend to provide a wrapper-function combined with a conditional import. - Does repozo have any tests? If so, how do I run them. Looking around I found this: scripts/manual_tests/testrepozo.py Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs for noobs
On 11/17/2009 04:46 AM, Shane Hathaway wrote: James Bergstra wrote: I'm surprised by the complete silence... was this post sent to the wrong list? What would be a better one? This is the right list. I don't know why you didn't get an answer earlier. Where can I find information on using BLOBs outside of plone? I use ZODB directly from my python programs and I would like to store large files across ZEO. I'm using a recent ZODB (3.9.3) - I use runzeo.py to start my server, how do I tell it where on the filesystem to put blob data? Create a ZConfig file that specifies the blob-dir option. Does anyone know where to find documentation of the ZConfig format? I always cobble something together using component.xml and examples. Yes, same here. Actually it even has readable documentation about the various options in there already. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Using a storage that is also handled by StorageServer
Hi, in ZEORaid we're using the storage (a ZEORaid instance) as it's being by ZEO for recovering data from the good storages into one we're recovering. For finishing the recovery exactly on transaction boundaries and to avoid transactions slipping in when we think we're done we acquire the commit lock for a short period of time. If a client tries to write in that period of time the transaction will be put into the StorageServer's waiting list and sit there either indefinitely or until another client comes around calling abort() or finish(). So this is basically an issue because StorageServer assumes that it or another instance of StorageServer is ever talking to that storage, so it implements a secret protocol (the waiting list) which is not officially visible to other code. I can hack around this for now by making the waiting list visible to my code and notify the waiting storages from there, but, I'd like to propose an extension to the storage API. We basically need the ability to register callbacks with a storage that will be called on transaction boundaries. With that we can redo the current _restart implementation so that it can be triggered from code that doesn't need to know that a storage is being wrapped in such a way. If you think we can go down that route in general, then I'll prepare a more detailed proposal including making a draft implementation. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Fwd: Packing and GC versus packtime
Posted to the wrong list initially. Sorry. Original Message Subject: Packing and GC versus packtime Date: Mon, 09 Nov 2009 13:22:04 +0100 From: Christian Theune c...@gocept.com To: zope-...@zope.org Newsgroups: gmane.comp.web.zope.devel Hi, I'm wondering: Scenario 1: I pack a storage to a time T and append X transactions after that Scenario 2: I append X transactions to a storage and then pack it to time T. Will those scenarios end up with the same result? I can imagine GC to get in the way, but I'm not sure about how it does its thing exactly. The question can also be phrased as: Are the operations pack and append transaction for storages commutative? Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ Zope-Dev maillist - zope-...@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope ) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] StorageServer's waiting list
Hi, in ZEORaid recovery we acquire the commit lock of our RAIDStorage from a thread to be able to finish recovery atomically. That RAIDStorage is also being served from a StorageServer which causes some trouble because it assumes that the only way for a transaction on the served storage to end is via the StorageServer's tpc_abort or tpc_finish method. Thus, when a transaction comes in while the recovery has a transaction going on the StorageServer puts it into the waiting list but never gets the signal to continue. Reading the code talking to tpc_transaction I found that this seems to be merely an optimization (which I can disable by just letting tpc_transaction return None all the time). Why is the waiting list necessary? And why does it work alright in a ZEO fan-out scenario? Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb 3.8.3 strip_versions missing some versions?
On 11/02/2009 10:04 AM, Chris Withers wrote: Hi All, I'm upgrading a Zope 2.9.8 project to Zope 2.12 + zeoraid. As I've done with previous projects, I run the .fs files through zodb 3.8.3's strip_versions script. For both the storages in this project, the script found no version records. As a first step, I'm currently running the 2.9.8 app server against a pair of zeo storage servers running zodb 3.9.3 and zeoraid 1.0b3 However, when I try and view the history tab of *any* page template within the .fs files, even a newly created one, I get an assertion error from zeoraid as gocept.zeoraid.storage.RAIDStorage's history method is getting called with version passed as an integer 20. Anyone have any idea what that 20 means and why it's being passed like that? There's a size argument following the version which probably is not being passed as a keyword argument so it looks like a slight API incompatibility. I wonder whether ZEORaid should hack around that or whether the caller should be changed. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
On 11/02/2009 10:47 AM, Chris Withers wrote: Christian Theune wrote: There's a size argument following the version which probably is not being passed as a keyword argument so it looks like a slight API incompatibility. I wonder whether ZEORaid should hack around that or whether the caller should be changed. Hmm... looking more closely at the traceback, the caller is: File ZODB3-3.9.3-py2.6-linux-i686.egg/ZEO/StorageServer.py, line 1382, in history return self.storage.history(oid, size) Now, ZODB 3.9 doesn't support versions, which is why the versions parameter has vanished, I guess? However, I wonder if maybe this call should be: self.storage.history(oid, size=size) ...to support storages which still accept a version parameter *and* newer ones that don't? Looking at RelStorage: http://svn.zope.org/relstorage/trunk/relstorage/storage.py?rev=105120view=auto ...I can see it's going to have the same problem. Would anyone object if I wrote a test and fixed this on the 3.9 branch of ZODB? I'm not really sure what to do about the trunk... I think that's the best idea. It's not exactly part of the policy of supporting old protocols but if we want to support and encourage 3rd party storage implementations we should leverage this simple fix for better backwards compatibility. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Test time on 3.9 branch - windows?
On 11/02/2009 12:59 PM, Chris Withers wrote: Jim Fulton wrote: Should I really expect the tests to take an hour to run on Windows? Yes. Erk. Is there a subset I can look to run when looking to patch the ZEOStorage308Adapter? Feel my windows pain. Do the tests take much less time to run on Linux? If so, any idea as to why? I remember running them in less than 20 minutes. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Why is CHANGES.txt in src/?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/15/2009 01:12 PM, Jim Fulton wrote: On Tue, Sep 15, 2009 at 7:03 AM, Christian Theune c...@gocept.com wrote: is there a reason why ZODB's CHANGES.txt lives in ZODB/src/ instead of ZODB/ as in other packages? Yes, because the ZODB project has multiple Python packages. The entries in CHANGES.txt don't only pertain to the ZODB package. It would be silly to maintain separate CHANGES.txt file for each package. Right. Other projects still keep the CHANGES.txt in the very top level folder where the egg is defined, not in the Python package itself. - -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqvgnIACgkQdUt9X/gknwLj5QCg2dgPF0LznOO9WtU6vGJ/RvjY ZRYAn3+UPMaIG+2sPMsHJ1t0+2gd9Z64 =PZ3s -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Why is CHANGES.txt in src/?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, is there a reason why ZODB's CHANGES.txt lives in ZODB/src/ instead of ZODB/ as in other packages? I keep tripping over this and wonder whether it's accidental or intentional. Christian - -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqvdGwACgkQdUt9X/gknwKehwCeN5ZKYbDkJK/vD3Fx/WruGUrO NUMAnjzqFo9S7lWnniwjjG8cLWsyNHJZ =n9O2 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Utility to strip version records from storages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/14/2009 09:23 PM, Jim Fulton wrote: I'm working on a utility to strip version records from storages. I plan to release it in a bug fix 3.8 release, probably 3.8.3. It will have some limitations: - It will only convert storages that have an iterator method. I suspect that FileStorages are the only storages that have iterator methods in 3.8 and sooner. (Maybe RelStorages do too.) - It won't give blob records any special treatment. I think it's very unlikely that there will be databases that have both version record and blobs. In any case, it should be possible to copy a blob directory separately after copying the database records. If someone was insane enough to have blobs in versions, then the database records will be stripped but the associated blob files will be left. This is so unlikely that I don't intend to spend time dealing with this possibility. Hmm. I guess that those would vanish automatically after packing that database... - -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqGcrwACgkQdUt9X/gknwK7hQCgnXnfg6mt6lpInlyZvx1FGXWS 3OcAn00QEUcpEeOLf8Rg1lTAySbvepQI =B8+A -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 8-byte oids
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/13/2009 10:42 PM, Shane Hathaway wrote: Jim Fulton wrote: IPersistent specifies OIDs to be non-empty strings and reserves the 8-byte OID consisting of 8 null characters. Lots of ZODB infrastructure assumes that oids are 8 bytes. I'm tempted to say that oids must be 8-byte strings. +1. The simplified standard promotes interoperability. The difference to the spec before would be that previously strings with less than 8 bytes were allowed? - -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqFBfAACgkQdUt9X/gknwLO9QCeNuLCMk1qElAGJLFGMyagoFZT djMAn3OAPe3BD++9UqzdPMTLwaky0idH =mYEo -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 8-byte oids
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/14/2009 08:36 AM, Christian Theune wrote: On 08/13/2009 10:42 PM, Shane Hathaway wrote: Jim Fulton wrote: IPersistent specifies OIDs to be non-empty strings and reserves the 8-byte OID consisting of 8 null characters. Lots of ZODB infrastructure assumes that oids are 8 bytes. I'm tempted to say that oids must be 8-byte strings. +1. The simplified standard promotes interoperability. The difference to the spec before would be that previously strings with less than 8 bytes were allowed? Never mind, reading one of your mails in the other thread actually cleared this up. - -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqFBkgACgkQdUt9X/gknwLJVACgiF5Mk4o++FpLBXft4VGQUYpF oMUAoLoGs1BnnkU6m7ZVhg8xQAmF9Mys =yzsl -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo cache files : how to estimate the space?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/11/2009 03:32 PM, Wichert Akkerman wrote: On 8/11/09 15:29 , Dominique Arnoult wrote: Hello, i deal with a problem of /tmp. I have a 1 Go partition and cannot easily change it. I run a Zeo cluster of 7 clients. Since the lsof command, I can see 7 process and each process has 4 files opened of 100 Mb max. That corresponds of the cache-size directive of my zopecore.conf. But, I don't undertstand why there are 4 files and not only one by zeo client. Any can help me? Each zserver thread has its own cache. Live object cache: one per thread ZEO cache file: one per storage Christian - -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkqBdCYACgkQdUt9X/gknwIlAACfSWjrZ2PECPG4f3DIzx66cT9n 2MgAoOWX2HFE+T0gUe1fuhL8TF8vkYkg =tpax -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] how can I get the size of ZODB.Blob object ?
On 07/07/2009 04:47 PM, Andreas Jung wrote:
How about seek()ing to the end of the blob and then using tell() for getting
the position?
Yup.
An opened blob is just a regular file handle. So the same applies as to
normal files:
f = open('asdf')
t.seek(0, 2)
t.tell()
35L
Christian
--
Christian Theune · c...@gocept.com
gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Fan-out support / horizontal cache sharing for blobs
Hi there, Zagy and I have been discussing an issue that we have with fan-out setups and blobs. So, assume the following setup: - A main ZEO server which has the blob storage locally available - A fan-out ZEO server which runs on a different machine and has 4 clients connected Now, the fan-out ZEO server pulls the blobs via ZEO RPC and stores them in a blob cache directory. The same is true for the 4 clients which also create a cache directory and store blobs there. We now have 5 cache directories on the second machine. The cache of the fan-out server will be approximately the combined set of blobs from the other clients. As all those 5 directories live on the same disk, we'd like them to be shared between the various clients. This currently doesn't work because we apply coarse grained locking to the whole directory instead of individual files. We propose to change this so that locking will become fine-grained and allow multiple ZEO clients to work on the same blob cache directory. Comments? Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Blob Question
Hi, Just found this in my inbox. On Wed, 2008-10-15 at 12:18 +0200, Andreas Zeidler wrote: On Oct 14, 2008, at 5:43 PM, Christian Theune wrote: On Tue, 2008-10-14 at 10:09 -0500, Alan Runyan wrote: Why must the file be closed before calling consumeFile? And would you go as far as saying that if you *do not close* the file before calling consumeFile - all bets are off. You are not using Blob as intended? You're definitely not going to be able to work on Windows, which is the source of the requirements to close handles before moving/deleting files. alan and i hit that point when trying to avoid copying the uploaded data before handing it to `consumeFile` and were wondering if the requirement was only necessary because of that or if perhaps other unexpected things would creep up later. so, would you say it's generally okay to leave the file open (until the end of the request) on a posix system? Yeah, posix systems generally allow you to operate on the file(name) while open file handles are still referencing it. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Proposal (version 2): cross database reference seat belt
Hi, On Tue, 2009-04-28 at 13:54 -0400, Jim Fulton wrote: Thanks again! (Note to everyone else, Shane and I discussed this on IRC, along with another alternative that I'll mention below.) I like version 2 better than version 1. I'd be inclined to simplify and it and skip the configuration flag and simply publish an event any time we see a cross-database reference when saving an object. Here's proposed solution 3. :) - We add a flag to disable new cross-database references unless they are explicitly registered. - We add a connection method to register a reference: def registerCrossDatabaseReference(from_, to): Register a new cross-database reference from from_ to to. - We arrange that connections can recognize old cross-database references. If someone accidentally creates a new reference and the flag is set, then transaction will be aborted. An interim step, if we're in a hurry to get 3.9 out, is to simply add the flag. This would disallow cross-database references in new applications. These applications could still support multiple databases by providing application-level traversal across databases. I think I'm reading something incorrectly: is there an emphasis on *new* applications? The flag would disallow the creation of cross-database references for a given DB -- independent of whether the app is new or old, right? Only depending on whether the application uses a ZODB that has the feature and has it enabled. Right? Otherwise, I'm +1. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] merging databases containing multidatabase references
On Thu, 2009-04-23 at 15:02 -0700, David Glick wrote: I have a multidatabase of 3 databases containing a handful of multidatabase references. (This was created out of a combination of naivety along with a desire to be able to work with smaller files if I need to pull down a local copy of the database from the server.) Of course, now I am running into trouble because I cannot pack one of the databases without occasionally having an object get garbage collected that is still referenced from another database, which leads to POSKeyErrors when I try to load the object containing that reference. Having now realized the folly of trying to split things into multiple databases, I would really like to merge these 3 databases back into 1. Any ideas for how I could go about this? Based on http://docs.zope.org/zope3/Code/ZODB/cross-database-references.txt/index.html and some experimentation, multi-database references are not supported by the import/export code. Do I have other options? I think it should be possible to write a tool that merges one database into another. This needs some processing on the database objects as their OIDs (and thus the internal references) need to be reorganized. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On Thu, 2009-04-02 at 10:19 +0200, Adam GROSZER wrote: Hello, What's a good practice to create unique container keys for a heavily loaded application? (I mean lots of writes to the same container) Obviously having a counter on the container and incrementing and using that for key gives write conflicts. What do the experts use? If uniqueness is your only concern then using a large number range and consuming numbers randomly will perform better than incrementing. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On Thu, 2009-04-02 at 09:22 -0500, Alan Runyan wrote: It seems like such an easy goal: autoincremental integers for a container. Is this such a problem because of the ZODB architecture? or lack there of? There are two database primitives that everyone appears to want: - autoincrementing integers for containers (tables) - indexes (not in application) The new generation of databases (couchdb, tokyo cabinet, hypertable, etc) autoincrementing is usually absent. Though these data containers usually have index support. ZODB has autoincrement support for one type: OIDs. The problem of autoincrement is that this needs to be handled outside the scope of transactions. In a distributed fashion this seems rather hard to do in comparison to just buying into conflicts. The indexes are hard too. We have gocept.objectquery which is (still) in a proof-of-concept state, although I'm getting really motivated WRT using it for migrations. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On Thu, 2009-04-02 at 16:41 +0200, Wichert Akkerman wrote: On 4/2/09 4:36 PM, Christian Theune wrote: ZODB has autoincrement support for one type: OIDs. The problem of autoincrement is that this needs to be handled outside the scope of transactions. In a distributed fashion this seems rather hard to do in comparison to just buying into conflicts. I suspect it might be faster than the penalty of retrying a request on conflict if used properly. Sure, hard problems might still have fast solutions. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] _p_oid
On Wed, 2009-04-01 at 18:28 +0200, Adam GROSZER wrote: Hello, Does anyone know how _p_oid is generated? I'm asking because we would like to use _p_oid as a unique key to store BLOB-like files on the filesystem. Of course more threads/processes/machines are in the picture. So the question is once an object gets a _p_oid assigned, is it guaranteed to be unique over all instances accessing the same DB? Yes. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Relstorage and ZEO
On Mon, 2009-03-30 at 08:05 -0700, Stephan Richter wrote: On Monday 30 March 2009, Anthony Gerrard wrote: The impression I get is that using both ZEO and Relstorage are recommended. You cannot use both at the same time. Either you use Filestorage/ZEO or Relstorage, as in Relstorage the underlying DB takes care of the replication features. You can decide for each storage in your application server. At least in principle you should. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] FileStorage iteration data records missing data_txn
Hi, On Fri, 2009-01-09 at 11:39 -0500, Jim Fulton wrote: On Jan 9, 2009, at 7:41 AM, Christian Theune wrote: Hi, in ZEORaid we switched from using `restore` to `store` for replaying transactions from another storage and found that the data records that FileStorage returns aren't filling in the `data_txn` fields correctly, but end up writing `None` all the time. What makes you think this is incorrect? data_txn should: 1. Not be in the interface. It is an optimization that depends on a peculiarity of FileStorage of a case that only arises (today) with undo. I'm inclined to remove this optimization. In fact, in the presence of blobs, the optimization is dubious. This implies that the prev_txn argument to restore should be ignored. 2. data_txn should only be other than None for a record that was part of an undo. It is used to avoid copying data. Rather than copying data, we simply record the position of the earlier record that contains the data. My thought was based on the presumption that the DataRecord interface was described correctly (which it isn't as you point out below). I'm not exactly sure how to fix this, but attached is a patch that makes it work for the cases I could come up with. I'd like to add this change to the trunk (I'll write tests if this change gets accepted). Your patch is incorrect. data_txn is not the previous record transaction id. It is the transaction id of the transaction containing data when the current record doesn;t contain any. It is used as an optimization by referring to data stored in an earlier record rather than copying data to the current record. The relevant interface didn't originally mention this field. The documentation you added is incorrect. I'm going to remove the field (along with version and tid) from the interface. Ah. That's the source of the misunderstanding, thanks for clearing that up. This sounds like I should also clean up the DataRecord in the BaseStorage because that also exposes data_txn, and storage iteration protocol on ZEO transports this attribute as well. Looks like I've got to use loadBefore() to find the transaction that an object's modification was based on. Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] svn.zope.org up again?
On Tue, 2008-11-11 at 12:35 +0200, Marius Gedminas wrote: On Tue, Nov 11, 2008 at 08:51:28AM +0100, Adam GROSZER wrote: Hello Stephan, svn: Can't open file '/svn/repos/main/db/revs/70320': No such file or directory program finished with exit code 1 Seems that one revision did not get restored. It breaks also on show log, where other folders with older revisions seem to work. I've a full svn mirror up to rev 92837 if it turns out to be needed. Same here. :) It's publicly available at: http://svn.zope.de/zope.org -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: External Garbage Collection
On Fri, 2008-11-07 at 09:15 -0500, Jim Fulton wrote: On Nov 7, 2008, at 1:40 AM, Christian Theune wrote: On Thu, 2008-11-06 at 17:46 -0500, Jim Fulton wrote: I've posted a new proposal: http://wiki.zope.org/ZODB/ExternalGC That addresses multi-database garbage collection and can also be useful in other situations. Comments are welcome. Absent objections, I may start working on this fairly soon. +1 as far as I can see. One little thing: I find the interface a bit of a misnomer as all functions are IMHO useful on their own. The fact that you're going to instrument them to implement GC might not need to go into that interface name. shrug Thought so. :) I'm seriously contemplating a ZODB 4. That might be an opportunity collapse a number of the storage interfaces into a new IStorage. W00t. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with broken
On Thu, 2008-11-06 at 11:20 +0200, Marius Gedminas wrote: On Wed, Nov 05, 2008 at 03:59:07PM +0100, Christian Theune wrote: On Wed, 2008-11-05 at 16:00 +0200, Marius Gedminas wrote: On Wed, Nov 05, 2008 at 11:48:35AM +0100, Christian Theune wrote: On Wed, 2008-11-05 at 11:31 +0100, Laurence Rowe wrote: Broken objects occur when the class for a pickled object cannot be imported. To change the location of a class, you need to provide an alias at the old location so that the object can be unpickled, i.e. MyOldClassName = MyNewClassName. You can only remove MyOldClassName after you have updated all of the pickles (with your code below). Note: We made a relatively good experience by removing those symbols from their original modules and provide a generation which first sets up dummies, then runs migration code to remove/rebase those objects and then remove the dummies again. How do you do that? Monkey-patching? Yes. Monkey-patching *and* monkey-unpatching. Thought so. By the way, this has a potential failure mode when you need to evolve more than one generation, and a previous generation script needs to access those objects in the old location before your new generation script sets up the dummies. Yes. Our deployments are under relatively tight control and we check that generations work on a copy of a database before doing the actual roll-out. In this case we currently estimate that we'll need to change that earlier generation. Generations assume that the code looks like what it did when the generation was written -- which certainly doesn't hold. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: External Garbage Collection
On Thu, 2008-11-06 at 17:46 -0500, Jim Fulton wrote: I've posted a new proposal: http://wiki.zope.org/ZODB/ExternalGC That addresses multi-database garbage collection and can also be useful in other situations. Comments are welcome. Absent objections, I may start working on this fairly soon. +1 as far as I can see. One little thing: I find the interface a bit of a misnomer as all functions are IMHO useful on their own. The fact that you're going to instrument them to implement GC might not need to go into that interface name. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with broken
On Wed, 2008-11-05 at 11:31 +0100, Laurence Rowe wrote: Broken objects occur when the class for a pickled object cannot be imported. To change the location of a class, you need to provide an alias at the old location so that the object can be unpickled, i.e. MyOldClassName = MyNewClassName. You can only remove MyOldClassName after you have updated all of the pickles (with your code below). Note: We made a relatively good experience by removing those symbols from their original modules and provide a generation which first sets up dummies, then runs migration code to remove/rebase those objects and then remove the dummies again. That way you can get your source code clean of 'BBB' code in a single generation. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with broken
On Wed, 2008-11-05 at 16:00 +0200, Marius Gedminas wrote: On Wed, Nov 05, 2008 at 11:48:35AM +0100, Christian Theune wrote: On Wed, 2008-11-05 at 11:31 +0100, Laurence Rowe wrote: Broken objects occur when the class for a pickled object cannot be imported. To change the location of a class, you need to provide an alias at the old location so that the object can be unpickled, i.e. MyOldClassName = MyNewClassName. You can only remove MyOldClassName after you have updated all of the pickles (with your code below). Note: We made a relatively good experience by removing those symbols from their original modules and provide a generation which first sets up dummies, then runs migration code to remove/rebase those objects and then remove the dummies again. How do you do that? Monkey-patching? Yes. Monkey-patching *and* monkey-unpatching. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with broken
On Wed, 2008-11-05 at 16:02 +0200, Marius Gedminas wrote: On Wed, Nov 05, 2008 at 01:15:58PM +0100, Adam GROSZER wrote: Thanks for the advices on the generation script, but I think the problem lies in that the broken object is loaded fine but cannot be written back again to ZODB. Yes. You can't do anything with broken objects. They're there only so that you don't get an exception when loading an object that refers to a broken object. Or do I misunderstand that broken is a sort of proxy for the missing class that allows it to be around? Yes, but being around is a rather narrow goal that doesn't allow you to do anything with it. You do can inspect the unpickled state. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] New gc argument for pack
On Sat, 2008-10-25 at 12:59 -0400, Jim Fulton wrote: I propose to add a new optional gc keyword argument for database packing. It will default to true. If given and false, then pack non- current database records will be removed, but no garbage collection will be performed. Any objections? Mechanics sound good. However, if I got you right, the description contained a double negative that I'd like to avoid. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] New gc argument for pack
On Sat, 2008-10-25 at 14:06 -0400, Jim Fulton wrote: On Oct 25, 2008, at 2:03 PM, Christian Theune wrote: On Sat, 2008-10-25 at 12:59 -0400, Jim Fulton wrote: I propose to add a new optional gc keyword argument for database packing. It will default to true. If given and false, then pack non- current database records will be removed, but no garbage collection will be performed. Any objections? Mechanics sound good. However, if I got you right, the description contained a double negative that I'd like to avoid. no no. :) That means yes? :) Seriously, I don't see the double negative. Now I don't see it anymore too. :) /me shuts up and goes off for weekend relaxation. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. They were factored in and we have tests. However, the initial pickle will empty them: copying blobs this way including their content isn't currently supported. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote: Christian Theune wrote: On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. They were factored in and we have tests. However, the initial pickle will empty them: copying blobs this way including their content isn't currently supported. That doesn't sound like factored in to me, that sounds like bodged and ignored ;-) No. I was pointing out that Jeff first does a pickle and then the export. *His* pickle breaks it. The export works: it exports the (then empty) blob. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Zope-dev] Blobs and modes
On Fri, 2008-10-24 at 15:50 -0400, Benji York wrote: On Fri, Oct 24, 2008 at 3:28 PM, Jim Fulton [EMAIL PROTECTED] wrote: On Oct 24, 2008, at 3:24 PM, Benji York wrote: Is there a good reason blobs don't support b or t in the mode strings passed to open? I'm refactoring some code that expects a file-like object to use blobs and it wants to pass wb as the mode (which is a sane thing to do). Blobs are implicitly binary. (Note the B in Blob.) I'll add support for b and t to the blob code if no one objects. I object. BTW, this questions should have been asked o zodb-dev. [yep, copied on this message] For perpetuity: In a private discussion Jim explained that the blob .open() method returns things intended to be file-like, but the method itself isn't intended to mimic Python's open function, and as-such doesn't support the additional mode variations. It's intended to be similar but not identical. We originally did support 'b' and 't' but decided, as Jim pointed out, hat we only care for binary data and thus only the binary modes. I can't remember the reason that drove us to actually rip it out. Maybe the archive of zodb-dev knows. This distinction is similar to why __init__ methods aren't described in (zope.interface) interfaces; how you construct an object isn't prescribed by the interface exposed by the object itself. I'm not sure what you're saying. It's not the __init__ problem, as .open() is described on the Blob class. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB4 4.0a2
On Mon, 2008-10-20 at 11:12 -0400, Jim Fulton wrote: I'm not an owner of that. I wonder who on this list is. I guess it's nobody, except if Jeremy is still reading. (Jeremy? Are you out there?) I guess we could ask a PyPI admin to give us ownership. Someday, there will be a ZODB 4. It will be named ZODB and will have version numbers starting with 4.. *dream* :) -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Blob Question
On Tue, 2008-10-14 at 10:09 -0500, Alan Runyan wrote: Hi guys. At the plone sprint we started reviewing some code that uses Blob. The code that we reviewed is working in production although I wanted to be clear about the expected usage of Blob. ZODB/tests/blob_consume.txt says that you *must* close a file before calling consumeFile(). It seems the current Plone Blob implementation *does not close the file* before calling consumeFile. And all appears to be working. Why must the file be closed before calling consumeFile? And would you go as far as saying that if you *do not close* the file before calling consumeFile - all bets are off. You are not using Blob as intended? You're definitely not going to be able to work on Windows, which is the source of the requirements to close handles before moving/deleting files. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] What's best to do when there is a failure in the second phase of 2-phase commit on a storage server
On Fri, 2008-10-03 at 10:51 +0200, Dieter Maurer wrote: That's true for a FileStorage -- but it may not be that easy for other storages (e.g. BSDDB storage). Those storages using another system in the backend have to rely on them providing a two-phase commit API which needs to implement the second phase guarantee, or not? -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problems w/ Pickle based deep-copy of Blob objects
On Mon, 2008-09-15 at 13:19 -0700, Rob Miller wrote: Christian Theune wrote: On Tue, 2008-08-19 at 12:05 -0700, Rob Miller wrote: hi all, i've been experimenting w/ getting ZODB 3.8's blob storage support to work w/ the openplans.org stack, and have hit a snag about which i could use some feedback. There's another bug around the corner: If CMFEditions causes new OIDs be handed out to those objects (or even remove them) then the blob data will disappear. this is because the blob file path is constructed from the OID, right? Yes. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problems w/ Pickle based deep-copy of Blob objects
On Tue, 2008-08-19 at 12:05 -0700, Rob Miller wrote: hi all, i've been experimenting w/ getting ZODB 3.8's blob storage support to work w/ the openplans.org stack, and have hit a snag about which i could use some feedback. There's another bug around the corner: If CMFEditions causes new OIDs be handed out to those objects (or even remove them) then the blob data will disappear. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Backing up Data.fs and blob directory
Hi Laurence, On Wed, 2008-09-03 at 08:06 -0700, Laurence Rowe wrote: Backing up a ZODB has always been fairly easy in the past, but with the introduction of blobs things have got a little more complex. How should I create a consistent backup of my Data.fs and blob directory? My inital guess would be to take a copy of the Data.fs, then take a copy of the blob directory to ensure I have all blobs referenced in the Data.fs. Would I be able to restore from such a backup safely? (it may contain blobs from transactions that were newer than the backed up Data.fs). This should be safe because committed blobs are immutable and any dangling blobfiles would not interfere with the creation of blobs from new transactions in the restored zodb, as transaction ids would not overlap. I would be greatful if anyone could point out holes in my reasoning or has experience of this. Snapshotting a blob directory after taking a copy of your Data.fs should be safe, as long as you don't pack in between. Note that at the design stage we imagined that blob directories might become really large making backups unfeasable. For those situations we handwaved a very reliable storage for this directory, like a self-contained SAN/NAS solution that keeps your data safe. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] New to ZODB, how to make a db efficently?
Hi, On Mon, 2008-08-18 at 20:17 +0300, Markus wrote: I'm new here, so hi! ;-) I'm looking to create a database of persons and events, later to search persons by names, events by dates and locations (participants of events are already in an attribute of the event and instances of Person, which inherits from Persistent) At first I made a PersistentList of all the events and a PersistentMapping of all the people by an id, but later found out, that searching through a list with a for-loop is very slow (there are about 200 000 people and 100 000 events). And so as I've looked around here a bit (the docs and the wikis are mostly outdated or empty -- there's also talk about the bad documentation in this mailinglist) I've found, that I should be using OOBTree for making the indexes. Yes, the documentation situation is less than desirable for beginners. :/ So what I'm asking is, is it reasonable to create the db like this: persons in root['persons'], which is a OOBTree, mapping names to Person-objects and events in root['events'], which also an OOBTree, mapping dates to Event-objects? And if I want to map locations to events, I should do it at the same time, when creating the events, so I don't have to loop through all of them again? Here's what I do: Create a physical structure that models your data in a 'natural' way. This can e.g. be: - A root object representing the application, in case you may want to hold multiple instances of your application within a single database. - BTrees for storing large lists of objects, like you do. But mainly with a single lookup direction, e.g. for you the name-to-person mapping. Some times, those lists just work with arbitrary IDs for the objects, much like primary keys in tables. Alternatively, if you have a VFS-like structure, you might want to use the folder/item metaphor for the main structure of your database. - Add an indexing/searching framework for orthogonal queries. This is called `cataloging` in the Zope/ZODB universe. Some (more or less) standalone solutions are found in the proximity of `zope.catalog`. Use those to create tabular views on your data (independent of the physical structure) that are queryable by indexed arguments. Those are fast. If I have a OOBTree-mapping of dates to events, what should the values of it be? PersistentLists? I've read something about Buckets or Sets, but I'm not sure what they are good for, Bucket seems to behave like the equivalent BTree (OO, or IO or OI or IF or ), but Set seems to be a set... Is that true? I'd go with a flat structure. See my note on 'arbitrary' IDs above. What's the difference between a PersistentMapping and a OOBTree or OOBucket? Only the back-end, because on the front they all seem like dictionarys? Should I be using OOBTrees and OOBuckets for what I'm doing, because strings and dates are Os and not Is or Fs or... A PM is a persistent dictionary that loads all of its data at once. A bucket is an internal node of a BTree. A BTree is a (key-)sorted(!) data structure that provides a key/value interface like dictionaries do. Due to that, the lookup of items in a BTree is fast and also memory efficient, as only individual buckets of the BTree need to be activated for a lookup (optimally only O(logn) buckets). Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] what could cause this kind of cache corruption?
Hi, On Mon, 2008-08-11 at 09:37 -0400, Jim Fulton wrote: On Aug 3, 2008, at 4:03 AM, Christian Theune wrote: On Fri, 2008-08-01 at 09:21 -0400, Jim Fulton wrote: On Jul 31, 2008, at 1:53 PM, Chris Withers wrote: What I'd *really* like is a stable zodb release with Christian's patches for zeoraid and Shane's patches for RelStorage that then feeds through into a stable release of Zope 2. I'm not familiar with Christians patches. If they are bug fixes, they could make it into 3.8. I plan to include Shane's patches in 3.9. I'm not sure when there will be a stable release of 3.9. My patches aren't bug fixes and the plan was never to include them to ZODB 3.8. However, I was hoping for inclusion of at least the `storage-iterator-branch` in 3.9 and I asked a few times over the past months already for review but haven't gotten any responses at all so far. This branch has been used in production for a few month already. I haven' t had time to review this yet. I'll get to it eventually. I'm sure I'll get to it before 3.9 is released. \o/ In addition there's the `bushy-directory` branch which introduces a new blob directory structure to avoid scalability issues with filesystems like ext2/3 that only accept limited numbers of entries in a directory. Hm, this sounds like a bug fix if it prevents failures. It is. We ran into a failure on a production system that ran ext(2|3). This branch is being used in production but needs a little which should happen in the next days. (The packing is reported to block.) Is the packing blockage unique to this branch? Blocking currently happens on both trunk and 3.8. At last there's a small `ctheune-betterimport` branch which fixes ZEXP import for weakref objects and has been sitting around for almost a year now too. Is this a bug fix for 3.8? Or a feature for 3.9? I can't remember when 3.8 came out. Back than it was a feature. OTOH you could call it a bugfix because you could create ZEXP files that could never be imported again. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] what could cause this kind of cache corruption?
On Fri, 2008-08-01 at 09:21 -0400, Jim Fulton wrote: On Jul 31, 2008, at 1:53 PM, Chris Withers wrote: What I'd *really* like is a stable zodb release with Christian's patches for zeoraid and Shane's patches for RelStorage that then feeds through into a stable release of Zope 2. I'm not familiar with Christians patches. If they are bug fixes, they could make it into 3.8. I plan to include Shane's patches in 3.9. I'm not sure when there will be a stable release of 3.9. My patches aren't bug fixes and the plan was never to include them to ZODB 3.8. However, I was hoping for inclusion of at least the `storage-iterator-branch` in 3.9 and I asked a few times over the past months already for review but haven't gotten any responses at all so far. This branch has been used in production for a few month already. In addition there's the `bushy-directory` branch which introduces a new blob directory structure to avoid scalability issues with filesystems like ext2/3 that only accept limited numbers of entries in a directory. This branch is being used in production but needs a little which should happen in the next days. (The packing is reported to block.) At last there's a small `ctheune-betterimport` branch which fixes ZEXP import for weakref objects and has been sitting around for almost a year now too. Christian ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB fails on test
On Sun, 2008-08-03 at 11:36 +0300, Malka Cymbalista wrote: We are trying to install ZODB3-3.4.2 on a Linux machine running Red Hat Enterprise Linux AS release 4 and python 2.5.2. We ran python setup.py build which seemed to run ok except that we got very many warnings of the type In file included from Dependencies/BTrees-ZODB3-3.4.2/BTrees/BTreeModuleTemplate.c:335, from Dependencies/BTrees-ZODB3-3.4.2/BTrees/_IFBTree.c:34: Dependencies/BTrees-ZODB3-3.4.2/BTrees/BTreeItemsTemplate.c:389: warning: initialization from incompatible pointer type Dependencies/BTrees-ZODB3-3.4.2/BTrees/BTreeItemsTemplate.c:391: warning: `intargfunc' is deprecated (declared at /usr/local/include/python2.5/object.h:133) We then ran python test.py and got the results that are in the attached file. This does not look right. Can we continue with the installation? Any help will be appreciated. ZODB 3.4 is rather old and it doesn't surprise me the tests won't even really start up with Python 2.5. Try .. uh .. Python 2.4 or maybe even 2.3. Christian ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Blob savepoints leaving garbage?
Hi, I think I didn't totally grasp the way that blob savepoints work correctly. At one of our installations we have lots of .spb files left around. Looking at the code I don't see where .spb files get cleaned up again on savepoint aborts. On the bushy branch I found that having the TmpStore create directory structures is probably a bad idea. I propose to have it create a directory `savepoints` in the temporary directory and create files called `oidrepr-tidrepr.spb`. Comments? Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob savepoints leaving garbage?
On Thu, Jun 26, 2008 at 08:32:25PM +0200, Christian Theune wrote: Hi, I think I didn't totally grasp the way that blob savepoints work correctly. At one of our installations we have lots of .spb files left around. Looking at the code I don't see where .spb files get cleaned up again on savepoint aborts. I reviewed the savepoint tests and see that they don't check for cleanup: neither on rollback, nor on abort. On the bushy branch I found that having the TmpStore create directory structures is probably a bad idea. I propose to have it create a directory `savepoints` in the temporary directory and create files called `oidrepr-tidrepr.spb`. I implemented this on the branch and it seems to work well. I find it easier to inspect on the filesystem than having savepoint state being distributed over the whole structure. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Branch screwup, I used the trunk
Grrr. I screwed up my branching for the directory layout and at least one checkin went to the trunk. I'm cleaning it up right now, please give me some time to see what happened. I'll report back in a while. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Branch screwup, I used the trunk
On Sat, Jun 21, 2008 at 02:30:54PM +0200, Christian Theune wrote: Grrr. I screwed up my branching for the directory layout and at least one checkin went to the trunk. I'm cleaning it up right now, please give me some time to see what happened. I'll report back in a while. Looks like I forgot to switch my working copy when I branched off. To clean this up I removed the branch, branched the trunk off again and backed out my changes from the trunk afterwards. Sorry, Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
On Fri, Jun 20, 2008 at 07:35:10PM +0200, Christian Theune wrote: [...] and I'm working on an offline migration script that should be completed by tomorrow. The offline migration script is completed. I'm currently trying to make the tests run on windows. The code probably does run, but I get lots of errors due to path names in doctests and bogus permission information. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
Me again, On Thu, Jun 19, 2008 at 01:38:38PM +0200, Christian Theune wrote: Hi, We propose to introduce a new mode for the blob storage which breaks the directory structure into one level per byte of the oid. This would lead to directories 0x00-0xFF nested in 8 levels. I implemented a `bushy` layout and started calling the old layout `lawn` in reference to the DirectoryStorage terminology. The branch is 'ctheune-bushy-directory`. The layouts are pluggable now, provide 100% compatibility to the current implementation and I'm working on an offline migration script that should be completed by tomorrow. I'd like to get feedback and eventually the allowance to merge this into the trunk, we need this rather urgently so I'd be happy to fix any issues quickly that hinder the merge if they get pointed out. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
Hi, On Fri, Jun 20, 2008 at 07:55:17PM +0200, Andreas Jung wrote: --On 19. Juni 2008 14:31:47 +0200 Martijn Faassen [EMAIL PROTECTED] wrote: Hi there, On Thu, Jun 19, 2008 at 1:38 PM, Christian Theune [EMAIL PROTECTED] wrote: [snip] We propose to keep both implementations around and allow to select which one to use. We would extend the FileSystemHelper to abstract the two strategies. Could the default be to choose the more scalable one, or would this lead to backwards compatibility issues? Good point especially because Zope 2.11 is now inofficially released and people will start using blobs based on the limited directory layout...so how would a migration look like when people do encounter the current limitations? It's 100% backwards compatible. When the old layout is used a warning is issued with a hint to migrate but I continue to fully support the old data structures. When a new directory is started, the new layout becomes the default. Migration comes in the form of a command line script that can convert the old structure to the new one. It requires the database to be offline, but I'm working hard to make it reasonably fast. The script will be available ... later today. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Blob directory structure scalability limits
Hi, one of our installations hit a scalability limits with the current blob directory structure. The current structure looks like: \blobs\ oid\ tid.blob tid.blob ... oid\ ... ...\ We hit a limit with a database that contains more than 32k blob objects because ext3 doesn't allow more than 32k entries in a directory. We propose to introduce a new mode for the blob storage which breaks the directory structure into one level per byte of the oid. This would lead to directories 0x00-0xFF nested in 8 levels. The last directory denotes the blob itself and looks like the current directory: a list of blob files named by the tids they were committed for. We propose to keep both implementations around and allow to select which one to use. We would extend the FileSystemHelper to abstract the two strategies. We would also provide a migration tool that can convert the old format to the new format. Comments? Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
Hi, On Thu, Jun 19, 2008 at 09:29:13AM -0400, Jim Fulton wrote: We propose to introduce a new mode for the blob storage which breaks the directory structure into one level per byte of the oid. This would lead to directories 0x00-0xFF nested in 8 levels. +1 ... We propose to keep both implementations around and allow to select which one to use. We would extend the FileSystemHelper to abstract the two strategies. -1 to making this selectable. We would also provide a migration tool that can convert the old format to the new format. Maybe we can support, but deprecate the old layout and provide a conversion scriptt. Good for me as well. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Ruby/Smalltalk OODB
Hi, this might be interesting to ZODB users and developers: http://rss.slashdot.org/~r/Slashdot/slashdot/~3/302177093/article.pl -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb does not save transaction
On Wed, May 28, 2008 at 07:55:19PM -0700, tsmiller wrote: ZODB list, I have a bookstore that uses the ZODB as its storage. It uses qooxdoo as the client and CherryPy for the server. The server has a 'saveBookById' routine that works 'most' of the time. However, sometimes the transaction.commit() does NOT commit the changes and when I restart my server the changes are lost. This sounds like you are using mutable data types (like lists or dicts) in the non-persistence aware variants. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and BLOBs
On Wed, May 07, 2008 at 08:33:34AM -0300, David Pratt wrote: Hi Christian. Curious, what were you using for postgres replication? Many thanks. The only solution that I know of is slony (which is also sold in a commerical wrapping by EnterpriseDB) which is very awkward. I'm not aware of other solutions although people say they exist. -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and BLOBs
On Tue, May 06, 2008 at 01:47:45PM -0600, Shane Hathaway wrote: David Durham, Jr. wrote: My questions is whether or not RelStorage supports BLOBs. My situation is that I have a Plone site with user uploaded images and other files, that should presumably be stored as BLOBs with RelStorage. I'm open to placing BLOBs outside of RelStorage if that's reasonably accomplished. I think an NFS mount is still a better place for BLOBs than a relational database. If we put BLOBs in a relational database, we'd have to carefully dance with inter-thread connections and simulated file handles that may stay open for a long time. With NFS those concerns are already taken care of. The normal way to set up BLOB storage in zope.conf is to create a blobstorage wrapper around a storage configuration. That might just work with RelStorage, but I haven't tried it yet. At leasts its a scenario that is intended to work. I'd be happy about feedback of people who use it. The whole point of using RelStorage, for me, is reliability since I can use Oracle features to improve failure scenarios. I realize that Zope sells a replication product, but I will not be able to obtain funding for this. I'm also curious if something like ZeoRAID would be more appropriate for my usage scenario. Well, I think RelStorage is up to the task, though the capability has not yet been proven. I'm kind of curious what a match of RelStorage and ZEORaid may give. FWIW my experiences with Postgresql replication aren't that good so you could still go that route using two RelStorages with one (or more) ZEORaids in front. Christian -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Hi, On Fri, Mar 21, 2008 at 09:08:28PM +0100, Dieter Maurer wrote: Chris Withers wrote at 2008-3-20 22:22 +: Roché Compaan wrote: Not yet, they are very time consuming. I plan to do the same tests over ZEO next to determine what overhead ZEO introduces. Remember to try introducing more app servers and see where the bottleneck comes ;-) We have seen commit contention with lots (24) of zeo clients and a high write rate application (allmost all requests write to the ZODB). I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we should look into a technique called `group commit` to get rid of the commit contention. Does anybody know this technique already and maybe has a pointer for me? Christian -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo Server as a single point of failure
Hi, On Mon, Mar 10, 2008 at 08:28:48PM -0600, Shane Hathaway wrote: Christian Theune wrote: Hi, On Mon, Mar 10, 2008 at 05:34:59PM -0500, Kenneth Miller wrote: Hello all, I'm currently developing my application ontop of zodb/zeo, and the idea of having the Zeo server the only single point of failure still bothers me. My question is, has anyone thought of any solutions for this. My idea was to have a few or more background processes that kept multiple servers synced and concurrent. Any thoughts? Just to make sure that all of the authors involved with anything here spoke up, please refer to the other two mails by Gary and Shane. ;) Heh. How is the ZEO RAID project going, anyway? It seems like a good idea and I don't think it actually depends on ZEO, does it? For example, I wonder if it's possible to create a RAID storage around a set of RelStorage instances. We have alpha1 out there (see the zope.org repository) with full fail-over and recover facilities. We made some changes to the ZODB trunk to clean up some interfaces and make ZEO support the more elaborate protocol details. I got a report from one person who put RelStorage behind a ZEO server. Apparently it works fine, though I haven't tried it myself. Maybe we're finally reaching the point where we can choose from several ZODB storage components and connect them with ease. Yes, reaching that stage would be nice. Christian -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: become a ZODB mentor in the Google Summer of Code!
Martijn Faassen schrieb: Hey, Martijn Faassen wrote: It would be great if we put our community's secret gem, the ZODB, into the limelight more, and the Google Summer of Code would be a great opportunity. We need mentors, and fast, so if you want to mentor someone, please sign up in this wiki page here: http://wiki.zope.org/gsoc/SummerOfCode2008 No takers? It's on my list, I'll get around to it tomrrow, I guess. (need to get a google login. i hate that.) -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
