Re: [ZODB-Dev] Indexing and dates/times
On Tue, Jul 13, 2010 at 4:35 AM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Hello, I am currently trying to devise a way to index and retrieve some millions of objects according to their modification date/time. So, in a relational DB i would do something like: SELECT * FROM table WHERE timestamp = X AND timestamp = Y Since I cannot do this with ZODB, I'd have to have a BTree, indexed by timestamp... however, as you said, if I want to the second granularity, I will rarely have two items with the same key (which makes it pretty useless). If you use the timestamp as the key and you want to retrieve all values between two timestamps (inclusive), you can do my_btree.values(min=start, max=end) -- Benji York ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
On Tue, May 11, 2010 at 7:34 AM, Jim Fulton j...@zope.com wrote: [...] The best I've been able to come up with is something like: t = ZODB.transaction(3) while t.trying: with t: ... transaction body ... I think you could get this to work: for transaction in ZODB.retries(3): with transaction: ... transaction body ... ZODB.retries would return an iterator that would raise StopIteration on the next go-round if the previously yielded context manager exited without a ConflictError. -- Benji York ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Automating retry management
On Tue, May 11, 2010 at 10:08 AM, Jim Fulton j...@zope.com wrote: This is an improvement. It's still unsatisfying, but I don't think I'm going to get satisfaction. :) Given that PEP 343 explicitly mentions *not* supporting an auto retry construct, I should think not. :) -- Benji York ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
On Wed, Dec 2, 2009 at 5:34 PM, Godefroid Chapelle got...@bubblenet.be wrote: As far as I can understand, THE reason for repozo is to backup without stopping the ZODB. [I'm a little late to this conversation.] Filestorages can be backed up without stopping anything. Any incomplete writes that are in progress at the time of the copy will simply be ignored when the resulting backup is opened later. -- Benji York ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] client cache bugs when a client switches from one zeo server to another
On Tue, Nov 24, 2009 at 9:02 AM, Chris Withers ch...@simplistix.co.uk wrote: - have there been any client cache bugs fixed with Zope 2.9.8 that would cause this kind of behaviour when a client has multiple zeo servers and switches between the two due to failure of one of them? If I recall correctly, one or more (re)connection-time cache corruption bugs were fixed in ZODB 3.8.1. -- Benji York ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO Cache inconsistency
On Tue, Oct 20, 2009 at 5:41 AM, Vincent Pelletier vinc...@nexedi.com wrote: I'm facing a cache consistency problem in an old Zeo running on 2.8.8 products. I don't know what version of ZODB those Zope versions use, but version 3.8.1 of ZODB included several bug fixes in the ZEO cache code. -- Benji York ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO and relstporage performance
On Wed, Oct 14, 2009 at 3:38 PM, Ross J. Reedstrom reeds...@rice.edu wrote: On Wed, Oct 14, 2009 at 02:20:50PM -0400, Benji York wrote: We're actually set up w/ squid in front of the zope FEs, using IPC to talk to them all. The default behavior is just to respond w/ a CACHE MISS and use network access timings to select. This is non-optimal, since it does little true load-balancing, until the FE is completely hammered (very non-linear response-time curve). I'd love to see an example where someone replaced the default response w/ something more meaningful. You'd probably be interested in the last four paragraphs of http://benjiyork.com/blog/2008/03/icp-for-faster-web-apps.html. The shoal that replying HIT' for ZEO cached data breaks on is that the IPC request contains a URL, not ZODB object refs. And converting one to the other is what the whole dang machine _does_. Right, but in many situations you can guess effectively. See below. And you have very little time to answer that IPC query, lest you destroy the gains you hope to get from having a hot-cache. So some ad-hoc approximation, like keeping the last couple hundred URLs served, and responding 'HIT' for those, might get some part of the benefit. I'm thinking about something more specific. For example, lets say that your data is a dictionary application and URLs are of the structure: my-site.com/WORD, where word is the word you want to show the definition of. If you generate an IPC HIT if the process has served a word beginning with the same letter recently, then particular letters will begin to be associated with particular servers, increasing the ZEO (and ZODB) cache hit rates. If you take this a step further and design your app so that you load all the words that start with a letter as one giant object, you'll get an even greater effect. (Note that in real life that object would be too giant, but you get the idea.) This is probably the wrong list for it, but does anyone know of a published example of replacing Zope's default icp-server response? Last time I looked, I couldn't find one. Other than my blog post and zc.icp the only other ICP and Zope info I know of is http://www.zope.org/Members/htrd/icp/intro (very old). Note that I haven't used zc.icp in a long time because we decided to move away from ICP and (eventually) move to a load balancer that implements our desired policy instead. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
On Mon, Sep 14, 2009 at 2:44 PM, Shane Hathaway sh...@hathawaymix.org wrote: People have been looking at these performance charts for quite a while: Are the benchmarks publicly available? I looked through the blog posts and svn.zope.org but couldn't find them. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems packing Zope 2.9.8 storage
On Mon, Sep 14, 2009 at 3:52 PM, Shane Hathaway sh...@hathawaymix.org wrote: Benji York wrote: On Mon, Sep 14, 2009 at 2:44 PM, Shane Hathaway sh...@hathawaymix.org wrote: People have been looking at these performance charts for quite a while: Are the benchmarks publicly available? I looked through the blog posts and svn.zope.org but couldn't find them. [snip answer] Thanks! -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Temporary files not closed
On Thu, Jun 25, 2009 at 4:46 AM, Izak Burgeri...@upfrontsystems.co.za wrote: Hi all, This is a zope issue that I think is probably zodb related. I've been seeing this behaviour for a couple of years now. When I inspect /proc/PID/fd (on a linux machine, this lists the open file descriptors held by the process with process id PID), I see many descriptors that point to open files, eg: $ ls -go /proc/15431/fd | grep deleted lrwx-- 1 64 2009-06-25 10:09 13 - /var/tmp/tmptBEWCx.tbuf (deleted) lrwx-- 1 64 2009-06-25 10:09 14 - /var/tmp/tmpKSuzMN (deleted) lrwx-- 1 64 2009-06-25 10:09 6 - /var/tmp/tmp2HUAnn.tbuf (deleted) lrwx-- 1 64 2009-06-25 10:09 7 - /var/tmp/tmpex5QmJ (deleted) Have you looked at the contents of the files? Once you have an idea of what is being written you might be able to determine what is doing the writing. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB3 3.9.0b2 released
On Fri, Jun 12, 2009 at 5:22 AM, Jim Fultonj...@zope.com wrote: I didn't get these and I tested on Ubuntu, although a pretty old release. Tests pass on a more recent release as well (Python 2.4, Ubuntu 8.04.2) -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.0b1 released
On Mon, May 4, 2009 at 3:03 PM, Jim Fulton j...@zope.com wrote: I've just released the first beta of ZODB 3.9. Please try this out and report bugs found. Is the fact that http://pypi.python.org/pypi/ZODB3/3.9.0b1#compatibility mentions ZODB 3.8 and not 3.9 a bug? -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Very large amounts of output from fsrefs.py
On Thu, Aug 28, 2008 at 1:04 AM, Paul Winkler [EMAIL PROTECTED] wrote: On Wed, Aug 27, 2008 at 11:03:54AM -0400, Marius Gedminas wrote: ZEO had a bug, fixed in rev 73871, where it would spend up to 10ms (your OSes scheduling timeslice) in idle sleep for every object received from the ZEO server, resulting in very long request processing times on an otherwise idle server, if a request had to load hundreds or thousands of persistent objects. No kidding?? If you're interested in the background, the original thread is at http://mail.zope.org/pipermail/zodb-dev/2007-March/010846.html. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB fails on test
On Sun, Aug 3, 2008 at 6:42 AM, Malka Cymbalista [EMAIL PROTECTED] wrote: We need ZODB3.4 because we are installing the Cern Document Server Indico package and this needs python 2.4 or later. We already had python 2.3.4 but we installed pythin 2.5 because of Indico. Do you think it would be possible to build ZODB3.4 with python 2.3.4 and build indico with python 2.5? Would this work or would it clash? We have no experience with python so we're not sure what to do. The above sounds to me like you have two different apps that require different versions of Python. If so, you can install two Pythons, and the correct Python for each app. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Checkins] SVN: zope.component/trunk/src/zope/component/registry.py Correct error in persistance-bbb logic spotted by Jacob Holm
On Thu, Jul 24, 2008 at 1:44 PM, Wichert Akkerman [EMAIL PROTECTED] wrote: Log message for revision 88796: Correct error in persistance-bbb logic spotted by Jacob Holm It seems strange (and a bit frightening) that -- ostensibly -- the tests pass either way the code reads. Is there a missing test here? Changed: U zope.component/trunk/src/zope/component/registry.py -=- Modified: zope.component/trunk/src/zope/component/registry.py === --- zope.component/trunk/src/zope/component/registry.py 2008-07-24 17:32:08 UTC (rev 88795) +++ zope.component/trunk/src/zope/component/registry.py 2008-07-24 17:44:53 UTC (rev 88796) @@ -75,7 +75,7 @@ if provided is None: provided = _getUtilityProvided(component) -if (self._utility_registrations.get((provided, name)[:2]) +if (self._utility_registrations.get((provided, name))[:2] == (component, info)): # already registered return -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] large database files
On Wed, Jul 23, 2008 at 2:01 AM, Sean Allen [EMAIL PROTECTED] wrote: step 1, move about 10% of our data over to get an idea about speed of searching 1.5 million entries etc. standard file storage... the csv file i dumped all the info to is about 190 megs, the resulting file storage file is 750 megs ( 1.8 gigs before packing ). Does this mean, that I should expect the entire database to close in at something like 7.5 gigs? Not a bad guess. If yes, are there problems that can arise for a database of that size? You should be fine. That's a small database by some standards. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] large database files
On Wed, Jul 23, 2008 at 2:01 AM, Sean Allen [EMAIL PROTECTED] wrote: i'm starting to do some serious playing around with zodb. step 1, move about 10% of our data over to get an idea about speed of searching 1.5 million entries etc. standard file storage... the csv file i dumped all the info to is about 190 megs, the resulting file storage file is 750 megs ( 1.8 gigs before packing ). Does this mean, that I should expect the entire database (After reading your other message about indexing); if you aren't doing any indexing yet, then you should expect a larger database once the indexes are added. Some advocate keeping indexes in a separate database, especially if they are a significant fraction of the total database size. That way you can have different packing schedules, backup plans, storage devices, etc. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob directory structure scalability limits
On Thu, Jun 19, 2008 at 7:38 AM, Christian Theune [EMAIL PROTECTED] wrote: We propose to introduce a new mode for the blob storage which breaks the directory structure into one level per byte of the oid. This would lead to directories 0x00-0xFF nested in 8 levels. I suppose the directories would be built on-demand, right? -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Tue, Jun 3, 2008 at 1:13 PM, Gary Poster [EMAIL PROTECTED] wrote: What you *can't* do out of the box is ask hey, what products have an attribute that points to this brand?. That's a back-reference, and that needs solutions like the ones to which I was referring. In other words: if you want an index of references (or any attribute for that matter), you have to set it up (just as you would for a RDBMS). -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Mon, Jun 2, 2008 at 12:31 PM, Tim Cook [EMAIL PROTECTED] wrote: I benefit from Zope and the ZODB especially. I know that I am a little better at PR than I am at the deep technical aspects (not very good) so I will initially gather (I already have lots of bookmarks) all the info I can find and start building something coherent out of it. Great! Can I have a few minutes of attention to emails now and then from experts? I'm not a ZODB expert, but I'll do what I can to help you. I am NOT good at the flashy layout and graphics stuff that attracts the attention of people. I will needs some help in that respect. organize information into something that new-comers can lear Hopefully the already underway design effort will help there. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO Client deadlocking in asyncore.poll - how to I debug
On Fri, May 23, 2008 at 5:45 AM, Anton Stonor [EMAIL PROTECTED] wrote: Last month I ask for help about hanging zeo clients. I promised to follow up. And thank you for doing so. It's very helpful (for me at least) to see these sorts of things come to closure. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problems installing ZODB under Windows
Andreas Holtz wrote: Running zope.interface-3.4.1\setup.py -q bdist_egg --dist-dir c:\docume~1\holtza\locals~1\temp\easy_ install-9zm01c\zope.interface-3.4.1\egg-dist-tmp-8pfxjw I believe that if there were a zope.interfaces 2.4.1, Python 2.5 egg for Windows available on PyPi, setup_tools wouldn't try to build it for you. A request for such an egg would be best directed to [EMAIL PROTECTED] -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Christian Theune wrote: I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we should look into a technique called `group commit` to get rid of the commit contention. Does anybody know this technique already and maybe has a pointer for me? I'd never heard the phrase until reading your message, but I think I got a pretty clear picture from http://forums.mysql.com/read.php?22,53854,53854#msg-53854 and http://archives.postgresql.org/pgsql-hackers/2007-03/msg01696.php. Summary: fsync is slow (and the cornerstone of most commit steps), so try to gather up a small batch of commits to do all at once (with only one call to fsync). Somewhat like Nagle's algorithm, but for fsync. The kicker is that OSs and hardware often lie about fsync (and it's therefore fast) and good hardware (disk arrays with battery backed write cache) already make fsync pretty fast. Not to suggest that group commit wouldn't speed things up, but it would seem that the technique will make the largest improvement for people that are using a non-lying fsync on inappropriate hardware. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Dieter Maurer wrote: We do not yet precisely the cause of our commit contentions. Hard to tell what'll make them better then. ;) Almost surely there are several causes that all can lead to contention. We already found: * client side causes (while the client helds to commit lock) - garbage collections (which can block a client in the order of 10 to 20 s) Interesting. Perhaps someone might enjoy investigating turning off garbage collection during commits. - NFS operations (which can take up to 27 s in our setup -- for still unknown reasons) Not much ZODB can do about that. ;) - invalidation processing, espicially ZEO ClientCache processing Interesting. Not knowing much about how invalidations are handled, I'm curious where the slow-down is. Do you have any more detail? * server side causes - commit lock hold during copy phase of pack - IO trashing during the reachability analysis in pack The new pack code should help quite a bit with the above (if you're saying what I think you're saying). - non deterministic server side IO anomalities (IO suddently takes several times longer than usual -- for still unknown reasons) Curious. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Roché Compaan wrote: On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote: Maybe if you set up a ZODB cache that allows just over 10 million objects, the lookup time will drop to microseconds. You might need a lot of RAM to do that, though. Maybe, but somehow I think that disk IO will prevent this. I'll check. If you're on Linux, you can tweak swappiness (/proc/sys/vm/swappiness; http://lwn.net/Articles/83588/) to affect how much RAM is used for the page cache and how much for your process. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO+MultipleClients+ConflictErrors
Kenneth Miller wrote: Is there a standard bit of documentation as to how to handle a ZEO server with multiple clients writing to the same data structure at the same time? Not documentation, but there is a standard bit of snark that might be helpful: don't do that. ;) In other words, the best way to deal with a hot spot is to eliminate it. That may or may not be possible, but it's a good thing to try. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO+MultipleClients+ConflictErrors
Alan Runyan wrote: Could you provide a bit more guidance? Maybe an example. Here's one: in certain circumstances people want to aggregate lots of incoming data, but doing the obvious thing of updating a data structure as new data comes in turns out to be too slow. One way to handle that situation is to buffer incoming data and do a single aggregate write periodically (the approach taken by QueueCatalog). Another simple one is wanting to keep up with a count, the naive way will get you in trouble (incrementing an attribute on an object), but using a the BTree.Length object helps a lot. Most people come at ZODB with previous experience in RDBMS. How do they map SQL INSERT/UPDATE activities to ZODB data structures? In a way that does not create hotspot. A few years ago people would say pick a BTree implementation that maps to your data structures and use that. But that is naive and pushes the hotspot into the BTree where concurrent updates are happening. Naivety is relative. Most people don't push their RDBMS or ZODB to their limits, so a straight-forward approach is often best. Inserting object references into an IOBTree, for example, seems reasonable, being careful to choose good integer keys so there's little chance of keys conflicting. If your insert rate is high enough that bucket splits cause too many conflicts, then you'll have to deal with that. As I understand it one option is to choose your keys such that it is unlikely that two insertions happen in the same bucket. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
Flavio Coelho wrote: Actually what I am trying to run away from is the packing monster ;-) Jim has done a great deal of work on packing (that will go into 3.9 I presume) that should make your pack 3 to 6 times faster (depending on if you do garbage collection at pack time or not). I want to be able to use an OO database without the inconvenience of having it growing out of control and then having to spend hours packing the database every once in a while. (I do a lot of writes in my DBs). Do this Holy grail of databases exist? :-) Why not put the pack in cron? -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Getting started with ZODB
Jim Fulton wrote: On Sep 18, 2007, at 6:46 AM, Manuzhai wrote: ... What's new in ZODB4? The ZODB project has been abandoned. Much of the work done in that project was folded back into ZODB3. Jim means ZODB *4*. :) -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB 3.8.0b1 is released
Christian Zagrodnick wrote: In my case I shared the blob storage with the zeo client w/o telling the zeo client that it actually was shared. I would imagine that to be a common mistake. It would be nice (as you suggest) for that particular misconfiguration to be detected and warned against. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Concurrent transactions
Kai Diefenbach wrote: Within an adapter I increase a counter (e.g. for every download of an file) by doing: self.counter[0] += 1 self.counter is a PersistentList stored as annotation. Note that if you use a PersistentList, then the entire list must be written to the database each time a non-Persistent element changes (or an item is added or removed from the list). If the list is of any appreciably size, that will be bad. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2
Alan Runyan wrote: The website got slashdoted this morning [...] Just FYI: Varnish didnt go over 3% CPU during the traffic surge; over 200 req/second. Off topic: 200 requests a second seems a bit light for a slashdotting, any more details you can divulge there? -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO client cache tempfile oddness
Paul Winkler wrote: Our experiments suggest that ext2, ext3, and reiserfs optimize for sparse files so there is no such guarantee. AFAICT from some quick googling and wikipediaing, the same is true for NTFS, XFS, JFS, ZFS. I suspect we've accounted for the majority of the production Zope installations in the world. In that case it would seem better to just remove the ineffectual code altogether. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] two level cache
Atmasamarpan Novy wrote: A current ZODB cache will become a first level cache (L1), each ZODB connection maintains its own copy. When an object is needed it reads it from L2 cache instead from a storage. L2 cache is a common for all ZODB connections. Objects are stored there in the state where references are not yet resolved from their persistent form. If you're using FileStorage without ZEO, then your operating system's disk cache will act in a way similar to what you're describing as your L2 cache. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob test failure on FreeBSD
Gary Poster wrote: FWIW, It smells more like a 64 bit vs. 32 bit machine/OS question, but that's just on the basis of fly-by-diagnosis. :-) Yep, looks like it. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob test failure on FreeBSD
Christian Theune wrote: Who's the owner of the machine? You might know him. A guy by the name of Thomas Lotze. :) -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob test failure on FreeBSD
Christian Theune wrote: Am Dienstag, den 27.03.2007, 12:44 -0400 schrieb Benji York: Christian Theune wrote: Who's the owner of the machine? You might know him. A guy by the name of Thomas Lotze. :) Nope. That's the buildbot that has been offline for ages. ;) The failing one is 'tmiddleton'. :) Indeed! I read the wrong column. I guess I need to ping Thomas about restarting his slave, or just remove it. To answer your original question: that machine is 64-bit. tmiddleton is Tim Middleton ([EMAIL PROTECTED]). I've copied him on this message. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: PyPI-Entry
Jim Fulton wrote: Does anyone *really* want an exe installer? Regressing to a previous life: it would be more inviting to Windows-folk to have an executable installer; at least until easy_install permeates the community some more. That assumes that inducing uptake is a major goal for the ZODB project. It might well be possible that the cost of executable installers is greater than their value. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Invalidating caches for module refresh doesn't seem to work
Jim Fulton wrote: Python modules were not designed to be reloadable. If you want to change that, I suggest taking that up on the Python 3000 list. (Maybe someone already has. Regrettably, I don't have time to watch that.) I follow it pretty well and don't recall that being discussed. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HOWTO recover from Data.fs.tmp
Russ Ferriday wrote: But for an app to be writing into thin air without complaining - THAT's a bug! Even having a backup of the Data.fs would not help! Not picking on zope, but if that's what's happening it has to be called a bug. True. It is much more likely that the Data.fs was replaced about a week ago (say an attempted backup restore) and the Zope process wasn't restarted. That is one way to evoke the observed behavior. Since this is apparently a flavor of Unix, the output of lsof might be helpful. It should show you if you have a Zope process that is holding onto a deleted file. If this is the case you'll have to get imaginative to recover the file. Killing the process and immediately attempting to undelete it might work (for some file systems, apparently not ext3). -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: HOWTO recover from Data.fs.tmp
Tres Seaver wrote: Rather, I would verify ths by looking at /proc/PID_OF_ZOPE/fd. If it looks like the following: $ ls /proc//fd Spired by Tres' comment, I dug around in /proc/ and found the link associated with Data.fs of a running ZEO on my machine. I copied the file to /tmp. The original Data.fs and the recovered one are identical. This makes me wonder if the same will work for the OP. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HOWTO recover from Data.fs.tmp
Juan Pablo Giménez wrote: El lun, 02-10-2006 a las 14:00 -0400, Benji York escribió: Data.fs.tmp usually only contains the last transaction (although, it may not free unused space at the end). but... that's a bug... why zope keep writing in .tmp file if we can't recover nothing from there?!?!... Why is it a bug? A .tmp file is just that, temporary. It's not a backup. ok... but, if you delete Data.fs zope keep running, writing data into Data.fs.tmp, and don't log nothing about it. Sorry, but that's a bug... That's the way Unix-like file systems work. Perhaps you should switch to Windows, it doesn't (easily) allow you to delete in-use files. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Newbie questions re persistent objects and connections: [Was: can Leo...]
Edward K. Ream wrote: This is a repost of my original: Can Leo use zodb as a file system. Yes (s/file system/persistent store/). Start here: http://www.zope.org/Wikis/ZODB/guide/zodb.html This might also be helpful, although potentially slightly out of date: http://www.zope.org/Members/adytumsolutions/HowToLoveZODB_PartI -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Clustering ZEO Server with drbd
Christian Theune wrote: In the end, this might not be exactly HA because you'll have a period of time that isn't covered and may stretch for quite a while. An interesting idea: use drdb to create an HA pair for the read/write ZEO service, and also use a read-only ZRS secondary. That way when the primary r/w ZEO server went down, read transactions could still complete until the second r/w ZEO server could start. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Heads-up: Test failure on trunk/Python 2.4.2/OS X
Jim Fulton wrote: I'll update the buildbot setups to run all tests. That would be a good interim solution; long term I'd like to get dedicated ZODB buildbot slaves so they can be set up to react to check ins to ZODB, instead of coincidentally being run when Zope is changed. I'll add that to my list of things to do next time I have a buildbot day. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
