Re: [ZODB-Dev] [Persistent] STICKY mechanism unsafe
On 8/20/07, Jim Fulton [EMAIL PROTECTED] wrote: On Aug 18, 2007, at 5:01 AM, Dieter Maurer wrote: Excellent analysis snipped 1. and 3. (but obviously not 2.) could be handled by implementing STICKY not by a bit but by a counter. This has been planned for some. :/ I think Jim mentioned this problem to me when I started working on ZODB :-). Jeremy Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
On 3/20/07, Chris Withers [EMAIL PROTECTED] wrote: Dieter Maurer wrote: Chris Withers wrote at 2007-3-16 08:45 +: ... Is there any way an object could be invalidated without there being a non-current revision to read? Sure (through a call to ZODB.DB.DB.invalidate), although usually it is done only after the object changed. OK. I'm still not clear on whether what you describe is a bug or simply what might happen now? The MVCC implementation might not be prepared to cope with an explicit call to invalidate. Yould also achieve this by calling some method on the object, right? _p_invalidate()? I don't remember the details. In either case, I suspect that is a bug in the implementation, but it's not clear if it explains your problem. Jeremy We are using it to cause objects to be flushed from all ZODB connection caches (because we know that their _v_ might contain an inconsistent state). Fortunately, we did not yet met the loadBefore problem you have hit. Oh, so it's just me who's lucky? ;-) What are the possible causes of invalidation? A change of the object or some explicit call for invalidation. I don't think any explicit calls are involved here... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mvcc related error?
On 3/15/07, Chris Withers [EMAIL PROTECTED] wrote: Dieter wrote: Unfortunately, neither of these means anything to me ;-) That is because you did not look at the code :-) Much as I wish I had time to read and learn the whole zodb code base, I don't. It wasn't clear what that code did and what those assertions really meant... The code in question has some docstrings that explain the basic idea. You certainly don't need to read the whole codebase. _setstate_noncurrent(obj) attempts to load the state of obj that was current before the transaction started (technically, before _txn_time). loadBefore() returns a 3-tuple including the transaction ids the delimit the lifetime of this particular revision of the object. It was written by transaction start and was current until transaction end committed. If end is None, it implies that the revision returned by loadBefore() is the current revision. There is an assert here, because the _setstate_noncurrent() is only called if the object is in the invalidated set, which implies that there is a non-current revision to read. If I had to guess, I'd say it was a bug in loadBefore(). It looks like the only ways for loadBefore() to return None for end are - The very first record for the object has a transaction id less than the tid argument. If so, end_tid is never set. Not sure this is compatible with the object being in the invalidated set. - Something is happening with versions. Are you using versions? It seems likely that there are bugs here. - There's a bug in the code that reads the data record from the storage where it reads None for a transaction id. That seems very unlikely. Perhaps the reasoning about invalidated sets and transaction ids is wrong in the presence of versions. MVCC should not work with versions, but I don't see code that will abort the loadBefore() call if the Connection has a version. You aren't using versions, are you? Jeremy Jeremy Jim wrote: I'm glad you brought that up. I'd like to set up a project in Launchpad. https://bugs.launchpad.net/zodb/+bug/92507 cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Where did self._p_jar._storage._serial go?
On 7/4/06, Jean Jordaan [EMAIL PROTECTED] wrote: Answering myself .. FileStorage in 2.8.7 doesn't have '_serial' any more. Is there something equivalent I can get hold of and use? It looks like '_tid' fits the bill. It's not available when using ZEO though, which took me a while to figure out. _tid is what you want. tids and serial numbers were merged a few years ago. jeremy -- jean ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeoup.py writes to Data.fs?
On 3/8/06, Chris Withers [EMAIL PROTECTED] wrote: Is it just me or does zeoup.py write a transaction to the end of Data.fs containing a MinPO object? That sounds right. What does the source say? Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On 3/2/06, Chris McDonough [EMAIL PROTECTED] wrote: On Mar 2, 2006, at 10:48 PM, Gary Poster wrote: On Mar 2, 2006, at 10:03 PM, Chris McDonough wrote: persistence-3.6.0.egg (which would include persistent and BTrees modules). zodblib-3.6.0.egg (which would include ZODB and ZODB packages). transaction-3.6.0.egg (which would include only the transaction package). zconfig-2.3.1.egg (which would include only the ZConfig package). zodb-3.6.0.egg (which might be an empty dependency package that depended on persistence, transaction, zodblib, and zconfig). Anyone care? First, rah rah!. Go go! I don't understand the eggs philosophy. (I succeeded in missing the eggs talk twice at PyCon.) I don't think any of the pieces of ZODB are very useful in isolation. You can't use persistent without transaction, and there isn't much that uses transaction other than ZODB. Same with BTrees, you can't use them without ZODB and they're probably the most useful data structure in ZODB. Software like DirectoryStorage, BDBStorage, or IndexedCatalog seem like good candidates for separate packaging. ZConfig, Jeremy Second, zodblib seems unlikely to be useful alone, but maybe I'm wrong, and even if I'm right there's precedent for that sort of thing in packaging land, so what the hey. Well, I'm only an egg, as Michael Valentine would say, so I'm not sure what the right thing is. Third, I'd guess that zodb would contain the stitching for zconfig- to-zodb. It makes me wonder if a zodb-without-zconfig egg would be feasible/useful, but that's probably for later navel-staring. Yeah, that'd be good. It's currently pretty dense in there. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Possible contributions to ZODB
On 2/3/06, Gary Poster [EMAIL PROTECTED] wrote: On Feb 3, 2006, at 9:12 AM, Florent Guillaume wrote: Gary Poster wrote: On Jan 26, 2006, at 10:16 PM, Gary Poster wrote: I'm a bit surprised there was so little discussion: ZC has needed a persistent queue several times in the past. Whether the queue would be based on what I proposed or on something else someone else has, it seems like it would be a good thing to have. However, with only Julien replying I don't think that's enough to push it further. I'll keep it in zasync until/unless some better option comes around. Like Julien I'd like to encourage you to put it in the core. I'd say, in the persistent module. OK: I'll push this a bit more then. Thanks to you and Julien for your encouragement, and for Julien's offer of help. As I recall from conversations a few years back, a persistent queue is indeed useful and surprisingly difficult to get right (or, at least, get right with decent performance). Sounds like a good candidate for ZODB. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Another interesting ZODB cache inconsistency
Is the problem with consistency of results served across the ZEO clients or by consistency of the database itself? It seems like it must be the former. In the case of an intolerable ZEO failure, I would expect to lose execution time consistency among peers but preserve consistency of committed state. ZEO can't really provide consistency across the clients anyway, since one client could be executing before a particular transaction commits and another after it commits. If two web clients talk to the two different ZEO clients, they'll see different results. A big transaction exacerbates the problem, because its takes longer to do everything (including the underlying commit on the storage). A few thoughts about the effects: - Each client should process all of the invalidations from a transaction or none. If a client loses contact with the server while invalidations are being sent, it should not process any of them. Maybe there's a bug in the code here? I haven't looked at the code lately. - If a client is disconnected, regardless of the state it was in with respect to this one transaction, it should revalidate its cache and invalidate and stale data that it held as a result of the disconnect. Jeremy On 1/13/06, Dieter Maurer [EMAIL PROTECTED] wrote: We recently observed another ZODB cache inconsistency: The commit of a huge transaction caused our ZEO server to be late in responding to the HA monitoring probe. The HA monitor responded with a SIGTERM targeted to the ZEO server. ZEO restarted. The ZEO client performing the huge transaction reported an error in the second phase of its commit state. The ZODB states of other ZEO clients were inconsitent: some of them had received invalidation messages and saw the objects modified by the huge transaction with their new values. Others had not yet received the invalidation messages and treated the objects as still unchanged. This means that interrupting ZEO while it is sending invalidation messages can cause inconsitent states in the ZODB caches of its clients. I do not know what can be done about it... -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Query Regrading ZODB FileStorage(.fs file)
On 12/29/05, Monica chopra [EMAIL PROTECTED] wrote: I have recently jumped into python and ZODB . I am quiet familar to syntax and everything but there is one Issue . when we create a .fs file or say data.fs and saves in it some objects.If we open this file in notepad or other editior. It shows the data about objects everything its name its address or whatever information an object has.you can search particular property in that file. What should be done to hide that data. You need to be more careful in formulating your question. What data are you attempting to hide? Who you are attempting to hide it from? And so. Neither ZODB nor FileStorage were designed with a thought towards encrypting the persistent representation of the data. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] More exciting ZODB errors ;-)
On 12/14/05, Tim Peters [EMAIL PROTECTED] wrote: Jeremy, the code at the start of _setstate() looks fishy: any idea what the intent was?: # There is a harmless data race with self._invalidated. A # dict update could go on in another thread, but we don't care # because we have to check again after the load anyway. if (obj._p_oid in self._invalidated and not myhasattr(obj, _p_independent) and not self._invalidated ): # If the object has _p_independent(), we will handle it below. self._load_before_or_conflict(obj) return I don't have any idea what the intent of not self._invalidated was. I looked back at 3.3a2, which I think was the first MVCC release. It doesn't have the and not self._invalidated line , and I can't think of why it would. The code is just an optimization; it's trying to avoid loading the current version if it knows that it needs a non-current version. I don't have a checkout of ZODB, so I can't check the svn blame. Does it say who wrote the code and what change it was related to? Jeremy Specifically, how could: obj._p_oid in self._invalidated and: not self._invalidated be true at the same time? The latter says _invalidated is empty, but the former can't be true if _invalidated is empty. Offhand, then, unless we're _counting_ on a thread race here ;-), it looks impossible that the body of this `if` could execute. I don't have a reason to believe it's directly relevant to Chris's problem, it's just that I got baffled by the first thing I looked at ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] More exciting ZODB errors ;-)
On 12/14/05, Tim Peters [EMAIL PROTECTED] wrote: I don't have a checkout of ZODB, so I can't check the svn blame. Does it say who wrote the code and what change it was related to? Congratulations on weaseling out of another one ;-)! It was much harder to weasel out of them when I actually had commit privileges. Turns out this all happened after your statutory blame period, as part of a large merge to fix ZClasses: http://svn.zope.org/ZODB/branches/3.4/src/ZODB/Connection.py?rev=29891view =rev That introduced the and not self._invalidated clause. We'll fix that one way or another (it doesn't make sense), but I'm pretty sure it's irrelevant to Chris's symptom (it stops an optimization rather than creates a problem). Right. It's certainly going to hurt the performance of ZODB in a high-write situation, because it will always read the current revision of an object, even when it needs a non-current revision. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PersistentMapping
On 11/17/05, Thomas Lotze [EMAIL PROTECTED] wrote: Hi, I just noticed two things about persistent.PersistentMapping: - It inherits from UserDict.UserDict. Is there any reason not to inherit from dict directly, given that this has been possible since Python 2.3 IIRC? It has been possible to inherit from dictionary since Python 2.2, but it is not possible for a persistent object and it would not do what you expect even if it were possible. A persistent object has a custom C layout and so does dict, so it is not possible to have them both as base classes. (TypeError: multiple bases have instance lay-out conflict. (I don't know why there is a hypen in lay-out.)) If it were possible, it wouldn't work anyway. If you inherit from dict and override its builtin methods, like __setitem__, your overridden method will be ignored by C code using methods like PyDict_SetItem(). - Not all methods of the mapping interface are handled. In particular, there's no reason not to handle pop() as popitems() is handled. Unhandled methods that change the content of the dict lead to especially nasty bugs as they seem to work OK during a transaction while their effect is not permanent. On the other hand, I think that this omission is just an oversight. It would probably be a good idea to change PersistentDict/Mapping to use the DictMixin instead of UserDict. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
On 11/9/05, David Binger [EMAIL PROTECTED] wrote: On Nov 9, 2005, at 9:21 AM, Syver Enstad wrote: What I want to do is just update the persistent instance so that it will load from the new module/ new class after I have done a one time update of the database. I fear this will not work. For efficiency reasons, the class designator (usually a module, classname tuple) is stored both in the object itself and in persistent references. While you will be able to modify the object's class the persistent references will only change when the containers are stored into the ZODB. Is there something to prevent your update script from setting _p_changed = 1 on every persistent object that contains a reference to an instance of the moved class? I think that if you do that and commit and pack the database, the old module/class references will be purged. If you do this, the sys.modules hacking only has to happen in the update script. How do you find all the references? It seems like you would need to execute a transaction that iterated over every object in the storage and searched the pickle for references to the class. I can imagine the update transaction taking a really long time to run, which increases the possibility that it will get a conflict error. Maybe it's a resolvable conflict, but not using the current mechanism. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
On 11/9/05, David Binger [EMAIL PROTECTED] wrote: On Nov 9, 2005, at 10:37 AM, Jeremy Hylton wrote: How do you find all the references? It seems like you would need to execute a transaction that iterated over every object in the storage and searched the pickle for references to the class. You do need to do this to find the references (using referencesf()?), but it doesn't necessarily need to happen in one giant transaction. The script just needs to (load and) re-pickle every instance with a reference while the sys.modules hack is in place. Commit each referring instance in a new transaction if you like, and conflict errors should not be a problem. I suppose you could run a separate thread that performed this conversion in parallel with other activities. There's a bit of a coordination problem to make sure you're done. It seems other application code would need to use the old class definition until the update was completed, but that means new references can be created while the update is running. So you'd need some sort of coordination to atomically switch from old class to new class when the update was completed. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Multi-Threading Transactions
On 10/13/05, Chris Spencer [EMAIL PROTECTED] wrote: I understand that, but my point was when you call transaction.commit(), you don't necessarily know what you're committing. One thread may be ready to commit. Another may not be. If one thread calls transaction.abort(), they may be aborting changes made by another thread. You'd have to write a complicated coordination mechanism to handle multi-threaded transactions to be sure. If threading is limited to connections, then wouldn't in make sense if transactions were as well? The default transaction manager maintains a separate transaction for each thread. When you call transaction.commit() from thread T1, it commits the transaction for thread T1. It has no effect on other thread's running transactions. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Ordering before commit hooks
On 8/22/05, Tim Peters [EMAIL PROTECTED] wrote: Jim still wonders, and he got me wondering too, whether the `order=` gimmick is really needed. For example, you could have gotten to the same end here with the old method, by registering your actions with an object of your own creation, and registering just one commit hook with the transaction, where that one hook looked at the actions you registered with your own object and ran them in whatever order _it_ determined was best. The ordering logic would have been out of ZODB then, not limited to what an integer `order` can express, and might even benefit from ah, if I have to run A, then there's no need to also run B or C kinds of optimizations. I think that's the right reasoning. I agree with Jim. The transaction manager coordinates the actions of unconnected resource managers. If there are several transaction participants that are all part of the same software package, they can provide their own internal ordering as you suggest. If they are not related, then there's no reason to think they care about their order relative to other participants they know nothing about. To the extent that software cares about order, there is likely a simple partial order (run before X) rather than the total order that order= suggests to me. I'm inclined to agree with Jim that `order=` wasn't needed; that it was too general for the specific use case we've seen; and that it's not general enough for plausible other use cases. Should this really go into ZODB 3.5? The method name change and robustified signature were good improvements, and I'd certainly like to keep them. I think the jury is still out on `order`, though. Anyone else have strong feelings for or against it? Keep it out. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Savepoints are invalidated once they are used
On 7/12/05, Tim Peters [EMAIL PROTECTED] wrote:
[Jeremy Hylton]
IIRC, the old implementation of savepoints kept a copy of the index at
the time the savepoint was taken so that you could rollback to it
multiple times. I don't think there's any way to avoid such a copy.
Right, and the current implementation did that too. The surprise was that
it wasn't enough. Sketch:
1. Modify object 0.
2. Make savepoint 1.
It makes a copy of the current index, say {0: 0}, and remembers
the TmpStore size, say 100.
3. Fiddle around.
4. Rollback to savepoint 1.
This sets the TmpStore index to the saved {0: 0}, and truncatss
TmpStore to size 100. So far so good -- or so it seems.
5. Modify object 0 again, and make savepoint 2.
This changes the TmpStore index to {0: 100}, makes of a copy of
{0: 100} in savepoint 2, and increases TmpStore size to 200. This
just did something horribly wrong too, although it's subtle.
6. Rollback to savepoint 1 again.
Because a copy of savepoint 1's index wasn't _also_ made in
step #4, the index savepoint 1 is holding onto mutated to
{0: 100} during step #5 (object sharing). This (#6) step
leaves TmpStore with (the mutated) index {0: 100} and size 100.
7. Reference object 0.
Oops. The index tells us to seek to pos 100, but TmpStore has
been truncated to 100. We get a low-level exception from
struct.unpack() about not enough bytes to unpack the data record
header.
You can guess that I saw that happening wink. Step #4 also needs to copy
the index (from the savepoint to TmpStore) instead of sharing a reference,
although this wasn't needed so long as a savepoint could be used at most
once (then mutating the savepoint's index after a rollback had no ill
effect, as the savepoint's index could never be referenced again).
I understand. The further invariant is that the index captured when a
savepoint created is immutable.
Jeremy
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Savepoints are invalidated once they are used
IIRC, the old implementation of savepoints kept a copy of the index at the time the savepoint was taken so that you could rollback to it multiple times. I don't think there's any way to avoid such a copy. Jeremy On 7/11/05, Tim Peters [EMAIL PROTECTED] wrote: [Tim Peters] ... The good news is that, while it was hard to find, it's a one-line repair. Alas, that wasn't the end of it either. I think I'm at the end now, and all the tests are passing again (including new tests to provoke new problems I found). A savepoint (of the data manager Connection flavor) remembers the TmpStore index at the time the savepoint was made. When a savepoint could be used only once, the savepoint only needed to make sure it had a copy of the index at the time the savepoint was _made_. But when a savepoint can be reused, even the TmpStore.reset() line self.index = index was a source of subtle bugs (later mutations to `self.index` showed up in `index` too, and `index` there is typically taken from a savepoint's this is what the world looked like when I was new state -- mutating that was harmless before because the rollback could never be used again, but became disastrous when allowing re-use). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: Re: [ZODB-Dev] Handling more databases with zeo
On 6/29/05, Tim Peters [EMAIL PROTECTED] wrote: [Tim Peters] As before, I'd run a different ZEO server for each database. I'm not sure that what you're doing here will be supported for much longer (or really even _is_ supported anymore -- see my last msg). [Dieter Maurer] I do not know whether it is supported but it works in ZODB 3.2. Why do you want to cancel this? I didn't say I would cancel it / rip it out. I said it's undocumented, untested, and that its status is unclear; and I quoted a comment from current ZODB source that strongly seemed to imply its author (probably Jeremy) believed it was already dead meat (This argument is primarily for backwards compatibility with servers that supported multiple storages -- why did the comment use past tense if the current code still supports multiple storages? I used the past tense because I thought we had decided to cancel the feature at some point. The feature itself has been around and undocumented for much longer. The reason I want to remove the feature is that it adds complexity to the software and configuration without providing much real benefit. The benefit is that you get to run several storages using a single ZEO server process and TCP port. It's probably not a good idea to use a single process for many servers, although it might be convenient to use a single port. Reasons to get rid of it (recalling these from the distant past): - People were confused about what the feature actually did. I helped people several people debug problems that were caused by confusion around this feature. - It's probably better to run separate ZEO processes (possibly on different machines). - There would be less code to maintain and few features to test. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO client hangs when combined with other asyncore code
On 6/22/05, Jim Fulton [EMAIL PROTECTED] wrote: Tim Peters wrote: IMO/IME, asyncore is a poor fit for applications where the callbacks are fancy, or even where they may just take a long time to complete (because the asyncore mainloop is unresponsive for the duration). So if I had to use asyncore (I've never done so on my own initiative wink), I'd gravitate toward a work-queue model anyway, where threads unfettered by asyncore worries do all the real work-- especially on Windows, which loves to run threads --and where asyncore callbacks do as little as possible. Agreed. This is exactly the model that Zope uses. ZEO also runs several potentially slow operations in separate threads. I think we've wondered in the past whether the tpc vote should be another of those operations as the disk IO for a large transaction is non-trivial. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)
On 5/29/05, Shane Hathaway [EMAIL PROTECTED] wrote: Would a multi thread ZEO server improve anything here? Especially with concurrent access? It's possible. Although ZEO talks over the network using async sockets, it reads files synchronously, so I suspect it will frequently sit around doing nothing for 10 ms, waiting for the disk to read data. If your ZEO server has a load of 1.0 or more but low CPU usage, this is likely happening. The easiest way to overcome this is to buy gigabytes of RAM for the ZEO server--ideally, enough gigabytes to hold your whole database. A related problem is that the ZEO cache on the client is on disk, too. You may end up waiting for a disk seek to get it off disk on the client. If you've got it in memory on the server and if the ZEO protocol were more efficient, that would be a drag. Also, the design of ZEO clients tends to serialize communication with the ZEO server, so the throughput between client and server is likely to be limited significantly by network latency. ping is a good tool for measuring latency; 1 ms is good and .1 ms is excellent. There are ways to tune the network. You can also reduce the effects of network latency by creating and load balancing a lot of ZEO clients. It's really too bad that ZEO only allows a single outstanding request. Restructuring the protocol to allow multiple simulatenous requests was on the task list years ago, but the protocol implementation is so complex I doubt it will get done :-(. I can't help but think building on top of an existing message/RPC layer would be profitable. (What's twisted's RPC layer?) Or at least something less difficult to use than asyncore. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)
On 5/21/05, DJTB [EMAIL PROTECTED] wrote: [posted to comp.lang.python, mailed to [EMAIL PROTECTED] [Following up to both places.] I'm having problems storing large amounts of objects in a ZODB. After committing changes to the database, elements are not cleared from memory. Since the number of objects I'd like to store in the ZODB is too large to fit in RAM, my program gets killed with signal 11 or signal 9... The problem here is a common one with a first attempt at using ZODB. The problem is that ZODB manages memory at the granularity of first-class persistent objects -- that is, instances of classes that inherit from Persistent. ZODB can move such objects in and out of memory at transaction boundaries, which allows your application to use many more objects than it has physical memory for. It looks like your application has a single persistent instance -- the root ExtendedTupleTable -- so there's no way for ZODB to manage the memory. That object and everything reachable from it must be in memory at all times. You need to re-structure the program so that is has more first-class persistent objects. If, for example, the ExtendedTuple objects inherited from Persistent, then they could reside on disk except when you are manipulating them. The ObjectInterning instance is another source of problem, because it's a dictionary that has an entry for every object you touch. The various other dictionaries in your program will also be memory hogs in they have very many entries. The typical way to structure a ZODB program is to use one of the BTrees implementation types instead of a dictionary, because the BTree does not keep all its keys and values in memory at one time. (Its internal organization is a large collection of first-class persistent objects representing the BTree buckets and internal tree nodes.) You must use some care with BTrees, because the data structure maintains a total ordering on the keys. (And a dictionary does not.) The ZODB/ZEO programming guide has a good section on BTrees here: http://www.zope.org/Wikis/ZODB/guide/node6.html Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ConflictErrors won't clear
On 4/15/05, Chris Withers [EMAIL PROTECTED] wrote: Jeremy Hylton wrote: It's mentioned in the documentation -- see section 3.4 ZEO programming notes -- and it's been discussed on this list many, many times. Where are these notes? In the ZODB ZEO programming guide that's packaged with ZODB and available in the Wiki. Perhaps you've read it before? It sounds like the simplest approach for your application is to do like Zope and start a separate thread that runs an asyncore mainloop. Then your application threads will see updates when the commit and abort transactions, just like the would in Zope. I'd really prefer not to do that unless absolutely necessary: http://mail.zope.org/pipermail/zodb-dev/2004-June/007554.html It sounds like the answer here it so avoid fork, rather than asyncore. If you don't run an asyncore mainloop, you'll be responsible for manually sync-ing all the storages/connections that asyncore would handle automatically. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zodb-checkins] SVN: ZODB/branches/3.4/
On 4/14/05, Tim Peters [EMAIL PROTECTED] wrote: Beats me -- the ZEO version number has been one less than the corresponding ZODB version number (e.g., if ZODB is 6.7.8q12, ZEO is 5.7.8q12) as far back as my knowledge goes, and I can't find an explanation. The version number got added when we moved to ZEO 2 IIRC. At the same time, I started incrementing the minor version number along with the ZODB minor version number, because they were always released in tandem. Does anyone object to my changing the ZEO version number to match the ZODB version number? Concretely, that means: import ZODB import ZEO assert ZODB.__version__ == ZEO.version would no longer fail. +1 I've seen code and docs that actually reference ZEO.version (and why that isn't spelled __version__ is also unknown to me), Because of silly developer preferences. I don't like to use an __variable__ unless that __variable__ has an actual meaning to the Python interpreter. The __variable__ namespace is supposed to be reserved for Python (sort of like names beginning with _P are reserved in std C). Also, it's not a private variable, so I didn't see any point to have any underscores in it. There's also a file, ZEO/version.txt, that repeats the ZEO version number (ZEO.version is set up by ZEO/__init__.py). I have no idea why that exists either. Does any one here use ZEO/version.txt? Someone asked for it so that it would be easy to check the version of ZEO from a shell script. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
