Re: [ZODB-Dev] RFC: Pre-commit hooks

2005-04-08 Thread Chris Withers
Tim Peters wrote: Very nice, Florent -- thank you! I added some stuff (trust me, you won't object wink), and merged it to the 3.4 branch and to the ZODB trunk. I'll remove ZODB/branches/efge-beforeCommitHook next. One of the nice things about SVN is that retired branches and tags don't have to

[ZODB-Dev] ConflictErrors won't clear

2005-04-14 Thread Chris Withers
Hi there, I have a non-zope zeo client that pumps data into a storage server for later consumption by a zope zeo client. Everything is Zope 2.7.5. The non-zope client has logic that looks roughly like: for work in queue: try: get_transaction().begin() # do work, change zodb objects,

Re: [ZODB-Dev] ConflictErrors won't clear

2005-04-15 Thread Chris Withers
Christian Robottom Reis wrote: On Fri, Apr 15, 2005 at 12:57:07PM +0100, Chris Withers wrote: Okay, where in the above should I be calling sync()? Where do I get sync from? get_transaction() doesn't have a synch attribute.. On the Connection, of course wink And how do I get hold of the connection

Re: [ZODB-Dev] [Proposal] Size controlled ZODB cache

2005-06-03 Thread Chris Withers
Dieter Maurer wrote: Currently, the ZODB cache can only be controlled via the maximal number of objects. This makes configuration complex as the actual limiting factor is the amount of available RAM and it is very difficult to estimate the size of the objects in the cache. I therefore propose

Re: [ZODB-Dev] Re: How expensive are savepoints?

2005-07-10 Thread Chris Withers
Christian Heimes wrote: I'm migrating CMF objects to Archetypes objects including metadata, security and so on. Surely you mean hideous AT object to efficient CMF objects? 0.8 wink The migiration of a typical object takes about 0.2 to 1 sec including catalog updates. A folderish object with

Re: [ZODB-Dev] Try and fail connect strategy for zeo/zrpc

2005-07-12 Thread Chris Withers
Sidnei da Silva wrote: | Does't the *already included* zeoup.py do exactly what you are trying | to do? See the bin directory of your zope home. Unfortunately no. It uses ClientStorage, which goes through the 1000's of lines of the connection dance using connect threads and large timeouts.

Re: [ZODB-Dev] Reloading product in 2.8

2005-07-16 Thread Chris Withers
Dieter Maurer wrote: Let a bug occur in some component and then, instead (or in addition) of fixing the bug, we say often: rip the component off Zope. That was threadened for Versions, and ZClasses and now for Refresh. Well, they all don't work right, confuse people are aren't

Re: [ZODB-Dev] Reloading product in 2.8

2005-07-16 Thread Chris Withers
Chris McDonough wrote: Refresh was never much of a timesaver for me. Pressing ctrl-C - up arrow - return on the console from which I run Zope is my answer. I know lots of people say that this is too slow, but on my most recent project from the time I type runzope to the time I see ready to

Re: [ZODB-Dev] multi-level undo on a single object

2005-08-30 Thread Chris Withers
Arthur Peters wrote: Probably the easiest way to provide this functionality would be to implement a kind of undo that undoes all transactions back to a given point. Zope's management interface certainly lets you do this, might be good if you poked under the hood there to see how it's done...

Re: [ZODB-Dev] afterCommitHook

2005-09-26 Thread Chris Withers
Victor Safronovich wrote: Hello Chris Withers, Friday, September 23, 2005, 9:12:19 PM, you wrote: CW Why not just use MaildropHost? 1. Because it is *nix only, it use os.fork(), using Thread.setDaemon(1) is more friendly for me. I KNOW Jens will accept good patches ;-) 2. Because

Re: [ZODB-Dev] Re: Multi-Threading Transactions

2005-10-13 Thread Chris Withers
Chris Spencer wrote: I understand that, but my point was when you call transaction.commit(), you don't necessarily know what you're committing. Yes you do, each thread has its own connection to the database, and this connection has an independent view of the database from any other thread.

Re: [ZODB-Dev] Fixing Broken Objects?

2005-10-13 Thread Chris Withers
Andreas Jung wrote: Sounds like a total insane idea to me to use ZEO to distribute code :-) Yeah, 'cos ZODB-based Script (Python)'s and ZPT's aren't code :-P Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk

Re: [ZODB-Dev] know the state of an object

2005-10-14 Thread Chris Withers
Tim Peters wrote: I'm developing a ZODB based Collection Management software, and, for a bunch of reasons, i have to know the list of modified objects before the current transaction commit. Looking around seems there is no a public API to obtains this list That's true. How hard would it be

Re: [ZODB-Dev] Weird errors with Zope 2.7.7

2005-10-18 Thread Chris Withers
Hi Tim, Thanks for the feedback, [Chris] AttributeError: ClientCache instance has no attribute '_get' Both at random times while the servers are running... Tim Peters wrote: Then-- sorry --I don't have a clue. Like I said, I don't see any code capable of removing a ClientCache's `_get`

Re: [ZODB-Dev] Weird errors with Zope 2.7.7

2005-10-19 Thread Chris Withers
Hi Tim, Tim Peters wrote: As before, this Shouldn't load state ... error is almost certainly due to a logic error in some product you're using, or in Zope. Take this message as meaning exactly what it says: something is trying to work with a persistent object after the Connection it came from

Re: [ZODB-Dev] Weird errors with Zope 2.7.7

2005-10-20 Thread Chris Withers
Dieter Maurer wrote: First question, would you mind if I added oid_unrepr to ZODB.utils? I Why do you not like p64 (which is already there)? p64(0xhex digits) behaves very similar to oid_unrepr('0xhex digits') If that's the case, then great :-) I was just asking 'cos I couldn't

[ZODB-Dev] Shush ClientStorage

2005-10-21 Thread Chris Withers
Hi All, ZEO's got rather noisy in Zope 2.8 :-S Anyone mind if I make the following change to the ZEO trunk? cheers, Chris PS: What branches should I merge this to to get it into the next 2.8 release, assuming it is okay? Index: ZEO/ClientStorage.py

Re: [ZODB-Dev] ZConfig Extension?

2005-10-25 Thread Chris Withers
Dieter Maurer wrote: And I posted a ZConfig extension that allows to read the environment (thus using environment variables in the configuration file). Did you talk to the ZConfig maintainer about merging this? I know he wasn't keen but I think it's great extra functionality to have... Chris

Re: [ZODB-Dev] Weird errors with Zope 2.7.7

2005-10-25 Thread Chris Withers
Dieter Maurer wrote: -self._log.error(msg) +self._log.error(msg,exc_info=True) raise ConnectionStateError(msg) I mind -- the change is stupid: Because you are not in an exception handler, exc_info=True cannot provide meaningful information -- there is

Re: [ZODB-Dev] Shush ClientStorage

2005-10-25 Thread Chris Withers
Tim Peters wrote: ZEO's got rather noisy in Zope 2.8 :-S Offhand I didn't find any message logged at INFO level by ClientStorage.py in ZODB 3.4 (Zope 2.8) that wasn't also logged at INFO level by ClientStorage.py in ZODB 3.2 (Zope 2.7). What specifically do you find noisier under 2.8 than

Re: [ZODB-Dev] oids, numbers, hex and 8-byte strings

2005-10-25 Thread Chris Withers
Dieter Maurer wrote: def p64(v): Pack an integer or long into a 8-byte string Yes, but for that to be meaningful, you have to know that: - 0xSomething is an integer or long (it read hex number in my head - oids are 8-byte strings Neither of things that sit in my brain :-( But,

[ZODB-Dev] Re: Weird errors with Zope 2.7.7

2005-10-25 Thread Chris Withers
Florent Guillaume wrote: I guess I'm looking for a show_stack option... import traceback; traceback.print_stack() Yes thankyou, my egg sucking is quite proficient ;-) Tim, how would I go about getting one of those added to Python? (It's not much of a code change, but the process of getting

Re: [ZODB-Dev] oids, numbers, hex and 8-byte strings

2005-10-27 Thread Chris Withers
Tim Peters wrote: no inverse for oid_repr, and there isn't a need for one. For any 8-byte string oid S, p64(int(oid_repr(S), 0)) == S so that's how to get an inverse of oid_repr if you really want one (although I don't know why anyone would). ...'cos oid_repr is what gets used to log

Re: [ZODB-Dev] request on restrictedTraverse

2005-10-31 Thread Chris Withers
Daniel, I think you should try the plone-users list... Daniel wrote: Hi, It seems that the restrictedTraverse method changes the request, doesn't it? No. Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk

Re: [ZODB-Dev] ZConfig Extension?

2005-11-02 Thread Chris Withers
Dieter Maurer wrote: I know he wasn't keen but I think it's great extra functionality to have... Then, maybe, you lobby a bit? That's what I'm doing. Fred Drake is Mr ZConfig, no? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting -

[ZODB-Dev] Different weird errors with Zope 2.8.4

2005-11-29 Thread Chris Withers
Hot on the heals of my posts about Shouldn't load state for comes the eagerly awaited sequel Couldn't load state for ;-) Here's the errors: 2005-11-29T14:24:24 ERROR ZODB.Connection Couldn't load state for 0x6a7ed9 Traceback (most recent call last): File lib/python/ZODB/serialize.py, line

Re: [ZODB-Dev] Different weird errors with Zope 2.8.4

2005-12-01 Thread Chris Withers
Yay! Where's the correct place to report these nowadays? cheers, Chris Dieter Maurer wrote: Chris Withers wrote at 2005-11-29 14:33 +: Hot on the heals of my posts about Shouldn't load state for comes the eagerly awaited sequel Couldn't load state for ;-) ... File lib/python/ZEO/zrpc

Re: [ZODB-Dev] Re: [Zope] DateTime mess

2005-12-01 Thread Chris Withers
Tim Peters wrote: Sorry, I couldn't find a comprehensible question here after reasonable effort to extract one. Clearly, Zope2's DateTime.DateTime.DateTime objects are neither persistent nor do they define any mutating methods. Are those relevant? If not, try to ask a question directly,

[ZODB-Dev] More exciting ZODB errors ;-)

2005-12-14 Thread Chris Withers
Hi All, We recently upgraded to Zope 2.8.4 and have been seeing some different and special errors every so often (why does this always and only ever happen to me? ;-) Nothing noticeably bad appears to result from these, but, as always, I loose hair over them and would like ot know what's

Re: [ZODB-Dev] Dynamic Wrapper?

2005-12-16 Thread Chris Withers
Chris Spencer wrote: I understand this has some drawbacks. Namely, it will only work for new-style classes, but for a large code base this might be easier than manually writing _p_changed = 1 everywhere. The number of times you end up actually having to write this is pretty minimal ;-)

Re: [ZODB-Dev] More exciting ZODB errors ;-)

2005-12-16 Thread Chris Withers
Tim Peters wrote: This is on a cluster of machines, with the errors not coming from any one machine as far as I can see... Networking gear and cables are also hardware ;-) large bank datacenter - not a chance in hell of sensibly testing this :-( No need, you already did, and I added

Re: [ZODB-Dev] RE: lost implementer

2006-01-23 Thread Chris Withers
Jim Fulton wrote: 2. I think a real packaging system, like eggs would have helped here. Eggs in particular would have allowed multiple versions of zope.interface to be installed. Zope would have gotten the version it needed and ZODB would have gotten the version it needed. (Hm, maybe

Re: [ZODB-Dev] RE: lost implementer

2006-01-24 Thread Chris Withers
Chris McDonough wrote: See the egg intro doc at http://peak.telecommunity.com/DevCenter/ PythonEggs . I've scanned that before... Jim's mentioned some pretty deep and interesting stuff, I wondered if he'd found more in-depth docs or whether I was just missing stuff on that page... Chris

Re: [ZODB-Dev] RE: lost implementer

2006-01-24 Thread Chris Withers
Jim Fulton wrote: That page is part of a collection of pages and it links to other pages to provide details on various subjects. One of the cool things about the web is that you don't have to put everything in a single page. One of the non-cool things though, is that Firefox doesn't download

Re: [ZODB-Dev] RE: lost implementer

2006-01-24 Thread Chris Withers
Tino Wildenhain wrote: apt-get install wwwoffle :-) C:\Zopeapt-get install wwwoffle 'apt-get' is not recognized as an internal or external command, operable program or batch file. :-( Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk

[ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)

2006-01-24 Thread Chris Withers
Okay, now that we have 2.8.4 in place, we get proper reporting of ConflictErrors and today we started seeing one happening over and over again which looked roughly as follows: Traceback (most recent call last): File lib/python/Products/Transience/Transience.py, line 844, in new_or_existing

[ZODB-Dev] cache not minimized at transaction boundaries?

2006-01-26 Thread Chris Withers
Hi All, This is with whatever ZODB ships with Zope 2.8.5... I have a Stepper (zopectl run on steroids) job that deals with lots of big objects. After processing each one, Stepper does a transaction.get().commit(). I thought this was enough to keep the object cache at a sane size, however

Re: [ZODB-Dev] cache not minimized at transaction boundaries?

2006-01-30 Thread Chris Withers
Hi Tim, Tim Peters wrote: Do: import ZODB print ZODB.__version__ to find out. Good to know, thanks... I have a Stepper (zopectl run on steroids) job that deals with lots of big objects. Can you quantify this? 60,000 File objects of the order of 2Mb each. It does not do

Re: [ZODB-Dev] cache not minimized at transaction boundaries?

2006-01-31 Thread Chris Withers
Tim Peters wrote: [Chris Withers] ... ...oh well, if only the ZODB cache was RAM-usage-based ratehr than object count based ;-) Ya, that's not feasible. More plausible would be to base ZODB cache targets on aggregate pickle size; ZODB _could_ know that, and then it would also be strongly

Re: [ZODB-Dev] cache not minimized at transaction boundaries?

2006-02-01 Thread Chris Withers
Hey Toby, Long time no see ;-) Toby Dickenson wrote: On Tuesday 31 January 2006 11:41, Chris Withers wrote: Tim Peters wrote: ...base ZODB cache targets on aggregate pickle size All good, but still never gonna happen, right? *wry grinz* Sounds like a reasonable evening project (Ive

Re: [ZODB-Dev] cache not minimized at transaction boundaries?

2006-02-01 Thread Chris Withers
Dieter Maurer wrote: I plan to implement that -- waiting for the new style contributor agreement from the Zope Foundation ;-) Yay! Go Dieter! cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk

Re: [ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)

2006-02-01 Thread Chris Withers
Hi Chris, Chris McDonough wrote: Rather than chucking this in the collector as-is, it would be useful if you could get me the output of the ZEO log rather than the output of the client log. Right now, the actual traceback is in the ZEO log; the client log only shows that a conflict error

Re: [ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)

2006-02-01 Thread Chris Withers
Chris McDonough wrote: What front end do you use to do the request distribution? Pound. *barf* ew.. Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information

Re: [ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)

2006-02-02 Thread Chris Withers
Chris McDonough wrote: Pound. *barf* ew.. Works great. Except it doesn't balance load and can't do SSL correctly, right? ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk

Re: [ZODB-Dev] Conflict Error

2006-02-27 Thread Chris Withers
Antonio Beamud Montero wrote: Where Status is a Persistent Class, with an OOBTree attribute called _dict, and serveral methods wrapping this _dict, like add, remove, etc.. OOBTree's have conflict resolution code which will try and do the right thing instead of raising a ConflictError...

[ZODB-Dev] zeoup.py writes to Data.fs?

2006-03-08 Thread Chris Withers
Hi all, Is it just me or does zeoup.py write a transaction to the end of Data.fs containing a MinPO object? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information

Re: [ZODB-Dev] zeoup.py writes to Data.fs?

2006-03-09 Thread Chris Withers
You live and learn, thanks Tim and Jeremy! :-) Chris Tim Peters wrote: [Chris Withers] Is it just me or does zeoup.py write a transaction to the end of Data.fs containing a MinPO object? Make sure a ZEO server is running. usage: zeoup.py [options] The test will connect to a ZEO server

Re: [ZODB-Dev] Extenuated memory

2006-03-22 Thread Chris Withers
Antonio Beamud Montero wrote: But I can't minimize more the cache... What I need to do to free more memory space, because this always grows... This is a problem with python, I hear rumours it'll be fixed in Python 2.5. cheers, Chris -- Simplistix - Content Management, Zope Python

[ZODB-Dev] Conflict resolving persistent queue?

2006-03-23 Thread Chris Withers
Hi All, I was wondering whether anyone had implemented a FIFO persistent queue class which has the following conflict resolution strategy: two concurrent adds: adds both new items to the end of the queue in a time-based order one add and one remove happening concurrently: add the new item

[ZODB-Dev] edge case question about _p_deactiveate and modified objects

2006-03-24 Thread Chris Withers
Hi All, What happens if you call _p_deactivate() on a modified object but before the modifications have been committed? Do the modifications get lost or will they still be saved when the next commit happens after the _p_deactivate()? cheers, Chris -- Simplistix - Content Management, Zope

Re: [ZODB-Dev] Weird ZODB error on Zope startup :-(

2006-03-27 Thread Chris Withers
Tim Peters wrote: It means that some object in the Connection was in a modified state when an attempt to close the Connection was made. The current transaction must be committed or aborted first -- ZODB can't guess whether the pending changes should be committed or thrown away, so it won't let

Re: [ZODB-Dev] Question about BTree transaction data

2006-04-13 Thread Chris Withers
Dieter Maurer wrote: Pascal Peregrina wrote at 2006-4-12 08:35 +0100: I use FileStorage (via ZEO). I have switched a big dictionary from PersistentMapping to BTree. In the past, it was easy to compute added/deleted keys from states (cause PersistentMapping state contains the whole

[ZODB-Dev] Re: [Zope-dev] Re: 64-bit BTrees

2006-04-18 Thread Chris Withers
Philipp von Weitershausen wrote: in memory. Dieter estimates 20% to 35% slowdown for the C algorithms (whatever that means), Tim seems to think it won't have such a big effect. I guess we'll only know after some benchmarks. Can we please not make any definite decisions until this issue has

Re: [ZODB-Dev] Re: [Zope-dev] Re: 64-bit BTrees

2006-04-21 Thread Chris Withers
Dieter Maurer wrote: Chris Withers wrote at 2006-4-18 08:34 +0100: ... If having two isn't acceptable, then why do we have an I and O BTree's, not to mention the special ones used for in-memory ZODB indexes? Surely we should just have one BTree class? Using I versus O BTrees makes a huge

Re: [ZODB-Dev] Corrupted Data Error?

2006-04-25 Thread Chris Withers
Sidnei da Silva wrote: Got the following exception while doing some work on a Zope instance here. It's the first time I see such error. * Module ZEO.ClientStorage, line 746, in load * Module ZEO.ClientStorage, line 769, in loadEx * Module ZEO.ServerStub, line 192, in loadEx *

[ZODB-Dev] tid error with ClientStorage cache in Zope 2.9.2

2006-04-26 Thread Chris Withers
Hi All, Anyone else seen this? (864) CW: error in notifyConnected (('x', y)) Traceback (most recent call last): File C:\Zope\2.9.2\lib\python\ZEO\zrpc\client.py, line 506, in notify_client self.client.notifyConnected(self.conn) File C:\Zope\2.9.2\lib\python\ZEO\ClientStorage.py, line

[ZODB-Dev] Changes in _p_changed behaviour between Zope 2.7 and 2.9

2006-04-27 Thread Chris Withers
Hi All, I'm trying to fix this bug: http://www.zope.org/Collectors/Zope/2062 And I've narrowed it down to the following lines in History.py: if serial != self._p_serial: self.manage_beforeHistoryCopy() state=self._p_jar.oldstate(self, serial) print

Re: [ZODB-Dev] tid error with ClientStorage cache in Zope 2.9.2

2006-04-27 Thread Chris Withers
the problem with the two tid's being the same and do you have any idea why this could be occuring in production code on Zope 2.9.x? cheers, Chris Chris Withers wrote: Hi All, Anyone else seen this? (864) CW: error in notifyConnected (('x', y)) Traceback (most recent call last): File C:\Zope

Re: [ZODB-Dev] tid error with ClientStorage cache in Zope 2.9.2

2006-04-28 Thread Chris Withers
Not sure if Jeremy still on this list so CCing... Tim Peters wrote: No, I don't. The internal docs/comments are inconsistent on this point. FileCache.settid() starts with ## # Update our idea of the most recent tid. Is this the most recently used or the most recently available?

[ZODB-Dev] Re: Changes in _p_changed behaviour between Zope 2.7 and 2.9

2006-04-28 Thread Chris Withers
Florent Guillaume wrote: base._p_changed=0 Marks the object not changed, to allow ghostifying. base._p_deactivate() Ghostifies the object. base.__setstate__(state) Updates the object's dict directly. This really shouldn't be called on a ghost object,

Re: [ZODB-Dev] tid error with ClientStorage cache in Zope 2.9.2

2006-05-01 Thread Chris Withers
Tim Peters wrote: it knew about. To support this, a persistent ZEO cache stores the value of the largest tid the ZEO client knew about in the cache file. Hmmm, didn't think I was using a persistent client cache here... ...well, there are .zec files in the var directory, so I guess I must

[ZODB-Dev] Re: Changes in _p_changed behaviour between Zope 2.7 and 2.9

2006-05-01 Thread Chris Withers
Florent Guillaume wrote: base._p_activate() # make sure we're not a ghost base.__setstate__(state) # change the state base._p_changed = True # marke object as dirty OK, this is the code I went with. Well the C code is pretty clear, it does a PyDict_Clear before doing

Re: [ZODB-Dev] tid error with ClientStorage cache in Zope 2.9.2

2006-05-02 Thread Chris Withers
Dieter Maurer wrote: Chris Withers wrote at 2006-5-1 11:34 +0100: ... ...well, there are .zec files in the var directory, so I guess I must be. What controls whether a persistent or temporary client cache is used? Your Zope configuration file, of course ;-) Yeah, I knew that bit

Re: [ZODB-Dev] Re: tid error with ClientStorage cache in Zope 2.9.2

2006-05-02 Thread Chris Withers
Jens Vagelpohl wrote: Any recent zope has this in its default zope.conf: Come on, you know Chris *never* takes the attempt to look something up myself step ;) Actually, I think that's a little unfair. This is a particularly meaninglessly named key... Chris -- Simplistix - Content

Re: [ZODB-Dev] Undo differences between Z2 and Z3

2006-05-17 Thread Chris Withers
Hi Jim, Jim Fulton wrote: BTW, I strongly discourage use of Undo except in emergencies. Sadly, except when undoing the last (non-undo) transactions in a database can lead to inconsistency. What sort of inconsistencies are you referring to? Undo can be a very attractive feature although

Re: [ZODB-Dev] Undo differences between Z2 and Z3

2006-05-18 Thread Chris Withers
Jim Fulton wrote: Logical inconsistencies. Consider two transactions, T1 and T2 that run the same code: def f(): if 'x' not in somefolder: somefolder['x'] = 42 The postcondition of the transaction is that somefolder has an item named 'x' with the value 42. Now, when T2

Re: [ZODB-Dev] Undo differences between Z2 and Z3

2006-05-19 Thread Chris Withers
Jim Fulton wrote: Chris Withers wrote: Jim Fulton wrote: Even if you did track reads, how would you distinguish an unsafe read as above from a normal read that shouldn't cause a conflict? A write (or the undo of a write) would conflict with any reads in later transactions. Wouldn't

[ZODB-Dev] ZEO cache bug? (again)

2006-05-31 Thread Chris Withers
Hi All, I mentioned this before: File C:\Zope\2.9.2\lib\python\ZEO\cache.py, line 151, in setLastTid self.fc.settid(tid) File C:\Zope\2.9.2\lib\python\ZEO\cache.py, line 1060, in settid raise ValueError(new last tid (%s) must be greater than ValueError: new last tid

Re: [ZODB-Dev] ZEO cache bug? (again)

2006-06-01 Thread Chris Withers
Chris Withers wrote: File C:\Zope\2.9.2\lib\python\ZEO\cache.py, line 151, in setLastTid self.fc.settid(tid) File C:\Zope\2.9.2\lib\python\ZEO\cache.py, line 1060, in settid raise ValueError(new last tid (%s) must be greater than ValueError: new last tid (244828509247418982) must

[ZODB-Dev] ZEO LoadEx?

2006-06-19 Thread Chris Withers
Hi All, Having problems with a storage/client combo that have started having really bad performance problems since moving to Zope 2.9.3. The client becomes totally unresponsive, to the point where I had to install DeadlockDebugger to get any sense out of it :-/ Anyway, with that in place,

Re: [ZODB-Dev] ZEO LoadEx?

2006-06-19 Thread Chris Withers
Jim Fulton wrote: I don't know the details of this method and don't have time to look atm, But I'll take a shot in the dark and see if I get lucky. :) Are you running the same versions of ZODB on the client and the server? Yup, Zope 2.9.3 on both. curiously, Chris -- Simplistix - Content

Re: [ZODB-Dev] ZEO LoadEx?

2006-06-20 Thread Chris Withers
Tim Peters wrote: Sure, but no way to guess from here. The only thing I can really guess from the above is that your client is going to the server a lot to get data. Well, the client and the server are on the same machine, which isn't load or memory bound, and doesn't seem to be i/o bound

Re: [ZODB-Dev] ZEO LoadEx?

2006-06-20 Thread Chris Withers
Pascal Peregrina wrote: This reminds me something I noticed when we migrated from 2.7 to 2.8 Well, it's 2.7 to 2.9 here, but yeah, it's the same big jump ;-) Our issue was a very big PersistentMapping based tree of objects, which was involved in a lot of RW and RO transactions from

Re: [ZODB-Dev] ZEO LoadEx?

2006-06-21 Thread Chris Withers
Pascal Peregrina wrote: We located it by hacking the ClientStorage code in order to display the real load operations (oids) from ZEO server (cache misses). This sounds like pretty useful code... Is it in a how-to somewhere or otherwise online? cheers, Chris -- Simplistix - Content

Re: [ZODB-Dev] Re: What makes the ZODB slow?

2006-06-26 Thread Chris Withers
Jean Jordaan wrote: The ZODB is actually very fast. [...] So you're probably observing slowness in the frameworks on top of it. I'll believe this anytime :-] ;-) In our case, a transaction may be a workflow state change on say 50 objects. Two or three people try a transaction like that

Re: [ZODB-Dev] Re: What makes the ZODB slow?

2006-06-26 Thread Chris Withers
Florent Guillaume wrote: BTrees perform best when keys' prefixes are randomly distributed. So if your application generates keys like 'foo001', 'foo002',... you'll get lots of conflicts. Same for consecutive integers in IOBTree. Tempted to call bullshit on this, since there's code in the

Re: [ZODB-Dev] Re: What makes the ZODB slow?

2006-06-26 Thread Chris Withers
Andreas Jung wrote: BTrees perform best when keys' prefixes are randomly distributed. So if your application generates keys like 'foo001', 'foo002',... you'll get lots of conflicts. Same for consecutive integers in IOBTree. Tempted to call bullshit on this, since there's code in the catalog

Re: [ZODB-Dev] Re: What makes the ZODB slow?

2006-06-26 Thread Chris Withers
Florent Guillaume wrote: I can comment, I have a big brain too: the code in the catalog uses per-connection series of keys, so no conflicts arise. Really? I thought they were per-thread... wasn't aware that each thread was tied to one connection indefinitely... I thought the connections were

[ZODB-Dev] Re: What makes the ZODB slow?

2006-06-27 Thread Chris Withers
Florent Guillaume wrote: Chris Withers wrote: Florent Guillaume wrote: I can comment, I have a big brain too: the code in the catalog uses per-connection series of keys, so no conflicts arise. Really? I thought they were per-thread... wasn't aware that each thread was tied to one connection

Re: [ZODB-Dev] Re: More exciting ZODB errors ;-)

2006-06-27 Thread Chris Withers
Florent Guillaume wrote: The AssertionError makes it back to the user, as it's not a ConflictError. That's a big problem. Oh, you've actually had real user complaints :-/ We haven't yet, but I guess we've just been lucky... If anyone has a means to reproduce these errors, I'd gladly

Re: [ZODB-Dev] Re: What makes the ZODB slow?

2006-06-27 Thread Chris Withers
Dieter Maurer wrote: PostGres does use looks, lots of them and for different purposes. Could ZODB use locks to gain a similar performance boost? The only thing for which Postgres does not use locks is reading. For this is uses MVCC (which we meanwhile adapted for the ZODB to get rid of

Re: [ZODB-Dev] Re: What makes the ZODB slow?

2006-07-02 Thread Chris Withers
Dieter Maurer wrote: If for example 10 Zope objects are modified and this cause the full text indexes to be updated then this can cause more modifications than the update of hundreds of Postgres rows (as such rows cannot contain mass data -- due to the restriction

Re: [ZODB-Dev] Where did self._p_jar._storage._serial go?

2006-07-06 Thread Chris Withers
Jean Jordaan wrote: It looks like '_tid' fits the bill. It's not available when using ZEO though, which took me a while to figure out. Might be good to check the relevent apis. A lot of this has got much clearer in recent zodb releases. If you end up using variables starting with

Re: [ZODB-Dev] Re: Mirrored ZODBs

2006-07-13 Thread Chris Withers
Tres Seaver wrote: Zope Corporation's Zope Replication Services products operates along those lines: http://www.zope.com/products/zope_replication_services.html Yeah, but you can only write to one of the storages, right? Chris -- Simplistix - Content Management, Zope Python Consulting

Re: [ZODB-Dev] ZEO cache bug? (again)

2006-08-17 Thread Chris Withers
Jim Fulton wrote: I'll note that I have been able to provoke the error message, but, for me at least, the result was non-fatal. The error occurs during cache verification and causes the connection to fail. The connection thread keeps running and tries again. Okay, with the proviso that

Re: [ZODB-Dev] ZEO cache bug? (again)

2006-08-17 Thread Chris Withers
Hi Jim, Sorry for the delay, was on holiday in Canada... Jim Fulton wrote: On May 31, 2006, at 4:03 AM, Chris Withers wrote: File C:\Zope\2.9.2\lib\python\ZEO\cache.py, line 151, in setLastTid self.fc.settid(tid) File C:\Zope\2.9.2\lib\python\ZEO\cache.py, line 1060, in settid

Re: [ZODB-Dev] ZEO cache bug? (again)

2006-08-18 Thread Chris Withers
Jim Fulton wrote: Jim Fulton wrote: On May 31, 2006, at 4:03 AM, Chris Withers wrote: File C:\Zope\2.9.2\lib\python\ZEO\cache.py, line 151, in setLastTid self.fc.settid(tid) File C:\Zope\2.9.2\lib\python\ZEO\cache.py, line 1060, in settid raise ValueError(new last tid (%s) must

Re: [ZODB-Dev] Problems with ReadConflictError

2006-08-22 Thread Chris Withers
Andrew McLean wrote: Any advice gratefully received. I'd suggested moving to a ZODB version new enough to have MVCC support, which will likely make your problem go away... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk

Re: [ZODB-Dev] ZODB Conflict Error

2006-08-22 Thread Chris Withers
Wohnlich, Eric (IMS) wrote: http://www.y.com/cbctr ... Could faulty rewrite rules cause it to fail the retry? No. xx.xxx.xx.xxx - rickerw [15/Aug/2006:13:44:06 -0400] POST /VirtualHostBase/https/xxx.xxx.xxx.xxx:443/db2/tissueresources/VirtualHo

Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage

2006-09-25 Thread Chris Withers
Patrick Gerken wrote: system with ZEO for an ERP5 deployment. In my case I don't need to care for data replication, all is stored on a SAN considered HA by the customer already. You run your live Data.fs off a SAN? That usually makes for interesting performance problems in the best case! So

Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage

2006-09-27 Thread Chris Withers
Patrick Gerken wrote: You run your live Data.fs off a SAN? That usually makes for interesting performance problems in the best case! Well, yes, that is/was the idea when suddenly HA requests popped up. You might be right, Painful experience has taught me that I am ;-) need to access many

Re: [ZODB-Dev] HA Setup and zodb index file of FileStorage

2006-09-27 Thread Chris Withers
Dieter Maurer wrote: Why do you think so? Almost everything is run off SANs. Why should Data.fs make a problem? Under high load, an awful lot of data is read from this file. SANs have higher latency than local disks. In some apps this can make a noticeable performance difference. cheers,

[ZODB-Dev] Why DO we use timestamps as transaction ids?

2006-10-04 Thread Chris Withers
...rather than just incrementing integers? I'm asking 'cos I've just started having time-stamp reduction errors on a production system where a contingent system is having a .fs file that's been re-constituted from repozo backups tested with fstest.py... cheers, Chris -- Simplistix -

[ZODB-Dev] zodb time-stamp reduction error without time chaning on server

2006-10-04 Thread Chris Withers
Hi All, One of my customers has a large (21GB) production zodb which they back up onto a contingency server using repozo and rsync. The process is roughly as follows: 1. pack the production database to 3 days once a day. 2. create a full backup with repozo and rsync this to the contingency

[ZODB-Dev] Re: [Zope-dev] zodb time-stamp reduction error without time chaning on server

2006-10-04 Thread Chris Withers
Dieter Maurer wrote: 3. If repozo is not to blame, what could be? One possibility would be a bad call. A bad call? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more

Re: [ZODB-Dev] Why DO we use timestamps as transaction ids?

2006-10-05 Thread Chris Withers
Dieter Maurer wrote: You should be happy about the much more explicit information. It may allow you to analyse your problem better. This question has nothing to do with that problem, it just came up as a result of once again being reminded that we use timestamps as transaction ids. For

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-05 Thread Chris Withers
Dieter Maurer wrote: A new proposal: http://www.zope.org/Wikis/ZODB/MemorySizeLimitedCache It outlines how to implement a ZODB cache limited not be the number of containing objects but by their estimated memory size. Feedback welcome -- either here or in the Wiki. I think any work in this

Re: [ZODB-Dev] [Enhancement Proposal] Memory size limited Cache

2006-10-06 Thread Chris Withers
Dieter Maurer wrote: The pickle size is a *VERY* rough estimation (probably wrong by a factor of 5 to 15) But, as you point out, much better than a hard coded 1 ;-) We probably would get a much better estimation using PySizer but probably at a significantly higher cost. Right, I guess

Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-09 Thread Chris Withers
Jim Fulton wrote: - I wonder if an argument could be made than we shouldn't implicitly deactivate an object that has been accessed in a a transaction while the transaction is still running. Would this prevent ZODB from ever promising not to use more than a certain amount of memory? The

Re: [ZODB-Dev] [Enhancement Proposal] Garanteed lifetime for volatile variables

2006-10-10 Thread Chris Withers
Jim Fulton wrote: Chris Withers wrote: Jim Fulton wrote: - I wonder if an argument could be made than we shouldn't implicitly deactivate an object that has been accessed in a a transaction while the transaction is still running. Would this prevent ZODB from ever promising not to use more

  1   2   3   4   >