Re: [Zope-dev] Help! Need to extract files out of Data.fs!
Kris, I assume you have a backup of the Data.fs. If you do, nothing is really lost. Have you compressed the Data.fs? There is an option in the control panel to do this. I can be run on a live machine, but you may not want to do so given the size of stuff. I'd do it on a quiet machine. This will remove any cruft you have in the Data.fs and will improve your speed of access somewhat, Things are probably slow because of swapping. Reducing the size of Data.fs may speed things up. Take a look at the documentation for the ZODB (Google is your friend). You could write a program in Python to simply open the ZODB and extract the files you need. It might save a bit of overhead. Remember 1.3TB is a large file and is going to take some time to process no matter what. On Mon, 28 Aug 2006, Kris Adcock wrote: Hiya, My company has been using a Zope server to store some reference material, but it has grown beyond belief, and I need to extract all the collected files (about 1.3 terabytes!) from the Data.fs into distinct files onto a different server. Unfortunately, I've run into some problems. Firstly, the FTP transfer process seems to be extremely slow, and if I try and transfer too large or deep a folder structure in one go then things grind to a halt. To beat this I've been carefully recreating the folder structure manually, and copying things in easier-to-digest chunks. Unfortunately, because I was a bit rushed over the weekend I have accidentally deleted about 130,000 files! I've tried going through the Undo procedure, but the idea of manually ticking 130,000 boxes (in pages of 20) and undoing them fills me with dread. So ... Please please /please/ does anyone happen to have the file format for the Data.fs file so that I could write a C program to extract and recreate the files and folders to another server? Or has anyone written such a program already? If so, I'll happily have your children if you can help me out! As it stands, I'm going to have to break it to the librarian (the poor man who has been collating all the reference material for the last six months) that I've lost half of his work, and blaming it on the poor FTP client in Windows seems a bit lame! If it helps: I'm running Zope 2.9.2 on a SuSE Linux box. I have full access to the Linux box ... and a large hammer, if I need it! :( Help! Many thanks in advance, Kris (considering giving up this computing job and going to live in a cave somewhere). ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help! Need to extract files out of Data.fs!
One more thing, what does the disk where the Data.fs resides look like? On Mon, 28 Aug 2006, Dennis Allison wrote: Kris, I assume you have a backup of the Data.fs. If you do, nothing is really lost. Have you compressed the Data.fs? There is an option in the control panel to do this. I can be run on a live machine, but you may not want to do so given the size of stuff. I'd do it on a quiet machine. This will remove any cruft you have in the Data.fs and will improve your speed of access somewhat, Things are probably slow because of swapping. Reducing the size of Data.fs may speed things up. Take a look at the documentation for the ZODB (Google is your friend). You could write a program in Python to simply open the ZODB and extract the files you need. It might save a bit of overhead. Remember 1.3TB is a large file and is going to take some time to process no matter what. On Mon, 28 Aug 2006, Kris Adcock wrote: Hiya, My company has been using a Zope server to store some reference material, but it has grown beyond belief, and I need to extract all the collected files (about 1.3 terabytes!) from the Data.fs into distinct files onto a different server. Unfortunately, I've run into some problems. Firstly, the FTP transfer process seems to be extremely slow, and if I try and transfer too large or deep a folder structure in one go then things grind to a halt. To beat this I've been carefully recreating the folder structure manually, and copying things in easier-to-digest chunks. Unfortunately, because I was a bit rushed over the weekend I have accidentally deleted about 130,000 files! I've tried going through the Undo procedure, but the idea of manually ticking 130,000 boxes (in pages of 20) and undoing them fills me with dread. So ... Please please /please/ does anyone happen to have the file format for the Data.fs file so that I could write a C program to extract and recreate the files and folders to another server? Or has anyone written such a program already? If so, I'll happily have your children if you can help me out! As it stands, I'm going to have to break it to the librarian (the poor man who has been collating all the reference material for the last six months) that I've lost half of his work, and blaming it on the poor FTP client in Windows seems a bit lame! If it helps: I'm running Zope 2.9.2 on a SuSE Linux box. I have full access to the Linux box ... and a large hammer, if I need it! :( Help! Many thanks in advance, Kris (considering giving up this computing job and going to live in a cave somewhere). ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
On Fri, 24 Mar 2006, Jim Fulton wrote: Andreas Jung wrote: I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import. Comments? I filed a bug report (2051: xml import export fail) which was triggered by the need to move material from a recent zope (2.9) to an older zope (2.7.4) where the zexp format was incompatible. Eventually, I was able to make the XML work. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZSyncer
Paul, Just moved all our systems to ZSyncer 0.7.0 and have encountered a problem related to authentication. In our past setup, using ZSyncer 0.5.1, we use the ability to specify a user:password to provide a single authorization mechanism that could be used by all of our developers. Now we've stumbled onto the fact that ZSyncer 0.7.0 has eliminated the optin al user:password specification. Some of our users can use ZSyncer and others cannot. It's not clear what authorization is being used -- I suspect it is ownership, but I have not investigated. So, how do I configure things to allow a subset of my developers to use ZSyncer transparently? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZSyncer
Yes, it is bad form to answer ones own questions, but ... I had not read the documentation carefully. The answer is that the specifications for targets is user:[EMAIL PROTECTED]:port/path-to-ZSyncer-instance and is as elegant as it gets. It uses Basic Authentication and is documented in the README I neglected to read as this is the fifth version of ZSyncer I've installed and I thought I knew the product. On Wed, 1 Feb 2006, Dennis Allison wrote: Paul, Just moved all our systems to ZSyncer 0.7.0 and have encountered a problem related to authentication. In our past setup, using ZSyncer 0.5.1, we use the ability to specify a user:password to provide a single authorization mechanism that could be used by all of our developers. Now we've stumbled onto the fact that ZSyncer 0.7.0 has eliminated the optin al user:password specification. Some of our users can use ZSyncer and others cannot. It's not clear what authorization is being used -- I suspect it is ownership, but I have not investigated. So, how do I configure things to allow a subset of my developers to use ZSyncer transparently? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: sessions in the presence of conflicts
Chris McDonough identified a persistence problem with the routine(s) that
manage sessions variables. (Thanks Chris) I have put the correction in
place which resolved some (but not all) of the problems.
There are still problems which are apparently due conflicts in accessing
the session variables. To minimize frequency of conflicts, I am rewriting
several routines using Dieter's rules of the thumb (Thanks Dieter).
One routine being modified is a Script(Python) that initializes a number
of session variables. I am collecting the session values in a dictionary
and then use update to set their value, for example:
s = {}
s['alpha'] = 'a'
s['beta'] = 'b'
request['SESSION'].update(s)
Is the persistence machinery smart enough to detect this as a change? I
suspect that it has to be flagged since the assignment won't be seen.
Usually this means setting the _p_changed=1 attribute, but it is not clear
to me where to set it in this particular context.
--
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Re: [Zope-dev] Re: sessions in the presence of conflicts
The interaction between sessions, conflicts, and persistence is a bit
confusing. I am still trying to understand the code in depth.
One thing is for sure, request.SESSION and/or request['SESSION'] must be
persistent for things to work. Mutable objects in the session variable
set (dictionaries and lists) have to be handled specially to get the
persistence machinery to recognize they have been changed.
In this case, I am trying to ensure that the session variables get
identified as persistent. My question is whether using update (an
implicit assignment) triggers the persistence mechanism. It is the
moral equivalent of writing
request['SESSION']['alpha'] = 'a'B
request['SESSION']['beta'] = 'b'
but I am unsure whether the persistence mechanism will recognize it as
such.
Doing session variable initialization in a Script(Python) object has a
downside because one cannot set a _p_changed attribute and so must rely on
the assignment paradigm. Perhaps the interface should be in a Product or
External Method which is less constrained.
Anyhow, David, thanks for the assist.
On Mon, 19 Dec 2005, David H wrote:
Dennis Allison wrote:
Chris McDonough identified a persistence problem with the routine(s) that
manage sessions variables. (Thanks Chris) I have put the correction in
place which resolved some (but not all) of the problems.
There are still problems which are apparently due conflicts in accessing
the session variables. To minimize frequency of conflicts, I am rewriting
several routines using Dieter's rules of the thumb (Thanks Dieter).
One routine being modified is a Script(Python) that initializes a number
of session variables. I am collecting the session values in a dictionary
and then use update to set their value, for example:
s = {}
s['alpha'] = 'a'
s['beta'] = 'b'
request['SESSION'].update(s)
Is the persistence machinery smart enough to detect this as a change? I
suspect that it has to be flagged since the assignment won't be seen.
Usually this means setting the _p_changed=1 attribute, but it is not clear
to me where to set it in this particular context.
Dennis,
Did you means request['SESSION']['someDictionary'].update(s)?
Anyway your idea seems correct - The SESSIONS chapter (at least on
plope.com) addresses SESSION staleness and mutable objects.
1) someDict = SESSION['someDict']
2) someDict['someKey'] = newValue
But (2) does not guarentee that a subsequent lookups like:
SESSION['someDict'] will return newValue.
But this WILL:
3) SESSION['someDict'] = someDict.
Which looks like your example. How this connect to your primary issue:
*conflicts* is not clear to me. :-\
David
--
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [RfC] Removal of old stuff in Zope 2.10
+1 on the lot if you are looking for votes. On Tue, 20 Dec 2005, Andreas Jung wrote: Hi, for next release we plan to replace several parts with the corresponding components from Zope 3 (e.g. ZPT´). Philipp is working on a proposal on that issue. In addition I would like to get rid of some old stuff that is no longer maintained and buggy: - ZopeTutorial (could be ripped off without implications and made available for download on zope.org) - HelpSys - from a programmers view pretty much useless and not very helpful. I consider to replace it with something more useful (not sure we can re-use apidoc from Zope 3 in some way, perhaps the inclusion of Dieter's Docfinder might be more useful for programmers) - Gadfly(DA) - do we really need this? We discussed this already. In my opinion the purpose of Gadfly is only educational but nothing that one really needs or uses for production. It could be removed and made available for download on zope.org. And my favourite enemy in Zope: ZClasses :-) I would like to mark them _clearly_ as an obsolete feature (DeprecationWarning, Warnings in the ZMI and the Zope Book). I _don't_ propose to remove them at some point but ppl should be aware that they are using one of the most-scary components in Zope (please no further discussion about the pros and cons of ZClasses..this discussion took already place already a bunch of times on the list). -aj -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: sessions in the presence of conflicts
Zope 2.8.4, ZODB 3.4.2 Chris, I'm pretty sure that I mentioned having done that in one of my postings. I have followed your recommendations, but the problem remains. (um... persists grin) The systems are running a Zope/ZEO combination with a store configuration of: # zodb_db temporary # Temporary storage database (for sessions) temporarystorage name temporary storage for sessioning /temporarystorage mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer /zodb_db # # ZEO client storage: # zodb_db main mount-point / # ZODB cache, in number of objects cache-size 5000 zeoclient server 192.168.0.92:8301 storage 1 var $INSTANCE/var # ZEO client cache, in bytes cache-size 20MB # Uncomment to have a persistent disk cache client group1-zeo /zeoclient /zodb_db # Although the connection to ZEO is via a network port, it runs on the same physical hardware. TemporaryStorage is not transactional. Does it need to be under MVCC? TemporaryStorage does provide a conflict cache to do rudimentary conflict resolution. There are several timing and scaling parameters that need to be considered: CONFLICT_CACHE_MAXAGE = 60 (seconds) CONFLICT_CACHE_GCEVERY = 60 (seconds) RECENTLY_GC_OIDS_LEN = 200 Entries in the recently gc's oids list are those which may be resolvable by a retry. These numbers may be too small given the loads we see and the number of accesses made to the session variables. I plan to incrase them to see if there is any impact. MAYBE CONFLICTS AND THEIR RESOLUTION ARE NOT THE ROOT CAUSE OF THE SESSION VARIABLE PROBLEM. The observed problem is that session variables suddenly disappear. At the point of failure due to a KeyError, inspecting the SESSION object shows two failure modes: either all the session variables are gone and only the container remains or most of the session variables are gone and a few remain. 74769573A2H7SIH2AKo=id: 11343269231636975299, token: 74769573A2H7SIH2AKo, contents keys: ['currentTab', 'calendarPage', 'currentCourse', 'currentTextbook'] and 77307574A2HTTdXCYYg=id: 11343267811075063138, token: 77307574A2HTTdXCYYg, contents keys: [] Access to the session variables are almost alwsys through a pair of Scripts(Python). Occasionally a session variable is read with an expression of the form REQUEST['SESSION']['key']. ## Script (Python) getSessionVariable ##bind container=container ##bind context=context ##bind namespace= ##bind script=script ##bind subpath=traverse_subpath ##parameters=varname ##title= ## request=container.REQUEST session=request['SESSION'] return session[varname] # Script (Python) setSessionVariable ##bind container=container ##bind context=context ##bind namespace= ##bind script=script ##bind subpath=traverse_subpath ##parameters=var, val ##title= ## request = container.REQUEST RESPONSE = request.RESPONSE session=request['SESSION'] session[var]=val request.set( 'SESSION', session ) This all seems right to me. Any suggestions as to how to localized when the session variables get lost? That might help localize the root cause. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: zLOG deprecation?
We probably want an ALL level as well which would map to the NOTSET of the Python logging code and log everything. Florent, I don't see a TRACE level in this list. Did you think one was needed? On Mon, 5 Dec 2005, Florent Guillaume wrote: Chris Withers wrote: # In the days of zLOG, there were 7 standard log levels, and ZODB/ZEO used # all of them. Here's how they map to the logging package's 5 standard # levels: # #zLOG logging #---- #PANIC (300) FATAL, CRITICAL (50) #ERROR (200) ERROR (40) #WARNING, PROBLEM (100) WARN (30) #INFO (0) INFO (20) #BLATHER (-100) none -- defined here as BLATHER (15) #DEBUG (-200) DEBUG (10) imnsho, blather and debug are the wrong way round. blather should go away completely, it's a stupid name anyway :-( Just because you don't see the use in something doesn't mean it has to go away. In my book BLATHER is used for verbose INFO and is different than DEBUG and TRACE (both levels are useful too). INFO and BLATHER are for the administrator. DEBUG and TRACE are for the developer. Florent -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Logging of ConflictError
Dieter's point about not includeing the traceback makes sense if all it does is report on the reporting code. Wlorent, do you envision a single ConflicError or two -- one which succeeds on retry and another where the retry fails? On Sun, 4 Dec 2005, Dieter Maurer wrote: Florent Guillaume wrote at 2005-12-2 22:59 +0100: ... If you look at the way their purpose is explained in zLOG, you'll see that level INFO is reserved for things like server startup and shutdown. Or, as shown above, initial mounting of databases. Anything recurring that can happen many times in the life of the server but does not pose any problems should *not* be visible at INFO. Really? You infer that from the INFO For things like startup and shutdown, do you? That's very weak reasoning... The level obviously has an importance association: higher values indicate higher importance. If fact, the ConflictError messages should not be reported at INFO but at level PROBLEM because they are not causing any immediate problems, but deserve attention. On the other hand, that's exactly what BLATHER is for folks! Use it! You see it this way. I do not: ConflictErrors are definitely as important as mount messages and therefore should be reported on at least the level of mount reports: i.e. at INFO or above. -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Please vote about conflict errors logging
1. INFO 2. Yes, with traceback (necessary as these are heissenevents) 3. ERROR 4. don't care -- we can adapt Thanks Florent for taking the lead on this. I think it's a great help. On Fri, 2 Dec 2005, Florent Guillaume wrote: Please vote for the level at which you want to log retried conflict errors. These are the ConflictErrors that aren't returned to the user but automatically retried by the Zope publisher. 1. Do you want these ConflictErrors retried logs to be at level: - INFO - BLATHER - DEBUG - not logged - other 2. In addition, please specify if you feel those retried ConflictErrors should have their full traceback logged? - Yes, with traceback - No, without traceback 3. Finally, please tell us if the ConflictErrors that *can't* be retried (and are returned to the user as an error, and are also logged to the error_log) should be additionally explicitely logged to the event log, and at which level: - ERROR - not logged - other (Also, if you feel the logging should be different between 2.8 and 2.9, please say so.) I'll wait until Wednesday morning to collect results. Thanks, Florent -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: How bad _are_ ConflictErrors
Thanks, I'll take a look. On Wed, 23 Nov 2005, Michael Dunstan wrote: On 11/22/05, Florent Guillaume [EMAIL PROTECTED] wrote: Dennis Allison wrote: *** you are correct -- this is the easy hack on the event.log. It's much harder to know how many make it out to the user. We have an associated bug in the MySQL interface which generates threading errors, apparently triggered by a conflict error and the subsequent backout. These occur with most conflicts which involve the database--almost every conflict with our system structure. I'm actually not sure what's logged when a Conflict Error makes it back to the users, offhand I don't see anything in my logs. Can someone confirm or infirm that fact? If nothing is logged, I'll add something at level ERROR. BTW does someone have a handy script to provoke conflict errors on a naked Zope? There is a doctest that might be useful here. See the last half of testPublisher() in lib/python/ZPublisher/tests/testPublish.py which tests the behavior of the publisher in the face of one or more ConflictErrors. -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How bad _are_ ConflictErrors
Conflicts and how they interact with the database and sessioning machinery is my hot button right at the moment )-: I Hope I have not included too much information. I ran a quick report and we see about 1000 conflicts per hour at about 12 hits per hour. These are order of magnitude numbers and are highly variable. The 1% number is way bigger than I am comfortable with although I have no basis to scale my expectations. I'd be much happier were it a couple of orders of magnitude smaller. Conflict errors are not always errors. As I understand it, Zope retries when a conflict occurs and usually is able to commit both sides of the conflicting transaction. Sometimes Zope cannot commit conflicting transactions--and it is at that point that an error occurs. There are supposed to be significant changes in the Zope 2.8.4/ZODB 3.4.2 system. Read-read conflicts no longer generate conflict errors and the retry mechanism has been reworked at the ZODB level to retry once and then raise a POSKEY exception. The optimistic locking used by Zope can cause problems, particularly when the conflicting method changes external state. We have seen instances where an action was taken multiple times due to conflicts and their resolution. In one instance, we had an infinite loop in the conflict resolution. The interactions which can cause conflicts are not always obvious. I am still learning. We do have occasional instances where unresolved conflicts raise user visible diagnostics. These are real errors. While I have not explored the reasons why, it appears that at least some of these errors are not logged in event.log but only displayed to the user. I asked the list the other day whether anyone had prepared a set of best practice guidelines on the techniques to use to minimize conflicts? Dieter Maurer responded: * Localize out into separate persistent objects attributes with high write frequency. E.g. when you have a counter, put into its own persistent object (you can use a BTrees.Length.Length object for a counter). * Implement conflict resolution for your high frequently written persistent objects. Formerly, TemporaryStorage had only very limited history information to support conflict resolution (which limited the wholesome effect of conflict resolution). Rumours say that this improved with Zope 2.8. * Write only when you really change something. E.g. instead of session[XXX] = sss use if session[XXX] != sss: session[XXX] = sss (at least, if there is a high chance that session already contains the correct value). Session variable present a particularly vexing problem since they may trigger writes even though they are apparently read-only. Chris McDonough [EMAIL PROTECTED] wrote in response to my posting: On Nov 20, 2005, at 12:16 PM, Dennis Allison wrote: [...] Looking at the code, I don't understand why I am seeing conflicts. As I understand things, neither variables in the dtml-let space nor the REQUEST/RESPONSE space are stored in the ZODB so modifications to them don't look like writes to the conflict mechanism. Am I incorrect in my understanding? Yes, but that's understandable. It's not exactly obvious. The sessioning machinery is one of the few places in Zope where it's necessary for the code to do what's known as a write on read in the ZODB database. Even if you're just reading from a session, looking up a session, or doing anything otherwise related to sessioning, it's possible for your code to generate a ZODB write. This is why you get conflicts even if you're just reading; whenever you access the sessioning machinery, you are potentially (but not always) causing a ZODB write. All writes can potentially cause a conflict error. While this might sound fantastic, it's pretty much impossible to avoid when using ZODB as a sessioning backend. The sessioning machinery has been tuned to generate as few conflicts as possible, and you can help it by doing your own timeout, resolution, and housekeeping tuning as has been suggested. MVCC gets rid of read conflicts. But it's not possible to completely avoid write conflicts under the current design. Here's why. The sessioning machinery is composed of three major data structures: - an index of timeslice to bucket. A timeslice is an integer representing some range of time (the range of time is variable, depending on the resolution, but out of the box, it represents 20 seconds). This mapping is an IOBTree. - A bucket is a mapping from a browser id to session data object (aka transient object). This mapping is an OOBTree. - three increasers which mark the last timeslice in which something was done (called the garbage collector, called the finalizer, etc). The point of sessioning is to provide a writable namespace
Re: [Zope-dev] How bad _are_ ConflictErrors
On Mon, 21 Nov 2005, Chris McDonough wrote: On Nov 21, 2005, at 2:10 PM, Dennis Allison wrote: Conflicts and how they interact with the database and sessioning machinery is my hot button right at the moment )-: I Hope I have not included too much information. I ran a quick report and we see about 1000 conflicts per hour at about 12 hits per hour. Is this the number of log messages that indicate a conflict error occurred (e.g. x conflict errors since DATE messages in the event log) or the number of conflict errors that are retried more than three times and thus make it out to the app user? I'm guessing the former. *** you are correct -- this is the easy hack on the event.log. It's much harder to know how many make it out to the user. We have an associated bug in the MySQL interface which generates threading errors, apparently triggered by a conflict error and the subsequent backout. These occur with most conflicts which involve the database--almost every conflict with our system structure. These are order of magnitude numbers and are highly variable. The 1% number is way bigger than I am comfortable with although I have no basis to scale my expectations. I'd be much happier were it a couple of orders of magnitude smaller. I would be too. It's considerably difficult when ZODB is used as the sessioning backend. A lot of effort has been put in to reducing the potential for conflicts already. It could of course be better if more time was put in, but there hasn't been any reason (besides a sense of accomplishment and contribution to the greater good, anyway ;-) to put in that effort since the last time this machinery was overhauled. *** I've moved from a ZODB sessioning backend to local sessioning. There has not been a significant change, I think because the MySQL problem dominates at the moment. That said, if no conflict errors actually bubble up to the user using the application, the penalty is just app performance and knowledge expense (e.g. you can't use a nontransactional mailhost, you can't use a nontransactional database table, etc). You've already paid for the latter the hard way. ;-) I can't judge the expense of the former to you but I assume that's what you're primarily worried about now. *** Right now, we have major problems with our transactional database and locks. Once that gets resolved, we will address how to refactor to minimize the cost of transactions and ensure correctness in the presence of conflicts. Correctness is already pretty much guaranteed with our current systems structure. Conflict errors are not always errors. The real reason they're called errors is only because they're implemented as Python exceptions. They are implemented as exceptions because it was the easiest mechanism to use (exceptions are already built into Python). As I understand it, Zope retries when a conflict occurs and usually is able to commit both sides of the conflicting transaction. There can be more than two sides (actually there always are... there are three.. the two conflicting in-progress connection states and the database state). Sometimes Zope cannot commit conflicting transactions--and it is at that point that an error occurs. An exception occurs, yes. Oops, I just realized Tim responded to the rest of these points, so I won't go on. *** Yes, he did. THANKS TIM for your comments and help. (And you too Chris) We do have occasional instances where unresolved conflicts raise user visible diagnostics. These are real errors. While I have not explored the reasons why, it appears that at least some of these errors are not logged in event.log but only displayed to the user. To be pedantic, if you're right about conflict error tracebacks being shown to end users, it's not because they are unresolved (in the sense that 'application-level conflict resolution' could have prevented them), it's because a request was issued that resulted in a conflict error, which was retried, and then that retried request raised a conflict error, and then twice more. The only way to figure out what's going on here is to see the traceback. IIRC, Zope logs conflict error tracebacks at the BLATHER log level (as well as a deluge of other ancillary info). However, even if BLATHER logging mode is not on, if no obvious error is put in the event log when a conflict error is relayed to a user, that's definitely a bug. I'd believe it in a second! ;-) *** have done that but no helpful results as of yet. The Zope conflict exception catching code is written in such a complicated way (and without the benefit of any automated tests) that tracking that down could take an entire day which I don't have to burn ATM. So I'm afraid the status quo will prevail until someone gets so indignant about
Re: [Zope-dev] Disabling FTP by default
That's what I have been doing. -d On Wed, 30 Mar 2005, Florent Guillaume wrote: I'd like to disable FTP by default, by commenting it out in skel/etc/zope.conf. The reason is that it's better to open as few ports as possible by default. Opinions ? Florent -- Dennis Allison * Computer Systems Laboratory * Gates 227 * Stanford University * Stanford CA 94305 * (650) 723-9213 * (650) 723-0033 fax * [EMAIL PROTECTED] * [EMAIL PROTECTED] ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] ANNOUNCE: Zope 2.6.3 Release and Security Update
Brian -- Does this mean that Zope 2.6.3 is compatible with Python 2.3.3? I would be nice to retire 2.1.3. -dra On Thu, 8 Jan 2004, Brian Lloyd wrote: Zope 2.6.3 Release and Security Update Zope 2.6.3 contains a number of security related fixes for issues resolved during a comprehensive security audit conducted in Q4 2003. You may download Zope 2.6.3 from Zope.org: http://www.zope.org/Products/Zope/2.6.3/ [...] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: ANNOUNCE: Zope 2.6.3 Release and Security Update
Glad to hear the news. I tried using Python 2.2 and 2.3 with Zope 2.6.2b3 with little success in the early Fall without success--but the many problems were RH9 related. We've since gone back to RH7.3 (the last stable RH release in my book) and used Python 2.1.3 for Zope (and Python 2.3.3 for everything else except RedHat tools). On Thu, 8 Jan 2004, Tres Seaver wrote: Dennis Allison wrote: Does this mean that Zope 2.6.3 is compatible with Python 2.3.3? I would be nice to retire 2.1.3. A significant part of the effort for 2.6.3 (which was a backport from the original 2.7 work) lay in ensuring that the issues were fixed under all three Pythons (2.1.3, 2.2.3, 2.3.3). While we won't change the officially supported Python at this point in the 2.6 release cycle, you should know that our own projects have been running 2.6.x on 2.2.3 since at least June. After this effort, I have no reservations about recommending that those same customers begin evaluating the use of 2.3.3 for their 2.6-based Zope sites. In addition, we have a major support customer who *must* upgrade to Python 2.3.3, in order to allow Data.fs on Windows to grow beyond 2 Gb; they are not, however, ready to upgrade Zope (yet) to 2.7. We therefor have incentive to fix at least *major* issues related to running 2.6.3 under Python 2.3.3 (cosmetics are a different story, of course). Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] Re: ANNOUNCE: Zope 2.6.3 Release and Security Update
Tried to do the former, but Python 2.3.1 would not build on RH9 with significant brain surgery. Updated RH9 to the bleeding edge and got things mostly working except for some subsystems adn supporting systems which use threading and would not work under the new threading model without significant rework. Hence the decision to revert to RH7.3. Eventually we plan to move to a Gentoo system--I've been experimenting with Gentoo and have found it to be fairly easy to construct a customized, fast, and clean system although the time-to-build can be daunting. After some more testing I plan to move to Gentoo for production, a move motivated by the bad experience I've had with RH9 and RedHat's new business focus on the enterprise. One point of information, Tres. Was your positive experience over a range of machines. We've pretty much standardized on dual processor Athlon machines, 4GB memories, and hardware raid controllers in a RAID-10 configuration. It's possible that our problems with RH9 may be tied to some problem with their Athlon SMP systems. Thanks for your comments and advise. On Thu, 8 Jan 2004, Tres Seaver wrote: Dennis Allison wrote: Glad to hear the news. I tried using Python 2.2 and 2.3 with Zope 2.6.2b3 with little success in the early Fall without success--but the many problems were RH9 related. We've since gone back to RH7.3 (the last stable RH release in my book) and used Python 2.1.3 for Zope (and Python 2.3.3 for everything else except RedHat tools). RH9 has been rock solid for us, given two choices we made: - *Never*, *ever* run Zope in production with the OS's version of Python: it *won't* be built optimized for Zope, and it *will* change unexpectedly (from the perspective of the appserver developer). The SA will feel free to change the OS Python, including adding potentially destabilizing extensions, or *upgrading* it, and will be unrepentant when you (the appserver developer) complain that he broke your Zope. - *Run*, don't walk to get access to the updated kernel and glibc; the kernel, in particular, which RH9 installs out of the box is pathetically useless for running a memory-hungry appserver. We have found both yum and the apt-rpm port useful for keeping servers up to date. Tres. -- === Tres Seaver[EMAIL PROTECTED] Zope Corporation Zope Dealers http://www.zope.com ___ Zope maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] Zope Binary Release Changes for Zope 2.6
Matt-- I've run both RH7.2 and RH7.3 and IMHO RH7.3 is more stable, etc etc. -dra ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
