[Zope-dev] Documentation
Im a little out of touch with Zope development these days; Ive been focussing on non-zope projects for the last year or so... I have some Zope documentation on http://www.zope.org/Members/htrd that I am no longer able to maintain, but wich generates a steady stream of emails with questions and change suggestions. In particular, my notes on using squid with zope, and the changes I originally drafted for the Zope Developers Guide on using unicode in Zope. Is there now a more approporiate place to hold this documentation where it can be maintained? Thanks in advance, -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] any limits on object number?
[resend] We are builiding a large portal using Zope. We need to create a large number of objects. How large? in one case, we want the metadata of all the wikipedia contents, that will be around 300,000 objects. The data component of the objects is small, but each object carries lots of metadata. My question is: Is there any limit on the number of objects in a given folder? No, but you should use BTreeFolder2 in any case and don't use the standard Zope folder! 'Folders' provide an easy gui for managing sub-items one at a time, and a single interface for accessing sub-items that enables things like the tree view (in the left pane of the management interface) to work with different folder implementations. Im not sure any of that is a benefit to you, and Folders arent the only kind of ZODB container. I suspect you would be better off directly using a BTree of some kind. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] RFC: Multi-Databases
(reply-to set to zodb-dev) In the long term, this will allow is to *finally* support cross database references. Your proposal describes an unfamiliar kind of reference. In unix filesystem terms, it mixes characteristics of hard links and symbolic links. Like hard links it directly references the inode/oid. The reference doent need to have a filename/object reference from the root of the referenced filesystem/database. Like symbolic links it does not prevent garbage collection from freeing the inode/oid. So there does need to BE a filename/object reference from the root of the referenced filesystem/database it needs to exist, but the reference doesnt need to have it. (by 'object reference' above I mean a chain of normal object references like we have in normal python and zodb today) Right now I cant think of a use case for this new type of reference (either in a filesystem or zodb database). Can you give some examples? -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Unicode in Zope 2 (ZMI, Archetypes, Plone, Formulator)
On Monday 26 April 2004 09:27, Bjorn Stabell wrote: While we're all waiting for Zope 3 and Plone 3, I'd like to know what the standard practice way of using Unicode with Zope 2. In particular, we'd like to store all text as Unicode in the ZODB, and have Zope do the encoding/decoding as automatically and transparently as possible. I added most of the Unicode support in core Zope in 2.6. Transparency was my goal too. For the applications I was working on at the time this goal was achieved. Naturally I stopped without converting all the components that I dont use, and sadly noone has picked up this work in a consistent manner. * in PropertyManager you can add ustrings, but since it doesn't add :ENCODING to the field names, you get a Unicode error when trying to save since it tries to decode the text assuming ASCII (big problem) It works for me in 2.6. Note that it only adds the :ENCODING after adding your first unicode property. This is ugly, but necessary for legacy applications using 8-bit string properties to hold pre-encoded text. I suggest this way of dealing with Unicode right now in Zope 2: good advice. None of the above components handles Unicode in this way, but it seems to be how the Unicode support in Zope 2 was meant to be used. That is correct. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] performance tuning of ZODB
On Thursday 22 April 2004 10:42, Syver Enstad wrote: I've used the cache_size paramter to the constructor of the DB to good effect. Are there any further gotchas for ensuring that the ZODB stays in memory as much as possible? Memory usage scales proportional to the number of threads. Reducing the number of worker threads lets you increase the per-thread cache size. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] performance tuning of ZODB
On Thursday 22 April 2004 11:43, Syver Enstad wrote: cache_deactivate_after sounds interesting. Since I am running ZODB in a web server I don't want the data to timeout and disappear from memory since every object I have should be loaded at all times. Thats why ZODB ignores that parameter now ;-) -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Display Files from directory
On Monday 15 March 2004 12:59, Darshan Preet Singh Manku wrote: file = 'C:\tmp' try 'C:/tmp' or 'C:\\tmp'. unless your file name really does contain a tab character ;-) -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)
On Thursday 11 March 2004 12:19, Chris McDonough wrote: What if you want to make a change in the error handler? If anyone had been depending on this behavior in their current error handlers, they would have seen weirdnesses in their applications like those being reported now. I have applications that send emails in error handlers. MailHost isnt transactional today, but Ive always wanted it to be. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Proposal (was How to make Zope fail nicely under high load?)
On Thursday 19 February 2004 11:18, Bjorn Stabell wrote: I wrote up a simple proposal for how to change Zope to work better under heavy load: http://zope.org/Members/bjorn/proposals/MakeZopeFailNicelyUnderHeavyLoad The solution is a bit different from what we discussed on the list before; basically do away with the Medusa request queue altogether and use listen()'s backlog instead. Last week you were keen that medusa keep processing requests so that it can return a 503 error quickly. I guess your view on that requirement has changed? -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Thursday 12 February 2004 00:57, you wrote: One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog. I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances. (my apologies if this is obvious) -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope, forking and cpu-affinity
On Thursday 12 February 2004 08:26, Dario Lopez-Kästen wrote: Dieter Maurer wrote: Zope itself does not fork (unless you use system, popen or things like that; in these cases, you do not want to bind the children to the same processor). Well, I had that impression as well, but here is the output of ps on my machines (edited for cruft-removal): UID PID PPID C STIME TIME CMD zope 32538 14379 0 Feb09 00:02:32 python2.1 z2.py -X -w zope 32539 32538 0 Feb09 00:00:00 python2.1 z2.py -X -w zope 32541 32539 0 Feb09 00:00:09 python2.1 z2.py -X -w zope 32542 32539 0 Feb09 00:00:11 python2.1 z2.py -X -w zope 32543 32539 0 Feb09 00:01:52 python2.1 z2.py -X -w zope 32544 32539 0 Feb09 00:01:41 python2.1 z2.py -X -w that is more than 2 processes for a single instance... one of them is the supervisor process that restarts the main zope process if it crashes. the others are all threads of one process. one main medusa thread, and 4 publisher threads. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Wednesday 11 February 2004 09:16, Jamie Heilman wrote: Bjorn Stabell wrote: Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog. You might get some benefit from capping the length of that queue. The idea is that you want tweak it so some of your users get everything, and some get nothing, instead of everybody getting partials. This means you have to track sessions though... Before creating a session, check the size of the ZRendezvous backlog. might work. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
On Tuesday 27 January 2004 19:08, Tim Peters wrote: Maybe Toby remembers which release(s) of ZODB the current cache implementation first appeared in Zope 2.6 -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
On Tuesday 27 January 2004 09:39, Mario Lorenz wrote:
Given that this property is not that widely published (in the various
tutorials etc.), I wonder if it might be a good idea to improve that loop
check code, and walk through the ring not more than once, using a counter,
and not the comparison vs. the ring start, which, as we have seen, may be a
target that moves too quickly.
You got me curious
The attached patch to 2.6 uses an extra temporary ring node to mark where
scanning should stop. Normally this will be right next to the home node, and
behaviour is unchanged. Any of these troublesome objects that move themselves
to the top of the LRU list *during* the scan will end up *after* the scan
terminator node, and therefore do not get rescanned.
This works for me on 2.6, which is the only branch Im set up to use at the
moment.
--
Toby Dickenson
? diff
? diff2
Index: cPickleCache.c
===
RCS file: /cvs-repository/Zope/lib/python/ZODB/Attic/cPickleCache.c,v
retrieving revision 1.68.10.3
diff -c -4 -r1.68.10.3 cPickleCache.c
*** cPickleCache.c 30 Apr 2003 17:03:34 - 1.68.10.3
--- cPickleCache.c 27 Jan 2004 23:02:06 -
***
*** 216,224
#endif
static int
! scan_gc_items(ccobject *self,int target)
{
/* This function must only be called with the ring lock held,
because it places a non-object placeholder in the ring.
*/
--- 216,224
#endif
static int
! scan_gc_items(ccobject *self,int target,CPersistentRing *terminal)
{
/* This function must only be called with the ring lock held,
because it places a non-object placeholder in the ring.
*/
***
*** 265,281
return -1;
}
#endif
! /* back to the home position. stop looking */
! if (here == self-ring_home)
! return 0;
/* At this point we know that the ring only contains nodes
! from persistent objects, plus our own home node. We know
! this because the ring lock is held. We can safely assume
! the current ring node is a persistent object now we know it
! is not the home */
object = OBJECT_FROM_RING(self, here, scan_gc_items);
if (!object)
return -1;
--- 265,288
return -1;
}
#endif
! /* reached the end position. stop looking */
! if (here == terminal)
! return 0;
!
! /* back to the home position. should not happen */
! if (here == self-ring_home) {
! PyErr_SetString(PyExc_RuntimeError,
! reached home before terminal, in scan_gc_items);
! return -1;
! }
/* At this point we know that the ring only contains nodes
! from persistent objects, plus our own home node, plus the
!terminal node. We know this because the ring lock is held.
!We can safely assume the current ring node is a persistent
!object now. */
object = OBJECT_FROM_RING(self, here, scan_gc_items);
if (!object)
return -1;
***
*** 325,332
--- 332,343
static PyObject *
lockgc(ccobject *self, int target_size)
{
+ int error;
+ CPersistentRing terminal;
+
+
/* This is thread-safe because of the GIL, and there's nothing
* in between checking the ring_lock and acquiring it that calls back
* into Python.
*/
***
*** 338,350
if (IS_RING_CORRUPT(self, pre-gc))
return NULL;
ENGINE_NOISE();
self-ring_lock = 1;
! if (scan_gc_items(self, target_size)) {
! self-ring_lock = 0;
! return NULL;
! }
self-ring_lock = 0;
ENGINE_NOISE(\n);
if (IS_RING_CORRUPT(self, post-gc))
return NULL;
--- 349,376
if (IS_RING_CORRUPT(self, pre-gc))
return NULL;
ENGINE_NOISE();
self-ring_lock = 1;
!
! /* insert a placeholder just before the head node. This will be used to
! * terminate the scan_gc_items loop. Items touched during the
! * deactivation run will get inserted between the head and this terminal
! * node, and will not get re-scanned. */
! terminal.next = self-ring_home;
! terminal.prev = self-ring_home.prev;
! self-ring_home.prev-next = terminal;
! self-ring_home.prev = terminal;
!
! error = scan_gc_items(self, target_size, terminal);
!
! /* unlink the placeholder from the ring */
! terminal.next-prev = terminal.prev;
! terminal.prev-next = terminal.next;
!
self-ring_lock = 0;
+ if (error)
+ return NULL;
+
ENGINE_NOISE(\n);
if (IS_RING_CORRUPT(self, post-gc))
return NULL;
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists
Re: [Zope-dev] cPickleCache endless loop...
On Friday 23 January 2004 17:08, Tim Peters wrote: It looks like ghostifying your self triggers self.__del__(). Then the __del__ method unghostifies self, which has the side effect of moving self to the MRU end of the ring, which in turn means the list traversal will visit self *again*. When it does, same thing happens all over again, ad infinitum. Not necessaralily ad infinitum. It will only run forever if the number of __del__-resurrecting objects in the cache is larger than the cache target size. Does that fit with your scenario? 2. If you need a __del__ method (it's hard to imagine why, since it will get called whenever the object is ghostified, and has nothing to do with the object's actual lifetime), don't reference any persistent objects (and esp. not self) within it. or 2b as jeremy suggested, put your __del__ on a non-persistent sub object. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZODB load state error
On Monday 26 January 2004 12:08, Thyb wrote: What could be the problem? That indicates that you are out of memory. If that doesnt seem right, maybe your storage is corrupt in a manner that the unpickler interprets as something with a huge memory footprint. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cPickleCache endless loop...
On Monday 26 January 2004 17:22, Tim Peters wrote: It's actually that the number of __del__-resurrecting objects *plus* the number of non-ghostifiable objects in cache is larger than the cache target size, right? Yes, Right. That is more achievable than I thought. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] 2.7 management_page_charset cannot be callable anymore
As the originator of management_page_charset, I should have jumped into this thread earlier. My appologies for my late arrival. On Thursday 15 January 2004 17:22, Martijn Faassen wrote: it assumes the only place management_page_charset can be coming from is a property, which is not true. Ive missed the use case that needs this. management_page_charset was added as a transition aid for people who already had sites where their use of the management interface assumed something other than latin-1 was used for plain string properties. That is, a transition towards using unicode objects for storing all natural language strings. I wouldnt like to see management_page_charset grow into something bigger than this - its too ugly. Considering the amount of rewriting necessary to make this saner and the scariness of doing that just before a Zope 2.7 release, I'll skip it and live with my Formulator hack solution. Can you point me to a desicription of this hack please. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZServer HTTP 1.1 support
On Thursday 11 December 2003 21:39, Paul Winkler wrote: In particular, some people on my team asked me if zserver supports persistent connections, and I didn't know how to answer. I'm looking now at HTTPServer.py and the docstrings suggest that it does ... yes? no? If you have a squid proxy in front of your zope, then squid will manange both HTTP 1.1 persistent connections to your browsers, and persistent connections to your Zope. The squid-to-zope persistent connections are not in the style of HTTP 1.1 though. Squid uses slightly different headers - I forget the details. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.7.0 b3 regressions
On Monday 08 December 2003 21:21, Paul Winkler wrote: On Mon, Dec 08, 2003 at 08:24:04PM +0100, Dieter Maurer wrote: Maybe, my contribution has not been read. Thus, I try again: / + absolute_url(1) should implement the notion of absolute-path URL reference (see RFC2396 section 5). This means, that the receiving browser should be able to retrieve the object correctly given this URL reference. Yup. But while we're on the subject... Why doesn't absolute_url(1) include a leading slash? I don't think I've ever seen a use of absolute_url(1) that didn't have to add the slash. What was the rationale originally? Because dtml-var BASEPATH1/dtml-var absolute_url(1) looks nicer than without the slash ? -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Post-mortem [Was: Zope 2.7.0 b3 regressions]
On Tuesday 09 December 2003 16:24, Evan Simpson wrote: Summary: absolute_url(1) Looks good. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.7.0 b3 regressions
On Monday 08 December 2003 11:35, Stefan H. Holek wrote: Note that this is one of my main points: It will be of little use to document usage of BASEPATH1+absolute_url(1) when '/'+absolute_url(1) appears to work (until it is far too late). We can fix this social problem by providing an easy way for product developers to run their development zope server with the virtual path equivalent to an inside-out hosting configuration. easy means not needing apache/squid. Our staging server is uses an inside-out virtual host configuration (to simplify ssl certificate management) so we hit all these problems early enough to fix the damage cheaply. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Unicode id's and utf8 url encoding
On Tuesday 11 November 2003 14:23, Tres Seaver wrote: On Mon, 2003-11-10 at 22:13, Stuart Bishop wrote: Hi. I think the last thread on this issue was: http://aspn.activestate.com/ASPN/Mail/Message/zope-Dev/1843637 From my reading of RFC2396 and RFC2277, the character set encoding for URLs is UTF8. This is confirmed by http://www.w3.org/International/O-URL-code.html . Does this settle the issue on how to handle non-ascii strings in URLs? if only browser always worked that way :-( If so, is their anything stopping us allowing Unicode ids in Zope? This would involve patching OFS.ObjectManager.checkValidId to accept strings not in [0-9a-zA-Z\$\-_\.\+!\*'(),], and writing replacement urllib.quote and urllib.unquote methods for use by HTTPRequest.py. +1 on starting a prototype now, but I cant see this landing before 2.8. This is going to break every use equivalent to getattr(some_object, id). Its not obvious to me how this can be cleanly resolved in zope 2. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] _p_deactivate() and _v_ variables?
On Thursday 23 October 2003 08:07, Chris Withers wrote: Toby Dickenson wrote: Apart from the most trivial cases, it would allow _v_ attributes to disappear at random. Its a similar problem to the one that makes it hard to write an optimiser for python code, and I am unconvinced that this is sane. Which, unfortunately, then leaves us with the problem of how to stop Zope using up an undeterminable amount of memory... No, we just exclude objects with _v_ attribute from mid-transaction deactivation. There arent many objects in that category, but they do need protection. But, your proposal means we would improve the situation for transactions that read from an undeterminable number of persistent objects. It does not help for transactions that touch an undeterminable number of non-persistent objects, or transactions that change an undeterminable number of persistent objects. Is the gain big enough to justify the effort? -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] cvs.zope.org down
On Tuesday 21 October 2003 18:08, Jens Vagelpohl wrote: Just a quick heads-up: Then we will start restoring the data from the old drives. That makes me nervous. How will you know that the sources in cvs havent been compromised? -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: _v_ variables and cache garbage collection
On Thursday 23 October 2003 18:52, Chris Withers wrote: What in the ZODB cache or other ZODB code could be causing _v_ variables to stick around after they've been set to None in their containing objects? reference cycles -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Making Zope accept international characters in Id's
On Monday 20 October 2003 15:01, Lennart Regebro wrote: The fact that litmus complains about Zope's inability to accept UTF-8 filenames is something I view as a bug. does webdav specify how utf-8-encoded dav filenames correspond to file names used in URLs? Either the already existing, but seemingly undocumented setting management_page_charset could be used to check that the given Id is acceptable in that charset. That feature is intended only as an aid for the transition to unicode for users who have content that is stored in an encoded form. OR, a separate setting could be used, maybe allowed_id_charset? If we need to support unicode ids, then the right approach is to allow them to be of unicode type. It looks like OFS has the beginnings of supporting this (as your proposed code fragment pointed out), but Im not aware of it being used. I'm gonna go with that last one if nobody complains. consider this a complaint. character encodings should be applied at zopes boundary, not in the middle of its processing. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] _p_deactivate() and _v_ variables?
On Wednesday 15 October 2003 12:47, Chris Withers wrote: Casey Duncan wrote: I would argue that a better plan would be to only use _v_ vars for completely disposable data only. The application should expect that this values will be gone at any random time, not just at transaction boundaries. Agreed. Are there any situations, apart from the already discussed CMF skindata, where this currently isn't the case? every database adapter? (I guess, but havent checked) -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] _p_deactivate() and _v_ variables?
On Wednesday 15 October 2003 14:53, Casey Duncan wrote: Agreed. Are there any situations, apart from the already discussed CMF skindata, where this currently isn't the case? I'm a bit puzzled - of what use is a variable which may disappear at any random time? It's not exactly random. It would happen when the object was deactivated (removed from cache). The proposal earlier in the thread was aiming towards allowing objects to get deactivated at any time if the cache was overfull, not just at transaction boundaries. This is desirable from a cache management point of view. Apart from the most trivial cases, it would allow _v_ attributes to disappear at random. Its a similar problem to the one that makes it hard to write an optimiser for python code, and I am unconvinced that this is sane. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] _p_deactivate() and _v_ variables?
On Wednesday 15 October 2003 15:51, Casey Duncan wrote: As it is, the hard cache implementation, although beneficial from a memory management perspective cause loaded servers to do alot more work because they are constantly pruning the cache and then reloading objects again immediately thereafter. If you are seening that then I think you need a bigger cache. And possibly fewer publisher threads. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-CMF] Re: [Zope-dev] _p_deactivate() and _v_ variables?
On Friday 10 October 2003 18:34, Dieter Maurer wrote: Toby Dickenson wrote at 2003-10-10 07:54 +0100: ... A while ago there was a discussion on zodb-dev about _v_-like attributes that would be automatically cleared at the end of a transaction. Do we need something similar that guarantees it will _not_ be cleared until the end of the transaction? This definitely is necessary for the _v_ attributes that hold connections to a relational database. If such a _v_ attribute is flushed, the next access to the DA (in the same request) reopens the database. As this is a new connection, it does not see the changes made by the previous connection (in the same request). Thats how alot of code works today, and how it had to be done in the past. Today we have the 'transaction participant' interface. That would be a better place to hold these things, allowing the DA object itself to be deactivated if necessary. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] _p_deactivate() and _v_ variables?
On Thursday 09 October 2003 14:01, Florent Guillaume wrote: I agree with this. How do we go about find code that uses the assumption that _v_ stuff won't change unless it's at a transaction boundary? Note that we had a problem related to this with a client recently: In CMF, skin data is stored in portal._v_skindata, and is actually needed for the whole request, but in some ZEO setting this go cleared by a get_transaction().commit(1) which was unexpected and led to breakage because in that batch treatment we used some skin methods too... Something after the subtransaction commit must be tickling the cache garbage collector. Thats generally what subtransactions are used for. A while ago there was a discussion on zodb-dev about _v_-like attributes that would be automatically cleared at the end of a transaction. Do we need something similar that guarantees it will _not_ be cleared until the end of the transaction? -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Re: [Zope-dev] Using 2.3.2 for Zope 2.7
On Friday 03 October 2003 15:33, Chris McDonough wrote: Jim is keen to get an audit going quickly before a 2.7 final release, and the audit would be performed against Python 2.3.2. Does anyone else have an interest in blessing Zope 2.6.x with Python 2.2/2.3 ? -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] ZODB 3.2 feedback
On Monday 29 September 2003 22:25, Christian Reis wrote: On Mon, Sep 29, 2003 at 05:06:52PM -0400, Jeremy Hylton wrote: Has anyone had a chance to test ZODB 3.2b3? We're hoping to do a release candidate tomorrow, and it would be good to know if anyone has tried to the beta release. If you've got a current CVS checkout of Zope I'm unable to test because of the Python 2.2 requirement :-( Im in the same position, also :-( -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] _p_deactivate() and _v_ variables?
On Friday 26 September 2003 09:32, Chris Withers wrote: Toby Dickenson wrote: On Thursday 25 September 2003 11:51, Chris Withers wrote: Hmmm, does _p_deactivate() clear the contents of the object's _v_ variables? Yes Then given your earlier comment that _v_ variables are supposed to last at least teh length of the request, John's idea of using _p_deactivate() to reduce memory usage for large ZCatalog results sets could be seen as playing with fire, right? He had code that would only deactivate objects at the end of a loop if they were not active at the start of the loop. Its not entirely safe, but I think it mitigates the most serious risks here. ZCatalog (and others) have used this same technique for a while. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] _p_deactivate() and _v_ variables?
On Thursday 25 September 2003 11:51, Chris Withers wrote: Hmmm, does _p_deactivate() clear the contents of the object's _v_ variables? Yes -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Catalog performance
On Thursday 11 September 2003 03:03, John Barratt wrote: I think ghosts are only 'removed' after a restart, fyi, ghosts are removed from memory using reference counting. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Cache growing during single REQUEST
On Wednesday 03 September 2003 12:15, Chris Withers wrote: If the object load would cause the cache to go above it's maximum number, *Number* isnt the the right parameter to control here. We need to limit the total amount of RAM. Objects are of variable size, and the largest ZODB objects are very much bigger than the average. One task I would like to find time for is making the cache aware of this. For example, this would fix the problem where the current cache unfairly penalises ZCatalog operations because its BTree nodes are so small. then boot an object out of the cache in order to make room for the new one. That would have a bad effect on ReadConflictErrors. Cache purging should only happen on transaction boundaries for storages where ReadConflictErrors are possible. control the amount of memory Zope actually uses and other requests would stand a chance of beind processed normally. I havent seen a mention of ulimit or autolance earlier in this thread They are mostly adequate protection against the work problems. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Cache growing during single REQUEST
On Friday 05 September 2003 14:21, Chris Withers wrote: Hmmm, maybe not... could we make a note of the pickles size when the data is loaded and update that size when it's comitted? Is this the same as the in-memory size? Yes, I hope its close enough for most purposes. It would be nice to have a way for the object to override that in the cases where it is badly wrong. then boot an object out of the cache in order to make room for the new one. That would have a bad effect on ReadConflictErrors. Don't follow, can you explain? You get a ReadConflictError when loading an object if it has been modified since the start of the transaction. This exception therefore becomes increasingly likely as time progresses since the start of the transaction. Today you can minimise the probability of a read conflict by touching the objects you need (at least the ones likely to change) near the start of the transaction. This works because, once loaded, the objects stay in memory until the end of the transaction. Another problem I didnt mention is that _v_ attributes are supposed to last at least until the end of the transaction. I havent seen a mention of ulimit or autolance earlier in this thread They are mostly adequate protection against the work problems. Do they kill the thread or the whole Zope? whole zope of course there isnt a great overhead in using twice as many zopes, each with half as many publisher threads. (assuming you already have zeo, of course) I'm also keen for users to nto get MemoryErrors, but to just have their request take much longer ( cache thrashing and the like...) use squid, and it will retry the request. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] UndoSearch Timeout
On Wednesday 06 August 2003 22:05, Brian Brinegar wrote: Hello, We have a ZODB which is about 12 - 14 gigs when packed. We use FileStorage, and when anyone clicks Undo eventually times out. This is better now that the undoLog does not lock the database. However if there are not enough transactions to fill the batch Undo is useless. So, everytime we upgrade to a new version of Zope I patch FileStorage.py to add a timeout. Used to be 3 seconds when the ZODB locked, now I've bumped it up to 10 seconds. Has anyone else had this problem? Would adding the timeout, possibly set by an environment variable, be considered by the maintainers? DirectoryStorage has always provided a timeout, set by config file. There have been occasions where I wanted to wait for the full detail, and a global timeout setting was inconvenient. Is it worth adding it as a new parameter to these functions? -- Toby Dickenson - http://www.geminidataloggers.com/people/tdickenson Want a job like mine? http://www.geminidataloggers.com/jobs for Software Engineering jobs at Gemini Data Loggers in Chichester, West Sussex, England ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] segfaults in cPersistence under 2.6
On Friday 25 July 2003 08:30, Anthony Baxter wrote: More information: I don't see the failure with a fresh Data.fs. The Data.fs in question is my 2.5 one. Do you have any custom persistent extension classes stored in that Data.fs that might not have been updated to the 2.6 persistence C API? -- Toby Dickenson - http://www.geminidataloggers.com/people/tdickenson Want a job like mine? http://www.geminidataloggers.com/jobs for Software Engineering jobs at Gemini Data Loggers in Chichester, West Sussex, England ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: FHS, zopectl, #925, Re: [Zope-dev] 2.7 installation
On Thursday 19 June 2003 15:58, Shane Hathaway wrote: Has anyone tried extracting Gentoo's build system and using it for partial software distributions? That might be the way to go. I am now using Gentoo on all my servers and on this workstation. I find value in being able to manage Zope stuff as a non-root user. Portage was designed to allow this, but I doubt it is well exercised. Trying this has been on my to-do list for a while. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] version status
On Tuesday 17 June 2003 09:01, Oliver Bleutgen wrote: I don't quite understand the nature of this DOS attack after the patch. You do requests with REQUEST['Zope-Versiom'] == big string. If I understand your code correctly (it was bash and perl afterall ;)) you create version i with a version name str(i)*50. It seems (to me) that the sole cause for this DOS is that zope stores the version names in memory, that means you get a memory consumption for all version name strings of 10*50 + 90*50*2 which is 95.000.000 bytes, which is roughly the 90M you reported. The connection cache will also store a cached connection for each version. The connection is opened to *read* from the storage; no writes are needed. A more 'efficient' attack would be to use a tiny (but unique) Zope-Version string to request a page that loads alot of zodb objects into the connection cache, for example as a seach page. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZMI tweak
On Saturday 14 June 2003 16:53, Christian Theune wrote: When catching up with the mail in the mailinglist, I remembered a small annoyance I'm having with the ZMI. I really appreciate the errorlog and am using it frequently. But if you don't put it it's own window (or tab) you always need two clicks to reach it. I once put it's icon in the upper ZMI bar to move me to the servers central error log. Is this something that others would like too? I have a number of objects that provide site-global information like the error log. I have occasionally considered developing a mechanism to allow them to hook into the global gui; either the top bar or Control_Panel. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZMI tweak
On Monday 16 June 2003 08:05, Toby Dickenson wrote: On Saturday 14 June 2003 16:53, Christian Theune wrote: When catching up with the mail in the mailinglist, I remembered a small annoyance I'm having with the ZMI. I really appreciate the errorlog and am using it frequently. But if you don't put it it's own window (or tab) you always need two clicks to reach it. I once put it's icon in the upper ZMI bar to move me to the servers central error log. Is this something that others would like too? I have a number of objects that provide site-global information like the error log. I have occasionally considered developing a mechanism to allow them to hook into the global gui; either the top bar or Control_Panel. To throw in an extra idea... For a long time I have been using a patch that puts BASE0 and the title of the root folder in the ZMI top bar. The reasons for wanting this should be appreciated by anyone who has ever had two browser windows open for two different zope sites, and made a change in the wrong one. The patch is in toby-zmi-title-branch (the branch tag is only on the lib/python/App/dtml directory) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] version status
On Sunday 15 June 2003 08:11, Jamie Heilman wrote: Whats the status of versions for 2.6.2 and 2.7? Have there been any decisions reached? I saw Jim's code get checked in but it won't stop the DoS I posted. Ive not tested Jims code, but it looks to me like it *should* stop that attack. Have you tested it? -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: small summary and big plea was:(Re: [Zope-dev] Versions: should they die?)
On Friday 06 June 2003 21:28, Jamie Heilman wrote: Quick way to add 100 zodb connections and ~90M to the memory footprint with relatively little clue of who is responsible assuming traditional logging; presumeably one would get much trickier if they really wanted to obfuscate the source of attack, slowly crawling the site, changing the user-agent string, etc. Attached is a temporary patch to block this denial of service attack. This patch applies cleanly to the trunk and the 2.6 branch. This patch works by blocking all access to versions in the publisher, so dont apply it if you cant afford to stop using versions -- Toby Dickenson http://www.geminidataloggers.com/people/tdickensonIndex: lib/python/ZODB/ZApplication.py === RCS file: /cvs-repository/Zope/lib/python/ZODB/ZApplication.py,v retrieving revision 1.13 diff -c -2 -r1.13 ZApplication.py *** lib/python/ZODB/ZApplication.py 8 Apr 2003 18:48:22 - 1.13 --- lib/python/ZODB/ZApplication.py 9 Jun 2003 22:58:43 - *** *** 42,49 def __bobo_traverse__(self, REQUEST=None, name=None): ! db, aname, version_support = self._stuff ! if version_support is not None and REQUEST is not None: ! version=REQUEST.get(version_support,'') ! else: version='' conn=db.open(version) --- 42,53 def __bobo_traverse__(self, REQUEST=None, name=None): ! # Disable nasty insecure version support. Thanks to ! # Jamie Heilman and everyone one zope-dev ! # ! # db, aname, version_support = self._stuff ! # if version_support is not None and REQUEST is not None: ! # version=REQUEST.get(version_support,'') ! # else: version='' ! version = '' conn=db.open(version)
Re: small summary and big plea was:(Re: [Zope-dev] Versions: should they die?)
On Tuesday 10 June 2003 09:32, Jamie Heilman wrote: Toby Dickenson wrote: ! # Disable nasty insecure version support. Thanks to ! # Jamie Heilman and everyone one zope-dev Unless you're damning me with faint praise for posting an exploit, (which is fine) No criticism was implied public exploits are valuable part of the security process. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: small summary and big plea was:(Re: [Zope-dev] Versions: should they die?)
On Friday 06 June 2003 15:04, Shane Hathaway wrote: I think 2.6 ought to fix this by disabling recognition of the Zope-Version cookie Setting this individually for each http port would better support existing happy users of this feature. (Im sure there must be some ;-) Being able to set up a port that ignored cookies but always used a specific Version would avoid many of the existing problems too. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Versions: should they die?
On Wednesday 04 June 2003 17:40, Brad Clements wrote: Sorry if is OT. I'd like ZODB and Zope to support Revisions. That is, historical copies that do not get removed when ZODB is packed. DirectoryStorage allows you to designate classes that should have all their history kept indefinitely. That may not have the flexibility that you need... Does the Version mechanism contribute to this kind of functionality? I'd like to be able to tag revisions of wiki pages and other documents, and then be able to diff them later without having to create a special kind of product for each object type that I'd like to use revisions for. The accepted wisdom is that feature like this should be implemented in the application, above zodb. This has come up a few times on zodb-dev list. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZCM] [ZC] 869/ 5 Comment Broken transaction handling in case of exceptions
Lets take this back to the list Ive trimmed the cc list to people I think might be interested. http://collector.zope.org/Zope/869 I am not in line with your principle the error report is part of [the output of] the first transaction. The error report tells you what went wrong. You can only generate an accurate report if you see the objects in the error state. Zope is saying to your application: your transaction is doomed. tell me why. As you said: Drawback: you may see newer data in the error report than has been seen by the erroneous transaction. I think thats a *major* drawback. The risk of zodb badness is a major drawback too. Small risks add up over time. But, even with Zope, when the error report should do something persistently, it must do it in a new transaction as the old state is inconsistent and can not be committed. I agree that it is unacceptable to commit changes made during the error report without starting a clean new transaction. I agree my proposal is inadequate if we need to allow this. Currently Zope doesnt allow the error report to do something persistently, and the use case for this is not obvious to me. Can you explain the use case? I am hoping that there may be a better solution that avoids the problems I raised. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [ZCM] [ZC] 869/ 5 Comment Broken transaction handling in case of exceptions
The view has access to the original request that ended in the error. The view is looked-up and rendered in a new transaction. A new transaction means that *arbitrary* changes may have occurred to the application state in between: 1. the transaction that raise the error, and 2. the transaction that reports it In principal it is posssible to delete the database state and import a whole new application between these two points. The object containing the originally published method might no longer exist. I guess that might not matter if your error page just says sorry an internal error has occurred - if you want your users to be shielded from the details of any problem. I have some applications at the other extreme, where the users are all capable of debugging any problem and I want the error page to be a core dump of application state. Can you give an example of the kind of situation where you'd need access to objects in a doomed transaction in their doomed state, in order to make an error report? Access to the doomed state at the end of the transaction is what zope 2 gives us today. Access to the objects restored to their state as they were at the beginning of the failing transaction would be ok too (maybe even better) but zodb cant do that today. Either of these two are necessary for easily providing an error report that displays application state explaining why the error occurred. Under the Zope 3 model I think I would need to take a non-persistent copy of the application state in an exception handler towards the end of the first transaction. The second, error-reporting transaction would dump the copy, rather than dump what it sees to be the current world state. It is important that the objects passed from one transaction to the next are non-persistent, otherwise bad zodb-level things can happen. Does Zope 3 guard against this? -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Errors (read conflicts)
On Tuesday 18 March 2003 3:24 pm, Chris McDonough wrote: Our live sessions code uses the sessions about half to two-thirds of the way through the transaction. Given what can happen in that first half, there is easily plenty of time for read conflicts. I think I might be able to move our session use to the beginning of the transaction (just storing stuff in the REQUEST object for later usage) and hopefully fix our issue. Depending on what your sessioning involves, you might not need to change when you do all the work. Possibly just touching (an attribute of) the persistent session object will be enough. But I'm not sure I understand why touching an object at the start of a transaction would fix the consistency problem. If a object A is read at the beginning of a transaction that uses the data from object A during write of object B and later object A changes and is invalidated by another transaction (before the first finishes), no conflict errors will be raised. Isn't there the potential for the first transaction to write inconsistent data into object B? Today you are guaranteed that your transaction reads a consistent initial state of A and B. That is, there is no chance that you only see half the changes of a recent transaction that modified both. Yes, it is possible for one transaction to modify A and a concurrent transaction to modify B. This is what Deiter describes as not safe in that zodb-dev thread. IMO, it is only not safe if you are not aware of it. And it is critical for performance with concurrency - consider these two transactions occuring on different zeo nodes. In my mind, this is the same thing as using a low conflict connection (ie. you are essentially turning off read conflicts, albeit by circumstance rather than by code). There is a significant difference. The low conflict connection permits the possibility of seeing half the changes of a recent transaction that modified both objects. low conflict connection is a misleadnig name. The conflicts are still there - just not reported as exceptions. low consistency connection would be more accurate ;-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Errors (read conflicts)
On Tuesday 18 March 2003 6:12 pm, Chris McDonough wrote: I'm thinking that I also may just need to move the housekeeping duties to a separate scheduled thread that only happens when the system is not busy (e.g. when the asyncore poll select timeout is reached maybe) in order to reduce the potential for conflicts. Thats hard to define, when zeo is installed. But I'm not sure what else to do, and I still don't understand what is causing the KeyErrors in the storage code. Sigh. Ive never looked at nor used your Sessioning support, but I am interested in things that can cause (POS)KeyErrors in Storages. This might not be a storage bug. There are several known corner cases in zodb that mean semi-legitimate applications can cause KeyErrors. DirectoryStorage tries to guard against them, and will raise a DanglingReferenceError rather than commit a transaction that will cause a POSKeyError when read. Can the problem be reproduced using DirectoryStorage? (without the low consistency connection) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Errors
On Saturday 15 March 2003 5:40 pm, Chris McDonough wrote: - The second is an unidentified (yet) bug in TemporaryStorage. As a cause of the KeyError exception? Could be I have seen equivalent exceptions with plain FileStorage in applications that do not use sessions, so I suspect there might be a hard-to-hit zodb bug that can cause this too. (In FileStorage the equivalent exception is POSKeyError, a subclass of KeyError) If I find and fix the TemporaryStorage bug I will let you know. In the meantime, you can probably work around this by using a different non-undoing storage to put your session data in (e.g. Berkeley, or maybe DirectoryStorage?). It would be interesting to hear if this exception is repeatable with the session state stored in a FileStorage too. You may see many more conflicts with this running. But maybe the data structures will not become desynchronized. You weren't kidding about the increase in conflict errors. Read conflicts occur if a change is committed in between the start of a transaction, and the transaction needing to load the object. A workaround to reduce the number of read conflicts is to touch the objects that are likely to change early in the transaction. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How (in)secure is Zope?
On Thursday 13 March 2003 5:21 am, Shane Hathaway wrote: The only vulnerability would involve trusted users who want to vandalize Zope. So even though there have been many hotfixes, they are irrelevant--Zope is still secure. (Unless you can't trust your trusted users, which is a different problem.) Of course you cant *completely* trust your trusted users. Thats why we have seperate user accounts, and seperate roles. IMO: Zope is sufficiently vulnerable to abuse from trusted users to justify concern. The normal zope development model is to consider normal python code as trusted - normal python code can do anything without security checks. Zope has many normal python methods that can be called by through-the-web code (after permission checking). In unix terms this is equivalent to having many setuid root programs. IMO concern can be justified without needing to find a specific exploit. From this point of view, Jamies advocacy of using Unix mechanisms to restrict this 'trusted' python code is valuable. On Thursday 13 March 2003 1:58 am, Jamie Heilman wrote: I will go on record as saying that, recently, response times to security related issues in the Zope2 tree have been disapointing. Construe from that what you will. It is hard to find time for security work among the feature rush of the cvs trunk, and without compromising the stability of the maintenance branch. Would there be any interest from other developers in addressing these potential security issues in a *fork* starting with the 2.6 maintenance branch? (reply-to set to zope-dev) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How (in)secure is Zope?
On Thursday 13 March 2003 9:25 am, Lennart Regebro wrote: 5. Protecting yourself against denial of service: Zope does not seem to crash if you send random data to it, and I have in logs seen attemps to overflow buffers and the like that obviously are attempt to crash or break in to other (MS) servers, without this affecting Zope at all. There is evidence that this is not true. If you don't trust Zope in this, you can put Apache in front of it. In this sense Zope is again VERY secure. Zope is insecure Zope+Squid(or Apache or Pound)+OS resource limits+careful choice of products is secure (Note that I dont consider this a flaw in Zope.) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Re: [Zope-dev] Proposed installation changes for review
On Tuesday 11 March 2003 10:48 pm, Jamie Heilman wrote: You'd probably still want a single master config file for the whole thing, and a tool to check the configuration is valid separate from the process that uses the file to configure itself. Not I. Large applications with a master config file are to be held with suspicion. Their longevity inevitably suffers because they are difficult to adapt to new situations. Im not sure the big config file approach is necessarily less adaptable than the big /etc directory approach. It is the details that make the difference - both approaches can be done well. Another way to ease configuration is to make things modular so its easier to visualize the flow of data. There is no amount of reconfiguration that can improve this in Zope2. Zope3 promises to fix this, but with modular python components rather than modular unix components. I would be interested in your thoughts on whether this makes a difference. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Prevayler and Object Prevalence
On Monday 03 March 2003 3:48 pm, John Ziniti wrote: Any ZODB developers read this article at /. and the accompanying one at developerWorks? Would anyone care to comment about the relevance/comparisons of prevalence to the concepts of persistence used in ZODB and/or ZEO? I had looked at this (and the python port) briefly last year The main advantage of ZODB over a relational approach is that it can *automaticly* add ACID conditions to application logic objects, without major code changes. It is possible to think about your application logic and ignore the ODB aspects. Until recently, using a relational store for objects meant adding intrusive persistence code and breaking encapsulation. As I understand it, prevayler preserves encapsulation, but still requires the intrusive code changes. It relies on separating appliction logic from the object store. Application logic can only read and write to the object store using remote method calls, which prevayler calls persistent command objects. Commands are processed by a single threaded server process. Command objects have to be persisitent because durability is provided by writing them to a log file, and replaying commands on recovery. There is no way to apply transactional semantics to application logic, except by cramming all the work into one big command object. (I hope that helps. Appologies in advance for any inaccuracies) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] support for low level HTTP Response logging?
On Tuesday 25 February 2003 12:08 pm, Romain Slootmaekers wrote: We need this in our 3th line support, were we want to be able to follow all ins and outs for singled out user (fi filtered on cookie). We also need to be able to turn this on/off at runtime. I would do this in the front-end proxy. Set up a squid acl to catch this one user, and send his requests via tcpwatch (or similar) rather than direct to zope. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Bare except dangerous to ZODB?
On Thursday 20 February 2003 5:17 pm, Jeremy Hylton wrote: There are application-level reasons to mark a transaction as doomed, and I would like to keep *that* code looking similar ;-). The transaction states approach would work in that context too, right? Here's a late answer: If an application needs to mark a transaction as doomed, it is supposed to call get_transaction().abort(). If a transactional resource manager, like a database connection, needs to mark a transaction as doomed, it: - returns False from prepare() -- the ZODB4 spelling - raises an exception in tpc_vote() -- the ZODB3 spelling Thanks for that idea. Normally in the Zope context, transaction and request boundaries are co-aligned. This assumption runs deep in zope. I can see problems because an application calling abort wont stop the publisher calling commit at the end of the request. Having application state revert mid-request to the pre-transaction state seems like a bad idea. Commiting application changes made in the second half of the request seems bad too. (All from theory - I ve not tested this) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How can I tell what a Zope Process is doing?
On Tuesday 18 February 2003 4:04 pm, Chris Withers wrote: Hi, I have a Zope instance whose memory usage is ballooning every so often (to 800Mb or so :-S) Now, I can see using top that one Zope process is using a lot of CPU and I'm guessing it's doing some silly iteration that's dragging huge numbers of objects into memory. If you have got a live one then Control Panel / Debug Information will show current requests. As a related question, does the line get written to Z2.log before or after serving the request has been completed? You mean the log that has a field for the number of bytes sent back to the client? ;-) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: Bare except dangerous to ZODB? was Re: [Zope-dev] Accept-Charset hearders causing 500 internal server error.[correct but not lenient]
On Monday 10 February 2003 8:47 pm, Shane Hathaway wrote:
Great, so there's at least 133 things to examine to see if they
could catch a ConflictError. And I only wrote about 15 of those.
The rest could be very time-consuming to audit.
tal:on-error also catches all exceptions. It could be made to catch
all exceptions except ConflictError, but I don't feel like that's the
right solution.
A while ago I tracked down a bug in one of our products to a case where a
mutator method failed half way. It raised an exception and left the object in
an invalid state. This would be safe except for a dtml-try that swallowed the
exception.
Recently I have been experimenting with this attached module to ensure that an
application-level object can veto a transaction. I guess it would be
applicable to ConflicctErrors too.
from zLOG import LOG, WARNING, ERROR
class _veto:
def __init__(self,msg):
self.msg = msg
def sortKey(self):
return repr(self)
def _do_nothing(self,*args):
pass
tpc_finish = tpc_abort = abort = abort_sub = _do_nothing
def _veto(self,*args):
try:
LOG('Veto', ERROR, self.msg)
except:
pass
raise TranactionVetoError(self.msg)
__getstate__ = tpc_begin = commit = tpc_vote = commit_sub = _veto
class TranactionVetoError(Exception):
pass
def veto(msg=None,ob=None):
Call this if your object is in an invalid state,
it is about to raise an exception, and you want to be sure that
the current transaction does not complete even if something
swallows that exception.
msg - An error message describing the problem. optional.
ob - The object which is now invalid. optional. If specified,
the transaction will only abort if this object is
persisted.
m = 'This transaction had been vetoed following a previous error'
if ob is not None:
try:
m += ' in '+str(ob)
except:
m += ' in unprintable object'
if msg is not None:
try:
m += ':\n'+str(msg)
except:
m += ':\nunprintable error'
print 'nnn',m
if ob is not None:
try:
ob._transaction_veto_attribute_that_can_not_be_persisted = _veto(m)
except:
pass
else:
return
# Drop through if we can not set our veto attribute on that object,
# or if no object was specified
get_transaction().register(_veto(m))
Re: [Zope-dev] [Bug] Zope's transaction behaviour flawed
On Sunday 02 February 2003 3:40 pm, Dieter Maurer wrote: This is flawed as error handling is done outside of a transaction. Excellent analysis. A futher problem is that this could cause dangling references, and a subsequent POSKeyError, since persistent objects can be passed from one transaction to the next inside the exception and traceback. The same applies to your prorosed fix. Is there a need to allow the error handling transaction to commit? I propose it always be aborted. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [Bug] Zope's transaction behaviour flawed
On Sunday 02 February 2003 3:40 pm, Dieter Maurer wrote: Zope's current transaction behaviour is essentially: 1 ## request starts 2 transaction.begin() 3 try: 4object= REQUEST.traverse(...) 5mapply(object,...) 6transaction.commit() 7 except: 8transaction.abort() 9handle_error() 10 ## request ends This is flawed as error handling is done outside of a transaction. Potential changes during the error handling spill over uncontrolled into another request and are there either committed or aborted as part of this request. A visit to the dentist has given me some time to think about this, and I think there is a flaw in this explanation. Surely any changes that leak out from one request will will be be aborted before they can do any damage, during the call to transaction.begin() on line 2. The doc string from transaction.begin is: '''Begin a new transaction. This aborts any transaction in progres. ''' (I should have a chance to experiment with this tomorrow) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope-Coders] FYI: Zope 2.6.1 b2 scheduled for friday
On Wednesday 29 January 2003 8:50 pm, Brian Lloyd wrote: (I saw Toby had committed some things, and it looks like that merge is complete). Yes, that is complete -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] product refresh and random number generation.
On Tuesday 28 January 2003 11:31 am, Romain Slootmaekers wrote: I'm having trouble understanding what is happening: You are storing an instance of the Random class in ZODB, but not telling ZODB about changes to its internal state. def next(self): returns next random number. self._p_changed = 1 return self.__random.random() -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Sunday 26 January 2003 7:25 pm, Kazuya FUKAMACHI wrote: Have you decided which version the patch should be included? I combined 4 of the patches and Yousei Tahara's patch into single patch and published for japanese users on Jan 15. http://www.atransia.co.jp/home/ZenKai/Members/kafka/patch/Zope261b.patch Thanks to everyone who has helped on this. This patch is in cvs ready for 2.6.1 beta 2. Please give the beta a good work out. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Sunday 26 January 2003 7:25 pm, Kazuya FUKAMACHI wrote: On Tue, 14 Jan 2003 10:33:40 + Toby Dickenson [EMAIL PROTECTED] wrote: I am currently looking at getting this into 2.6.1 or 2.6.2. I would appreciate confirmation that you are all happy with this combination of patches. Have you decided which version the patch should be included? I am trying to get confirmation of the release schedule for 2.6.1 beta 2. If only I had known it would be delayed this long back in December... I've announced the patch by Japanese Zope Users Group Mailing list, which has more than 1000 subscribers. More than 100 people downloaded it, and any no complaints heard. Many said they were happy with the patch. Thats good news. Thankyou. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] 2.6.1b2?
On Saturday 25 January 2003 1:07 am, Guido van Rossum wrote: Guido van Rossum wrote: If only it were that easy. Everybody wants no new features except for the one feature they desperately want to have without upgrading to the next feature release. :-( Still, we're back to the situation where it's been weeks since the last release, and there's nothing we (the public) can do about that :-( If you want to make a rogue release, that will be available until Brian deems the time ready for the next release, be my guest. Is there an estimate for this? afaict, the public schedule since December has been that this is imminent any day now. I have some important bug fixes that I would like to include in 2.6.1, and I would like some assurance that beta 2 will not be released mid-merge. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope reference counts
On Wednesday 22 January 2003 3:01 pm, Ivo van der Wijk wrote: It soudns asif a further upgrde is reasonable. I think you will have much more joy with 2.6 Also, the 'activity' tab is new to me (looks interesting ;), and caching settins seem to be ignored mostly until 2.6, right? it worked in a way that lead to strange behavior. 2.6 is more predictable, and understandable. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope reference counts
On Tuesday 21 January 2003 4:14 pm, Ivo van der Wijk wrote: Does this mean each thread keeps it's own cache pool of objects? yes Do these references point to the same object instances, or are these copied as well? They are independant copies I have the impression the memoryconsumption grows with the number of threads - is this true? yes Cheers I hope this helps -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope and Python compatibility
On Monday 13 January 2003 7:46 pm, Casey Duncan wrote: I personally feel like Zope should run on 2.2.x whether required or not. Requiring 2.2.x does have the advantage of limiting the possible blessed configurations on which Zope will run, thereby making development, support and bug fixing that much easier. Im not sure thats true. Having less strict version requirements means that there will be *more* *people* doing the development, support, and bug fixing. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Saturday 21 December 2002 10:54 pm, Kazuya FUKAMACHI wrote: I applied Toby's patch + Yousei's patch, and it works. Thanks to your neat piece of work, I can start to work on Zope 2.6.0 now, and go farther. I am currently looking at getting this into 2.6.1 or 2.6.2. I would appreciate confirmation that you are all happy with this combination of patches. thanks, -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope and python compatibility
On Saturday 11 January 2003 4:06 pm, Jamie Heilman wrote: Guido van Rossum wrote: Without python 2.2 zope will continue to harbor remotely exploitable zlib-based memory exhaustion attacks. FWIW There are workarounds that work in 2.1.x, and I dont think any uses in Zope are performance-critical. I will be happy to push through any patches for any other Zopes uses. Can you explain? Where does Zope even use zlib? dtml-tree for one, That is the least of your worries for dtml-tree. Please try the patch at: http://www.zope.org/Members/htrd/tree.diff This has a chance for 2.6.2 if I get enough positive feedback from people who actually use dtml-tree. (and a sufficently paranoid review would be nice too.) -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] zope and python compatibility
On Sunday 12 January 2003 8:16 pm, Guido van Rossum wrote: On Saturday 11 January 2003 4:06 pm, Jamie Heilman wrote: Guido van Rossum wrote: Without python 2.2 zope will continue to harbor remotely exploitable zlib-based memory exhaustion attacks. FWIW There are workarounds that work in 2.1.x, and I dont think any uses in Zope are performance-critical. I will be happy to push through any patches for any other Zopes uses. I expect I'll be releasing Python 2.1.4 in the next month or so. Can you submit a patch and assign it to me? The fix in python 2.2 is in rev 2.44 of zlibmodule.c. It involves an API addition to the zlib module, which I understand is discouraged for a Python bug fix release? I think the workarounds in Zope are a better solution for Zope 2.6.x. We can gradually migrate these to the new zlib API once Zope has other dependencies on Python 2.2 -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Python method wrapping around DTML method
On Friday 03 January 2003 10:13 am, Bjorn Stabell wrote:
Right now, I'm struggeling because I cannot get the
namespace to be passed to PlainFunc, therefore PlainFunc cannot pass the
namespace on to DTMLFunc. Any clues?
There isnt one true namespace. You can create your own as easy as { }, and it
will be as good as any other.
If you want an authentic ZPublisher namespace then you need
http://www.zope.org/Members/htrd/howto/FunctionTemplate
--
Toby Dickenson
http://www.geminidataloggers.com/people/tdickenson
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Friday 20 December 2002 6:26 pm, Yusei TAHARA wrote: 1. browsers autodetection make a mistake. Not a browser mistake - this is *my* mistake. thanks. 2. management_page_charset_tag really needs? Yes, I agree with your analysis. I made a patch for fix those problems. It looks good work on my zope. Good work. This definitely looks viable for 2.6.2 -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Collector 697 should be critical!!!
from ages ago In OFS/dtml/properties.dtml I replaced select name=xyz:utf8:list multiple with: select name=xyz:utf8:list:string multiple It works for me, but I'm not sure, if this is a good approach on this problem (think of the integers as items for the multiple-select). I havent looked in detail yet, but my first impression is that this is a good approach. Well, I think that this is just a workaround, but not a bugfix ;) This patch was committed on bug day Assume that ulines property is used to define the set of possible values for a multiple selection property. In this case one would expect to get a ustring vaule back. And as Maik already mentioned, it is even possible to use something like [1, 2, 3,4] as the possible values of the multiple selection . OK, this case does not work very well with older version of Zope too, but I think that this shows the general problem: Currently (including older Zope versions), there absolutely no type conversion for multiple selections. Correct. That means that multiple selection can never work 100% with ulines without risking breaking old code. Something like your suggested improvement would be one way to improve this. I think this would make a good fishbowl proposal -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Wednesday 11 December 2002 2:15 am, Kazuya FUKAMACHI wrote: 5) Toby's proposal I hope +1. I'm not satisfied with (1)-(4). So, I would like to wait for Toby's implementation. Ive not had as much time for this as I hoped, so please excuse any rough edges in these patch. All comments gratefully accepted. http://collector.zope.org/Zope/737 -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Friday 20 December 2002 2:22 pm, Kazuya FUKAMACHI wrote: if all of them are included in Zope 2.6.1. The final beta is due today. If that goes to plan then it is already too late. 2.6.2 is a realistic target. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tuesday 10 December 2002 12:03 am, Kazuya FUKAMACHI wrote: On Mon, 09 Dec 2002 13:18:26 -0800 Heiichiro NAKAMURA [EMAIL PROTECTED] wrote: 2. Whenever any experimental enhancements to the ZMI which rely on using Unicode is to be integrated, create an new tab and put these features in that page in order: Sounds reasonable. USA is #1 in Internet Users with 160M. http://www.etforecasts.com/pr/pr1202.htm At least 21.91% of Internet users are CJK users. (Japan + China + South Korea + Taiwan) Since using UTF-8 for their pages is still not in common in these area, it is preferable to accomodate such workaround. I dont follow the logic here. Using unicode in the server doesnt force you to use utf8 in the browser. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Monday 09 December 2002 9:18 pm, Heiichiro NAKAMURA wrote: On Mon, 9 Dec 2002 10:13:16 + I agree with everything, except This is a tentative limitation on UI design, and should be removed after the Unicode Support gets matured enough to handle Unicode object with all languages/encodings. Thats not right. The core unicode support has been mature in this area for over a year. The incompatability with pre-encoded strings is a design choice - not something that will improve with time. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Monday 09 December 2002 2:20 am, Heiichiro NAKAMURA wrote:
On Sun, 8 Dec 2002 21:58:16 +
Toby Dickenson [EMAIL PROTECTED] wrote:
On Thursday 05 December 2002 9:36 pm, Toby Dickenson wrote:
Yes, I have an idea. I hope to find time to flesh it out early next
week.
I propose:
Here is my understanding of your proposal:
1. management_page_charset property for non-Latin-1
Affected application: ZMI only
Limitation:
- Standard Objects in Zope (such as title property) must be
Non-Unicode.
Yes, if using management_page_charset.
- All objects must be Non-Unicode in order to work in ZMI.
To be precise, all objects rendered on the management page must be plain 8bit
strings.
- If there is even one Unicode object there, all data will be
assumed as 'Latin-1' and non-Latin-1 data cannot be used in
any object.
yes
- REQUEST.set('management_page_charset','UTF-8') does not work
when 'management_page_charset' is set as global property.
That is the current, broken behaviour. I propose that REQUEST.set overrides
the global property. Note that no standard ZMI page will do so, except the
'properties' tab explained below.
Homework (new issues):
- define the safe steps of implementation of Unicode Support
- find a way of smooth migration of MBCS-to-Unicode
Yes. There is nothing zope-specific about this being hard ;-)
2. Behaviour of handling of text in ZMI
Not all of ZMI - this is just the 'Properties' tab.
When UnicodeType Properties are contained:
- internal representation: 'UCS-2/UTF-16' if UnicodeType(ex. ustring)
'Latin-1' if not UnicodeType(ex. string)
- encoding of output data: 'Latin-1'
No, output over http will be utf8.
- encoding of input data: 'Latin-1'
If you mean input from the browser when a property changes, no. it will use
utf8.
When UnicodeType Properties are NOT contained:
- encoding of input data: any
yes, the same as all other ZMI pages. latin-1 by default, and can be overidden
by management_page_charset.
- internal representation: same as input (no conversion)
- encoding of output data: same as internal (no conversion)
Limitation:
- Non-Latin-1 value cannot be used when creating the first Unicode
Property.
(Any correction is welcomed)
I'll ask power users of Japan-Zope-User-Group-ML about opinions
regarding to the proposal.
but I
suggest that compatability with this feature should not hold back any
future enhancements to the ZMI which rely on using unicode)
I guess this statement is somewhat ambiguous. Probably we could say
something like that? :
1. Any future enhancements to the ZMI which rely on using unicode
should be carefully defined, examined, evaluated and feasibility-tested
so that all issues/limitations can be clarified and the consequent
impact of these issues/limitation can be evaluated in order to
avoid hassle implementation integrated into the official version.
yes
2. Any experimental enhancements to the ZMI which rely on using unicode
must be integrated into the official version in the manner that
it doesn't affect the behaviour of 'legacy' applications,
I disagree. I dont want to block a good new feature from Zope 2.x (x=7)
simply because it uses unicode in a way that is incompatible with the
management_page_charset property. Thats what 'legacy' means.
until
the Unicode Support gets matured enough to handle Unicode object
with all languages/encodings.
Yes. There are several places in the ZMI that could benefit from
'unicodization'. At the moment I suspect it will be difficult to implement
these improvements while retaining support for management_page_charset.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non Bloating Page Counter
On Wednesday 06 November 2002 8:33 am, Janko Hauser wrote: Just an idea from the 'could-be-done-if-needed-department'. Generally there is quite often the need to have non-undoable properties in different objects of a site. There is one way to store it in the temporary storage, which is ram based, and not shared between ZEO instances. But if a second non-undoable storage is mounted it could be stored there and could be shared between ZEOC. To make this more transparent, one could perhaps implement something like a special propertysheetobject, which stores its properties in this mounted storage. Is there anything obviously wrong with this idea? the problem is that cross-storage references are tricky. Mixing undoable and non-undoable *objects* in the same storage is on the to-do list for DirectoryStorage, http://dirstorage.sourceforge.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thursday 05 December 2002 9:36 pm, Toby Dickenson wrote:
On Thursday 05 December 2002 8:41 pm, Heiichiro NAKAMURA wrote:
Does anyone have any other idea for the Collector 623 issue?
I hope better ideas will be posted..
Yes, I have an idea. I hope to find time to flesh it out early next week.
I propose:
1. Currently you can set a management_page_charset property to override the
global assumption that legacy ZMI pages are latin1. This is a hack that
happens to mostly work by accident, provided your ZMI pages do not encounter
a unicode object. I propose promoting this to a standard documented feature.
Is there anyone who actually uses this feature who can contribute some
documentation?
(Note that feature will never work with pages that do encounter unicode
objects. This means Zope should try to avoid gratuitous unicode usage, but I
suggest that compatability with this feature should not hold back any future
enhancements to the ZMI which rely on using unicode)
2. The documented way of setting a ZMI-page-specific encoding is
REQUEST.set('management_page_charset','UTF-8'). Currently
management_page_charset as a global property will override this page specific
setting. This is a bug that needs to be fixed.
3. I propose changing the encoding used by the standard 'Properties' ZMI page.
Currently it uses UTF8 always, and assumes that 8bit string properties are
latin1. I propose that this policy is used only if a unicode property has
already been defined on this object. If it has not, it uses the same encoding
policy as every other ZMI page - that is the value of the
management_page_charset property, or latin-1 if it has not been set.
A disadvantage of this change is that it will not be possible to set a
non-latin-1 initial value when creating the first unicode property. I think
this is just about acceptable.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] bug day
On Thursday 05 December 2002 5:14 pm, Chris McDonough wrote: Developers and any other interested parties should gather via IRC on the #zope-dev channel on irc.openprojects.net at the time and date above. I think its worth saying that the most important (IMO) outcome from the last two bugs days has been *concensus* about which bugs can be forgotten about, or rejected, and which are most important. You dont need to be a hardened zope developer to make a valuable contribution. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thursday 05 December 2002 8:41 pm, Heiichiro NAKAMURA wrote: Does anyone have any other idea for the Collector 623 issue? I hope better ideas will be posted.. Yes, I have an idea. I hope to find time to flesh it out early next week. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tuesday 03 December 2002 1:16 pm, Yusei TAHARA wrote: Hi. I made monkey patch for myself, when management_page_charset is not UTF-8, this patch remove :utf8: from non unicode type input field. because if input values are not latin-1, then unicode error raised. I think that every encoding can be used it satisfactory:-) Please consider whether it can merge into Zope2.6.1. There are some details missing from your explanation, but hopefully not from your patch. where do I find it? What thing is concretely some details? The fix to this situation is more complicated than removing a :utf8: from somewhere that it shouldnt be. Im sure you know this. I'm very interested in Zope development, especially i18n. So I would like to contribute something about it:-) The patch you mentioned? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Bugfix release?
On Sunday 01 December 2002 3:36 pm, Brian Lloyd wrote: FYI I'd like to have a 2.6.1 beta out next week. Jeremy is still looking at a few ZODB bug reports - as soon as he's done we'll make the beta. Im not sure this is a good plan. Jeremy's sortKey changes look like they deserve a longer beta testing period than I would have thought the demand for a DateTime fix would allow. My understanding was that objects would not be required to implement sortKey (and wouldn't be adversely affected for not implementing it) unless they cared about the specific issue that it addresses. I'll verify this w/Jeremy tomorrow, but last I heard ordering should only actually be an issue for Connection objects - other objects that play with the transaction mgr will generally not be subject to the deadlock issue that the sorting is designed to prevent (the deadlock is due to connection interaction w/a storage server). Yes, Jeremys changes should be entirely backwards compatible. I have commented on zodb-dev how well this has been handled. I am entirely confident that it can go into a 2.6.x fairly soon. However it is still a major change to a critical component, with some indications that things are not 100% right: http://collector.zope.org/Zope/701 http://lists.zope.org/pipermail/zodb-dev/2002-November/003810.html I am not normally shy of the bleeding edge, but I am still running the cvs dated 2002-11-11 in production. This one change makes me nervous. A bug in the transaction manager would be really bad news. I would have thought it appropriate to resolve these problems and leave at least a few weeks of beta test before release. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Collector 697 should be critical!!!
On Monday 02 December 2002 4:11 pm, Maik Jablonski wrote: I don't have much experience with the unicode-converters too... that's my problem for this issue...:-( I had seen this bug on the list, but hant spotted that it was a unicode-related. I will try to take a look sometime this week. Abel Deuring told me a patch, which solves the problem in the meanwhile: In OFS/dtml/properties.dtml I replaced select name=xyz:utf8:list multiple with: select name=xyz:utf8:list:string multiple It works for me, but I'm not sure, if this is a good approach on this problem (think of the integers as items for the multiple-select). I havent looked in detail yet, but my first impression is that this is a good approach. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Saturday 30 November 2002 5:39 am, Yusei TAHARA wrote: 2. ustring can not join or replace to 8-bit strings other than ascii. Yes, it is painful to work in a mix of pre-encoded 8 bit strings and unicode strings. Thats why Zope's unicode policy was intended to be entirely optional - at least in the short term - and provided you can avoid unicode objects entirely. Zope's approach is that your application (excluding the ZMI for now) should work as in 2.5, provided you dont use any unicode values. Can you confirm that this is working OK? yes. If I input values by PythonScript without using property tab, it works well. Hurrah. One area where this went wrong for you is the properties page, which is explicitly set to utf8 to allow unicode properties to be set. Im not yet sure on the best way to fix this for you, but I agree it needs fixing. I made monkey patch for myself, when management_page_charset is not UTF-8, this patch remove :utf8: from non unicode type input field. because if input values are not latin-1, then unicode error raised. I think that every encoding can be used it satisfactory:-) Please consider whether it can merge into Zope2.6.1. There are some details missing from your explanation, but hopefully not from your patch. where do I find it? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Friday 29 November 2002 5:44 am, Yusei Tahara wrote:
Hi.
The right approach is to make it possible to change the title property to
a unicode string. All my custom products have this already, but it is a
deficiency in the standard Zope types such as 'Folder' that their titles
type can not be changed.
This approach is not useful for me.
I often use zope with RDB like MySQL.
generally japanese encoding text is in RDB.
so if object properties are unicode string,
I always encode it before publishing.
title tal:content=python:here.title.encode('euc-jp')TITLE/title
because, it always mix in RDB data(japanese) and
properties(python unicode string) in one page.
certainly I will be faced with this ustring problem everytime.
Japanese charactor is not ascii,
so if I do not encode ustring before publishing,raise unicode error.
The unicode changes were designed to cope with your scenario, so I would like
to explore the limits of where this approach has gone wrong.
As I understand it, you dont want to do anything in unicode. You store,
manipulate, process, and output strings as 8-bit pre-encoded strings. You
are making an assumption that these 8-bit strings use some specific character
encoding. The scope of this assumption is quite broad. It has to cover:
1. All strings stored by your ZODB
2. All strings stored by your RDB.
3. All processing performed by Zope products. (must follow your assumptions,
or be encoding neutral)
Anything that beaks these assumptions will need special treatment.
Am I right so far?
Im sure you are aware of the limitations of this approach - they are the
limitations that unicode was designed to break (not Zope's unicode, but
unicode in general). Im not sure your approach is workable long term because
an increasing number of Zope products will naturally use unicode so that they
can store properties containing text across a range of scripts. Even if this
is not workable long term, it was supposed to work in 2.6.
Zope's approach is that your application (excluding the ZMI for now) should
work as in 2.5, provided you dont use any unicode values. Can you confirm
that this is working OK?
Zope 2.5 left ZMI character encoding down to browser autodetection, and as a
result most ZMI-controlled properties are encoding-neutral. For compatability
with unicode pages, which are seen as the long term future, the ZMI has to
specify *some* character encoding. By default in 2.6 this is latin-1, a
change which I think was announced mid-way through the 2.5 development cycle.
I understand that some users are very happy overriding this with a
management_page_charset property on their root folder. Ive never used this,
and it wasnt designed to work this way, but it looks like it works due to
happy coincidence. Note that this doesnt work for management pages which
explicitly set their own character encoding, or management pages which
'accidentally' encounter a unicode string.
(Although this management_page_charset feature was not planned, I guess I will
be supporting this feature if it is being used)
One area where this went wrong for you is the properties page, which is
explicitly set to utf8 to allow unicode properties to be set. Im not yet sure
on the best way to fix this for you, but I agree it needs fixing.
As far as I can tell this is the only area which should be causing you
problems. Am I missing something?
(The 'title' problem I mentioned in a previous post is also a problem that
needs fixing, but not one that affects you if you dont want to use unicode.)
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] post-publishing hook
On Friday 29 November 2002 12:07 pm, Carlo Giomini wrote: Dear all, I need a sort of post-publishing hook (so to say). I need Zope to call a function of mine as the very last action of publishing a request, i.e. after having built the response (and sent it to the browser possibly), but before starting publishing a new request. My Zope server is running as a single-threaded app. Does Zope support this any way? Yes, but its nasty. Look at ZPublisher.BaseRequest._hold ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Bugfix release?
On Wednesday 27 November 2002 5:50 pm, Brian Lloyd wrote: FYI I'd like to have a 2.6.1 beta out next week. Jeremy is still looking at a few ZODB bug reports - as soon as he's done we'll make the beta. Im not sure this is a good plan. Jeremy's sortKey changes look like they deserve a longer beta testing period than I would have thought the demand for a DateTime fix would allow. Sure, ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] consistent naming in ZEO releases
On Monday 25 November 2002 2:03 pm, Andrew Sydelko wrote: You'll see that when I switched to ZEO 2.0 I had to change the URL. No doubt you had to change alot more too. This was a major release. Lots of things changed between ZEO 1 and ZEO 2. I think you picked a bad example; I agree it should be consistent between minor releases of the same product. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] DB.close() needs to be called
On Tuesday 12 November 2002 7:16 pm, Barry A. Warsaw wrote: Looks like Toby's recent change to ApplicationManager.py causes DB.close() to never be called when you hit Shutdown in the Control Panel. Yes. This is a bad thing for the Berkeley storages because their .close() must get called or you'll end up with corrupt databases or worse wink. How much of that paragraph is covered by the wink? Even before my recent changes, there are plenty of other ways that Zope can open the storage, run a transaction, then hit an error before properly starting up which means it fails to close the database correctly. Would this be bad? So here's a patch to z2.py to fix this. I won't check this in because it looks a little ugly to me and I'm not sure what the right fix is, That is exactly what I was planning to commit. Before making that change, I need to remove a couple other calls to db.close() to prevent it being called twice on the same db. but we definitely need to fix this before Zope 2.6.1 is released. 2.6.1? This change has only been on the trunk, not the 2.6 maintenance branch. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] DB.close() needs to be called
On Wednesday 13 November 2002 4:01 pm, Chris McDonough wrote: Maybe normal shutdown should manually call the shutdown signal handler function and normal restart should manually call the restart signal handler function? We are pretty close to that now, which I agree is a good thing. The shutdown signal handler is now a one-liner that calls Lifetime.shutdown. The ZMI handler is a two-liner that calls the same function, then returns some html. Lifetime.shutdown causes the asyncore select loop to return (possibly after a brief pause to allow clients to disconnect cleanly), and z2.py closes the databases (and hence storages) as its very last action. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] DB.close() needs to be called
(cc zodb-dev, who may also be interested) On Wednesday 13 November 2002 4:18 pm, Barry A. Warsaw wrote: TD == Toby Dickenson [EMAIL PROTECTED] writes: worse part. If you've enable autopacking and you don't cleanly close the storage, you won't exit the process because the autopack thread won't get stopped and joined. TD Yes, I had the same problem with DirectoryStorage, which uses TD a seperate thread to move writes from its journal directory TD into the database directory. Ah. Is this transactional? What would happen if the thread got killed in the middle of the move? There are lots of ways to look at that question I am relying on the filesystem guarantee that the files are all either in their original directory, or their new directory. DirectoryStorage has its equivalent of BerkeleyStorage's autorecovery which, at startup, asynchronously resumes the file moving wherever it left off. A more interesting question is what happens if the thread is killed when it has moved some, but not all, of the files which relate to one transaction. In DirectoryStorage terminology this leaves the data directory in a state which is not a snapshot. The only disadvantage is that you can not use the tools which assume the data directory is a snapshot: the checking tool, the incremental back tool, and the replication tool. We could make the autopack thread a daemon process TD Thats how DirectoryStorage now works. Hmm, maybe we make it a daemon thread and put a timeout on the join, so we'll try to exit cleanly and won't hang if we can't. Because some of the operations performed in the daemon thread take a long time to complete? Would it be possible to break those operations into transactionally-coherent chunks which complete in a reasonable time? close() could wait for the current chunk to finish. Autopacking should be safe transactionally, but we'd have to do a recovery if it got killed uncleanly. Doesnt having a good checkpointing strategy mean that this should never take long? TD Last time I looked at your BerkeleyDB storages, the TD administrator needed to implement his own checkpointing TD policy. I always thought that was a disadvantage. Would it now TD be a good idea for the storages to trigger their own TD checkpointing inside the autopacker thread? Actually, now you can configure the storages to automatically checkpoint every nth ZODB transaction. Checkpointing in a thread may or may not provide additional benefit. Interesting. BerkeleyStorage's automatic checkpointing is currently triggered inside a commit or abort. This means the checkpoint overhead is incurred at the one time you dont want it - while a user is waiting for his transaction to commit. DirectoryStorage's main use for the thread is to perform its equivalent asynchronously. Assuming your storage is not saturated with writes (which only happens in benchmarks ;-) then checkpointing happens for free. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [ZODB-Dev] Re: [Zope-dev] DB.close() needs to be called
On Wednesday 13 November 2002 5:27 pm, Barry A. Warsaw wrote: Potentially yes, although the steps in the pack process are BerkeleyDB transactionally protected. I think the mark-and-sweep phases are examples of things that could take a long time. It should be possible to craft some escape hatches to do a more controlled shutdown. Yes, DirectoryStorage's referential integrity rules had to be carefully crafted to ensure that it can bomb out of a pack at short notice. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
