Re: [Zope-dev] limit to number of operations in a transaction?

2002-09-17 Thread Karl Anderson

Christopher N.Deckard [EMAIL PROTECTED] writes:

 I have roughly 1,500 people in the database.  I wrote a script to
 migrate the people.  Basically it does getProperty for each property
 on each person folder, then creates the new person object which uses
 XML.  Zope apparently cannot handle this number of operations in one
 transaction.  Zope gets slower and slower and eventually becomes on
 responsive.  It looks like the script has completed, but nothing is
 ever committed to the ZODB, and since Zope is unresponsive it must
 be restarted.  This, as expected, kills that entire transaction
 which was never committed.
 Is it known that large numbers of operations, such as above, in a
 single transaction can cause problems?  A transaction, of course,
 being a request, and an operation being something like

In practice I've only seen problems when I'm dealing with lots of data
and need to be thinking about this anyway :)

 I've solved the problem by using xmlrpc and for person in people
 calling my migrate_person script for only one person at a time. 
 This is SO MUCH FASTER.  I previously ran the script that migrates
 all of the people, and after 8 hours it still had not completed.

As Jens replied, this is because the transactions are getting
committed for each person, and you can do this without xmlrpc by
committing in your script.

The possible drawback of committing after each person is that you're
committing a new version of the object being modified.  Depending on
how you're storing the stuff this can grow your ZODB - if every person
is a node in a single ParsedXML document, you'd be storing 1500
versions, each one person bigger than the last.

You can use transactions and subtransactions to juggle database size,
memory usage, and temp file usage.

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

[Zope-dev] DAV collections, index_html, and HEAD

2002-06-06 Thread Karl Anderson

Objects that inherit from Folder (and therefore from Collection) and
also have an index_html that's a Python method (or othewise
nonvanilla, presumably) return a 405 (Method Not Allowed) for HTTP
HEAD requests.

Zope 2.5.1 raises on line 57 of  The object raises if
its index_html member doesn't have a HEAD attribute.

Surely it's a bug to only give HEAD for objects that return an
index_html via hasattr/getattr.  What about getitem,
__bobo_traverse__, and __call__?  Shouldn't we find the object to HEAD
via ZPublisher?  And aren't we bypassing the access control checks,

Would it be a good idea to try and make up a HEAD response for
arbitrary attributes that don't support it (such as Python methods)?
Perhaps by publishing them internally, and picking whatever we can
from RESPONSE, computing Content-Length from the returned stream if
necessary?  It's inefficient, but at least it's cache-friendly... and
better for tools (er, linbot) that assume that if it can't be found
via HEAD, it isn't there.

Excerpt from Collection.HEAD():

# Note that we are willing to acquire the default document
# here because what we really care about is whether doing
# a GET on this collection / would yield a 200 response.
if hasattr(self, 'index_html'):
if hasattr(self.index_html, 'HEAD'):
return self.index_html.HEAD(REQUEST, RESPONSE)
raise 'Method Not Allowed', (
  'Method not supported for this resource.'
raise 'Not Found', 'The requested resource does not exist.'

persephone ~/tmp= telnet 80
Connected to
Escape character is '^]'.
HEAD /blog HTTP/1.0
HEAD /blog HTTP/1.0

HTTP/1.1 405 Method Not Allowed
Date: Thu, 06 Jun 2002 02:22:40 GMT
Server: Apache/1.3.12 (Unix) mod_ssl/2.6.6 OpenSSL/0.9.5a
Bobo-Exception-File: /usr/local/zope/2-5-1/lib/python/webdav/
Bobo-Exception-Line: 57
Bobo-Exception-Type: Method Not Allowed
Bobo-Exception-Value: bobo exception
Connection: close
Ms-Author-Via: DAV
X-Powered-By: Zope (, Python (
Content-Length: 2106
Content-Type: text/html

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Parsed-XML-Dev] Re: [Zope-dev] ParsedXML in Zclass methods loses permissions on Zope restart

2001-10-30 Thread Karl Anderson

Martijn Faassen [EMAIL PROTECTED] writes:

 Brad Clements wrote:
  I'm still casting around for a suggestion on where I can go to fix this.
  I have a ParsedXML object in the methods list of a ZClass Product.
  The Access Contents Information Permission Mapping always get's reset to blank 
  the ParsedXML object when Zope restarts.
  Is this a problem with ZClasses, or with ParsedXML? 
 Taking a look at ParsedXML, it looks like it just does a very simple
 permission definition, so it doesn't *look* like there's something
 wrong there.
 What it sounds like is that something that should be persistent isn't
 for some reason. Either an object attribute is changed without 
 the permission mechanism being informed of the change, or it's 
 for some reason an attribute on a nonpersistent is changed. This at least
 would explain why the permission mapping magically goes away..

I don't understand the situation very well, but Martijn's explanation
sounds familiar.  Remember, there's only one persistent Zope object in
a ParsedXML instance, and that's the document proxy object.  All of
the proxy subobjects are transient, and that includes proxy documents
that are navigated to from proxy subobjects.  I.E. if
is the ParsedXML instance, then setting Zope attributes on and
won't persist.  Same goes if you traverse to these nodes via the
web-exposed URLs of these methods.  Perhaps you're trying to persist
changes to Zope attributes on those nodes?

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: oodb philosophics ;) was: Re: [Zope-dev] Experiments withORMapping

2001-05-14 Thread Karl Anderson

Casey Duncan [EMAIL PROTECTED] writes:

 I am not arguing necessarily for SQL as a query language for the ZODB.
 Although it is an accepted standard, but not a perfect one by any means
 especially for OODBs. Its appeal lies mainly in the high level of
 community familiarity and the plethora of SQL software to borrow from.

Does anyone have an opinion on the possible usefulness of XPath,
XQuery, and other XML standards for this?  Someone suggested (on the
zope-xml wiki) that it would be nice to be able to drop in a cataloger
that supported a presumably standard and presumably well-known XML
query syntax, and which would work throughout the database because
Zope objects would support DOM.

This is all speculation, and I personally don't know much right now
about XML database interfaces and how finished or well-regarded they

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

[Zope-dev] ZCatalog catalogable types and nonproducts

2001-04-18 Thread Karl Anderson

I have a Python Product with several subobjects which are not products
in their own right.  I'm trying to make these subobjects selectable in
the Find Objects tab of a ZCatalog, so I can restrict the cataloged
objects to these types.

Whats the right way to do this?

Since they're not addable products, they don't go in the "Find objects
by type" selection by default.  Rooting through the source, I see that
this selection is populated by all_meta_types, and ObjectManager
creates that list by appending a _product_meta_types attr onto an
acquired value.

I can't create a _product_meta_types property on the ZCatalog.  I can
add a _products_meta_type dict to the ZCatalog via Python:

app.catalogtest.catalog._product_meta_types = ({'permission': 'Dummy', 'name': 
'Manageable Element', 'action':'Dummy'},)

I added the dummy permission and action becuase the contents view
doesn't display without them.

This seems to work, and these meta types aren't displayed in the add
list for the container view (becuase I don't have the 'Dummy' perm),
which is what I want.  But should I be telling the users of my product
to do this?

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Zope-dev] memory leak

2001-04-13 Thread Karl Anderson

Marco Nova [EMAIL PROTECTED] writes:

 I've replaced ParsedXML's ExpatBuilder with the pyXML package and I used the
 sax parser without modifing the code (except for the import), this is the
 refcounts results.
 Class  April 12, 2001 11:55 am  April 12, 2001 12:00
 xml.dom.NodeList.NodeList   2103  3678  +1575  
 xml.dom.Text.Text   1263  2208  +945  
 xml.dom.NamedNodeMap.NamedNodeMap507   885  +378  
 xml.dom.Element.Element  499   871  +372  
 xml.dom.Attr.Attr339   591  +252  
 xml.dom.DocumentType.DocumentType  710  +3  
 xml.dom.Document.Document  710  +3
 xml.dom.Document.Document increments by 1 each time an xml is procesed (I
 tried to add a del doc at the end but it's ineffective).
 So the problem is not ParsedXML but Zope itself or my bad methods.

Not necessarily - it's been mentioned that PyXML requires releasing
nodes after they're done, so I'd expect this if you replaced
ParsedXML's DOM with PyXML's without adding the release.

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Zope-dev] Memory problems with ParsedXML ?

2001-04-06 Thread Karl Anderson

Godefroid Chapelle [EMAIL PROTECTED] writes:

 Karl Anderson a crit :

  Another thing to keep in mind is that the ParsedXML *product* is a DOM
  tree and a management interface around it.  The management interface
  only gives you a convenient UI and a place to store a few bits that
  the DOM doesn't know about (content type, namespace usage, etc.).
  It's always more efficient to just use the DOM tree and related
  utilities (parsing, printing), although it is less convenient - you
  have to know a little bit more about what you're doing.  Do you need
  each of those 2500 product instances?  See the createDOMDocument
  method to create a DOM document without the management interface.
 I am trying to use createDOMDocument, calling it from an external method:
 from Products.ParsedXML import ParsedXML
 def accessXml(document):
 return ParsedXML.createDOMDocument(document)
 When using the document returned in a python script,  it seems that Zope
 security prevents me from accessing any DOM attributes.
 Am I doing something wrong or am I forced to do everything through external
 methods ?

The DOM objects without the ParsedXML management proxies don't have
any Zope security information associated with them, so you have to
provide it yourself.  This is one of the things that the management
proxies provide.

I didn't mean to sell the proxies short; I should have mentioned that
you have to know about providing the features youself that they
provide, including security, so this is an area for Zope experts.
I've added this to the README.  Furthermore, if you're just calling
methods on the Document node, without traversing to its subnodes,
you're really not paying much of a price; you're basically just
putting more function calls between you and the DOM object until you
access a DOM subobject.

I don't use the python scripts much, but they have very restricted
security, being web-edited and all.  So yes, one way to give access is
to use external methods.  Another way is to use a python product and
the security machinery available to you there.

The unproxied DOM objects are similar to any non-Zope aware python
class instances with regards to security; if you're unfamiliar with
it, I'd play around with some non-Zope aware dummy class instances,
with nested subinstances and attributes, until you know how to make
those objects and attributes publishable.

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Zope-dev] FTP interface being worked on?

2001-03-27 Thread Karl Anderson

"Chris McDonough" [EMAIL PROTECTED] writes:

   Are you thinking that we would build client-side tools to recognize an
   representation of a subpart of a site?
  Client-side tools, no.  I'm thinking that exporting to XML would allow
  existing tools to recognize and manipulate a subpart of a site.
 Which ones?

Parsers, transformers, etc..  I don't know of any end-user tools (are
there any?), I'm talking about building blocks.

  Is there a particular set of tools or editing paradigms that we have
  in mind when we say that a non-XML representation is suited for client
  side tools?
 Yes.. anything that works well with diff and CVS and is recognizable by a
 human.  PythonScripts should be serialized to something that looks like a
 regular Python script.  Images should look like images, etc.  The directory
 tree generated should look as much as possible like a normal filesystem.

Sure, I would use that, too, probably more than an XML representation,
I'm referring to the ability to represent in other formats because of
their usefulness and scalability.

For example, if I were doing lots of random edits on a subtree of
DTML files and Python scripts, I'd want them to look like a filesystem
tree full of text files and I'd use emacs, grep, find, etc. on them.
If I wanted to be able to use an automated tool to, say, update the
spam and eggs fields of everyting in a subtree every friday, an XML
parser and/or DOM tree and/or transform working on a single XML
representation of the subtree checked out to the local filesystem
seems like the easiest entry point.

I'm probably straying from the proposal here.

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Zope-dev] FTP interface being worked on?

2001-03-26 Thread Karl Anderson

Chris McDonough [EMAIL PROTECTED] writes:

 I think the only good reasons we have right now for having
 filesystem-compatible serialization are to make Zope content editable via
 common tools in a way that makes sense to people not used to (or comfortable
 with) the object database, and to give people a plausible way to put a Zope
 site under source control.
 Are you thinking that we would build client-side tools to recognize an XML
 representation of a subpart of a site?

Client-side tools, no.  I'm thinking that exporting to XML would allow
existing tools to recognize and manipulate a subpart of a site.

I'm basically agreeing with you - "common tools" for manipulating text
sounds like parsers to me.

I'm not sure why XML is considered less readable than an unknown
format for Zope object serialization; I guess I haven't seen what's
being considered.  But it seems to me that for humans, XML might lose
by a little, but for any type of mediated or batch processing, XML
wins by a lot.  Parsers are standard enough that their APIs are easy
to learn if you've played with them before.

Random human-editable text formats sounds like StructuredText; when I
think of StructuredText I think "simple DOM serialization".

Is there a particular set of tools or editing paradigms that we have
in mind when we say that a non-XML representation is suited for client
side tools?

This is the way that Manila seems to be doing it:

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

[Zope-dev] ParsedXML 1.0 released

2001-03-23 Thread Karl Anderson

We've just released ParsedXML 1.0, the first stable release.  It can be
downloaded from at

Parsed XML allows you to use XML objects in the Zope environment. You
can create XML documents in Zope and leverage Zope to format, query,
and manipulate XML.

Parsed XML consists of a DOM storage, a builder that uses PyExpat to
parse XML into the DOM, and a management proxy product that provides
Zope management features for a DOM tree.

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Zope-dev] FTP interface being worked on?

2001-03-22 Thread Karl Anderson

"Chris McDonough" [EMAIL PROTECTED] writes:

 I don't think it's reasonable or wise to impose any "master
 structure" for filesystem serialization of bodies of
 objects.  Each instance (or perhaps each class) should
 define how best to serialize itself to disk.
 Representations between classes are likely to be radically
 different.  A place for standardization is in the
 "properties" file(s) which accompany each object rep... this
 is likely to be XML or another structured variant.

Is there a motivation for using serialization to provide an editable
"god's eye view" of all or part of a Zope site?

That is, provide a two-way XML serialization at any stage of the
managed hierarchy of a site?  That way, XML tools could be used as
stream editors at any level, to the extent that the serialization is

So, for example, we'd have ways to not just alter a template or
content that gets templatized, but the containers that organize them,
and related metadata or content that isn't as near.

This links well with the standardization that you mention - objects
can have arbitrary serialization formats, but if certain attributes
that we're interested in are recognizable, those attributes could be
edited on a containerwide level.  So, if we had arbitrary objects that
we wanted to have an effect on content or display, the same XML tools
could be used to manage them all, limited only by the ability of those
tools to slog through a level of the hierarchy.

This is kind of stream-of-consciousness talk, and might be more silly
than realistic; certainly, if the objects didn't guard how they could
be updated, a misconfigured tool could waste them without warning.

There's a similar project called psilib that was discussed in

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

[Zope-dev] acquisition interfering with []?

2001-02-21 Thread Karl Anderson

Hopefully this is enough context for this question.

When I subclass Acquisition.Implicit or .Explicit, I find myself
losing the foo["item"] and foo[slice:slice] syntax.  __getslice__ and
__getitem__ aren't affected.

When I don't subclass Acquisition, the [] syntax works fine.

I can get them back by explcitly redefining them __getslice__ and
__getitem__ (even though those methods never had any problems), but is
there a better way around this?

I'm doing a lot of other craaazzy stuff with these classes, I may be
causing this problem myself elsewhere.  I'm not defining
foo.__getslice__, rather I'm defining __getattr__ in a superclass, and
returning the result of a call on a proxied object for attr names that
match a list.

But this does look like the [] syntax is running into the fact that
you need to call inheritedAttribute rather than the superclass method
directly, or something.

Here's what I'm doing to make this work:

class ManageableNodeList(ManageableWrapper, DOMProxy.NodeListProxy, 

def __getslice__(self, i, j):
return self.__getattr__("__getslice__")(i,j)

def __getitem__(self, i):
return self.__getattr__("__getitem__")(i)

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

[Zope-dev] the Parsed XML project

2000-11-16 Thread Karl Anderson

The Parsed XML project is now public and ready for input.  Parsed XML
is the successor to XMLDocument.  We have a wiki at
[EMAIL PROTECTED] is also a good place for discussion at this time.

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Zope-dev] mailing list 'noise'

2000-10-02 Thread Karl Anderson

Ken Manheimer [EMAIL PROTECTED] writes:

 In fact, i'm *really* interested in "turning answers into stories".  That
 is, not just getting answers to questions, but preserving them in a way
 that makes them easy to find when they're next needed - organizing them so
 they collectively serve to describe the topic they're about, to make the
 topic, as a whole, discoverable.  While i think there are many modes of
 discussion that can serve this purpose, depending on the application and
 collaborative context, i think mailling list discussion threads need more.  
 They're a step towards that building-together, but fail to organize beyond
 that - so the answers they provide are fragmentary glimpses into the topic
 at hand.

 One key way wiki documents help bind the fragments is by providing more
 "fixed points" around which discussions can range.  The fixed points
 are not immutable - they can evolve - but they're easy to point at, and
 provide a definite manifestation of the topic at some stage of its life.

That's a good point.  Mailing list threads are great if you're around
when they come up.  After that, searching is doable, but not easy enough -
witness that the same threads tend to come up for any list.

One good quality of lists is that if you can tell your reader to
organize threads, then the discussion is broken up and shoved in your
face for you piece by piece.  In a wiki, on the other hand, you have
to return to the same page, find where you were last, and actively
look for changes.

 The proposals site is one example where definite subjects are
 at hand.  As someone behind the WikiNG proposal, who *wants* to be able to
 reap the suggestions and details from a discussion, but knows i won't have
 the time for a while to actually concentrate attention on it, i dread
 having to collect all the messages, for later review for harvesting.  
 Furthermore, messages on the mailling list tend to diverge more and
 farther from the topic, than they do when placed within the wiki.
 What i'd like the best, for now, is to have discussion happen on the
 mailling list *when someone wants to feel something out*, *and then
 they're responsible for summarizing in the wiki discussion page, if they
 have anything to harvest*.

Note that we keep on acknowledging that the different fora are better
in different ways, and that what we keep on wanting is the right way
to communicate and propagate between the fora.

Here, you want it to be easy to pop a thread into a wiki.  Something
like a thread-to-wiki feature would be nice - tell the wiki "flatten
this thread  make a page for it", then edit it by hand.  But it's
still a one-way link, really, the best you can do is post a final
message to the thread - 'see the wiki for further discussion'.  Which
isn't that bad, really.

What I really want is for the different fora to just be interfaces on
the information.  I'm not sure how, it isn't that realistic, I can't
think of an implementation without it getting overfeatured.  Something
like wiki edits being reflected in the mailing list archive.  Nah.

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Zope-dev] mailing list 'noise'

2000-09-28 Thread Karl Anderson

Ken Manheimer [EMAIL PROTECTED] writes:

 On Thu, 28 Sep 2000, Chris Withers wrote:
  Toby Dickenson wrote:
   I dont see this as a problem: You only create a new list when the
   traffic for that proposal gets too great for zope-dev. Threading is
   good enough before that point.
  Yes, but zope-dev has a relatively high traffic load... Why should you
  have to put up with all that 'noise' if you're only interested in posts
  for your comparatively small discussion?
 Yeah - maillists flow by, and not everyone can follow all the traffic all
 the time!! The cool thing about "content-based" mailling lists, where
 people can subscribe to notifications about changes in subthreads, is that
 you just subscribe to the part of the discussion that has your interests!!

I haven't understood this gripe ever since I started reading mail with
Gnus.  Before anyone groans, I'm not sure that Gnus is ready for
general use by anyone who doesn't want to learn elisp - but surely
there's anther reader with these features?

The point that I'm trying to make is that a mailing list has all the
strucure needed to keep abreast of an important thread.  I don't think
it's perfect when you can't afford to miss a single important article,
but it works great for general lists.

Gnus treats mail  news as the same, and allows you to score posters,
threads, messages, etc. both manually and adaptively.  Threads can be
presented by order of their score.  Adaptive scoring is what really
makes it work.  The normal reading commands - read article, kill
thread, save article, catchup (mark unread articles as read) can
affect the scores.  So, reading articles in a thread tends to make it
float to the top, and posters who contribute to well-read threads
elevate future threads that they contribute to.  The inverse for
killed threads, less so for caught up threads.

I can read comp.lang.python when it has 3000 unread articles, by
skimming 100-500 articles, reading some, catching up or killing the
rest, and saving other 2500 for later.  I draw in what I'm interested
in from the mass of unread articles first, each time, and the stuff
that I rarely get around to is the stuff that I don't miss.  Same with
rec.bicycles.soc - when the article count gets to 200-300, I read the
2-10 articles that I'm probably interested in, and miss the 95% which
is almost always noise.

That's why I resist moving to other fora.  I've never seen one that
lets me use better tools.

Okay, my download finished while I wrote this, back to work :)

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )

Re: [Zope-dev] Please HELP: Nested ZClass, Nested instance

2000-05-22 Thread Karl Anderson

"Park, Karl" [EMAIL PROTECTED] writes:

 I created a ZClass in the Product folder. And anther ZClass under the
 ZClass(thus nested class)

 To do it, I created a DTML Method which accepts user's input and then, if
 needed, it have to create an object.

 First part is to create the parent object, which works just fine. 
 But the sencond part to create the child object doesn't seem to work. 
 If I take the easy way, that is, creating othere ZClass on the top of the
 products folder of Control Panel, I can do it. 
 But I don't think it is a good desing from the perspective of OOP

I had some hassles with this recently.  There's a HOWTO at:

Can you create the child zclass from the manage screen inside the
parent zclass?

Does the parent zclass inherit from ObjectManager before any other

Karl Anderson  [EMAIL PROTECTED]

Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists - )