Re: [Zope-dev] Effect of DB Writes on Performance
Writes are slower than reads, but the real bottleneck is likely to be your application, not the ZODB. Commonly ZODB writes take place within a busy transaction, including catalog updates, transformations, etc. Brian R Brinegar wrote: Hello, What effect do ZODB Writes have on Performance? We use Zope in an environment where users are constantly updating and maintaining content within the ZODB. Do these writes to the Database slow down overall performance? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Chris Withers wrote: Shane Hathaway wrote: performance. There needs to be a way for applications that modify the database to tell Zope about the modification, so Zope can reset its caches. But, IIRC, the last time this was discussed on a mailing list you had some cool ideas to sovle the problem, right? Yes, but I want to hear other people's ideas first. What do you think? Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Can't return / publish object of a dictionary?
Sounds like an acquisition problem to me. Are the objects wrapped? Try def _getOb(..): ... return someobject.__of__(self) Lars Heber wrote: Hi zopers, my class has a list with several objects in it. When calling my self written _getOb() method, I want to return one of these objects. But I get an Unauthorized... When I put an object of the same type into a normal class attribute (self.dummyObject), return of that object from _getOb() works perfectly. What am I doing wrong? Thanks a lot! ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Can't return / publish object of a dictionary?
Lars, I believe that for security validation to work the object you return has to be acquisition wrapped. Try something like (untested): def _getOb(self, id): return self._secretList[id].__of__(self) HTH, Stefan --On Donnerstag, 16. Jänner 2003 16:51 +0100 Lars Heber [EMAIL PROTECTED] wrote: Hi zopers, my class has a list with several objects in it. When calling my self written _getOb() method, I want to return one of these objects. But I get an Unauthorized... When I put an object of the same type into a normal class attribute (self.dummyObject), return of that object from _getOb() works perfectly. What am I doing wrong? Thanks a lot! -- Those who write software only for pay should go hurt some other field. /Erik Naggum/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
seb bacon wrote: Shane Hathaway wrote: Chris Withers wrote: Shane Hathaway wrote: performance. There needs to be a way for applications that modify the database to tell Zope about the modification, so Zope can reset its caches. But, IIRC, the last time this was discussed on a mailing list you had some cool ideas to sovle the problem, right? Yes, but I want to hear other people's ideas first. What do you think? Isn't this a different problem for each kind of storage, e.g. MD5 hash for ext2, transaction ID for foo...? Or are you referring to a different aspect of the problem? I'm thinking about real-time updates. When the underlying data changes, you'd like Zope to see the change immediately. If indefinite delays are OK, then AdaptableStorage already does enough: it raises a ConflictError if you try to write changes based on old data. The idea I like the most for relational databases is to ask the RDBMS what the ID of the last transaction was. If Zope missed a transaction, it should flush all caches. This will work if the database is infrequently changed by external applications, or if Zope is accessed infrequently. If external applications make a lot of changes, however, and Zope needs good performance at the same time, then both Zope and the external applications need to update a per-object transaction ID. Then, at the beginning of transactions, Zope would invalidate only the recently updated objects. Hmm, perhaps smarter RDBMSs could make it easy to keep transaction IDs updated using triggers. (This solution could also replace both ZEO and ZRS, BTW. ;-) ) On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. For that case, I'm thinking that requiring external apps to touch a special file somewhere might be the right thing. At the beginning of each transaction, if Zope sees a change to the file, it flushes its cache. While reading the referenced thread on the subject, I found your description of the product design here: http://lists.zope.org/pipermail/zope-dev/2002-August/016981.html Could this go in the docs/ directory of the product? The design, while very clean, doesn't lend itself to immediate understanding on a cursory view of the source... I'm hoping to present a complete tutorial on AdaptableStorage at PyCon DC 2003. I'll integrate those notes. Thanks for pointing them out--I'd forgotten about them. The names are changed somewhat, but the basic design is the same. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Shane Hathaway wrote: Chris Withers wrote: Shane Hathaway wrote: performance. There needs to be a way for applications that modify the database to tell Zope about the modification, so Zope can reset its caches. But, IIRC, the last time this was discussed on a mailing list you had some cool ideas to sovle the problem, right? Yes, but I want to hear other people's ideas first. What do you think? Isn't this a different problem for each kind of storage, e.g. MD5 hash for ext2, transaction ID for foo...? Or are you referring to a different aspect of the problem? While reading the referenced thread on the subject, I found your description of the product design here: http://lists.zope.org/pipermail/zope-dev/2002-August/016981.html Could this go in the docs/ directory of the product? The design, while very clean, doesn't lend itself to immediate understanding on a cursory view of the source... seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] question: forcing https for authentication
Oliver Bleutgen wrote: One thing that bothers me is that I cannot reliably (as in in a generic way which always works) prevent users from sending their authentication unencrypted. Well its true you can't prevent users from compromising their credentials, but you can prevent users from coming in the wrong door, as it were. I'm not clear on which one you really hope to accomplish, though from your proposed modifications it looks like the latter. Preventing users from compromising their creds can only be effectively done through education. For example, even if you used client certificates (which, afaik, are only used once an encrypted channel has been established) instead of basic auth there's nothing to stop a user from giving their cert and any requisite ancillary usage information (like the passphrase to decrypt it) to somebody else. The only ideas I have to tackle this without modifying zope itself are - customize all pages which need authentication to check for https://; in one of the relevant REQUEST attributes and do a redirect if not. - use apache with some magic to trigger redirection if it encounters authentication headers in the request. - use apache with some rewrite magic trigger redirection when a substring like manage is found in the request. These alternatives are neither elegant, nor really secure. I would agree the first two aren't sane, the third however is as good and as secure as you're going to get. Techniques for accomplishing it have been discussed on the zope list before and are undoubtedly in the archives. The mods you've posted with respect to guarding the WWW-Authenticate header aren't likely to work, and certainly wouldn't be flexible enough to allow, for example, mandating SSL usage for all /manage* requests while allowing clear-text basic auth for other restricted areas. Apache rewrite rules OTOH are flexible enough to allow this. I'm unclear as to why you consider it to be insecure, care to elaborate? -- Jamie Heilman http://audible.transient.net/~jamie/ Most people wouldn't know music if it came up and bit them on the ass. -Frank Zappa ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Shane Hathaway wrote: On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. FWIW, since I had the same problem some time ago (which could be solved in another way), I dug out an url, which might be of interest - probably you already know about it: FAM, used by the two major open source desktop envs: http://oss.sgi.com/projects/fam/ It may at least help to make the whole problem more os independend. They have a lot of related pointers on their homepage. Btw. windows (=nt IIRC) already has the capability to notify on directory alteration events, without polling. cheers, oliver ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Oliver Bleutgen wrote: Shane Hathaway wrote: On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. FWIW, since I had the same problem some time ago (which could be solved in another way), I dug out an url, which might be of interest - probably you already know about it: FAM, used by the two major open source desktop envs: http://oss.sgi.com/projects/fam/ It may at least help to make the whole problem more os independend. They have a lot of related pointers on their homepage. I've seen it before, but I don't think FAM is able to monitor an entire directory tree. It only monitors individual files. I'd really like to be wrong. :-) Btw. windows (=nt IIRC) already has the capability to notify on directory alteration events, without polling. Do you know what API? That would sure help. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Shane Hathaway wrote: Oliver Bleutgen wrote: Shane Hathaway wrote: On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. FWIW, since I had the same problem some time ago (which could be solved in another way), I dug out an url, which might be of interest - probably you already know about it: FAM, used by the two major open source desktop envs: http://oss.sgi.com/projects/fam/ It may at least help to make the whole problem more os independend. They have a lot of related pointers on their homepage. I've seen it before, but I don't think FAM is able to monitor an entire directory tree. It only monitors individual files. I'd really like to be wrong. :-) I think you are wrong, because the manpage (for IRIX) says otherwise. Additionally, it wouldn't be of much use for kde etc. if it only could monitor files. I think a filemanager would mainly be interested in directory changes (files added/deleted). Then there's also dnotify (also reference from the FAM site) - there's hope that the d isn't an acronym for file ;). I remember someting about recent 2.4.x versions having the prerequisites to use that. Btw. windows (=nt IIRC) already has the capability to notify on directory alteration events, without polling. Do you know what API? That would sure help. I don't have any expirience on win32, but just searched google. There's Win32::ChangeNotify for perl, described here http://www.xav.com/perl/site/lib/Win32/ChangeNotify.html and this seems to use ReadDirectoryChangesW, decribed here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/readdirectorychangesw.asp cheers, oliver ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Rendering of objects in DTML
For the record, by following the same pattern it should be possible to implement a class that responds to dtml-in content too, by implementing __getitem__ On Wed, 2003-01-15 at 06:00, Bjorn Stabell wrote: Thanks Leo, this is very helpful. It should probably be in a HOWTO somewhere. :) -Original Message- From: Leonardo Rochael Almeida [mailto:[EMAIL PROTECTED]] On Tue, 2003-01-14 at 01:01, Bjorn Stabell wrote: Bjorn Stabell wrote: I would like dtml-var content dtml-with content... dtml-if content to call different functions. The first renders the object, the second returns a mapping of the content's attributes/properties, the third checks for trueness of the content. Rendering a content could be an expensive operation and I don't want to do it when doing dtml-with and dtml-if. you can do that by having your object be non-callable (or returning self on __call__) and implementing __str__ where you can have access to the REQUEST thru self.REQUEST. dtml-with content works as expected because you're getting the object itself. To make dtml-if content work you should implement __len__ (if your object implements sequence or mapping, which it probably doesn't) or __nonzero__. Read the python docs for details on the implementation of these methods. Cheers, Leo -- Ideas don't stay in some minds very long because they don't like solitary confinement. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) -- Ideas don't stay in some minds very long because they don't like solitary confinement. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Oliver Bleutgen wrote: Shane Hathaway wrote: I've seen it before, but I don't think FAM is able to monitor an entire directory tree. It only monitors individual files. I'd really like to be wrong. :-) I think you are wrong, because the manpage (for IRIX) says otherwise. Additionally, it wouldn't be of much use for kde etc. if it only could monitor files. I think a filemanager would mainly be interested in directory changes (files added/deleted). I checked again. It is still limited to 1000 files or directories at a time. It's not meant for entire subtrees, it has to run as root, and it requires portmap, making it less attractive. Then there's also dnotify (also reference from the FAM site) - there's hope that the d isn't an acronym for file ;). I remember someting about recent 2.4.x versions having the prerequisites to use that. Now this one is quite interesting. It requires at least kernel 2.4.19, so I guess I'm at the edge of kernel development. (!) It just might do the trick, and maybe even better than I hoped. Thanks. I don't have any expirience on win32, but just searched google. There's Win32::ChangeNotify for perl, described here http://www.xav.com/perl/site/lib/Win32/ChangeNotify.html and this seems to use ReadDirectoryChangesW, decribed here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/readdirectorychangesw.asp Yep, that's it. With some unicode update, it even works on Win95. But I'm going to leave the Windows work for someone else. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Effect of DB Writes on Performance
Brian R Brinegar wrote at 2003-1-15 16:37 -0500: What effect do ZODB Writes have on Performance? We use Zope in an environment where users are constantly updating and maintaining content within the ZODB. Do these writes to the Database slow down overall performance? When you are using FileStorage (below ZODB) then each write will append the modified object as a serialized bytestream to the file. With a high write frequency, your storage file may rapidly grow and may need to be packed frequently. Moreover, the ZODB uses an optimistic conflict resolution policy. Such a policy is only efficient when the conflict probability is low. With higher write frequency, the conflict probability tends to increase... Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] question: forcing https for authentication
Oliver Bleutgen wrote at 2003-1-16 15:42 +0100: One thing that bothers me is that I cannot reliably (as in in a generic way which always works) prevent users from sending their authentication unencrypted. The only ideas I have to tackle this without modifying zope itself are - customize all pages which need authentication to check for https://; in one of the relevant REQUEST attributes and do a redirect if not. - use apache with some magic to trigger redirection if it encounters authentication headers in the request. - use apache with some rewrite magic trigger redirection when a substring like manage is found in the request. You might use a SiteAccess access rule. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] question: forcing https for authentication
Jamie Heilman wrote: Well its true you can't prevent users from compromising their credentials, but you can prevent users from coming in the wrong door, as it were. I'm not clear on which one you really hope to accomplish, though from your proposed modifications it looks like the latter. Preventing users from compromising their creds can only be effectively done through education. For example, even if you used client certificates (which, afaik, are only used once an encrypted channel has been established) instead of basic auth there's nothing to stop a user from giving their cert and any requisite ancillary usage information (like the passphrase to decrypt it) to somebody else. Granted. I indeed want prevent users from coming in the wrong door. And it would really help if there wasn't a wrong door at all - therefore my question ;). The only ideas I have to tackle this without modifying zope itself are - customize all pages which need authentication to check for https://; in one of the relevant REQUEST attributes and do a redirect if not. - use apache with some magic to trigger redirection if it encounters authentication headers in the request. - use apache with some rewrite magic trigger redirection when a substring like manage is found in the request. These alternatives are neither elegant, nor really secure. I would agree the first two aren't sane, the third however is as good and as secure as you're going to get. Techniques for accomplishing it have been discussed on the zope list before and are undoubtedly in the archives. The mods you've posted with respect to guarding the WWW-Authenticate header aren't likely to work, and certainly wouldn't be flexible enough to allow, for example, mandating SSL usage for all /manage* requests while allowing clear-text basic auth for other restricted areas. Apache rewrite rules OTOH are flexible enough to allow this. I'm unclear as to why you consider it to be insecure, care to elaborate? The third alternative is indeed secure compared to the other two, in that it prevents plaintext credentials from going over the wire - as long as you remember to use always names like manage*. It's factually more insecure than my idea in that even ZMI's methods can be accessed with mangled names (e.g. http://server/manage/index_html). It's cumbersome because newly installed products always have to be checked for the method names. It's shaky, because if your filtering is too broad, better don't give the corporate management profile page an intuive id ;). And it's clearly worse performance wise for the typical use case, because you now get a load of rewrite stuff for any URI which has to be processed. All these complications and administrative burdens alone lead in effect to insecurity. The example code I posted was more or less to illustrate my intention, if I'd know where to implement such a hook, I'd probably try to make it more flexible. Make it dependend from a startup flag, and check for a magic attribute in the acquisition chain perhaps, which prevents the redirect. Also allow for redirects to a different host. cheers, oliver ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] question: forcing https for authentication
And it's clearly worse performance wise for the typical use case, because you now get a load of rewrite stuff for any URI which has to be processed. Using mod_ssl the following works pretty well: LocationMatch /manage SSLRequireSSL /LocationMatch No rewrite or mod_rewrite overhead. You simply get a 403 if you don't use https. Dunno if thats an option for you but its something to think about. All these complications and administrative burdens alone lead in effect to insecurity. I'll buy that, but I don't have a good way to fix it. The z object hierarchy just isn't as easy to secure as a filesystem. The example code I posted was more or less to illustrate my intention, if I'd know where to implement such a hook, I'd probably try to make it more flexible. Make it dependend from a startup flag, and check for a magic attribute in the acquisition chain perhaps, which prevents the redirect. Also allow for redirects to a different host. Well somebody mentioned Access Rules ... frankly I'm not sure what that buys you really, the problem seems to be its very difficult to classify what needs to be protected and what doesn't without trodding on somebody elses namespace or creating something overly fragile. -- Jamie Heilman http://audible.transient.net/~jamie/ I was in love once -- a Sinclair ZX-81. People said, No, Holly, she's not for you. She was cheap, she was stupid and she wouldn't load -- well, not for me, anyway. -Holly ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Can't return / publish object of a dictionary?
Thanks for the tips, it was the missing wrapper. I'm quite a newbie in Python programming, so please forgive me. I've got two other questions: 1. Say we have a Class A with an attribute myObjects which is just a list. Now, I'd like to write my own Class B(A) - it extends A. But, in Class B, myObjects has to be a method because it has to be rebuild everytime it is called. So, how can I do something like: myObjects = getMyObjects() ? I tried to do this in the class itself, didn't work. Also tried self.myObjects = self.getMyObjects() - result wasn't what I wanted, myObjects got (of course) the resulting list of getMyObjects(), but I need myObjects to be a reference to getMyObjects() How to do this? 2. Class A has another attribute, say data, which I want to control in Class B(A), i. e. everytime data is accessed (reading or writing), I want to intercept those actions. If it was just reading, I could use the strategy from 1., but I also want to control made changes to that attribute. Do I absolutely have to rewrite all the methods which access the wanted attribute, or is there another possibility with some kind of references, perhaps similar to software interrupts in DOS? Thanks for your patience! Lars -- Lars Heber T-Systems GEI GmbH Hausanschrift: Clausstrasse 3, 09126 Chemnitz Postanschrift: Clausstrasse 3, 09126 Chemnitz Telefon : (+49 371) 5359-271 Fax : (+49 371) 5359-133 E-Mail : [EMAIL PROTECTED] Internet: http://www.t-systems.de ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
