Re: [Zope-dev] App.Permission security hole
Jamie Heilman wrote:
Tooling through restructuring of my site I discovered a stupid
permissions problem. While App.Permission declares the 'Define
permission' perm it never gets initialized and thus
manage_addPermission{,Form} basically had weakened security. The
permission 'Access contents information' was still protecting the
method, but thats not adequate--that permission was never intented
(afaik) to represent write access to the zodb. By default, as the
'Access contents information' permission is granted to the Anonymous
user, anybody could fire off a request to
http://victimhost/Control_Panel/Products/x/manage_addPermission?id=footitle=bar
where 'x' is some installed product (hey, why not use the HelpSys vuln
to find one that fits your fancy!) to add a permission object to
Product x. Anonymous users being allowed to bloat the zodb at the
least, possibly other issues at the worst (I don't know, I don't use
the ZClass machinery).
Ok, I've looked into this. I don't think there is a serious hole. The
most people can do is add permission objects, but those permission
objects have no real effect on security until someone maps them to a
method. I'm going to check in a fix, but there's no need for a hotfix.
A quick refactor of App/Permission.py may be found at
http://audible.transient.net/zope/Permission.py which protects the
methods in question, however I have a hunch there may be more broken
here than this permission alone.
Anything called manage_* is automatically restricted to ['Manager'] by
default, as long as the class gets initialized. The problem, as you
said, was that the class was never initialized.
Shane
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] manage_addZClass* permission question
Jamie Heilman wrote: I can't fathom the ZClass code. Can somebody tell me if manage_addZClass, manage_addZClassForm, and manage_subclassableClassNames are supposed to be protected by the 'Add Zope Class' permission, or if the code in ZClasses/__init__.py is pure fluf? That permission never shows up in any folder's security settings that I can see. VerboseSecurity has this to say about manage_addZClassForm: Unauthorized: Your user account does not have the required permission. Access to 'manage_addZClassForm' of (Product instance at 89189e0) denied. Your user account, meh, exists at /acl_users. Access requires one of the following roles: ['Manager']. Your roles in this context are ['Authenticated']. So it doesn't look like there is a named permission associated with those methods. I have to wonder if thats intentional. It is. Older Zope code uses the manage_ prefix to require the Manager role by default. Needless to say, that strategy did not cope well with later enhancements to Zope. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] The Return of the Son of Session Errors, redux
On Wed, 2003-05-28 at 04:54, Jean Jordaan wrote: Hi all We're being chowed heavily by the KeyError discussed from the 13th to the 18th March in this thread: http://mail.zope.org/pipermail/zope-dev/2003-March/019081.html Has there been any definite resolution of that issue? We're running Zope 2.6.1, has the issue gone away on Zope 2.6.2? I just checked in a change that disables the low conflict connection feature of temporary storage. This fixes the KeyError issue out of 'get', but others have reported a problem for which the symptom is a a KeyError out of the storage's 'load' method. I have not been able to reproduce this problem. In that thread a patch to TemporaryFolder.py was suggested to deal with ReadConflictErrors, but we don't get those. The suggestions regarding the KeyErrors was: 1) to move to a different storage, such as BerkeleyStorage or DirectoryStorage, for the session data, or 2) to touch the objects that are likely to change early in the transaction. (Toby Dickenson). Our errors are triggered in the bowels of XMLWidgets, so I don't think 1) is a simple solution, and we don't want to do 2) if we can avoid it. Have specific alternative session storages been found to be free of this issue? You can give it a shot. The problem definitely seems to be related to the TemporaryStorage class, which won't be used if you use a different storage. The initial report has been logged as a 'critical' error as http://collector.zope.org/Zope/848 but it's still Pending. I have commented on the issue. Thanks! - C -- Jean Jordaan http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] manage_addZClass* permission question
Shane Hathaway wrote: It is. Older Zope code uses the manage_ prefix to require the Manager role by default. Needless to say, that strategy did not cope well with later enhancements to Zope. OK. So what about the stuff in ZClasses/__init__.py, pure fluf? After yesterdays App.Permission bug I did a quick grep over the Zope source to make a big list of all used Permissions, and the 'Add Zope Class' in __init__.py is the only one I can't account for. (The one in Draft.py had me confused for a few minutes too before I found out that Drafts are completely disabled at the moment.) -- Jamie Heilman http://audible.transient.net/~jamie/ We must be born with an intuition of mortality. Before we know the words for it, before we know there are words, out we come bloodied and squalling with the knowledge that for all the compasses in the world, there's only one direction, and time is its only measure. -Rosencrantz ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] The Return of the Son of Session Errors, redux
Jean Jordaan wrote at 2003-5-28 10:54 +0200: ... Session Errors ... Our errors are triggered in the bowels of XMLWidgets What have session errors to do with XMLWidgets? Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] WebDAV File Descriptor Leak
Andreas Jung wrote at 2003-5-27 18:11 +0200: Can you reproduce the behaviour with 2.6.2? We have the same problem with 2.6.1 without WebDAV. Thus, the problem is probably not WebDAV related (but rather related to file uploading). Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZEO install/runtime issues
Jeremy Hylton wrote at 2003-5-27 14:38 -0400: ... The problem with sharing software is that the ZEO server can load arbitrary modules when it attempts to perform conflict resolution. If there is a conflict for an instance of class A.B.C, then ZEO will load A.B.C and see if it has an _p_resolveConflict() method. If the modules A or B have any side-effects at import time, then those side-effects will occur in the ZEO server. I've seen this method lookup cause all of CMF to get imported and try to initialize itself. This ended up brining down the ZEO server. That sounds bad... Products may well have classes which require conflict resolution (see e.g. QueueCatalog). It would be very tedious to let ZEO see only the classes, keep them importable for ZEO but remove everything else. Can we find a better solution, maybe, some kind of configuration which lists are classes which may have conflict resolution? Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] [CRITICAL] Conflict Errors, Transactions, Retries, Oh My....
I need to know more about Conflict Errors. We're running into a lot of them lately, it seems, on production Zope 2.6.1 sites (running on FreeBSD). The primary culprit seems to be Temporary Storage/Sessions. Something that has happened, and is causing a small amount of alarm, is that a large method that interfaces to external non-transactional systems seems to (on occasion) send their information off to that external system twice, but there's only one matching set of Zope data. As the two writes to the non-transactional system are very close to each other and contain nearly identical data (except for one bit that gets regenerated in the method), and there are conflict INFO reports in the Event Log from around the same time, I'm assuming that a conflict error is happening somewhere in this method and causing the transaction to be retried (if I'm understanding how Conflict Errors work). Zope and the relational databases seem to do things fine with rolling back the data, but the non-transactional systems now have duplicate data that they **absolutely should not have**. This doesn't happen often, but (as stated), this is a critical operation that needs to be better protected. All other exceptions and bits and pieces in the block of code in question has been tested thoroughly and we have not had any other problems that cause erroneous writes. Is there a way I can protect against Conflict Error retries as well? Is there some sort of Try/Except or Try/Finally I can wrap around the code that won't interfere with the ZODB? Is there any other sort of best-practice here that could help me (and others) who might unknowingly trigger this problem? I know there are some fixes likely to be in Zope 2.6.2 that may help with the situation, but I'd like to put extra protections around this code regardless of what may be coming in the future. Thanks in advance, Jeffrey P Shell [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [CRITICAL] Conflict Errors, Transactions, Retries,Oh My....
On Wed, 2003-05-28 at 21:33, Jeffrey P Shell wrote: Something that has happened, and is causing a small amount of alarm, is that a large method that interfaces to external non-transactional systems seems to (on occasion) send their information off to that external system twice, but there's only one matching set of Zope data. As the two writes to the non-transactional system are very close to each other and contain nearly identical data (except for one bit that gets regenerated in the method), and there are conflict INFO reports in the Event Log from around the same time, I'm assuming that a conflict error is happening somewhere in this method and causing the transaction to be retried (if I'm understanding how Conflict Errors work). Zope and the relational databases seem to do things fine with rolling back the data, but the non-transactional systems now have duplicate data that they **absolutely should not have**. Within Zope, when a conflict error is raised, ZPublisher catches the exception and retries the request up to 3 times. This is why sometimes, for example, you'll see double email notifications from Wiki subscriptions on zope.org. This doesn't happen often, but (as stated), this is a critical operation that needs to be better protected. All other exceptions and bits and pieces in the block of code in question has been tested thoroughly and we have not had any other problems that cause erroneous writes. Is there a way I can protect against Conflict Error retries as well? Is there some sort of Try/Except or Try/Finally I can wrap around the code that won't interfere with the ZODB? Is there any other sort of best-practice here that could help me (and others) who might unknowingly trigger this problem? Not infallibly. You can really never know where a ConflictError will might be raised. Any concurrent access to a persistent object is a possible candidate. I know there are some fixes likely to be in Zope 2.6.2 that may help with the situation, but I'd like to put extra protections around this code regardless of what may be coming in the future. It will only get worse with 2.6.2: the number of conflict errors cause by the sessioning machinery in 2.6.2 is going to go up as compared to 2.6.1 and below. This is because the strategy used to reduce the number of conflict errors used currently causes data desynchronization problems. Some folks have created products that mimic transactional semantics (like Jens' MailDropHost) to avoid this kind of problem. You might want to try the same... - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZEO install/runtime issues
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wednesday, May 28, 2003, at 04:38 AM, Jeremy Hylton wrote: [Please followup to zodb-dev.] You made some changes to the mkzeoinst.py script in April. I was busy then, and I've just had a chance to look at the changes now. I'd like to discuss some of the changes, and I'm including a wider discussion list to make sure we include anyone else who is interested. A number of the changes are Zope specific. (For example, you can't even run mkzeoinst.py without having a directory named Zope hanging off of sys.path.) ZEO and ZODB are intended for use separately from the rest of Zope, so we need to find a way to factor this out into a generic configuration and a Zope-specific configuration. Go for it :) [I'm in hard-core product development mode for a few months, so apart from critical Zope bugfixes along the way I'll not really be much use, sorry ... even Roundup is taking the back seat for a while ;)] Perhaps Zope's mkzeoinstance should have all that Zope-specific stuff, and only hook into the mkzeoinst module for some of the generic functions. there may even be some more potential for sharing of code between mkzeoinstance and mkzopeinstance. The other question I have is about the organization of software into a Zope home and an instance home. I'm not sure what the history of this arrangement is, but I recommend that people do not configure their ZEO servers to share software with their Zope app servers. It can cause fairly severe problems! I was completely unaware of this, and have always run ZEO servers with the full complement of Products. I have no immediate suggestion for a solution to this problem. The big issue is really how are (potentially dumb, point-n-click) users to know that they need to install product X in their ZEO server but not product Y? Dieter's solution of some configuration variable controlling this sounds like a really good idea, if possible. Richard -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.1 (Darwin) iD8DBQE+1ZI4rGisBEHG6TARApA1AJ9V5Vy/FD8Dx7Nyp2lcXhTgDuAokwCeJqLJ gbubTTKmtV/Heyush75rW44= =qVyG -END PGP SIGNATURE- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] [CRITICAL] Conflict Errors, Transactions, Retries, Oh My....
On Wednesday, May 28, 2003, at 10:19 PM, Chris McDonough wrote:
This doesn't happen often, but (as stated), this is a critical
operation that needs to be better protected. All other exceptions and
bits and pieces in the block of code in question has been tested
thoroughly and we have not had any other problems that cause erroneous
writes. Is there a way I can protect against Conflict Error retries
as
well? Is there some sort of Try/Except or Try/Finally I can wrap
around the code that won't interfere with the ZODB? Is there any
other
sort of best-practice here that could help me (and others) who might
unknowingly trigger this problem?
Not infallibly. You can really never know where a ConflictError will
might be raised. Any concurrent access to a persistent object is a
possible candidate.
Thanks for the information. Is it safe at all to try to catch a
ConflictError during the critical part of the code, log some
information, and then reraise the error to let the system do what it
needs?
I guess you're right though - it's hard to know when it will occur.
In the production system, in this particular method, there are only two
known persistent object interactions. At the end of the entire method,
after a notification email has been sent, I have something like:
session['pieces'] = {}
(session['pieces'] was a dictionary of {item_id:integer} bits. It
never gets large for an individual user). I think that the one recent
case of desync'd data happened when we got to this point. Since it's
at the very end of the script (no more writes are expected beyond this
point), I imagine that a get_transaction().commit() might be OK to
precede this statement, just so that even if any conflicts happen when
trying to write back to the session, we at least have synchronized data
between the two systems. Although, prior to this, there are a few
reads of this session data. Might it be safer to do something like
this at the top of the method?:
pieces = session['pieces'].copy()
I apologize if this post is making little sense (or stupid sense) -
dealing with threads, locks, conflicts, etc, has been the part of Zope
I've understood the least. I like that for the most part I don't have
to think about it, but I don't know where to go for [fairly] current
documentation on how to deal with it for those rare times I do.
The other persistent data write occurs earlier in the method, an object
that generates serial numbers based off of some simple data in a
PersistentMapping gets updated. I think that PersistentMapping has
become fairly large by now. It maps the item_id referenced above to a
regular dictionary containing three key/value pairs each. I make sure
to follow the rules of persistence when dealing with these
dictionaries-with-a-PersistentMapping, but I'm guessing that an OOBTree
might be better instead. I still don't understand the potential
pitfalls of Zope/ZODB BTrees (I keep reading about 'bucket splits'
causing conflicts, and I don't know if that would be better or worse
than any pitfalls a PersistentMapping gives).
Finally, the system in question has a few (three? four?) public Zope
sites using the same session storage. Is there any documentation,
notes, etc, about fine tuning the default session storage set up to
handle large sites (or groups of sites) with less conflicts?
Thanks again for the help. I'll take a look at MailDropHost. Maybe
I'll have to wrap another gateway around the gateway to the external
system to try to catch these conflict situations. Fortunately, the
critical area only occurs once in the current copy of the code.
Hopefully that will make it easier to protect.
Thanks again,
Jeffrey
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] The Return of the Son of Session Errors, redux
Dieter Maurer wrote: What have session errors to do with XMLWidgets? XMLWidgets stores the state of the document that you're editing in the session. We don't want to hack and fork XMLWidgets, so we're not in a position to change the way it uses the session. -- Jean Jordaan http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] The Return of the Son of Session Errors, redux
Hi Chris I just checked in a change that disables the low conflict connection feature of temporary storage. We've patched, and so far it looks good. We've seen one KeyError, but that was after leaving the session to expire. Will 2.6.2 get the patch? Thanks! -- Jean Jordaan http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] The Return of the Son of Session Errors, redux
On Thu, 2003-05-29 at 07:56, Jean Jordaan wrote: Hi Chris I just checked in a change that disables the low conflict connection feature of temporary storage. We've patched, and so far it looks good. We've seen one KeyError, but that was after leaving the session to expire. Will 2.6.2 get the patch? Yes... hopefully I'll be able to track down the problem in TemporaryStorage, but probably not by 2.6.2. Thanks! - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
