[Zope-dev] [Problem] unordered entries in Z2.log
Apparently, entries in Zope's Z2.log are not chronologically (by causuality) ordered. In my concrete case, a form was POSTed that resulted in a redirected GET request. The entries in Z2.log, however, were in the order: GET ... 200 POST ... 302 This can be confusing (altough, I do not know whether it can be easily avoided). -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Bug in ZTUtil.Batch solved
Hi Zopers It seems that there is a bug in ZTUtil.Batch: when the batch should be the length of the batch and there are exactly as many orphans as there would fit on one page the length of the batch becomes size instead of size+orphans. Here are the numbers: no of batchesstartendorphanlengthsequence_length 1 112 3 1212 1 110 3 1013 WRONG * 2 110 3 1014 * end and length should be 13 in this case, 3 items are omitted length=12 length=13 length=14 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 10 10 10 11 11 | second batch 12 12 | 13 | 14 | So, there is a jump on the edge where there should be one batch of the length sequence_length == length+orphans. At that point 3 (in the above case) items disappear. I have looked at the code in ZTUtils.Batch.py and found the following solution (line 109): 94 def opt(start,end,size,orphan,sequence): 95 if size 1: 96 if start 0 and end 0 and end = start: 97 size=end+1-start 98 else: size=7 99 100 if start 0: 101 102 try: sequence[start-1] 103 except IndexError: start=len(sequence) 104 105 if end 0: 106 if end start: end=start 107 else: 108 end=start+size-1 109 #try: sequence[end+orphan-1] 110 try: sequence[end+orphan]# replace above with this 111 except IndexError: end=len(sequence) 112 ... Can somebody confirm that this solution is correct and modify it in the CVS, please? NB This has been run in Zope 2.7 with Python 2.3.3. thanks Andre -- -- The disclaimer that applies to e-mail from TNO Physics and Electronics Laboratory can be found on: http://www.tno.nl/disclaimer/email.html -- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Bug in ZTUtil.Batch solved
Please submit the patch (+ unittest) to the bug collector. Otherwise it might get lost. -aj --On Montag, 22. März 2004 13:09 Uhr +0100 AP Meyer [EMAIL PROTECTED] wrote: Hi Zopers It seems that there is a bug in ZTUtil.Batch: when the batch should be the length of the batch and there are exactly as many orphans as there would fit on one page the length of the batch becomes size instead of size+orphans. Here are the numbers: no of batchesstartendorphanlengthsequence_length 1 112 3 1212 1 110 3 1013 WRONG * 2 110 3 1014 * end and length should be 13 in this case, 3 items are omitted length=12 length=13 length=14 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 10 10 10 11 11 | second batch 12 12 | 13 | 14 | So, there is a jump on the edge where there should be one batch of the length sequence_length == length+orphans. At that point 3 (in the above case) items disappear. I have looked at the code in ZTUtils.Batch.py and found the following solution (line 109): 94 def opt(start,end,size,orphan,sequence): 95 if size 1: 96 if start 0 and end 0 and end = start: 97 size=end+1-start 98 else: size=7 99 100 if start 0: 101 102 try: sequence[start-1] 103 except IndexError: start=len(sequence) 104 105 if end 0: 106 if end start: end=start 107 else: 108 end=start+size-1 109 #try: sequence[end+orphan-1] 110 try: sequence[end+orphan]# replace above with this 111 except IndexError: end=len(sequence) 112 ... Can somebody confirm that this solution is correct and modify it in the CVS, please? NB This has been run in Zope 2.7 with Python 2.3.3. thanks Andre ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Bug in ZTUtil.Batch solved
Please post this in the Collector at http://zope.org/Collectors/Zope jens On Mar 22, 2004, at 7:09, AP Meyer wrote: Hi Zopers It seems that there is a bug in ZTUtil.Batch: when the batch should be the length of the batch and there are exactly as many orphans as there would fit on one page the length of the batch becomes size instead of size+orphans. Here are the numbers: no of batchesstartendorphanlengthsequence_length 1 112 3 1212 1 110 3 1013 WRONG * 2 110 3 1014 * end and length should be 13 in this case, 3 items are omitted length=12 length=13 length=14 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 10 10 10 11 11 | second batch 12 12 | 13 | 14 | So, there is a jump on the edge where there should be one batch of the length sequence_length == length+orphans. At that point 3 (in the above case) items disappear. I have looked at the code in ZTUtils.Batch.py and found the following solution (line 109): 94 def opt(start,end,size,orphan,sequence): 95 if size 1: 96 if start 0 and end 0 and end = start: 97 size=end+1-start 98 else: size=7 99 100 if start 0: 101 102 try: sequence[start-1] 103 except IndexError: start=len(sequence) 104 105 if end 0: 106 if end start: end=start 107 else: 108 end=start+size-1 109 #try: sequence[end+orphan-1] 110 try: sequence[end+orphan] # replace above with this 111 except IndexError: end=len(sequence) 112 ... Can somebody confirm that this solution is correct and modify it in the CVS, please? NB This has been run in Zope 2.7 with Python 2.3.3. thanks Andre -- --- --- The disclaimer that applies to e-mail from TNO Physics and Electronics Laboratory can be found on: http://www.tno.nl/disclaimer/email.html --- --- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) smime.p7s Description: S/MIME cryptographic signature ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Parameter referencing bug in Python Scripts?
Hmmm,
There seems to be a bug in the way parameters are referenced in python
scripts. I have a simple script that has a list as a default parameter.
The script then appends something to this list and returns the list.
Each time I call the script the list gets to be one element longer. Here
is an example script:
## Script (Python) pyFix
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=a=['a']
##title=
##
a.append('b')
return a
The output from the script is as follows:
1st call: ['a','b']
2nd call: ['a','b','b']
3rd call: ['a','b','b','b']
4th call: ['a','b','b','b','b']
etc.
It seems that I'm getting a reference to the default parameter which I
then modify for future calls. Currently I'm getting around this by
having list parameters default to none and then setting the defaults
within the script.
I have some security concerns since I can change the default parameters
for other users on the server. Or so it seems.
Suggestions?
Brian Brinegar
Engineering Computer Network
Purdue University
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Parameter referencing bug in Python Scripts?
Brian Brinegar wrote:
Hmmm,
There seems to be a bug in the way parameters are referenced in python
scripts. I have a simple script that has a list as a default parameter.
The script then appends something to this list and returns the list.
Each time I call the script the list gets to be one element longer. Here
is an example script:
## Script (Python) pyFix
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=a=['a']
##title=
##
a.append('b')
return a
The output from the script is as follows:
1st call: ['a','b']
2nd call: ['a','b','b']
3rd call: ['a','b','b','b']
4th call: ['a','b','b','b','b']
etc.
It seems that I'm getting a reference to the default parameter which I
then modify for future calls. Currently I'm getting around this by
having list parameters default to none and then setting the defaults
within the script.
I have some security concerns since I can change the default parameters
for other users on the server. Or so it seems.
Suggestions?
The behavior you are observing is not unique to PythonScripts; mutable
default values are a notorious bug magnet for Python applications in
general. Try the following in your Python interpreter:
def memoize(value, seen=[]):
... seen.append(value)
... return seen
...
memoize('a')
['a']
memoize('b')
['a', 'b']
memoize('b')
['a', 'b', 'b']
The classic advice here is, Don't do that. Instead, use a non-mutable
default value (e.g. None or the empty tuple), and special case it. Even
better, avoid writing to the default, e.g.,::
## Script (Python) pyFix
##parameters=a=('a',)
##
result.extend(a)
result.append('b')
return result
Tres.
--
===
Tres Seaver[EMAIL PROTECTED]
Zope Corporation Zope Dealers http://www.zope.com
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.7 memory behaviour
Arndt Droullier wrote: Hello, Hello Arndt I'm trying to find a memory leak in a zope appliction and started to test how zope memory consume behaves basically under load. I'm having exactly the same problem, but under linux using Python 2.3.3 and Zope 2.7.0. First time I noticed this behaving was with Zope 2.6.4 using Python 2.1.3. Than I recreated our internal zope package just updating Python from 2.1.3 to Python 2.3.3 but the problem remained. The current result is: Zope constantly consumes more memory. Jup, exactly, the memory consumption goes up and up and up till the machine starts swapping and gets deadly slow. The testcase is: 12 http clients request constantly request the same page template (it does not contain any tal statements and only 10 lines of HTML) in a simple folder. The increase is about 50 - 100 kb / minute. I´m using Windows 2000 as server platform. My test case is quiet similar. I wrote a Perl script which requests continuously the same page template (which contains some basic DTML stuff). The database cache and debug information on the control panel seem allright. The debug information looks rather strange to me, but I don't know anything about internals of Zope. Anyway I have here reference counts going up fast. Don't know whether this is normal or not. Class March 22, 2004 5:26 pm March 22, 2004 5:38 pm Delta Acquisition.ImplicitAcquirerWrapper 850 18136 +17286 ZPublisher.HTTPRequest.HTTPRequest 145 3026+2881 ZServer.HTTPResponse.ZServerHTTPResponse139 3019+2880 ZPublisher.BaseRequest.RequestContainer 135 3015+2880 DateTime.DateTime.DateTime 51 52 +1 I'll let you know, should I find some solution (apart from restarting :-)). Now I wonder if there´s basically a problem with Zope 2.7? Has anyone made the same experience? Or an explanation why this happens? Thanks, Arndt. Have nice day. Stef ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- - addr://Rathausgasse 31, CH-5001 Aarau fon://++41 62 823 9355 http://www.terreactive.com fax://++41 62 823 9356 - terreActive AG Wir sichern Ihren Erfolg - ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] TALES context
(please stop posting in HTML) Garito wrote: I try to create another context object for TALES like 'container' or 'here' Is there any mechanism to create these? Like PUT_factory for FTP/WevDav or meta_types for the products What extra context are are you looking to add? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] asyncore - twisted
The Zope2 sprint has been focusing on HTTPServer, and one of the interesting ideas that came up was to replace asyncore with twisted. Several people have worked on this in the past, and we were hoping that you could share your experiences, warnings, intuitions, and hopefully even code :-) We look forward to any input, - PyCon2004 Zope2 Sprint ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
AW: [Zope-dev] Zope 2.7 memory behaviour
Your debug information looks like severel objects referenced by acquisition remain in memory. You can be sure this is the case if you in the management interface to Control Panel/Database Management/main/Flush Cache and remove all objects from cache. If the reference count doesn´t change then some objects are leaking. The reference count is the number in the table, or in other words you have 18136 Acquisition.ImplicitAcquirerWrapper objects in memory. This value should normally be much less after flushing the cache. BTW, the effect I described is no real problem. After some time the memory usage gets stable. Greetings, Arndt. The debug information looks rather strange to me, but I don't know anything about internals of Zope. Anyway I have here reference counts going up fast. Don't know whether this is normal or not. Class March 22, 2004 5:26 pm March 22, 2004 5:38 pm Delta Acquisition.ImplicitAcquirerWrapper850 18136 +17286 ZPublisher.HTTPRequest.HTTPRequest 145 3026+2881 ZServer.HTTPResponse.ZServerHTTPResponse 139 3019+2880 ZPublisher.BaseRequest.RequestContainer135 3015+2880 DateTime.DateTime.DateTime 51 52 +1 I'll let you know, should I find some solution (apart from restarting :-)). Virus checked by G DATA AntiVirusKit Version: AVK 14.0.298 from 27.01.2004 Virus news: www.antiviruslab.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.7 memory behaviour
Stefan Doerig wrote at 2004-3-22 17:46 +0100: ... The debug information looks rather strange to me, but I don't know anything about internals of Zope. Anyway I have here reference counts going up fast. Don't know whether this is normal or not. Class March 22, 2004 5:26 pm March 22, 2004 5:38 pm Delta Acquisition.ImplicitAcquirerWrapper850 18136 +17286 ZPublisher.HTTPRequest.HTTPRequest 145 3026+2881 ZServer.HTTPResponse.ZServerHTTPResponse 139 3019+2880 ZPublisher.BaseRequest.RequestContainer135 3015+2880 DateTime.DateTime.DateTime 51 52 +1 Apparently, your application is leaking HTTPRequest instances. Are you storing acquisition wrapped objects in the REQUEST object? Do not do this! It forms cyclic garbarge that cannot be recycled by Python's cyclic garbage collector because the special Zope classes have not yet been made compatible with GC (this comes with Zope 2.8). -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] [Problem] strange state after SIGSEGV
Sorry, the message was intended for zope-dev. I have accidentally sent it to zodb-dev. Redirected... Shane Hathaway wrote at 2004-3-22 11:03 -0500: Dieter Maurer wrote: This problem report is for Zope 2.7.0, Python 2.3.3, Linux 2.4.19. After an application provoked SIGSEGV (caused by a C runtime stack overflow), my Zope process entered a strange (and unhealthy) state: Zope did not die completely (as it should have done) but only partially: One of the threads had disappeared, the others where in the following state: * their parent pid has been set to 1 * attaching with GDB was only allowed as root * at least two of the three remaining processes were waiting in accept * they would not die on SIGTERM but only SIGKILL Consequences: * Zope did no longer respond to requests * stop did not work (as SIGTERM was ineffective) * start did not work, as the dangling processes kept the HTTP port bound. Anyone with some understanding what can cause such a strange state? While developing, this happens all the time for me. The most reliable way to get there is to Ctrl-C out of a 'pdb' session. I can explain some of it. Python threads other than the main thread set a mask that blocks most signals, but SIGKILL (9) can't be blocked. You can find out the signal mask for a process by looking at the SigBlk line of /proc/(process_id)/status. I think Python freezes because a lock held by the dead thread never gets released--perhaps the storage's commit lock. The parent pid and gdb issues could be normal for Python threads. Thank you for your response! The commit lock is held only during the last phase of a request (the commit). It is unlikely that it is being held. It definitely was not held in my case. The parent pid and gdb issues are not normal for Python threads. It looks wrong that a SIGSEGV does not terminate the complete process. Such a behaviour interferes with what zdaemon is supposed to do (restart Zope in case of a problem). Python threads indeed block signal 11 (SIGSEGV). -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] asyncore - twisted
Duncan M. McGreggor wrote: The Zope2 sprint has been focusing on HTTPServer, and one of the interesting ideas that came up was to replace asyncore with twisted. Several people have worked on this in the past, and we were hoping that you could share your experiences, warnings, intuitions, and hopefully even code :-) Meh. The event handling and multiplexed IO infrastructure isn't the part of Zope that needs the most work. Sure, you can replace ZServer with Twisted, but in the long run you aren't solving the big problems by doing so. If it makes you happy, if you feel its a prerequisite to getting any real work done, then you might as well do it. Nevertheless, you don't need everything that twisted brings just to serve a web page, and for the vast majority of Zope's users, serving web pages is the only service that anybody cares about. -- Jamie Heilman http://audible.transient.net/~jamie/ I was in love once -- a Sinclair ZX-81. People said, No, Holly, she's not for you. She was cheap, she was stupid and she wouldn't load -- well, not for me, anyway. -Holly ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] PageTemplateFile vs. Bindings vs. Security
So... remember way back when Brian warned about the binding breakage due to the improved security checks? It just bit me in a way I didn't forsee. I forgot that PageTemplates use bindings too, which means those changes affect more than just PythonScripts they affect Products that use PageTemplateFile methods. My product had declared its object instances to be private, and in doing so it totally breaks the ability to use PageTemplateFile methods. This sucks. Using DTML for product methods goes against the fiber of my being, I just can't bring myself to do it; the namespace pollution as programing paradigm is so frustrating it just makes me want to throw things. But DTMLFile has a huge advantage ... it does the Right Thing with respect to Zope security. It ignores it completely. Paradoxically, by ignoring Zope's security framework in the context of on-disk methods this actually improves Zope's overall security. It allows you to fully take advantage of the API seperation available to Products. To briefly enumerate, you've got 3 basic types of methods: def _python_only_methods(): docstring doesn't matter as this can't be published or accessed TTW, security declarations are superfluous, can't be accessed by RestrictedPython either def python_or_RestrictedPython_methods(): # no docstring ensures its not publishable, security declarations # are necessary to dictate RestrictedPython's privileges pass def publishable_python_or_RestrictedPython_methods(): docstring dictates publishablility, security declarations are necessary and dictate both RestrictedPython's privileges and TTW privileges Now DTMLFile does its work with the privileges of python code, which is great because it means you can define methods which generate content fragments without any restrictions on what APIs it can and can't use. This is useful--more on why in a sec. Conversely the PageTemplateFile does it work with the privileges of RestrictedPython code, which might seem great at first, (More security? What can possibly be bad about that?!) but rapidly wears thin when you try to use PageTemplateFiles in API code, and screws Product authors who were trying to write secure code. To illustrate the first point where PageTemplateFile forces a weakened API, consider the case of Tabs.manage_tabs() ... this method must be declared *public* so that Products using PageTemplateFile can access it. The caller's security context is taken into account with PageTemplateFiles and as 2 different products using PageTemplateFile methods might require completely different permissions to access the method which in turn relies upon Tabs.manage_tabs() the only way to satisfy all possible permission combinations is to just give up and declare the method public so it always succeeds. Products using DTMLFile OTOH don't care, it could be totally private, as it *should* be, and DTMLFile methods would still be able to call it and use the results to display content without regards to callers security context, this makes DTMLFiles useful because you can maintain the integrity of your API and not expose stuff to RestrictedPython that true RestrictedPython has no business calling. If only DTML wasn't such a horrible, syntactically braindamaged, stack-bound HELL to work with! But DTML is hell, and thats not going to change, its better to focus energies on the future which is ZPT, and I assert that PageTemplateFile behavior needs to change. Back to the second point, that the current behavior screws Product authors trying to write secure code. My product called security.declareObjectPrivate() because this is the only way I know of to ensure a product and its attributes are safe from tampering, and until the Binding changes, I thought this was a great deal. Now that the hole in bindings have been fixed though it turns out that because my product uses PageTemplateFile methods which in turn pay attention to bindings, and, unfortunately, pay attention to the callers security context as well, none of those methods work anymore, they raise Unauthorized: Not authorized to access binding: container I can fix this by changing my object security declaration to be public or restricted to a permission, but that has side effects that I don't want, namely opening my object instances up to tampering from within RestrictedPython given the appropriate security context. Given that my need to have the object instance completely private and my desire to avoid DTML are now in direct conflict I'm basically forced to choose between a range of compromises and I don't like any of them. So here's the questions I have for you all... is there a way to declare appropriate security on the bindings that are screwing me right now from within my product code so that I can selectively poke holes to allow container access where needed, or am I to be forcibly coerced into exposing my object to restricted code? And two, assuming I haven't
[Zope-dev] ...but I want to access 'a particular tuple' in that context!
Try this in a PythonScript:
d = {1:2}
for t in d.iteritems():
pass
Annoyingly, it raises Unauthorized: You are not allowed to access 'a
particular tuple' in this context
Is there a reason for this, or is it just part of
AccessControl/RestrictedPython that hasn't been fleshed out yet?
--
Jamie Heilman http://audible.transient.net/~jamie/
I was in love once -- a Sinclair ZX-81. People said, No, Holly,
she's not for you. She was cheap, she was stupid and she wouldn't
load -- well, not for me, anyway. -Holly
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
