Re: [Zope-dev] Runaway processes
On Wednesday 05 December 2007, Stephan Richter wrote: > On Unix-like systems, we can use `os.fork()`. The advantage of this > approach is that I can use OS system calls to kill the process. However, > ZODB database storages cannot be shared between processes. Nikolay Kim has > done some preliminary experiments and found that `db.open()` locks the > system (for both, `FileStorage` and `ZeoClientStorage`). I have not > verified these results or tried to figure out why it is hanging, but I can > see the problem for `FileStorage`. Okay, I spent the rest of the day testing the waters. ;-) The results are somewhat discouraging but the situation is not hopeless. FileStorage - All the file handling works just right. However, the object index is kept in memory, and the original and forked process do not share the same memory space. Thus, once the child process is done doing its modifications to the database, the parent does not know about the updated index. I have done a small hack that reloads the index and it works. However, loading the index can take a long time for large databases. To make this approach feasible, we would need to find a way to describe changes in the index and send the result to the parent via a file. (ZEO) ClientStorage - I could not get this to work at all, because at various steps in the transaction process, the code tries to allocate a lock, but cannot get it, which causes infinite loops. I have attached a small package that demonstrates the behavior. You can use the usual bootstrap/buildout dance. Before running the zeo-based test, you need to start the ZEO server using "./bin/zeo-server fg". Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training forktest.tgz Description: application/tgz ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Runaway processes
> On Unix-like systems, we can use `os.fork()`. The advantage of this approach > is that I can use OS system calls to kill the process. However, ZODB database > storages cannot be shared between processes. Nikolay Kim has done some > preliminary experiments and found that `db.open()` locks the system (for > both, `FileStorage` and `ZeoClientStorage`). I have not verified these > results or tried to figure out why it is hanging, but I can see the problem > for `FileStorage`. i have to create new zeo storage for each child to make forking server work. i think for forking server we need long running childs ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: getting RestrictedPython working on Python 2.5
- "Benji York" <[EMAIL PROTECTED]> wrote: > Philipp von Weitershausen wrote: > > On 5 Dec 2007, at 15:46 , Chris Withers wrote: > >>> The neat thign about the buildout way is that > >>> - it's an absolute no-brainer > >> If you know what magic to incant ;-) > > > > All you have to remember is python boostrap.py and bin/buildout. Not > > > very hard. The advantage is that every single sandbox can be set up > > > like this. I deploy all my sites like this now, be it Grok, Zope 3 > or > > even Plone: checkout the project from SVN and enter the two command > - > > done! > > Maybe we should have a default GETTING-STARTED.txt that goes along > with > bootstrap.py. May be we can add it to top-level README.txt of all packages. Regards, Baiju M ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Runaway processes
Hi everyone, I have a problem and I am hoping that it has been solved already by someone or that I will get some input on at least.I apologize for the lengthy E-mail in advance, but I wanted to provide a detailed discussion as a starting point. Zope is designed to have very short-lived transactions. If transactions are long-living all sorts of problems arise, most notably: 1. We occupy one thread for a long time. 2. The chance of conflict errors increases. Problem 1 can be addressed by increasing the number of allowed threads or to simply add more Zope servers. But his has clearly its limits and is really just a work-around. Another way to solve the problem is to identify long-running operations and calling them asynchronous. Many of us have implemented solutions for this, one of which is lovely.remotetask. Problem number 2 can only be addressed by identifying the long-running tasks beforehand and move them into an async call, again via lovely.remotetask for example. But what happens, if a something unexpected happens and we have an unanticipated long-runnning process? The worst case being something runs forever. Then whenever this problem occurs, one thread will be locked forever, and we can have a total system lockdown in no-time. So how can this be solved? Effectively, from within Zope we cannot do anything, because (a) Zope makes no assumption about running in a thread, and (b) the application is stuck and won't have a hook to get unstuck. So we have to solve the problem from outside. Currently, Zope is commonly run from an application thread. At least both WSGI servers that we commonly use, twisted and zserver, are implemented this way. This means that by some criterion, probably some timeout, the thread should be killed. But hold on! In Python threads cannot be killed. :-( I have done some research and found issue 221115 [1], which discusses the shortcoming of not being able to kill a thread. The discussion ended in making a feature request in PEP 42 [2] which has not been implemented as far as I can tell. So I googled some more to find possible implementations. Here are two distinctively different solutions (others I have found are either obviously trivial and will not work, or are derivatives of these two): 1. A Python-only solution using sys.settrace [3]. Besides making everything very slow, sys.settrace() is only called when a new byte code instruction is executed. So in case a low-level call hangs up the process, then the trace intercept will never be called. 2. Use an exception to intercept execution on the C-level [4]. This looked very promising, until I read the following comment on the page: The exception will be raised only when executing python bytecode. If your thread calls a native/built-in blocking function, the exception will be raised only when execution returns to the python code. So my conclusion is that Python threads cannot be unconditionally killed. BTW, if a low-level call is blocking, then all Python threads are blocked. From the Python `thread` library documentation[5]: Not all built-in functions that may block waiting for I/O allow other threads to run. (The most popular ones (time.sleep(), file.read(), select.select()) work as expected.) In all fairness, though, those are very rare occurrences. Most libraries are non-blocking and the above solutions would be just fine. But in my case, I really need to find a way to kill a Zope execution environment when a C call hangs. So what other choices do we have? On Unix-like systems, we can use `os.fork()`. The advantage of this approach is that I can use OS system calls to kill the process. However, ZODB database storages cannot be shared between processes. Nikolay Kim has done some preliminary experiments and found that `db.open()` locks the system (for both, `FileStorage` and `ZeoClientStorage`). I have not verified these results or tried to figure out why it is hanging, but I can see the problem for `FileStorage`. Are there any known side-effects on what happens, if I fork after the connection has been made? Since I am using the original process merely as a control, I guess I should be fine. Of course, the interesting question is: what happens to the ZODB connection, not to mention to the DB, if it is in the middle of writing? I guess the safest solution would be to fork within the constraint of the transaction. Any comments will be very much appreciated. Once we decide on the forking approach, we have to solve the issue for Windows of course too. My googling did not turn out immediately successful, but I think if we use Windows' native threads they will provide us with the necessary API, since I can exit it at any time. .. [1]: http://bugs.python.org/issue221115 .. [2]: http://www.python.org/dev/peps/pep-0042/ .. [3]: http://www.velocityreviews.com/forums/t330554-kill-a-thread-in-python.html .. [4]: http://sebulba.wikispaces.co
Re: [Zope-dev] Re: getting RestrictedPython working on Python 2.5
Philipp von Weitershausen wrote: On 5 Dec 2007, at 15:46 , Chris Withers wrote: The neat thign about the buildout way is that - it's an absolute no-brainer If you know what magic to incant ;-) All you have to remember is python boostrap.py and bin/buildout. Not very hard. The advantage is that every single sandbox can be set up like this. I deploy all my sites like this now, be it Grok, Zope 3 or even Plone: checkout the project from SVN and enter the two command - done! Maybe we should have a default GETTING-STARTED.txt that goes along with bootstrap.py. -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: getting RestrictedPython working on Python 2.5
On 5 Dec 2007, at 15:46 , Chris Withers wrote: There's python setup.py develop, which will also install the checked out thing as a development egg, What does that mean? Which python files would I edit to make changes in this case? The ones you checked out. This isn't rocket science, you know :). Also, you don't get a test runner this easily (though we should probably make python setup.py test work at some point, it currently doesn't work). I might be interested in making that happen ;-) Cool. What still needs to be done? I'm imagining just plumbing the 'test' action through to running the zope.testing testrunner, or is there more to it than that? No idea. You'll presumably have to deal with the tests_require thing as well. The neat thign about the buildout way is that - it's an absolute no-brainer If you know what magic to incant ;-) All you have to remember is python boostrap.py and bin/buildout. Not very hard. The advantage is that every single sandbox can be set up like this. I deploy all my sites like this now, be it Grok, Zope 3 or even Plone: checkout the project from SVN and enter the two command - done! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: getting RestrictedPython working on Python 2.5
Philipp von Weitershausen wrote: ($z refers to svn+ssh://svn.zope.org/repos/main on my system) Where does this put the code that I'd be changing? Uh, you checked it out (first line: svn co). Sorry, I'm used to things like "python setup.py install" putting all manner of things in all manner of weird places ;-) Dependencies will be downloaded from PyPI (or any other index you specify) as eggs. Cool. There's python setup.py develop, which will also install the checked out thing as a development egg, What does that mean? Which python files would I edit to make changes in this case? Also, you don't get a test runner this easily (though we should probably make python setup.py test work at some point, it currently doesn't work). I might be interested in making that happen ;-) What still needs to be done? I'm imagining just plumbing the 'test' action through to running the zope.testing testrunner, or is there more to it than that? The neat thign about the buildout way is that - it's an absolute no-brainer If you know what magic to incant ;-) - it's completely self-contained (nothing is installed outside that one directory, unless you explicitly allow that). This *is* a good thing :-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Tests: 5 OK
Summary of messages to the zope-tests list. Period Tue Dec 4 13:00:00 2007 UTC to Wed Dec 5 13:00:00 2007 UTC. There were 5 messages: 5 from Zope Unit Tests. Tests passed OK --- Subject: OK : Zope-2.7 Python-2.3.6 : Linux From: Zope Unit Tests Date: Tue Dec 4 20:54:03 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-December/008746.html Subject: OK : Zope-2.8 Python-2.3.6 : Linux From: Zope Unit Tests Date: Tue Dec 4 20:55:33 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-December/008747.html Subject: OK : Zope-2.9 Python-2.4.4 : Linux From: Zope Unit Tests Date: Tue Dec 4 20:57:04 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-December/008748.html Subject: OK : Zope-2.10 Python-2.4.4 : Linux From: Zope Unit Tests Date: Tue Dec 4 20:58:34 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-December/008749.html Subject: OK : Zope-trunk Python-2.4.4 : Linux From: Zope Unit Tests Date: Tue Dec 4 21:00:05 EST 2007 URL: http://mail.zope.org/pipermail/zope-tests/2007-December/008750.html ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Re: MIME type syntax (was: Bug in zc.resourcelibrary?)
Christian Theune wrote: > IMHO we should assume whatever is given to us was unpacked by the > container format and is intended to be a valid MIME type. Meaning we > should not strip it. OK, then we agree. Good that I did zc.resourcelibrary 0.8, not 1.0 yesterday ;o) -- Thomas ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: MIME type syntax (was: Bug in zc.resourcelibrary?)
Am Mittwoch, den 05.12.2007, 11:22 +0100 schrieb Thomas Lotze: > Christian Theune wrote: > > > That depends on the container format and therefore it's the responsibility > > of the container (e.g. a HTTP header) to remove the whitespace. > > > > Note that AFAICT RFC 2045 section 5.1 (that's what you're referring to > > AFAICT) defines the MIME type specification as done with the Content-Type > > header for MIME messages without defining exactly how the actual type > > relates to the container. > > That's why I asked about the intended semantics of our MIME type strings: > should anything that's called a MIME or content type be strict according > to the spec, it being the responsibility of whatever creates those strings > to make them so, or should we have to be forgiving in all the places that > consume MIME types? > > > IMHO we should not pay attention to whitespace. > > I'm not sure whether you mean we should ignore whitespace in a MIME type > string or not have to pay attention to the issue when consuming one. IMHO we should assume whatever is given to us was unpacked by the container format and is intended to be a valid MIME type. Meaning we should not strip it. Christian -- gocept gmbh & co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: getting RestrictedPython working on Python 2.5
Chris Withers wrote: Philipp von Weitershausen wrote: $ svn co $z/RestrictedPython/trunk RestrictedPython (or a branch) $ cd RestrictedPython $ /your/development/python bootstrap.py $ bin/buildout $ bin/test ($z refers to svn+ssh://svn.zope.org/repos/main on my system) Where does this put the code that I'd be changing? Uh, you checked it out (first line: svn co). Are any dependencies brought out as svn checkouts or just got as stable releases? Dependencies will be downloaded from PyPI (or any other index you specify) as eggs. Note that this procedure is the canonical one for all packages maintained in svn.zope.org nowadays. OK, is there a non-buildout equivalent? There's python setup.py develop, which will also install the checked out thing as a development egg, but it will try and install the dependencies in the site-packages of the 'python' you used to call it with. This usually means isntalling things globally (unless you use virtualenv). Also, you don't get a test runner this easily (though we should probably make python setup.py test work at some point, it currently doesn't work). The neat thign about the buildout way is that - it's an absolute no-brainer - it's completely self-contained (nothing is installed outside that one directory, unless you explicitly allow that). ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: getting RestrictedPython working on Python 2.5
Philipp von Weitershausen wrote: $ svn co $z/RestrictedPython/trunk RestrictedPython (or a branch) $ cd RestrictedPython $ /your/development/python bootstrap.py $ bin/buildout $ bin/test ($z refers to svn+ssh://svn.zope.org/repos/main on my system) Where does this put the code that I'd be changing? Are any dependencies brought out as svn checkouts or just got as stable releases? Note that this procedure is the canonical one for all packages maintained in svn.zope.org nowadays. OK, is there a non-buildout equivalent? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: getting RestrictedPython working on Python 2.5
Chris Withers wrote: Hi All, I'm looking to get RestrictedPython working in Python 2.5. Has any work been done on this so far? My first question is about the "right" way to get a checkout that I can develop in now. In years gone by, I would have created a branch of the whole of zope 3, checked it out and then started work. Since eggification, I'm not sure what to do... What I was planning on doing was creating a branch of RestrictedPython, checking that out and then manually checking out any dependencies from svn. Is there a "better" way to do this with easy_install or some such? $ svn co $z/RestrictedPython/trunk RestrictedPython (or a branch) $ cd RestrictedPython $ /your/development/python bootstrap.py $ bin/buildout $ bin/test ($z refers to svn+ssh://svn.zope.org/repos/main on my system) Note that this procedure is the canonical one for all packages maintained in svn.zope.org nowadays. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] getting RestrictedPython working on Python 2.5
Hi All, I'm looking to get RestrictedPython working in Python 2.5. Has any work been done on this so far? My first question is about the "right" way to get a checkout that I can develop in now. In years gone by, I would have created a branch of the whole of zope 3, checked it out and then started work. Since eggification, I'm not sure what to do... What I was planning on doing was creating a branch of RestrictedPython, checking that out and then manually checking out any dependencies from svn. Is there a "better" way to do this with easy_install or some such? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: MIME type syntax (was: Bug in zc.resourcelibrary?)
Christian Theune wrote: > That depends on the container format and therefore it's the responsibility > of the container (e.g. a HTTP header) to remove the whitespace. > > Note that AFAICT RFC 2045 section 5.1 (that's what you're referring to > AFAICT) defines the MIME type specification as done with the Content-Type > header for MIME messages without defining exactly how the actual type > relates to the container. That's why I asked about the intended semantics of our MIME type strings: should anything that's called a MIME or content type be strict according to the spec, it being the responsibility of whatever creates those strings to make them so, or should we have to be forgiving in all the places that consume MIME types? > IMHO we should not pay attention to whitespace. I'm not sure whether you mean we should ignore whitespace in a MIME type string or not have to pay attention to the issue when consuming one. -- Thomas ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] MIME type syntax (was: Bug in zc.resourcelibrary?)
Am Mittwoch, den 05.12.2007, 10:20 +0100 schrieb Thomas Lotze: > Fred Drake wrote: > > > On Dec 4, 2007 5:55 PM, Thomas Lotze <[EMAIL PROTECTED]> wrote: > >> It wasn't even about whitespace around the / but leading whitespace in > >> front of the major type. > > > > Wow. It probably didn't occur to me that would be screwed up. > > I've now looked up RfC 2045 which states that there is to be no whitespace > within the MIME type specification, i.e. in particular none around the > slash. This requires a change to zope.publisher.contenttype. OTOH it's > probably up to us how to treat whitespace surrounding the string > specifying MIME type: > > - Either we decide that such a string should always follow the RfC exactly > and contain no whitespace at all (which makes the way zope.publisher.http > handles it the correct way), > > - or we allow for it to be some possibly whitespace-padded string that > contains a valid MIME spec, which would allow for surrounding whitespace, > but not space around the slash. > > I'd prefer the first option for clarity. Is there any reason to be > forgiving regarding whitespace in the first place? That depends on the container format and therefore it's the responsibility of the container (e.g. a HTTP header) to remove the whitespace. Note that AFAICT RFC 2045 section 5.1 (that's what you're referring to AFAICT) defines the MIME type specification as done with the Content-Type header for MIME messages without defining exactly how the actual type relates to the container. IMHO we should not pay attention to whitespace. Christian -- gocept gmbh & co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] MIME type syntax (was: Bug in zc.resourcelibrary?)
Fred Drake wrote:
> On Dec 4, 2007 5:55 PM, Thomas Lotze <[EMAIL PROTECTED]> wrote:
>> It wasn't even about whitespace around the / but leading whitespace in
>> front of the major type.
>
> Wow. It probably didn't occur to me that would be screwed up.
I've now looked up RfC 2045 which states that there is to be no whitespace
within the MIME type specification, i.e. in particular none around the
slash. This requires a change to zope.publisher.contenttype. OTOH it's
probably up to us how to treat whitespace surrounding the string
specifying MIME type:
- Either we decide that such a string should always follow the RfC exactly
and contain no whitespace at all (which makes the way zope.publisher.http
handles it the correct way),
- or we allow for it to be some possibly whitespace-padded string that
contains a valid MIME spec, which would allow for surrounding whitespace,
but not space around the slash.
I'd prefer the first option for clarity. Is there any reason to be
forgiving regarding whitespace in the first place?
As for capitalization, the RfC states clearly that major type, minor type
and parameter names are case-insensitive. This requires a change to
zope.publisher.http which currently says:
if not content_type.startswith('text/'):
raise ValueError(...)
I'll do all that once we have agreed on one way of treating the MIME type
specs.
--
Thomas
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
