RE: [Zope-dev] Logging for ZScheduler?
On Thu, 15 Jun 2000, Loren Stafford wrote: It would be a good idea if there was a field in the ZEvent that defined the subsystem used in the zLOG call. I didn't follow your point here. By "subsytem" do you mean which logger in the loggers tuple? Then do you mean that different ZEvents could log to different loggers? Why would this be a "good idea", I mean, do you have a use case in mind? from zLOG.py: def LOG(subsystem, severity, summary, detail='', error=None, reraise=None): The first argument specifies a subsystem, which is passed to the logging implementation. A logging subsystem may choose to ignore log messages from particular subsystems, or perform special actions (eg. if a critical error has occured in the ZScheduler subsystem, page the sysadmin). By allowing an individual ZEvent to override the subsystem reported, you can gain even more control. -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Request for comments: Directory storage
On Fri, 9 Jun 2000, Petru Paler wrote:
I'd love some sort of benchmarking tool for this (and posibly other
Storages). I guess the best way would a python script that uses urllib.
Something that would algorithmically pump up the DB to 1GB in size
and retrieve the URL's. Any volunteers or am I doing it in my
copious spare time (tm)?
It would be great if you could do it, but beware that you will be
benchmarking a lot of overhead if you only plan to measure storage
performance. Why not use ZODB directly ?
If I talk HTTP, it measures things fully - Python's interpreter lock
will mean a storage system written in python will benchmark better
without having to compete with ZServer, and vice versa for storage
systems with non-pythonic bits.
I've got a nice NetApp here to run some tests on.
What filesystem does that use ?
No idea :-) Something log based that is very fast and handles huge
directories happily. It also appears that another member of this
list has an EMC Symmetrix box to test on, which I believe is the next (and
highest) level up from a Netapp.
I've attached a prerelease alpha of zouch.py for giggles. Not even a
command line yet, so you will need to edit some code at the bottom.
The current settings generate about 360 directories and about 36000 files,
and proceeds to make about 18 reads. This bloated by test ZODB
to just over 200MB and took about 2.6 hours attacking my development Zope
server from another host on my LAN.
Todo:
tidy and vet ugly code
command line interface
dynamic option (do more intensive DTML stuff - currently just
standard_html_header/standard_html_footer)
catalog option (since DTML Documents arn't catalog aware, will need
to make two calls to make a new document)
upload larger documents and some binaries (200MB isn't great for
benchmarking when you might have a gig of ram doing caching for you)
standard test suite
better reporting
spinning dohicky so we know it hasn't hung without having to look
at log files
--
Stuart Bishop Work: [EMAIL PROTECTED]
Senior Systems Alchemist Play: [EMAIL PROTECTED]
Computer Science, RMIT University
#!/bin/env python
'''
$Id: zouch.py,v 1.3 2000/06/12 04:23:01 zen Exp $
Zouch - the Zope torture tester
'''
import whrandom
import sha
import threading
import ftplib
import httplib
from string import split,join,replace
from time import time,strftime,localtime,sleep
from StringIO import StringIO
from Queue import Queue
from threading import Thread,RLock
from urllib import urlencode
from urlparse import urlparse
from base64 import encodestring
retries = 10
retrysleep = 1
def debug(msg):
print 'D: %s - %s' % (threading.currentThread().getName(),msg)
# Fatal exceptions will not be caught
class FatalException(Exception): pass
class UnsupportedProtocol(FatalException): pass
class FolderLock:
def __init__(self):
self.locks = {}
self.sync = RLock()
def lock(self,dirs):
self._lock(self._mypath(dirs))
self._lock(self._parentpath(dirs))
def unlock(self,dirs):
self._unlock(self._parentpath(dirs))
self._unlock(self._mypath(dirs))
def _parentpath(self,dirs):
if len(dirs) == 1:
return 'root'
else:
return join(dirs[:-1],'/')
def _mypath(self,dirs):
return join(dirs,'/')
def _lock(self,d):
locks = self.locks
sync = self.sync
while 1:
try:
sync.acquire()
acq = 1
if locks.has_key(d):
l = locks[d]
sync.release()
acq = 0
l.acquire()
l.release()
else:
l = RLock()
l.acquire()
locks[d] = l
break
finally:
if acq: sync.release()
def _unlock(self,d):
locks = self.locks
sync = self.sync
sync.acquire()
try:
l = locks[d]
del locks[d]
l.release()
finally:
sync.release()
folderlock = FolderLock()
class HTTPMaker:
'Baseclass for HTTP Maker classes'
def __init__(self,queue,url,username,password):
purl = urlparse(url)
host,port = split(purl[1],':',1)
path = purl[2]
if port:
port = int(port)
else:
port = 80
if path[-1] == '/':
self.path = path
else:
self.path = path + '/'
self.queue = queue
self.ops = 0
if username is None:
self.auth = None
else:
if password is None: password = ''
self.auth = 'Basic %s' % \
[Zope-dev] ZPHP [was Re: [Zope] CHAT]
On Thu, 1 Jun 2000, [iso-8859-1] Gonçalo Gomes wrote: Silver_Surfer: how do you see the future of Zope fighting against PHP space on the web? Jim: I don't know much (enough) about PHP, but from what I vaguely know, PHP methods seem like a logical step. Hmm... I'm having to setup a PHP web environment for the students next semester, and have elected to run it as a CGI application (tie into existing suid security arcitecture). I'm told there won't be any functionality loss except, just a bit of performance (?). If we didn't care about persistance of the PHP engine, it would be a trivial task to create a PHP method object - just a small python stub using the CGI libraries to call PHP. Or even more trivial - I could hack my existing Logger product to create a dtml-php tag, so you would be able to preprocess your PHP code with DTML tags or embed PHP output in DTML for standard_html_header etc. (Is that my foot bleeding?). The standard response to a dtml-python tag is that it would make it too PHP like, so dtml-php seems very appropriate :-) And then the next step is to import IMP and other PHP apps into Zope land... -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Z SQL: optional isn't?
On Mon, 29 May 2000, Kyler B. Laird wrote: This is not the same functionality as is provided by the "optional" attribute. How would I use it with an integer, for example? I don't want a default value; I want the option of not having the value set at all. I like the optional attribute. I'd like to see it do more than sit idle in the code. You don't have to do the typechecking in the 'arguments' box if you don't want. eg. in your arguments section: course="" subject="" And your method: select * from enrolments dtml-sqlgroup where dtml-sqltest course type=int optional dtml-sqltest subject type=nb optional /dtml-sqlgroup The definition of the 'optional' tag from the docs might help here: '''A flag indicating if the test is optional. If the test is optional and no value is provided for a variable, or the value provided is an invalid empty string, then no text is inserted''' Hmm... looking at this definition, it seems you have found a bug (but setting the default value to "" in the arguments section will provide a workaround). Or maybe 'no value provided' means 'is None'. Anyone from DC want to provide a ruling? :-) -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Memory 2.1.4-2.1.6 a.k.a. how to get objects out ofthe cache
On Sat, 27 May 2000, Eric Sattler wrote: I do see GenericUserFolder and SQLSession objects with the Control_Panel_Debug screen, and they do not seem to go away. I wrote a simple python script to do nothing more than authenticate (log in) using the GenericUserFolder method docLogin. The memory usage quickly grows out of control. After waiting 15 minutes(my cookie timeout), no decrease in memory usage. The objects are still in the cache also. Phillip J. Eby identified a memory leak in GenericUserFolder over the weekend. I'm just downloading LoginManager now to see if I can steal their fix :-) (Oh... thats easy. Just search for 'Waaa!') There is also a good chance that this is also causing the SQLSession leak - GUF is maintaining a reference to REQUEST (in a nice circular way causing the memory leak), and REQUEST maintains a reference to the SESSION, so the SESSION won't be freed. I should have a patch available shortly. I would appreciate people who know how to drive the debuggers better than I confirming that the leak is gone. /me hops on the 'real garbage collection for Python' bandwagon -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] New GUF (1.2.3)
I think I've fixed the memory leak identified over the weekend in GUF.
This leak may also have caused {SQL,FS}Session to leak through no fault of
their own. Could people who are experiencing the problem try out the
new version and let me/zope-dev know the results. I'll announce the release
on zope.org soonish, giving the guinee^h^h^h^h^h^hzope-dev members a
chance to give it a poke.
--
Stuart Bishop Work: [EMAIL PROTECTED]
Senior Systems Alchemist Play: [EMAIL PROTECTED]
Computer Science, RMIT University
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Z SQL: optional isn't?
On Fri, 26 May 2000, Kyler B. Laird wrote: dtml-sqltest subject type=nb optional dtml-and dtml-sqltest course_nbr type=nb optional dtml-and dtml-sqltest version type=nb optional dtml-and dtml-sqltest campus_code type=nb optional /dtml-sqlgroup This works only if I provide at least an empty string for all values. If I don't provide a value at all for one of the arguments, I get In your 'Options' box for your ZSQL method, you can specify a default value for arguments if they were not passed: subject:string = '' course_nbr: string = '' Don't know if/where it is documented. -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Problems with LoginManager form-based login
On Wed, 24 May 2000, Phillip J. Eby wrote: This is a problem that apparently can only be solved by replicating ZPublisher's backward walk, which is the wrong thing to do because the traversal will be O(n^2). Bleah. I guess we'll have to do something like Stuart Bishop's backward walk in the GUF, since any enhancement of the ZPublisher architecture to handle login forms properly won't happen until at least 2.2. The backward walk stuff (which I don't think ever worked - every time I fixed one case it seemed I'd break another) has been torn out and replaced with code that does what you are after. Grab a copy of the latest version and search for 'WooHoo' in GenericUserFolder.py Meanwhile, I suppose Ty and I should try to come up with a proposal for revising ZPublisher to be able to walk back on the user lookups but still use the login form closest to the URL being accessed. -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Objects?
On Tue, 23 May 2000, Adam Pawliuk wrote: Although there is several user characteristics that we would like to store in a session object cached in server RAM, such as the user's unique system ID, greeting name, etc; rather than having to hit the DB for this information on each request. Basically we would like to use something similar to the session objects in ASP (yuck), or Java JSP/servlet model. I've seen several Zope session products although they all seem to hook directly to a DB and don't seem to provide memory persistence which is what we would like to use. Is there any existing products which do this? Boring old ZSQL Methods actually do exactly what you want - check out the caching options under the advanced tab (or the ZSQL docs on zope.org). The database will only be hit the first time you call the method with a given set of parameters, or when maximum time to cache expires. If your GUF hooks make use of ZSQL methods, you should have a look at tuning these too as it can be a big performance improvement. Actually - I tell a slight lie. The DB might be hit once for each thread, as the ZSQL cache is not shared between Zope threads. You need to be aware of this as a cached result may be refreshed in one thread but not yet expired in another and can give ambiguous results and unreproducable bugs if you are not aware of it when you are developing. -- ___ // Zen (alias Stuart Bishop) Work: [EMAIL PROTECTED] // E N Senior Systems Alchemist Play: [EMAIL PROTECTED] //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
