RE: [Zope-dev] Logging for ZScheduler?
On Thu, 15 Jun 2000, Loren Stafford wrote: > > It would be a good idea if there was a field in the ZEvent that defined > > the subsystem used in the zLOG call. > > I didn't follow your point here. By "subsytem" do you mean which logger in > the loggers tuple? Then do you mean that different ZEvents could log to > different loggers? Why would this be a "good idea", I mean, do you have a > use case in mind? from zLOG.py: def LOG(subsystem, severity, summary, detail='', error=None, reraise=None): The first argument specifies a subsystem, which is passed to the logging implementation. A logging subsystem may choose to ignore log messages from particular subsystems, or perform special actions (eg. if a critical error has occured in the ZScheduler subsystem, page the sysadmin). By allowing an individual ZEvent to override the subsystem reported, you can gain even more control. -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Logging for ZScheduler?
On Thu, 15 Jun 2000, Loren Stafford wrote: > Aha! Even tho a cohosted account might not be able to modify the source of > the Zope implementation directly, installing a CustomLog product could do so > indirectly, by importing ZLogger and modifying the loggers tuple. To know > where to write the log file, CustomLog would have to either know or be > configurable enough to adapt to the ISP's Zope configuration. I suppose an > absolute path property would be suffient. An ISP could preconfigure and > preinstall the CustomLog product for all users, or each user could do it. Dump it Globals.data_dir unless a given environment variable is set. (Sorry bout the double reply) -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Logging for ZScheduler?
On Wed, 14 Jun 2000, Loren Stafford wrote: > I need some advise for designing a flexible system for logging the > activities of scheduled events. You could just call Zope's logging system and move it out of the scope of your product. Have a look at zLOG.py or LOGGING.txt if you havn't already. Means people can extend their logging system to their hearts content with no effect on your product. > But ZScheduler also needs to be ISP independent. That is, you should have > complete scheduling capabilities for your cohosted Zope applications without > having to call up the ISP and ask for system configuration changes. > ZScheduler partially achieves this level of independence, by eliminating the > need for cron. This could be achieved with a seperate product. A product that ties into Zope's logging infrastructure and allows viewing the log files through the web. All it would have to do is append itself to the ZLogger.loggers tuple, and the interface is pretty trivial. And of course, it means that other products can use it without modification (Logger product, Zope itself etc.) and can be extended as required without having to submit patches to you for inclusion in ZScheduler. It would be a good idea if there was a field in the ZEvent that defined the subsystem used in the zLOG call. > So, I need a more flexible structure for the logging aspect of ZScheduler. > While the ZEvent class is subclassable by users, I haven't made the > Scheduler class instantiable -- there should be only one instance -- so it's > not subclassable either, and therefore you can't override its logging > method. Do you see the problem? > Having little experience with Python and OO applications, I don't know how > others solve this problem. Can you give me any pointers? Its not a problem with ZScheduler, it a problem that no one has written a plug-in logging system that is good enough for what you are trying to do. The existing zLOG API is fine (well - it could be better), but just needs someone to write the relevant modules. Hmm... I think I'll add a section to the Interfaces Wiki... -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Request for comments: Directory storage
On Fri, 9 Jun 2000, Petru Paler wrote:
> > I'd love some sort of benchmarking tool for this (and posibly other
> > Storages). I guess the best way would a python script that uses urllib.
> > Something that would algorithmically pump up the DB to > 1GB in size
> > and retrieve the URL's. Any volunteers or am I doing it in my
> > copious spare time (tm)?
>
> It would be great if you could do it, but beware that you will be
> benchmarking a lot of overhead if you only plan to measure storage
> performance. Why not use ZODB directly ?
If I talk HTTP, it measures things fully - Python's interpreter lock
will mean a storage system written in python will benchmark better
without having to compete with ZServer, and vice versa for storage
systems with non-pythonic bits.
> > I've got a nice NetApp here to run some tests on.
>
> What filesystem does that use ?
No idea :-) Something log based that is very fast and handles huge
directories happily. It also appears that another member of this
list has an EMC Symmetrix box to test on, which I believe is the next (and
highest) level up from a Netapp.
I've attached a prerelease alpha of zouch.py for giggles. Not even a
command line yet, so you will need to edit some code at the bottom.
The current settings generate about 360 directories and about 36000 files,
and proceeds to make about 18 reads. This bloated by test ZODB
to just over 200MB and took about 2.6 hours attacking my development Zope
server from another host on my LAN.
Todo:
tidy and vet ugly code
command line interface
dynamic option (do more intensive DTML stuff - currently just
standard_html_header/standard_html_footer)
catalog option (since DTML Documents arn't catalog aware, will need
to make two calls to make a new document)
upload larger documents and some binaries (200MB isn't great for
benchmarking when you might have a gig of ram doing caching for you)
standard test suite
better reporting
spinning dohicky so we know it hasn't hung without having to look
at log files
--
Stuart Bishop Work: [EMAIL PROTECTED]
Senior Systems Alchemist Play: [EMAIL PROTECTED]
Computer Science, RMIT University
#!/bin/env python
'''
$Id: zouch.py,v 1.3 2000/06/12 04:23:01 zen Exp $
Zouch - the Zope torture tester
'''
import whrandom
import sha
import threading
import ftplib
import httplib
from string import split,join,replace
from time import time,strftime,localtime,sleep
from StringIO import StringIO
from Queue import Queue
from threading import Thread,RLock
from urllib import urlencode
from urlparse import urlparse
from base64 import encodestring
retries = 10
retrysleep = 1
def debug(msg):
print 'D: %s - %s' % (threading.currentThread().getName(),msg)
# Fatal exceptions will not be caught
class FatalException(Exception): pass
class UnsupportedProtocol(FatalException): pass
class FolderLock:
def __init__(self):
self.locks = {}
self.sync = RLock()
def lock(self,dirs):
self._lock(self._mypath(dirs))
self._lock(self._parentpath(dirs))
def unlock(self,dirs):
self._unlock(self._parentpath(dirs))
self._unlock(self._mypath(dirs))
def _parentpath(self,dirs):
if len(dirs) == 1:
return 'root'
else:
return join(dirs[:-1],'/')
def _mypath(self,dirs):
return join(dirs,'/')
def _lock(self,d):
locks = self.locks
sync = self.sync
while 1:
try:
sync.acquire()
acq = 1
if locks.has_key(d):
l = locks[d]
sync.release()
acq = 0
l.acquire()
l.release()
else:
l = RLock()
l.acquire()
locks[d] = l
break
finally:
if acq: sync.release()
def _unlock(self,d):
locks = self.locks
sync = self.sync
sync.acquire()
try:
l = locks[d]
del locks[d]
l.release()
finally:
sync.release()
folderlock = FolderLock()
class HTTPMaker:
'Baseclass for HTTP Maker classes'
def __init__(self,queue,url,username,password):
purl = urlparse(url)
host,port = split(purl[1],':',1)
path = purl[2]
if port:
port = int(port)
else:
port = 80
if path[-1] == '/':
self.path = path
else:
self.path = path + '/'
self.queue = queue
self.ops = 0
if username is None:
self.auth = None
else:
if password is None: password = ''
self.auth
Re: [Zope-dev] Request for comments: Directory storage
On Wed, 7 Jun 2000, Petru Paler wrote: > Comments ? Suggestions ? > > PS: a new DirectoryStorage release will be done today, with bugfixes and > new features. I'd love some sort of benchmarking tool for this (and posibly other Storages). I guess the best way would a python script that uses urllib. Something that would algorithmically pump up the DB to > 1GB in size and retrieve the URL's. Any volunteers or am I doing it in my copious spare time (tm)? I've got a nice NetApp here to run some tests on. -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ZPHP [was Re: [Zope] CHAT]
On Thu, 1 Jun 2000, [iso-8859-1] Gonçalo Gomes wrote: > Silver_Surfer: how do you see the future of Zope fighting against PHP space on the >web? > Jim: I don't know much (enough) about PHP, but from what I vaguely know, PHP methods >seem like a logical step. Hmm... I'm having to setup a PHP web environment for the students next semester, and have elected to run it as a CGI application (tie into existing suid security arcitecture). I'm told there won't be any functionality loss except, just a bit of performance (?). If we didn't care about persistance of the PHP engine, it would be a trivial task to create a PHP method object - just a small python stub using the CGI libraries to call PHP. Or even more trivial - I could hack my existing Logger product to create a tag, so you would be able to preprocess your PHP code with DTML tags or embed PHP output in DTML for standard_html_header etc. (Is that my foot bleeding?). The standard response to a tag is that it would make it too PHP like, so seems very appropriate :-) And then the next step is to import IMP and other PHP apps into Zope land... -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Z SQL: "optional" isn't?
On Mon, 29 May 2000, Kyler B. Laird wrote: > This is not the same functionality as is provided > by the "optional" attribute. How would I use it > with an integer, for example? I don't want a > default value; I want the option of not having > the value set at all. > > I like the optional attribute. I'd like to see > it do more than sit idle in the code. You don't have to do the typechecking in the 'arguments' box if you don't want. eg. in your arguments section: course="" subject="" And your method: select * from enrolments The definition of the 'optional' tag from the docs might help here: '''A flag indicating if the test is optional. If the test is optional and no value is provided for a variable, or the value provided is an invalid empty string, then no text is inserted''' Hmm... looking at this definition, it seems you have found a bug (but setting the default value to "" in the arguments section will provide a workaround). Or maybe 'no value provided' means 'is None'. Anyone from DC want to provide a ruling? :-) -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Z SQL: "optional" isn't?
On Sat, 27 May 2000, Kyler B. Laird wrote: > I hope it's not documented anywhere; it doesn't > work either. > > It *does* allow the method to run without > setting the arguments, but it generates SQL > like "where subject=''". > > Someone had a good vision when writing the > documentation for the sqltest optional flag. > I just want it to work. Oops the default argument looks like it needs to be in double quotes. So what I said before, but subject:string="" -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] New GUF (1.2.3)
I think I've fixed the memory leak identified over the weekend in GUF.
This leak may also have caused {SQL,FS}Session to leak through no fault of
their own. Could people who are experiencing the problem try out the
new version and let me/zope-dev know the results. I'll announce the release
on zope.org soonish, giving the guinee^h^h^h^h^h^hzope-dev members a
chance to give it a poke.
--
Stuart Bishop Work: [EMAIL PROTECTED]
Senior Systems Alchemist Play: [EMAIL PROTECTED]
Computer Science, RMIT University
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Memory 2.1.4-2.1.6 a.k.a. how to get objects out ofthe cache
On Sat, 27 May 2000, Eric Sattler wrote: > I do see GenericUserFolder and SQLSession objects with the > Control_Panel_Debug screen, and they do not seem to go away. I wrote > a simple python script to do nothing more than authenticate (log in) > using the GenericUserFolder method docLogin. The memory usage quickly > grows out of control. After waiting 15 minutes(my cookie timeout), > no decrease in memory usage. The objects are still in the cache also. Phillip J. Eby identified a memory leak in GenericUserFolder over the weekend. I'm just downloading LoginManager now to see if I can steal their fix :-) (Oh... thats easy. Just search for 'Waaa!') There is also a good chance that this is also causing the SQLSession leak - GUF is maintaining a reference to REQUEST (in a nice circular way causing the memory leak), and REQUEST maintains a reference to the SESSION, so the SESSION won't be freed. I should have a patch available shortly. I would appreciate people who know how to drive the debuggers better than I confirming that the leak is gone. /me hops on the 'real garbage collection for Python' bandwagon -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Z SQL: "optional" isn't?
On Fri, 26 May 2000, Kyler B. Laird wrote: > > > > > > > > > > This works only if I provide at least an empty > string for all values. If I don't provide a > value at all for one of the arguments, I get In your 'Options' box for your ZSQL method, you can specify a default value for arguments if they were not passed: subject:string = '' course_nbr: string = '' Don't know if/where it is documented. -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT University ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Problems with LoginManager form-based login
On Wed, 24 May 2000, Phillip J. Eby wrote: > This is a problem that apparently can only be solved by replicating > ZPublisher's backward walk, which is the wrong thing to do because the > traversal will be O(n^2). Bleah. I guess we'll have to do something like > Stuart Bishop's backward walk in the GUF, since any enhancement of the > ZPublisher architecture to handle login forms properly won't happen until > at least 2.2. The backward walk stuff (which I don't think ever worked - every time I fixed one case it seemed I'd break another) has been torn out and replaced with code that does what you are after. Grab a copy of the latest version and search for 'WooHoo' in GenericUserFolder.py > Meanwhile, I suppose Ty and I should try to come up with a proposal for > revising ZPublisher to be able to walk back on the user lookups but still > use the login form closest to the URL being accessed. -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Objects?
On Wed, 24 May 2000, Morten W. Petersen wrote: > Are there sheets with these (and similar) notes anywhere on zope.org? > If not, maybe there should be? There is the tips section - I've just added this. -- Stuart Bishop Work: [EMAIL PROTECTED] Senior Systems Alchemist Play: [EMAIL PROTECTED] Computer Science, RMIT ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Session Objects?
On Tue, 23 May 2000, Adam Pawliuk wrote: > Although there is several user characteristics that we would like to store in a >session object cached in server RAM, such as the user's unique system ID, greeting >name, etc; rather than having to hit the DB for this information on each request. >Basically we would like to use something similar to the session objects in ASP >(yuck), or Java JSP/servlet model. > > I've seen several Zope session products although they all seem to hook directly to a >DB and don't seem to provide memory persistence which is what we would like to use. > > Is there any existing products which do this? Boring old ZSQL Methods actually do exactly what you want - check out the caching options under the advanced tab (or the ZSQL docs on zope.org). The database will only be hit the first time you call the method with a given set of parameters, or when maximum time to cache expires. If your GUF hooks make use of ZSQL methods, you should have a look at tuning these too as it can be a big performance improvement. Actually - I tell a slight lie. The DB might be hit once for each thread, as the ZSQL cache is not shared between Zope threads. You need to be aware of this as a cached result may be refreshed in one thread but not yet expired in another and can give ambiguous results and unreproducable bugs if you are not aware of it when you are developing. -- ___ // Zen (alias Stuart Bishop) Work: [EMAIL PROTECTED] // E N Senior Systems Alchemist Play: [EMAIL PROTECTED] //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
