RE: [Zope-dev] How to make Zope fail nicely under high load?
On Thu, 2004-02-12 at 07:19, Bjorn Stabell wrote: Toby wrote: One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog. I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances. (my apologies if this is obvious) Okay, for example, since it's a BBS we need to have quick access to: the latest poster the latest posting the latest registered member Create a zope product that doesn't keep a history (otherwise the ZODB will grow like nuts) and store these values in it whenever there is a new port/registration. Then just read them when you need to. Anyone see anything wrong with that approach? All of these are complex queries (in the case of the latest registered member it's not even a query but a brute-force search). What I think most ASP/PHP boards do is that they store these values in the database instead of querying for them all the time. Since many of these can be found through queries, they don't really need persistent storage, but updates need to be seen by all threads; thus the requirement for ZODB if we use ZEO, but the possibility to use (even faster) shared memory if we don't use ZEO. In effect we want to do something like this: def getLatestPoster(self): if not hasattr(self, '_v_latestPoster'): self._v_latestPoster = whatever_query() return self._v_latestPoster return self._v_latestPoster def setLatestPoster(self, poster): self._v_latestPoster = poster But instead of having thread-local variables we wanted to use Dieter Maurer's SharedResource in order to share the cache between threads: http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html Regards, -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Thu, 2004-02-12 at 10:06, Jan-Wijbrand Kolman wrote: Bjorn Stabell wrote: (..)For ZEO, my understanding is that we would still have to store the values in the ZODB, though. You are able to store *and* share data over ZEO Clients in a TemporaryStorage by making the ZSS serve this TemporaryStorage to its ZCs. You could mount this storage as temp_folder I guess, but any other mount point would work too. Cool, didn't know that. Not sure, but I'd expect so, whether this has significant perfomance advantages over a 'normal' ZODB served by the ZSS. regards, jw -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Wed, 2004-02-11 at 13:09, Jamie Heilman wrote: Lennart Regebro wrote: OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? As I've been saying, if you do that, they will reload repeatedly making the problem worse. If the images are fluf, and the user knows they are fluf, then *maybe* they won't reload, and your pages will simply appear ugly. But then you have to ask yourself, why am I sending fluf images that degrades the overall user experience of my application? Clearly there's more optimization that could be done to your application. Without knowing the app I'd probably say there are many ways that it can be made more efficient. For instance are your image urls being generated using Zope's acquisition? If so, don't do that. Another alternative is to move the static/fluf images out of zope and just let apache serve them. That's just small thing that you can do to reduce the load on zope. Things like squid, zeo, better hardware, re-analyze long running routines, etc are all possible without having to do the other gymnastics that have been discussed. -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope] Proposed installation changes for review
On Tue, 2003-03-11 at 08:48, Chris McDonough wrote: On Tue, 2003-03-11 at 00:24, Edward Muller wrote: Once zope is installed in /opt/zope-2.7.0 can it be moved without damaging the install say to /home/virtual/some.host.name/opt/zope-2.7.0 ? Yes. Its location is only meaningful to the instance files that need to find it. In our hosting setup some things get run in a chroot, some things can't... Currently zope get's installed in a chroot environment for anyone who wants a zope install. It must be a complete install since when the user restarts it he will be in his chroot environment. So I'd ideally like to install zope in a way where all of the core of zope is in one place ... say ... /opt/zope/version # (/opt/zope/2.7.0, /opt/zope/2.7.1, etc...) This I can hardlink/symlink into each chroot and make permissions 755 root/root. I think this will work. The only thing that might be a little weird is tracebacks generated by pyc files, as they may report the filenames of the Python modules where they were originally installed, instead of where they live now. There is some contention about whether this happens under Python 2.2, but I know it's true for Python 2.1 and prior. Well I can install zope in /opt/zope/2.7.1 (in the real root) and then when I symlink/hardlink it into a virtual host I can link it into that hosts /opt/zope/2.7.1 ... So that's not a biggie From there I would like to be able to install an 'instance', which is ... in my case meaning the data.fs, /Products directory, log files, etc, etc. The stuff that make this users instance theirs. When the install is happening, the script executing it would most likely be outside of the chroot ... but I guess it could be configured to chroot as well.. You would need to chroot the run of makeinstance currently as it encodes paths to software within the instance files that start Zope. So if you ran it outside the chroot it would work, but when the user logged in to the chroot, the paths to the software would be wrong. That's not a problem ... at least IIRC. I can chroot when creating the account in a shell script and execute custom setup scripts. I think this might be made configurable with a switch to mkzopeinstance (--sw_location=/some/path), though. I will add this to the tentative TODO, thanks. all thought that would be nice. I already have start/stop scripts to go through the users that have a zope install and chroot into that users 'host' and then start zope as that 'hosts' administrative user. These scripts will unfortunately need to change for Zope 2.7 unless we create some sort of backwards compatibilty layer for startup. Yeah. Oh well. They aren't that complex. :-) I wouldn't worry about the backward compatibility layer myself. I don't know if there is a great value add to it, aside from keeping users from going 'WTF happened?' :-) -- Edward Muller Interlix - President Web Hosting - PC Service Support Custom Programming - Network Service Support Phone: 417-862-0573 Cell: 417-844-2435 Fax: 417-862-0572 http://www.interlix.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope] Proposed installation changes for review
new-install-branch -f Zope The installation instructions in doc/INSTALL.txt has been updated on the branch; that file provides detailed information on creating an installation and instance. (A copy has been attached for convenience.) A sample installation/startup runthrough --- The below is the output from a sample install session. During this session, we install the Zope software to /tmp/zopehome and install an instance home to /tmp/insthome. [EMAIL PROTECTED] software]$ cd NewInstallBranch/ [EMAIL PROTECTED] NewInstallBranch]$ ./configure --prefix=/tmp/zopehome Configuring Zope installation Testing for an acceptable Python interpreter... Python version 2.1.3 found at /home/chrism/bin/python Python version 2.1.3 found at /home/chrism/bin/python2.1 Python version 2.2.2 found at /home/chrism/bin/python2.2 The optimimum Python version (2.2.2) was found at /home/chrism/bin/python2.2. - Zope top-level binary directory will be /tmp/zopehome. - Makefile written. Next, run make. [EMAIL PROTECTED] NewInstallBranch]$ make /home/chrism/bin/python2.2 inst/setup.py -q build_ext -i ... compilation output elided ... Zope built. Next, do 'make install' (or 'make instance' to run a Zope instance directly from the build directory). [EMAIL PROTECTED] NewInstallBranch]$ make install ... installation output elided ... Zope binaries installed successfully. Now run '/tmp/zopehome/bin/mkzopeinstance' [EMAIL PROTECTED] NewInstallBranch]$ /tmp/zopehome/bin/mkzopeinstance mkzopeinstance requires exactly one argument mkzopeinstance: Create a Zope instance home. usage: mkzopeinstance [options] directory Options: -h/--help -- print this help text -u/--user NAME:PASSWORD -- set the user name and password of the initial user [EMAIL PROTECTED] NewInstallBranch]$ /tmp/zopehome/bin/mkzopeinstance \ --user=admin:123 /tmp/inst [EMAIL PROTECTED] NewInstallBranch]$ cd /tmp/inst/ [EMAIL PROTECTED] inst]$ ls bin etc Extensions import inituser log Products README.txt var [EMAIL PROTECTED] inst]$ bin/zopectl start . daemon process started, pid=2947 [EMAIL PROTECTED] inst]$ bin/zopectl stop . daemon process stopped Changing the configuration via the config file -- The Zope configuration file uses the ZConfig API and file format. See the ZConfig documentation PDF at http://cvs.zope.org/Packages/ZConfig/doc/zconfig.pdf for more information. The default Zope configuration file will live in the instance_home's /etc directory and will be named 'zope.conf'. The Zope configuration file allows you to change the following configuration variables: instance home client home debug mode effective user production installation (on/off) locale number of zserver threads the python check interval (thread switch timing) zserver read only mode pid file name lock file name structured text header level publisher profile file (debugging) cgi environment dns server (hostname resolving in logfiles) ip address (for zserver bindings) http realm automatic quotation of dtml request data security manager stack size security policy implementation (C, Python) skip authentication checking skip ownership checking max number of session objects session add notify script path session delete notify script path session timeout minutes access rule suppress site root suppression database quota size read only database zeo client name logging (access, event, debug, filenames, to syslog, etc). network servers (http, dav, ftp, monitor, etc) databases (storages, database classes, etc.) ___ Zope maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Edward Muller Interlix - President Web Hosting - PC Service Support Custom Programming - Network Service Support Phone: 417-862-0573 Cell: 417-844-2435 Fax: 417-862-0572 http://www.interlix.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] Re: [Zope-dev] Proposed installation changes for review
On Mon, 2003-03-10 at 17:07, Paul Winkler wrote: A few questions / concerns listed below, otherwise it looks fine to me... On Mon, Mar 10, 2003 at 04:41:48PM -0500, Chris McDonough wrote: debug mode does this still toggle a whole bunch of things? production installation (on/off) what's this mean? network servers (http, dav, ftp, monitor, etc) is this where you set the ports? One thing I didn't see in your list is a way to extend the products path (currently I do that with $PRODUCTS_PATH). We need that. One more question: Does zopectl.py always detach from the terminal (booo) or not (yay) ? Or does it behave like z2.py and this depends on debug mode (booo) ? Actually I like the way z2.py detaches or doesn't detach. Perhaps a separate config option would be good to control this. -- Edward Muller Interlix - President Web Hosting - PC Service Support Custom Programming - Network Service Support Phone: 417-862-0573 Cell: 417-844-2435 Fax: 417-862-0572 http://www.interlix.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
