6 months of labour
Hi all Well, my last 6 months of loving labour has finally drawn to a close with the launch of my site. http://www.traveljury.com is supposed to be an IMDB style travel site, which will contain the type of information you would find in Lonely Planet or Rough Guides, but contributed by a community. At the moment, it's pretty empty, but you've got to start somewhere :) Thanks to everybody who has pointed me in the right direction (and given their assurances about chosen directions!) I would appreciate any feedback, bug-reports and suggestions from all of you experienced webmasters out there, and any information or pictures you can contribute to traveljury would be greatly appreciated. I'm off to South America in a couple of weeks - laptop in hand - to start populating the database. thanks again Clinton Gormley
Sharing memory between children
I had a look at the memory usage of my apache/mod_perl 1 processes, and was alarmed to find that only 3Mb of 25Mb processes was being shared (and that's straight after startup) I have gone to great lengths to (1) Preload modules in my startup file (2) Load shared config data during server startup so that that info can also be shared The only module I load during child init is a wrapper to handle database connections so that database handles don't get shared between children. I've read the memory optimization bit on the mod_perl web site and am complying with all of those suggestions. My next step is to remove all of the modules and start adding them in again to see which has such poor memory usage. What numbers should I be expecting to achieve on a reasonably large web site (http://ww.traveljury.com - still a test site at the moment) and are there any gotchas that you know about that aren't mentioned on the mod_perl site? thanks Clint
Re: Sharing memory between children
On Mon, 2003-06-16 at 12:12, Clinton Gormley wrote:
I had a look at the memory usage of my apache/mod_perl 1 processes, and was alarmed to find that only 3Mb of 25Mb processes was being shared (and that's straight after startup)
I have gone to great lengths to
(1) Preload modules in my startup file
(2) Load shared config data during server startup so that that info can also be shared
i've pulled everything out and started adding it back in, and it looks like nothing much gets shared other than the perl interpreter.
When I load nothing, the size of my processes starts at 4204 (2504 shared) . When I load everything, the size rises to 17144 (3920 shared) - not much is being shared...
Am I being really dumb here? What am I missing?
thanks
Clint
My Apache compile options :
_
./configure --prefix=/usr/local/apache-1.3.27_perl_ssl_mm \
--activate-module=src/modules/perl/libperl.a \
--disable-module=userdir \
--disable-module=asis \
--disable-module=include \
--enable-shared=info \
--enable-shared=status \
--enable-module=so \
--enable-shared=env \
--enable-shared=setenvif \
--enable-shared=negotiation \
--enable-shared=autoindex \
--enable-shared=access \
--enable-shared=auth \
--enable-shared=cgi \
--enable-shared=actions \
--disable-module=imap \
--enable-module=ssl \
--enable-shared=ssl \
--disable-rule=SSL_COMPAT
_
mod_perl compile options
_
perl Makefile.PL \
APACHE_SRC=../apache_1.3.27/src \
DO_HTTPD=1 \
USE_APACI=1 \
PREP_HTTPD=1 \
EVERYTHING=1
_
My startup file (which is required in the apache config):
_
use strict;
use warnings FATAL = 'all', NONFATAL = 'redefine';
use Apache();
use Apache::DBI();
use lib Apache-server_root_relative ('projects/traveljury/modules');
$Apache::VMonitor::PROC_REGEX = join |, qw(httpd mysql);
## Set DB connection credentials
use TravelJury::Startup::InitDBI();
## Load constants before using Apache::DBI so that $dbh from parent process doesn't get shared
use TravelJury::Startup::Constants();
## Any initialisation which needs to occure during server startup
use TravelJury::Startup::ServerInit();
use TravelJury::Gateway();
Apache-push_handlers(PerlChildInitHandler ='TravelJury::Startup::ChildInit');
_
Re: Sharing memory between children
On Mon, 2003-06-16 at 13:03, Ged Haywood wrote: I had a look at the memory usage of my apache/mod_perl 1 processes, and was alarmed to find that only 3Mb of 25Mb processes was being shared (and that's straight after startup) I see about the same on my own server when processes get bloated, but I don't let the processes live to serve large numbers of requests so eventually they're killed off and replaced with new, smaller children. But this is happening right at startup... before any bloating. Sounds a little on the high side, but not outrageous. You're putting quite a lot of stuff into your httpd, do you really need it all? Sorry - that includes the perl processor. I don't think so. If it's a problem for your machines you might want to consider a light proxy front end + mod_perl backend. I will be using a lightweight frontend, but clearly still want to optimise my mod_perl processes. But I just don't get why I'm seeing the numbers I'm seeing. I preload everything, and yet the shared memory rises all of 1Mb, while the unshared memory rises 12Mb, before the first request has been served! Surely, at this stage, pretty much everything should stil be shared? yours bemusedly Clint
Cross Site Scripting
On Tue, 2003-03-11 at 06:03, Stas Bekman wrote: Changes since 0.7 * prevent cross-site scripting, now HTML-escaping the request field In Stas' Apache::VMonitor announcement, he mentions changes to prevent cross site scripting. This is a concern for me at the moment, because I'm building a site which will allow people to submit copy (to be displayed to other users) and I would like them to be able to use HTML and include links to other sites (much like slashdot). Do any of you have any ideas about good techniques to prevent CSS (and I don't mean those div elements) in this scenario? I've read the articles on cert.org (http://www.cert.org/tech_tips/malicious_code_mitigation.html) and apache.org (http://httpd.apache.org/info/css-security/encoding_examples.html) thanks Clinton Gormley
Optimising cache performance
I'd appreciate some feedback on my logic to optimise my cache (under mod_perl 1) I'm building a site which will have a large number of fairly complicated objects (each of which would require 5-20 queries to build from scratch) which are read frequently and updated relatively seldom. I'm planning a two level cache : 1) Live objects in each mod_perl process 2) Serialised objects in a database The logic goes as follows : NORMAL READ-ONLY REQUEST 1) REQUEST FROM BROWSER * Request comes from browser to view for object 12345 (responding to this request may involve accessing 10 other objects) 2) PURGE OUTDATED LIVE OBJECTS * mod_perl process runs a query to look for the ID's of any objects that have been updated since the last time this query was run (last_modified_time). * Any object IDs returned by this request have their objects removed from the in-memory mod_perl process specific cache 3) REQUEST IS PROCESSED * Any objects required by this request are retrieved first from the in-memory cache. * If they are not present, * the process looks in the serialised object cache in the database. * If not present there either, * the object is constructed from scratch the relational DB. and stored in the serialised object cache * retrieved object is store in the in-memory live object cache 4) TRIM LIVE OBJECT CACHE * Any live objects that are not in the 1000 most recently accessed objects are deleted from the in-memory cache UPDATE REQUEST Steps as above except : 3a) UPDATING OBJECT * Any objects that are modified * are deleted from the serialised object cache in the DB * and are deleted from the in-memory cache for this mod_perl process only This means that at the start of every request, each process has access to the most up to date versions of each object with a small (hopefully) penalty to pay in the form of the query checking for last_modified_time. Does this sound reasonable or is overkill many thanks Clinton Gormley
Re: Optimising cache performance
Thanks for your feedback - a couple more questions First, I'm assuming this is for a distributed system running on multiple servers. If not, you should just download one of the cache modules from CPAN. They're good. For now it's not a distributed system, and I have been using Cache::FileCache. But that still means freezing and thawing objects - which I'm trying to minimise. I suggest you use either Cache::Mmap or IPC::MM for your local cache. They are both very fast and will save you memory. Also, Cache::Mmap is only limited by the size of your disk, so you don't have to do any purging. When you say that Cache::Mmap is only limited by the size of your disk, is that because the file in memory gets written to disk as part of VM? ( I don't see any other mention of files in the docs.) Which presumably means resizing your VM to make space for the cache? You seem to be taking a lot of care to ensure that everything always has the latest version of the data. If you can handle slightly out-of-date Call me anal ;) Most of the time it wouldn't really matter, but sometimes it could be extremely off-putting If everything really does have to be 100% up-to-date, then what you're doing is reasonable. It would be nice to not do the step that checks for outdated objects before processing the request, but instead do it in a cleanup handler, although that could lead to stale data being used now and then. Yes - had considered that. If you were using a shared cache like Cache::Mmap, you could have a cron job or a separate Perl daemon that simply purges outdated objects every minute or so, and leave that out of your mod_perl code completely. I see the author of IPC::MM has an e-toys address - was this something you used at e-toys? I know very little about shared memory segments, but is MM used to share small data objects, rather than to keep large caches in shared memory? Ralph Engelschall writes in the MM documentation : The maximum size of a continuous shared memory segment one can allocate depends on the underlaying platform. This cannot be changed, of course. But currently the high-level malloc(3)-style API just uses a single shared memory segment as the underlaying data structure for an MM object which means that the maximum amount of memory an MM object represents also depends on the platform. What implications does this have on the size of the cache that can be created with IPC::MM thanks Clinton Gormley
Re: New module : Apache::Session::Manager
Hi Enrico After posting, I looked at Apache::AuthCookie for the first time - don't why I hadn't looked at it before - to find that I was, indeed, duplicating a lot of work that has been done before. Didn't I feel foolish! It looks like Perrin may have to write a ...why not to write your own session manager article... to compliment his Templating article. Apologies for this. I'll haul my sorry ass back to my machine now. Clint
New module : Apache::Session::Manager
I've written a new module as a wrapper around Apache::Session which provides short term (ie session) tracking (which is something that Apache::SessionManager and Apache::SessionX provide), but this adds long term user tracking, and methods to aid login, logout etc. It is meant to be subclassed, so that you can add your own database access methods to the module - full examples included! I would appreciate feedback, not least of all, what it should be called! Apache::SessionManager exists already, and I realise that calling this module Apache::Session::Manager will cause confusion, so please... README is below. The module is available from : http://cpan.perl.org/modules/by-authors/id/D/DR/DRTECH/Apache-Session-Manager-0.02.tar.gz thanks Clinton Apache/Session/Manager version 0.02 === Apache::Session::Manager provides a wrapper around Apache::Session which adds the following : * Short term session tracking using cookies or URLs (query string / POST data rather than URL munging) * Long term user tracking with cookies and query string * User recognition through query string (eg from an emailed link) * Login, logout, create new account, remove cookies - all operating with your favourite database and schema! It needs to be subclassed by your own module, which provides the methods to interact with your database - ie you can integrate this module into you own database schema. Apache::Session::Manager is not a drop in module - it requires a bit of work on your part to make it work - but once that work is done, it makes a lot of user and session management easy. Any website that wants to have any sort of long term relationship with a user needs to be able to track a user through a session, allow the user to register, and issue them with a password challenge before allowing them to view restricted pages... which is exactly what this module does. All you need to provide is 6 subroutines which speak to your database to perform a few simple functions (looking up a username etc).
Stylistically : To inherit, or to contain
When using Apache::DBI...
Am I correct in this: Apache::DBI can only really do its stuff when you perform a DBI-connect, so by calling $dbh = DBI-connect(..) during PerlChildInit and then never trying to reconnect, you are defeating the purpose of using Apache::DBI. To expand on this, when Apache::DBI intercepts a connection request: * it returns a stored live handle which matches the connection credentials * checks that the handle is indeed still live, and if it isn't, reconnects automatically (potentially destroying any $sth's that you have prepared and retained) So would this be the right balance between efficiency and safety: 1) use Apache::DBI 2) Do my $dbh = DBI-connect() at the beginning of each request (ie as early as required during each request) 3) use my $sth = $dbh-prepare_cached() once as early as required during each request This way, we're not expecting the DBI handle to die during a request, and so avoid calling a few lines of code to -connect and -prepare_cached several times during each request. Have I got the right idea? thanks Clint
RE: Excellent article on Apache/mod_perl at eToys
Thanks to all for the kind words. This article actually went up a little bit before it was supposed to, and there should be a revision going up soon with some grammatical fixes and a set of graphics to illustrate parts of it. I'll post a follow-up when that happens in case anyone wants to go and look at the pretty pictures. Hi Perrin What I'd love to see is the avg spec and numbers of machines in each section. So how many proxy, mod_perl and search servers were required to give the phenomenal performance you managed to achieve. thanks Clinton Gormley
RE: ANNOUNCE: Apache::OpenIndex
OpenIndex provides a file manager for an Apache modperl web site using a web browser. Looks good George any plans to add better authentication (eg certificates) and to set it up for secure SSL file transfers? Clinton Gormley
UK based modperl agency required for Virgin
Title: UK based modperl agency required for Virgin You have all supported me tremendously through building my first mod_perl web site : http://www.orgasmicwines.com - many thanks. Now Orgasmic Wines has raised its first round of capital from Richard Branson, and we are becoming Virgin Wines ltd. The existing web site and infrastructure is being viewed as a prototype, and everything needs to be redeveloped. We want to outsource as much as possible, and we are looking for agencies who will work as partners with us to create and run the new Virgin Wines. I would very much like to keep the site running on a Linux/apache/mod_perl platform - however, every major agency we talk to in the UK seems to use NT and ASP. So, calling any major UK based web agency with ecommerce experience building sites in mod_perl - I want to hear from you. please reply by email: [EMAIL PROTECTED] thanks Clinton Gormley Webmaster, Orgasmic Wines http://www.orgasmicwines.com
SegFaults caused by Apache::Cookie during ChildExit
I am using a home-baked session manager on my web site. I clean up expired sessions by called a child exit handlder and this all worked rather well. However, we have recompiled Perl, Apache, mod_perl and Perl modules with pgcc and a different configuration (removed all modules we didn't need), and now I get a SegFault when Apache::Cookie-new is called during a ChildExit. I use Apache::Cookie in Authorization and PerlHandler phases without a problem. Not sure whether this problem is caused by the compiler or the different configuration at compile. Any ideas of starting points? Thanks Clint
Compiling mod_perl with different versions of Perl
I have a working version of Perl and mod_perl on a live system at the moment. Since installation, I have installed pgcc. I want to recompile Apache and mod_perl to use a proxy front end, and would like to do this using pgcc and its optimisations. Stas says in his guide that Perl and mod_perl should be compiled with the same compiler, so it seems that I should recompile Perl with pgcc at the same time. I would prefer to compile Perl into a separate place to my current Perl. My question is, when I compile mod_perl, how do I specify which version of Perl it should refer to? Does it matter at compile time, or is it sufficient that the default Perl binary is the correct one at run time? Does the binary you use to run Makefile.pl have any influence? Any help greatly appreciated Thanks Clinton Gormley
RE: Memory problems
Why is it that my memory usage is going up and up, and shutting down the two major consumers of memory (Apache/mod_perl and MySQL) don't reclaim that memory? I am running RedHat 6, with Apache 1.3.9, mod_perl 1.21, and MySQL 3.23a. As an aswer to my question to anybody in a similar position, it turns out that Linux 2.2.11 (which I am using) is known to have a memory leak, which is sorted in 2.2.12. Clint
Memory problems
Hi I had huge problems yesterday. Our web site made it in to the Sunday Times and has had to serve 1/2 million request in the last 2 days. Had I set it up to have proxy servers and a separate mod_perl server? No. DOH! So what happened to my 1Gig baby? It died. A sad and unhappy death. I am in the process of fixing that, but if anybody could help with this question, it'd be appreciated. I am running Apache 1.3.9, mod_perl 1.21, Linux 2.2.11, mysql 3.23.3-alpha. What happened was this: My memory usage went up and up until I got "Out of memory" messages MySQL bailed out. Memory usage was high, and the server was swapping as well. So I thought - restart MySQL and restart Apache. But I couldn't reclaim memory. It was just unavailable. How do you reclaim memory other than by stopping the processes or powering down? Is this something that might have happened because it went past the Out of Memory stage? Thanks Clint
Logging Session IDs from environment variables
Hi all I have asked this before, but I still haven't managed to shed any light on it, so I was hoping that somebody might be able to shed some more light. (While you're about it, have a look at the site we have just launched : http://www.orgasmicwines.com - mod_perl mysql based site.) If I store my sessionID in an environment variable (or in $r-notes), I can see it in other Apache:: modules, but when I try to log it using the the CustomLog directive in httpd.conf, the environment variable (or $r-notes) is blank. Does this have anything to do with the fact that I'm setting the ENV variable in a PerlAuthenHandler, before %ENV is set up? And that shouldn't affect $r-notes, should it? Any help greatly appreciated Many thanks Clint
