Howdy-- This is an off-topic post-- feel free to ignore it. I know the folks on this list are active web developers and generally smart people and this is a problem that has been vexing me for some time now-- I am at the point of having to do _something_ and I could use some advice... I am working on a large application which is built on top of postgresql. The system is currently spread out over several hosts and will (relatively soon) grow to handling many many requests per short interval. (As an idea of what the site is/does, think "listings"). Overall everything works fine. An issue to contend with is fairly frequently portions of the data expire as old listings fall off, and new listings are added (in hundreds of categories). Another minor wrinkle is that the listings need to be accessible from one of about 5 sort orders. Currently the lists of listings are statically-cached on the system. I have a back-end process that runs periodically to rebuild the static caches which basically preprocesses the listings and prepares html static "chunks" that can then be included at the appropriate time. This I have found to be pretty much the fastest approach from the user's point of view, and probably, I'm thinking, the least of the evils as far as system load goes. But I'm concerned about several things and so I keep mulling over some other ideas and this is what I need advice on... At the moment, rebuilding the static caches is not a big issue- it doesn't take long and its possible to get it all rebuilt in less than half an hour. I'm concerned though what the next step will be when the number of listings grows to the point that half and hour (or an hour) isn't enough. One ameliorating factor, I suspect, will be that the number of hosts needed to serve users will continue to grow to the point that the sensible thing will be to break up the site to serve up only a subset of categories per host (or host cluster) and thus the load will be spread out that way. This will somewhat reduce the amount of overhead spent rebuilding category listings for a bit-- but ultimately, in my reasoning-- the system will reach a point where it is pretty much continuously rebuilding category listings and putting a constant strain on the main "live" database. So somewhere down the road another solution will have to be found. What other solutions are there? I have considered an alternative approach to maintaining category listings-- instead of pre-generating entire pages as static chunks- what about using something like gdbm (or whatever) to make tiny little category databases that store pre-generated __rows__ of each page, and then pull "x" of them at a time to make a page... has anyone tried something like this? What is the performance like? Is there much overhead associated with constantly opening and closing these types of database files (I'm not real familiar with non-postgres databases)...? Would this make it easier to deal with sorting? Ideally it would be nice to maintain only one body of category data and just resort it on the fly (or maintain indexes) as needed. How much efficiency would be lost versus time to serve up pages? I know there has to be a trade-off someplace but how bad will it be? Is there some other solution I'm missing? Is it faster to access something pre-generated from the disk (doing asynch system calls vs. lighter-weight network db access calls) to get data? I'm cool with the idea that the list of listings does not have to be up to the absolute minute accurate- reasonably accurate (say half-hour, hour, two-hours) is close enough... What are you experiences with this type of situation? Thanks for any help or guidance anyone can offer-- there don't seem to be a lot of places "out there" that one can go to get this type of information. Regards, John Whitten [EMAIL PROTECTED] Wizard.Org, Inc. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
