On Sun, Jul 29, 2001 at 11:28:01PM -0700, Bill Moseley wrote:
>
> I'm just looking for ideas:
>
> I want to build a server for the swish-e search engine. This would avoid
> the cost of forking and exec'ing a swish-e binary, plus avoid the startup
> costs of opening and parsing the header of the swish index file(s). The
> build process for swish-e builds a C library, and I have a working xs
> interface.
>
> A number of people at the open source conference recommended POE. But, I'm
> not sure if POE can help. One problem is that searches can take some time
> for large indexes. So the blocking could be an issue. Also, on errors
> swish calls exit(), so I'd still need to fork to protect the parent server.
The first probable solution that comes to mind-- and I have to admit I
don't know much about SWISH or any designs you've already worked out--
is several back ends spawned off the main server with POE::Wheel::Run.
P::W::Run does the usual pipe/fork/exec for stand-alone programs. It
also can execute functions in child processes with the usual stdio
pipes back to the parent.
In this model, the main process acts as a proxy between clients and
the back-end processes. That's extra overhead you may not want.
> So, I was thinking of a pre-forking (Apache like) design would be best,
> especially if there was a manager that would replace children that exit(),
> or could spawn/kill more children depending on load. That would avoid the
> forking for each request (search), yet still protect the parent from a
> child's death.
This sounds better. The child processes would answer requests
directly, letting the parent know which are busy so it can decide if
more need to be spawned or killed.
> I really like the looks of POE, but I'm not clear if POE can help.
>
> Any suggestions?
Build your dream design on paper, ignore system requirements and
limitations at first, and then find work-arounds or implementations
for the sci-fi stuff until it's feasable. :)
> I'm also haven't though about sharing or caching open index files between
> child processes. It would be nice to avoid separate copies of the same
> index file in different child processes.
Does SWISH-E load the entire index file into memory? The interface
seems to follow a pattern where data is iterated over on disk, but my
first exposure to it was just a few minutes ago.
-- Rocco Caputo / [EMAIL PROTECTED] / poe.perl.org / poe.sourceforge.net
