On Mon, Nov 14, 2011 at 5:15 AM, Marvin Humphrey <[email protected]> wrote: > On Mon, Nov 14, 2011 at 12:59:30PM +0200, goran kent wrote: >> If these two could be addressed: >> >> - LucyX::Remote::SearchClient to perform remote searches in parallel >> as opposed to serially, and > > As of this moment (r1201554), ClusterSearcher's interface, documentation, and > result aggregation logic are done.
Wow Marvin, that's fabulous! >> - LucyX::Remote::SearchServer to fork on each new client or otherwise >> allow multiple search clients at once (ie, typical TCP/IP >> client/server behaviour) > > Hacking fork() into the SearchServer#serve loop is probably the logical choice > to solve your immediate problem -- we aren't giving you much to work with, > after all! But hiding a fork() call in a library function can't be our final > solution for Lucy -- that takes SearchServer from dubiously architected to > laughable. :) Laughable, and as you point out not right for Core, but it might actually work just fine for 10-or-so of requests per second. Fork is fast on Linux, and we don't actually want too many concurrent requests happening at the same time. We're presuming that these searches are processor bound, so it would hurt us to have too many more than we have cores. But rather than forking in the handler, I think it might be easier to write a wrapper that does the fork. You might even be able to just configure "inetd" to launch a new server every time you get a search request. If that doesn't handle it, POE (as Goran mentioned earlier) wouldn't be a bad choice. Or perhaps http://search.cpan.org/~rhandom/Net-Server-0.94/lib/Net/Server.pm. If you use one of those two, you might as well prefork. The idea is that you fire off a bunch of servers, all listening on the same port, and they fight to see who accepts first. You need a manager process that makes sure you have the right number of children, and to restart them in case they die. It's very smooth, and should work out of the box with Marvin's code. Good luck (he calls from the sidelines while Marvin does the real work), --nate
