Hi,
I have the following requirement, and I am seeking your input.
1. web-based users make requests for data which are put
in a queue
2. these requests and their status need to be in a database
so that users can watch the status of the queue and their
requests in the queue
3. a set of servers process the requests for data from the
queue and put the results in a results table so that
users can view their data when their requests are done
QUESTIONS:
* What queue mechanism would you use, assuming all of the
writers and readers are on the same system?
(IPC::Msg? MsgQ?)
* How about if the queue writers were distributed, but the
queue readers were all on one machine? (RPC to insert into
the above-mentioned local queues?)
* How about if the queue writers and queue readers were all
distributed around the network? (Spread::Queue::FIFO?
Parallel::PVM? Parallel::MPI? MQSeries::Queue?)
* What Perl server-building software on CPAN do you recommend,
and why? (Net::Server? Net::Daemon? POE?)
I started working with Net::Server, but it seems focused
on being a network request multi-server, not a queue working
multi-server.
* Would you implement it as many peer-level servers waiting on
a single queue? or a single parent server waiting on the queue,
dispatching queued work units to waiting child servers?
QUICK AND DIRTY SINGLE-SERVER SOLUTION
I implemented a quick-and-dirty single-server solution, where
I use a single server to process requests. I simply poll the
request table in the database once a minute for new requests,
and if they exist, I process them.
Now I am looking to upgrade this for higher throughput (multiple
parallel servers), lower background load (no polling during quiet
periods), and lower latency (immediate response to queue insertion
rather than waiting for the next poll interval).
MY HUNCHES
I think I'll use IPC::Msg as the queue because the queue readers
will all be on one machine. I'll also have to implement a simple RPC
server (using Net::Server) to perform remote insertions into the
local queue. If this seems too rough, I'll probably install the
Spread Toolkit and use Spread::Queue.
I currently think I'll keep working with Net::Server to see if I
can use it to process a queue rather than listen on a network port,
but I'm not sure that this is the right use of the module.
I may end up ditching this effort and just have a set of parallel
servers all waiting on the queue. The queue mechanism itself will
work out who gets to work on which request.
Any input?
Stephen
