Echoing the others so far, I'd use JMS. But if you want a CF-only solution, I'd probably use a pub-sub model. Have each worker come up and register itself with the scheduling server saying "I'm ready to work and here is my contact url". The scheduler server keeps a local store of the known workers and the urls to send work notifications to.
When a job becomes available, queue it up locally and then broadcast the availability message to all the worker processes in parallel (cfthread to spawn cfhttp connections) with the id of the job to take and the url to get to take it. When a worker gets the message it looks to see if it is currently busy. If it is, it ignores the message. If it is not busy, it tries to contact the scheduler with the job token and "take" the job. The scheduler locks the job table, assigns it to one of the incoming requests, then moves it from the pool of available jobs. You would probably want to have a recheck interval to make sure that the job got picked up and if it didn't, broadcast it again and also have the worker services notify when the job was complete, success/failure, etc. One other thought I had was to use Railo's Server scope as a shared queue between scheduler and the workers instead of a database. The upside would be that it would avoid the db locking but on the downside it wouldn't necessarily have the state persistence in case of server errors, etc. I'd probably favor the db over the server scope for reliability but you might be able to find a decent mix of the two where you persist every change to the server scope queue into the database, etc. Judah On Mon, Aug 16, 2010 at 6:41 PM, Marc Esher <[email protected]> wrote: > Hi all, > Since it's been so long that cfcdev has had a message, I know you'll > all put down the beers, put the spouses and kids to bed, and jump at > the chance to pitch in here. > > I'm whipping up a prototype app -- not for work, not for homework -- > that I'm thought-experimenting about "how would I run this in a > multi-server environment?". I'm not looking for code samples or > anything really specific, just options. > > Imagine you have a *lot* of background work to do. You have an unknown > -- and potentially elastic -- set of servers with which to do the > work. It's probably easiest to think of it as if you were running this > thing on Amazon EC2 or another service. You have a "queue" of work to > be done. You have worker servers to do the work. > > What are your options for scheduling that work with ColdFusion? In an > ideal world, there'd be a shared queue, and workers would "take()" off > of that queue, with zero contention... i.e. as soon as one server > takes off the queue, that task is immediately unavailable to another > server that also attempted to take it. > > I'd normally use java for this, but I want to use CF to see how badass > it can get in a situation such as this one. And this means no java > Timers, TimerTasks, or my dear friends the ExecutorService and its > wonderful relations... straight CF. I'm not opposed to event gateways, > though I'd really like to stay away from a JMS server if I can help > it. Importantly, it needs to be fairly easy to debug, which is > always a problem in cases such as these. > > Thoughts? > > Oh, I know: this is like when the boss comes in and says "we need it > fast, we need it simple, and we need it now". I have a good idea of > how I'd do this with java, but I'm mostly interested in what CF could > provide. > > Thanks! > > Marc > > -- > You received this message because you are subscribed to the Google Groups > "CFCDev" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/cfcdev?hl=en. > > -- You received this message because you are subscribed to the Google Groups "CFCDev" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/cfcdev?hl=en.
