Hi, I thought that MQ is quite 'heavy' for my needs Basically the whole system allows users to request 'reports' and those reports can rung for days (or hours).
Instead of having 'one' persistent queue (or a service) such as provided for typical MQ implementations -- I thought just to have a 'little' message queue that for each user -- that would be visible from their profile (since there is no 'admin' interface to administer 'jobs' as in typical message queues) So I just simply planned a process scan each profiles queue, and send the job requests in round robin to available twisted servers. You might ask -- well you will store events to be dispatched in a table with profile IDs -- so that's what message queues do anyways. However, the system is built such there is more than one database instance (where users are destributed based on a hashing scheme acrsoss multiple databases) -- so it is not simply one table (that's also why there is a need for multiple Twsited servers as they would serve their own 'partitions' in the system) On the other hand ... may be I need to look at open source and free MQ solutions to see if I is easier to Thank you again for all the suggestions so far, I am writting up a small summary of the replies for furthere analysis. On Sat, 10 Jan 2009 14:27:18 -0800, "Don Smith" <donwsm...@gmail.com> said: > Why not use a Message Queue? I developed a very similar program to what > you > have described, only each agent would connect to the Queue to get their > data > to work on. However, I think something like Apache ActiveMQ or AmazonSQS > would be a better approach. > > =Don > > 2009/1/9 V S P <torea...@fastmail.fm> > > > Thank you for the replies. Wanted to make sure I was not missing > > a 'twisted goodie' > > > > What is ZeroConf? Can you please point me to the project page > > (I searched for some but did not find something twisted/python > > specific). > > > > my client will be a program that reads rows of 'requests' from the > > database and dispatch them to available twisted servers (I call it > > Dispatcher). Then twisted servers will use the pool of worker threads > > (with help of Ampula plugin I guess) to do the work (that will take > > hours > > of not more). > > > > So there will be just one program that needs to be > > aware of all the available servers in my scenario. > > > > Also I can see that some people might have needed a 'proxy' > > that sits infront of the cluster of twisted servers and that proxy > > has some balancing mechanism. And if there is any body that has that > > solution I would like to know as well. > > > > > > > > thank you again. > > > > > > On Fri, 9 Jan 2009 22:29:22 -0500, "Jarrod Roberson" > > <jar...@vertigrated.com> said: > > > 2009/1/9 V S P <torea...@fastmail.fm> > > > > > > > Hi, > > > > I am sorry I am reasking the same question I asked before > > > > (but I did not get any replies) > > > > > > > > Basically wanted to know -- for running Twisted on multiple servers > > > > > > > > is there such a thing as a "Naming" server that > > > > would register all the currently running twisted servers > > > > > > > > -- so that asynchronious event dispatching can be done in some > > > > round-robin to available twisted servers > > > > > > > > > > > > we used ZeroConf for autodiscovery that way the clients can decide for > > > themselves which server to connect to. > > -- > > V S P > > torea...@fastmail.fm > > > > -- > > http://www.fastmail.fm - And now for something completely different┘ > > > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python@twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > -- V S P torea...@fastmail.fm -- http://www.fastmail.fm - IMAP accessible web-mail _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
