On May 15, 4:15 pm, Kenneth Kalmer <[email protected]> wrote: > > Hi Marc
Hello Kenneth, and thank you for answering so quickly > This for me is the crux of your entire mail. I'm going to do my best to > answer everything, but I'm going to summarize here quickly. > > You're effectively looking for a way to distribute large number of files (or > "work") to a variable number processes. These processes will perform said > tasks upon receiving "work" and idle around waiting for more. You want each > process to quickly pick up the next available piece of "work" to prevent > being wasteful. You got it perfectly (it took me one hour before I could write something that sounded clear to me, but I wasn't sure it was clear enough for readers) > Over and above this, you're looking at ruote as a way to orchestrate all of > this, so it happens in an orderly fashion. > > How right/wrong am I? 100% right - I'm seeing ruote as the conductor that tells each participant what to play and when to play it (well, said like this, it sounds more like a superior officer than a conductor ;) ) > I deleted the rest of your mail on purpose, I feel the lines between the > workflow engine and the actual bodies of work you're describing is blurred > too much. It happened to me to when I started using ruote, so don't worry. I've only been using ruote for a few weeks, in-between other things at work, and being quite unaware of workflows when I started, it still is kind of blurry for me. But I'm working hard to change that :) > First off. Are you sure ruote is the right tool for the job? > > Unless I missed something obvious (very possible) it appears to me that you > want to use ruote as a delegation/communication mechanism, and not as a > business process engine. There is, at least in my honest opinion, a distinct > difference here, and it will make you love ruote or hate it. More at the end > of the mail... I'm opened to any suggestions. Actually, a few month ago, before I started working on this project, I was learning a bit of Python and Ruby by curiosity. When I started it, one of the first thing I was asked was to choose a workflow engine that would help us manage the flow of data between our applications (the computing tool I mentionned before) - indeed, the business process management doesn't seem to be relevant, but as John says in his message, there is (or will be) a bigger business process behind. Anyway, a quick "workflow engine" google/wikipedia search later, I narrowed my selection to a few names, and visiting OpenWFE, I saw it was beeing ported in Ruby, and that sounded really appealing to me. Here's for the story :) > Second off. Why isn't a message queue like AMQP sufficient? > > AMQP is an awesome message queueing and routing protocol. I'm using it > extensively in our systems with ruote. I'm cleaning up an AMQP > participant/listener pair that I'll pass to John for merging over the > weekend. I was totally ignorant of AMQP until you mentionned it, so be sure I'll give a look at it. > How would I do this? > > I'd start using a pure AMQP publisher/consumer model, without ruote (sorry > John). The flow might look something like this: > > = Publisher = > > 1. File accepted > 2. Prepare "work" (JSON) and push into AMQP durable exchange > 3. Listen on another durable queue for feedback > 4. React to feedback > 5. Rinse and repeat > > = Consumer = > > 1. Subscribe to a message queue > 2. Receive "work" (JSON) > 3. Process instructions > 4. Send feedback to publisher > > I've reduced the problem significantly, but hopefully makes it more clear. > Ruote will only benefit you on the publisher side when your process needs > more than "if respone == true; job_succeeded; else job_failed; end". Ruote > strengths (for me) lies in the ability to react on the responses. Failures > change the entire course of the process, people get notified, actions have > to be taken. Ruote isn't the right tool to use for "watching a directory" > and "starting processes". It is a decision maker, it calls the shots. > > I can ramble on here until I need an ISBN for the reply, so I'm gonna stop. Ok. It's gonna take me some time to re-read that and possibly take a look at AMQP, so I'll probably get back to you later. > Please give us some more info so we can help you disect the issue even > further. Sounds like an interesting problem you have to solve, curious to > learn more. Got it, I'll try to make things clearer and make a clear distinction between the workflow/flow of data and the computing part of my problem. Thanks again for your answer, and have a great week-end (mine will be 6 days long, it should give me enough time to freshen my brains ;) ) Marc --~--~---------~--~----~------------~-------~--~----~ you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en -~----------~----~----~----~------~----~------~--~---
