On Sat, Jul 17, 2010 at 10:22:20AM +0100, David Greaves wrote: > > Thanks for taking the time to consider this John - I'm trying to > explain why we need it and looking for the best way to keep it > upstream. > > As a focus: my objective is to have a participant-less Ruote at the > centre of a large group of machines; each remote machine will have > one or more Participants running in ruby, python or perl and > communicating with Ruote via AMQP. > > That group of machines/Participants will expand over time; some will > move from development to staging to production. I don't want the > codebase on the central machine to change at all as remote teams > deploy participants and processes.
OK. > On 17/07/10 02:59, John Mettraux wrote: > >On Fri, Jul 16, 2010 at 07:06:54PM +0100, David Greaves wrote: > >>On 15/07/10 15:31, John Mettraux wrote: > >> > >>>There is also the case where a participant emits work towards an engine > >>>(launch). > >>OK. This involves passing in a process definition which of course defines > >>and names a number of process steps. Less work-y but I concur. > > > >Hello David, > > > >Less "work" ? How do you launch workflows ? > Well, I see some reasonably distinct tasks. > 1) is passing a workitem; this is pure 'work'. > 2) is passing a process definition. This leads to 'work' being done but > does nothing tangible until a workitem is passed (1) When launching a process instance you are passing implicitely an empty workitem. Of course you can pass the fields (payload) of the initial workitem in the launch parameters. > 3) is passing a participant name mapping. This says where 'work' can > be done in a distributed system and steps in (2) direct (1) > > But as I said - I agree with you. > > (...) > > >Yes, this is were I don't want to wander (ie I don't want to have the > >responsibility to maintain it). > > > >Pushing messages over AMQP is simple enough. Building a RPC framework on top > >of it... > OK - that's 'done' over here and very separate. > It's also pretty trivial if you scope it to be service RPC and not object-RPC. So you have implemented a RPC framework on top of AMQP ? > Functionally it could be *identical* to the workitems queue after > all, I provided a 5-line patch to multiplex them. > > Either way it could be extended to use the 'reply_to' that is a > standard AMQP feature to publish a response. > > If it helps I'll volunteer to co-maintain the AMQP library. We're > pretty committed to it from a MeeGo PoV anyway. Do you mean the ruote-amqp library ? > I recognise that I've only been about for a few weeks so I know this > will take time where I hope you can build up some confidence and for > me to learn how best to tailor solutions for Ruote. > > >>There's still need for a cancel operation (and we've identified a possible > >>progress-query operation) on a wfid(?) in an AMQP Remote Participant and I > >>propose using a simple RPC over AMQP framework based on Kenneths design. > > > >I will add the "cancel" message to the "work" messages listed above. It is a > >must. > OK. Still have to work on that one. Sorry, I had/have other priorities. > I think this will look like this : http://wiki.meego.com/images/Boss9.png > > >>>Let's register a participant under a regex : > >>> > >>>engine.register_participant 'amqp-.+', RuoteAMQP::Participant, 'queue' > >>>=> 'y' > >>> > >>>All the workitems for a participant whose name begins with 'command-' > >>>will go there. > >> > >>No offence but ... ugh ! > > > >Replace "amqp-" with "do_" and there is no exposition of AMQP at all. > > But now you have 2 places which lookup and dispatch participants? The Engine > and > (in this case) the remote participant. > > So for do_onhost_a, do_onhost_b and do_onhost_c where the true > target is on different machines then the workitem must be pulled off > y, examined and pushed onto queue a, b or c. > > And if I decide to take a local participant and implement it remotely via AMQP > then I have to change any process that uses "notify customer" to "do_notify > customer" > So this pattern doesn't fit my distributed participant model. If you have a unique participant in ruote and all the multiplexing on the other side of AMQP, the "lookup and dispatch" activity on ruote gets seriously... > >Remember "engine.register_participant x, y, z" is not part of a process > >definition. It's configuration logic happening on a different timeline. > Yes. > I just don't see why it can't happen sporadically during the life of > the core deployment? I didn't mean it shouldn't happen during deployment, I'm trying to avoid to pollute the work channel that is AMQP with "configuration" messages/RPC. > >>... although, in the limit: engine.register_participant '.+', > >>RuoteAMQP::Participant, 'queue' => 'y' but it would be a bit blunt and I'd > >>just move the mapping to the thing handling queue y.... not nice. > > > >In my small one line example, I force the queue on the participant. You could > >write logic to determine the queue based on the workitem content (or simply > >the workitem.participant_name). > Yes. I could just force all AMQP participants to use the > workitem.participant_name as the AMQP queue name. > > (...) > > >OK, ok, replace 'amqp' by 'gucci' and you've got the elegance and no AMQP > >exposition (even loosely). > so > > sequence do > gucci variant => build > gucci variant => test > gucci variant => package > gucci variant => release > end > > vs > > sequence do > build > test > package > release > end > > Is that what you meant? If all the participants are AMQP participants it will be strictly the same thing. engine.register '.+', RuoteAMQP::Participant > >OK, "please have a smart mapping" vs "I just want to add a mapping". > So again... not 'smart'. > Dynamically adjustable Both approaches are dynamically adjustable. > On a remote machine in python I'd run: > p = MyPart(ruote_queue="package", amqp_host="amqpvm", amqp_vhost="ruote") > p.register("package", {'queue':'package'}) > p.run() This call is actually registering and running ? > I could then launch the 2nd pdef above. engine.register '.+', RuoteAMQP::Participant OK, I hereby declare that my command of the english language is not sufficient enough to let you see the benefits of my approach. Instead of losing time and energy, I will try to help you realize your stuff. But be warned, if your approach is too rococo, I won't let it go into the 'official' ruote-amqp, which doesn't prevent you from managing a fork. > >>>=== B.0 programmatically adding > >>> > >>>You have access to the engine. You can register participants on the fly > >>>: > >>> > >>>require 'my_participant' engine.register_participant 'toto', > >>>MyParticipant, 'flavour' => 'vanilla', 'position' => -2 > >> > >>How does this differ from: if item.has_key?('register') return unless > >>item.has_key?('name') register_participant(item["name"], > >>RuoteAMQP::Participant, item["options"]) other than the message transport? > >>I *think* it's very similar - I like it. > > > >As said, the need to build a custom RPC framework on top of AMQP. HTTP is > >sweet with its request/response system. > I could understand this if I was coming to you and asking for AMQP > and Ruote was an HTTP framework. > But all of this work is 100% inside RuoteAMQP.... I don't see the > need for an HTTP RPC solution inside the RuoteAMQP library? > > As an aside, AMQP is *explicitly* designed to support > request/response - RPC is built in. Not only that it doesn't matter > if the client/server are not running at the same time. Additionally > you can have 20 workers (on different hosts) and they failover and > load balance for free.... but enough evangelising ;) OK, I will look at this 'reply-to' thing. > >>>=== B.2 modifying the participant-list in the storage directly > >>[snip] > >>>Please note that the upcoming ruote 2.1.11 has methods that simplify > >>>that approach : > >>[snip] > >>>No need to restart the engine. > >>OK - but I wouldn't want that code replicate in every remote participant - > >>I'd pull it back to a service that would understand the storage I'm > >>using.... and we're back to B.0 > > > >That code would be needed in participants that need to register participants > >in the engine. > Well. I put it in the engine-side participant proxy and make it > simple for the remote participants to let the proxy know. It didn't > touch Ruote core. > > >I'd be glad to help with this control thing (read : [implement, ]write tests > >and maintain), but the RPC thing feels inelegant. Maybe this control thing is > >better under your control (in your fork / extension). > OK > I certainly don't want to aim to fork though. > > What I'd propose for now is that I extend my RuoteAMQP branch with > actual code and we work together to find a way to merge it that fits > with your design. Would that be a suitable approach? OK, I'll be watching. Cheers, -- John Mettraux - http://jmettraux.wordpress.com -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
