Thanks for taking the time to consider this John - I'm trying to explain why we
need it and looking for the best way to keep it upstream.
As a focus: my objective is to have a participant-less Ruote at the centre of a
large group of machines; each remote machine will have one or more Participants
running in ruby, python or perl and communicating with Ruote via AMQP.
That group of machines/Participants will expand over time; some will move from
development to staging to production. I don't want the codebase on the central
machine to change at all as remote teams deploy participants and processes.
On 17/07/10 02:59, John Mettraux wrote:
On Fri, Jul 16, 2010 at 07:06:54PM +0100, David Greaves wrote:
On 15/07/10 15:31, John Mettraux wrote:
There is also the case where a participant emits work towards an engine
(launch).
OK. This involves passing in a process definition which of course defines
and names a number of process steps. Less work-y but I concur.
Hello David,
Less "work" ? How do you launch workflows ?
Well, I see some reasonably distinct tasks.
1) is passing a workitem; this is pure 'work'.
2) is passing a process definition. This leads to 'work' being done but
does nothing tangible until a workitem is passed (1)
3) is passing a participant name mapping. This says where 'work' can be done in
a distributed system and steps in (2) direct (1)
But as I said - I agree with you.
Remember that you can also pass process definitions as URI references.
Not aware of this?
I'll google.
What you are asking about is adding "admin" operations over that channel.
I'm a bit reluctant.
Agreed and I see your point (I did actually wonder myself but it was needed
to make the PoC operate and it worked well).
It would seem trivial for me to offer a parallel "enginecontrol"
queue/service which would be a little more RPC-like (insofar as it would be
nice for a remote registrant to receive acknowledgement that a registration
has succeeded).
Yes, this is were I don't want to wander (ie I don't want to have the
responsibility to maintain it).
Pushing messages over AMQP is simple enough. Building a RPC framework on top
of it...
OK - that's 'done' over here and very separate.
It's also pretty trivial if you scope it to be service RPC and not object-RPC.
Functionally it could be *identical* to the workitems queue after all, I
provided a 5-line patch to multiplex them.
Either way it could be extended to use the 'reply_to' that is a standard AMQP
feature to publish a response.
If it helps I'll volunteer to co-maintain the AMQP library. We're pretty
committed to it from a MeeGo PoV anyway.
I recognise that I've only been about for a few weeks so I know this will take
time where I hope you can build up some confidence and for me to learn how best
to tailor solutions for Ruote.
There's still need for a cancel operation (and we've identified a possible
progress-query operation) on a wfid(?) in an AMQP Remote Participant and I
propose using a simple RPC over AMQP framework based on Kenneths design.
I will add the "cancel" message to the "work" messages listed above. It is a
must.
OK.
I think this will look like this : http://wiki.meego.com/images/Boss9.png
Let's register a participant under a regex :
engine.register_participant 'amqp-.+', RuoteAMQP::Participant, 'queue'
=> 'y'
All the workitems for a participant whose name begins with 'command-'
will go there.
No offence but ... ugh !
Replace "amqp-" with "do_" and there is no exposition of AMQP at all.
But now you have 2 places which lookup and dispatch participants? The Engine and
(in this case) the remote participant.
So for do_onhost_a, do_onhost_b and do_onhost_c where the true target is on
different machines then the workitem must be pulled off y, examined and pushed
onto queue a, b or c.
And if I decide to take a local participant and implement it remotely via AMQP
then I have to change any process that uses "notify customer" to "do_notify
customer"
So this pattern doesn't fit my distributed participant model.
I would like ruote's *beautiful* and readable process definitions that
don't care if amqp is involved. Lets move quickly on ... :)
Remember "engine.register_participant x, y, z" is not part of a process
definition. It's configuration logic happening on a different timeline.
Yes.
I just don't see why it can't happen sporadically during the life of the core
deployment?
... although, in the limit: engine.register_participant '.+',
RuoteAMQP::Participant, 'queue' => 'y' but it would be a bit blunt and I'd
just move the mapping to the thing handling queue y.... not nice.
In my small one line example, I force the queue on the participant. You could
write logic to determine the queue based on the workitem content (or simply
the workitem.participant_name).
Yes. I could just force all AMQP participants to use the
workitem.participant_name as the AMQP queue name.
=== A.1 command :variant => 'x'
engine.register_participant 'amqp', RuoteAMQP::Participant, 'queue' =>
'y'
then in your process definition
participant 'amqp', :variant => 'x' participant 'amqp', :variant =>
'y'
Same basic complaint.
OK, ok, replace 'amqp' by 'gucci' and you've got the elegance and no AMQP
exposition (even loosely).
so
sequence do
gucci variant => build
gucci variant => test
gucci variant => package
gucci variant => release
end
vs
sequence do
build
test
package
release
end
Is that what you meant?
OK, "please have a smart mapping" vs "I just want to add a mapping".
So again... not 'smart'.
Dynamically adjustable
On a remote machine in python I'd run:
p = MyPart(ruote_queue="package", amqp_host="amqpvm", amqp_vhost="ruote")
p.register("package", {'queue':'package'})
p.run()
I could then launch the 2nd pdef above.
=== B.0 programmatically adding
You have access to the engine. You can register participants on the fly
:
require 'my_participant' engine.register_participant 'toto',
MyParticipant, 'flavour' => 'vanilla', 'position' => -2
How does this differ from: if item.has_key?('register') return unless
item.has_key?('name') register_participant(item["name"],
RuoteAMQP::Participant, item["options"]) other than the message transport?
I *think* it's very similar - I like it.
As said, the need to build a custom RPC framework on top of AMQP. HTTP is
sweet with its request/response system.
I could understand this if I was coming to you and asking for AMQP and Ruote was
an HTTP framework.
But all of this work is 100% inside RuoteAMQP.... I don't see the need for an
HTTP RPC solution inside the RuoteAMQP library?
As an aside, AMQP is *explicitly* designed to support request/response - RPC is
built in. Not only that it doesn't matter if the client/server are not running
at the same time. Additionally you can have 20 workers (on different hosts) and
they failover and load balance for free.... but enough evangelising ;)
=== B.2 modifying the participant-list in the storage directly
[snip]
Please note that the upcoming ruote 2.1.11 has methods that simplify
that approach :
[snip]
No need to restart the engine.
OK - but I wouldn't want that code replicate in every remote participant -
I'd pull it back to a service that would understand the storage I'm
using.... and we're back to B.0
That code would be needed in participants that need to register participants
in the engine.
Well. I put it in the engine-side participant proxy and make it simple for the
remote participants to let the proxy know. It didn't touch Ruote core.
I'd be glad to help with this control thing (read : [implement, ]write tests
and maintain), but the RPC thing feels inelegant. Maybe this control thing is
better under your control (in your fork / extension).
OK
I certainly don't want to aim to fork though.
What I'd propose for now is that I extend my RuoteAMQP branch with actual code
and we work together to find a way to merge it that fits with your design. Would
that be a suitable approach?
David
--
"Don't worry, you'll be fine; I saw it work in a cartoon once..."
--
you received this message because you are subscribed to the "ruote users" group.
to post : send email to [email protected]
to unsubscribe : send email to [email protected]
more options : http://groups.google.com/group/openwferu-users?hl=en
