Re: [openstack-dev] [Mistral] Event Subscription

2014-12-09 Thread Renat Akhmerov
Ok, got it.

So my general suggestion here is: let's keep it as simple as possible for now, 
create something that works and then let’s see how to improve it. And yes, 
consumers may be and mostly will be 3rd parties.

Thanks

Renat Akhmerov
@ Mirantis Inc.



 On 09 Dec 2014, at 08:25, W Chan m4d.co...@gmail.com wrote:
 
 Renat,
 
 On sending events to an exchange, I mean an exchange on the transport (i.e. 
 rabbitMQ exchange https://www.rabbitmq.com/tutorials/amqp-concepts.html 
 https://www.rabbitmq.com/tutorials/amqp-concepts.html).  On implementation 
 we can probably explore the notification feature in oslo.messaging.  But on 
 second thought, this would limit the consumers to trusted subsystems or 
 services though.  If we want the event consumers to be any 3rd party, 
 including untrusted, then maybe we should keep it as HTTP calls.
 
 Winson
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Event Subscription

2014-12-08 Thread W Chan
Renat,

On sending events to an exchange, I mean an exchange on the transport
(i.e. rabbitMQ exchange
https://www.rabbitmq.com/tutorials/amqp-concepts.html).  On implementation
we can probably explore the notification feature in oslo.messaging.  But on
second thought, this would limit the consumers to trusted subsystems or
services though.  If we want the event consumers to be any 3rd party,
including untrusted, then maybe we should keep it as HTTP calls.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Event Subscription

2014-12-02 Thread W Chan
Renat,

I agree with the two methods you proposed.

On processing the events, I was thinking a separate entity.  But you gave
me an idea, how about a system action for publishing the events that the
current executors can run?

Alternately, instead of making HTTP calls, what do you think if mistral
just post the events to the exchange(s) that the subscribers provided and
let the subscribers decide how to consume the events (i.e post to webhook,
etc.) from these exchanges?  This will simplify implementation somewhat.
The engine can just take care of publishing the events to the exchanges and
call it done.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Event Subscription

2014-12-02 Thread Renat Akhmerov

 On 02 Dec 2014, at 23:53, W Chan m4d.co...@gmail.com wrote:
 
 On processing the events, I was thinking a separate entity.  But you gave me 
 an idea, how about a system action for publishing the events that the current 
 executors can run?

Yes, sounds great! I really like the idea.

 Alternately, instead of making HTTP calls, what do you think if mistral just 
 post the events to the exchange(s) that the subscribers provided and let the 
 subscribers decide how to consume the events (i.e post to webhook, etc.) from 
 these exchanges?  This will simplify implementation somewhat.  The engine can 
 just take care of publishing the events to the exchanges and call it done.

Yep. I understand the general idea but am still a little confused. Can you 
please share the details of what you mean by “exchange” and who is going to 
consume events? If, like previously said, it will be just sending actions to 
executors then it’s ok, I got it. And you can just use executor client from 
rpc.py to do the job. Or here you mean something different? Would be nice it we 
could present a communication diagram (engine - event occurred - … ).


Thanks___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Event Subscription

2014-12-01 Thread W Chan
Renat,

Alternately, what do you think if mistral just post the events to given
exchange(s) on the same transport backend and let the subscribers decide
how to consume the events (i.e post to webhook, etc.) from these
exchanges?  This will simplify implementation somewhat.  The engine can
just take care of publishing the events to the exchanges and call it done.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Event Subscription

2014-12-01 Thread Renat Akhmerov
Hi Winson,

 On 02 Dec 2014, at 09:10, W Chan m4d.co...@gmail.com wrote:
 
 To clarify on the shortcut solution, are you saying 1) we add an adhoc event 
 subscription to the workflow spec OR 2) add a one time event subscription to 
 the workflow execution OR both?

Not sure what you mean by “adhoc” here. What I meant is that we should have 2 
options:
Have an individual REST endpoint to be able to subscribe for certain types of 
events any time. For example, “Notifications about all workflow events for 
workflow name ‘my_workflow’” or “Notifications about switching to state ‘ERROR’ 
for workflow ‘my_workflow’”. Using this endpoint we can also unsubscribe from 
these events.
When we start a workflow (“mistral execution-create” in CLI and 
start_workflow() method in engine) we can configure the same subscription and 
pass it along with “start workflow” request. For such purposes, engine method 
“start_workflow” has keyword parameter “**params” that can take any kind of 
additional parameters needed for workflow start (for example, when we start a 
reverse workflow we pass “task_name”). This way we can start a workflow and 
configure our subscription with a single request. In the first approach we 
would have to make two requests.

 I propose a separate worker/executor to process the events so to not disrupt 
 the workflow executions.  What if there are a lot of subscribers?  What if 
 one or more subscribers are offline?  Do we retry and how many times?  These 
 activities will likely disrupt the throughput of the workflows and I rather 
 handle these activities separately.


Yeah, I now tend to agree with you here. Although it still bothers me because 
of that performance overhead that we’ll have most of the time. But generally, 
yes, you’re right. Do you think we should use same executors to process these 
notifications or introduce a new type of entity for that?

Thanks

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Event Subscription

2014-11-17 Thread Renat Akhmerov
Ok.

Here’s what I think.

Mostly I like the solution you’re proposing. Some comments/questions:
We discussed this with Dmitry about a couple of months ago and we concluded 
that we don’t necessarily need a separate endpoint, CRUD operations etc. (even 
though my initial BP had exactly this idea). One of the reasons was that we can 
just attach required information about events we’re interested in to “start 
workflow” request (technically it may be that “params” argument in 
“start_workflow” method) so that you don’t need to make 2 requests: 1) 
Subscribe to required events 2) Start workflow. However, I think we could 
combine these two things: having a separate endpoint and being able to attach 
“callback” information (the naming should be different though) in “start 
workflow” request. The first option is useful when I’m not interested in some 
particular workflow. For example, if I as a Mistral client just want to monitor 
what’s happening with my multiple workflows (not certain executions but rather 
all executions of specific workflows) and want to react on them somehow. The 
second one is just a shortcut solution to avoid one extra request to make event 
subscription.
Having only decorators is not enough. Even though I like that idea indeed. For 
instance, the event of completing workflow can’t be handled by a decorator I 
guess. But I’m not 100% sure, it may actually depend on how smart that 
decorator is. Anyway, just a thought.
Do you think we need a separate worker/executor connected via MQ to make all 
the notifications? Would it be simpler to just make those HTTP calls directly 
from engine w/o distributing them? Seems like distribution overhead may be 
higher than making the call itself. Btw, we’re now thinking about the concept 
of ‘local actions’ that don’t have to be distributed across executors. The 
reason is the same: distribution overhead is much higher than just running them 
right in the engine process. So I would like to know what you think here.

Thanks

Renat Akhmerov
@ Mirantis Inc.



 On 12 Nov 2014, at 22:21, W Chan m4d.co...@gmail.com wrote:
 
 Nikolay,
 
 You're right.  We will need to store the events in order to re-publish.  How 
 about a separate Event model?  The events are written to the DB by the same 
 worker that publishes the event.  The retention policy for these events is 
 then managed by a config option.
 
 Winson
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Event Subscription

2014-11-12 Thread W Chan
Nikolay,

You're right.  We will need to store the events in order to re-publish.
How about a separate Event model?  The events are written to the DB by the
same worker that publishes the event.  The retention policy for these
events is then managed by a config option.

Winson
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Event Subscription

2014-11-11 Thread Nikolay Makhotkin
Hi, Winson,

This is great!

But I have a question:

 9. Operation in python-mistralclient to re-publish events for a given
 workflow execution (in case where client applications was downed and need
 the data to recover).

How does Mistral know about all the events which it has already sent to
subscriber?

I guess you propose to store this data in EventSubscriber object. Correct
me if I mistaked.

On Tue, Nov 11, 2014 at 11:28 PM, W Chan m4d.co...@gmail.com wrote:

 Regarding blueprint to register event listeners to notify client
 applications on state changes (
 https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http),
 I want to propose the following.

 1. Refer this feature as event subscription instead of callback
 2. Event subscription supports only HTTP web hooks with retry policy in
 this first attempt
 3. Event subscription can be defined for a list of specific events,
 workflows, projects, domains, or any combinations (all if list is empty).
 4. Decorator to publish event (similar to logging) and place decorators at
 Engine and TaskPolicy methods @
 https://github.com/stackforge/mistral/blob/master/mistral/engine1/base.py
 .
 5. Events should be published to a queue and then processed by a worker so
 not to disrupt actual workflow/task executions.
 6. API controller to register event subscriber
 a. New resource type named EventSubscriber
 b. New REST controller named EventSubscribersController and CRUD
 operations
 7. DB v2 sqlalchemy model named EventSubscriber and appropriate CRUD
 methods
 8. Operations in python-mistralclient to manage CRUD for subscribers.
 9. Operation in python-mistralclient to re-publish events for a given
 workflow execution (in case where client applications was downed and need
 the data to recover).


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Best Regards,
Nikolay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev