Re: [openstack-dev] [trove][mistral] scheduled tasks
Greg,
Looks like I missed your last email in an email storm after NY holidays but now
was able to see it accidentally. So answering your question about Mistral
incubation your concern is valid. We’re currently working on moving Mistral
into incubation and we’ll keep you (and others) posted on that.
Renat Akhmerov
@ Mirantis Inc.
On 02 Jan 2014, at 09:52, Greg Hill greg.h...@rackspace.com wrote:
Renat,
Thanks for the additional information. I've been trying to put the pieces of
history together and it seems I missed some of it. I think I now understand
the evolution of things. Mistral does seem like it would work for what we
need, so I'll definitely be paying attention to it.
I'm curious about the general OpenStack precedent of having an incubated
project depend on one that isn't yet incubated. Does that make it a
hindrance for graduating to an official project since it would require
operators to deploy something that isn't yet officially part of openstack, or
does that not matter?
Greg
On Jan 2, 2014, at 12:21 AM, Renat Akhmerov rakhme...@mirantis.com wrote:
Greg,
This is really not a new discussion. Particularly, we’ve already discussed
EventScheduler thing before with heat folks
(http://lists.openstack.org/pipermail/openstack-dev/2013-November/019150.html).
We agreed that EventScheduler doesn’t make a lot of sense alone after
Mistral came into play. It’s because EventScheduler only describes calling
webhooks on schedule and nothing else. Mistral already has this
functionality just because it’s a part of its core idea. Timer events is
just one of the event types Mistral is going to support (there will also be
others like Ceilometer alarms). And one of the simple use case of Mistral
would be exactly “calling a webhook on schedule”. Since Mistral uses DSL as
a main interaction means with a user the team decided to implement ideas of
EventScheduler as a part of Mistral DSL as well. Here’s the simple Mistral
DSL that describes basically the same as described in EventScheduler:
Services:
MyRest:
type: REST_API
parameters:
baseUrl: http://localhost:8988/my_service
actions:
my-action:
parameters:
url: /my-action
method: GET
Workflow:
tasks:
my-task:
action: MyRest:my-action
events:
my-task:
type: periodic
tasks: my-task
parameters:
cron-pattern: */1 * * * *
I really hope it’s self-explaining enough (if not, let us know what is not
clear or what you don’t like, we’ll improve it). Basically, to make this
simple DSL snippet work we just need to upload it to Mistral via REST API.
Just want to repeat: this already works. To simplify this even more for use
cases like yours we also have an idea of providing some API sugar like
scheduling webhook calls using special REST API action like:
http POST /webhooks
{
“name”: “my every minute webhook”,
“webhook: “http://my.webhook.host/someurl”,
“schedule”: {
“cron”: “* * * * *
}
}
And under the hood Mistral would just generate a workflow definition similar
to provided above.
So my main point here that I was trying to make is that EventScheduler is
just a special case of Mistral capabilities. We can just provide a
convenience for scheduling webhook calls on the API (a couple of hours to
implement).
Thanks! Happy New Year to everyone!
Renat Akhmerov
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
Greg,
This is really not a new discussion. Particularly, we’ve already discussed
EventScheduler thing before with heat folks
(http://lists.openstack.org/pipermail/openstack-dev/2013-November/019150.html).
We agreed that EventScheduler doesn’t make a lot of sense alone after Mistral
came into play. It’s because EventScheduler only describes calling webhooks on
schedule and nothing else. Mistral already has this functionality just because
it’s a part of its core idea. Timer events is just one of the event types
Mistral is going to support (there will also be others like Ceilometer alarms).
And one of the simple use case of Mistral would be exactly “calling a webhook
on schedule”. Since Mistral uses DSL as a main interaction means with a user
the team decided to implement ideas of EventScheduler as a part of Mistral DSL
as well. Here’s the simple Mistral DSL that describes basically the same as
described in EventScheduler:
Services:
MyRest:
type: REST_API
parameters:
baseUrl: http://localhost:8988/my_service
actions:
my-action:
parameters:
url: /my-action
method: GET
Workflow:
tasks:
my-task:
action: MyRest:my-action
events:
my-task:
type: periodic
tasks: my-task
parameters:
cron-pattern: */1 * * * *
I really hope it’s self-explaining enough (if not, let us know what is not
clear or what you don’t like, we’ll improve it). Basically, to make this simple
DSL snippet work we just need to upload it to Mistral via REST API. Just want
to repeat: this already works. To simplify this even more for use cases like
yours we also have an idea of providing some API sugar like scheduling webhook
calls using special REST API action like:
http POST /webhooks
{
“name”: “my every minute webhook”,
“webhook: “http://my.webhook.host/someurl”,
“schedule”: {
“cron”: “* * * * *
}
}
And under the hood Mistral would just generate a workflow definition similar to
provided above.
So my main point here that I was trying to make is that EventScheduler is just
a special case of Mistral capabilities. We can just provide a convenience for
scheduling webhook calls on the API (a couple of hours to implement).
Thanks! Happy New Year to everyone!
Renat Akhmerov
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
Agreed taskflow doesn't currently provide scheduling as it was thought that reliable execution that can be restarted and resumed is the foundation that someone using taskflow can easily provide scheduling ontop of... Better IMHO to have a project doing this foundation well (since openstack would benefit from this) than try to pack so many features in that it does none of them well (this kind of kitchen sink approach seems to happen more often than not, sadly). But in reality it's all about compromises and finding the solution that makes sense and works, and happy new year!! :P Sent from my really tiny device... On Dec 30, 2013, at 9:03 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: Greg, Georgy is right. We’re now actively working on PoC and we’ve already implemented the functionality we initially planned, including cron-based scheduling. You can take a look at our repo and evaluate what we’ve done, we’d be very glad to hear some feedback from anyone potentially interested in Mistral. We were supposed to deliver PoC in the end of December, however, we decided not to rush and include several really cool things that we came up with while working on PoC, they should demonstrate the whole idea of Mistral much better and expose functionality for more potential use cases. A couple of days ago I sent out the information about additional changes in DSL that we want to implement (etherpad: https://etherpad.openstack.org/p/mistral-poc), so if you’d like please join the discussion and let us know how we can evolve the project to better fit your needs. In fact, even though we call it PoC it’s already in a good shape and pretty soon (~1.5 month) is going to be mature enough to use it as a dependency for other projects. As far as security, we thought about this and and we have a vision of how it could be implemented. Generally, later on we’re planning to implement sort of Role Based Access Control (RBAC) to, first of all, isolate user workbooks (definition of tasks, actions, events) from each other and deal with access patterns to OpenStack services. We would encourage you to file a BP with a description of what would be needed by Trove in that regard. I looked at https://wiki.openstack.org/wiki/Trove/scheduled-tasks and at the first glance Mistral looks a good fit here, especially if you’re interested in a standalone REST service with its capabilities like execution monitoring, history, language independence and HA (i.e. you schedule backups via Mistral and Trove itself shouldn’t care about availability of any functionality related to scheduling). TaskFlow may also be helpful in case your scheduled jobs are representable as flows using one of TaskFlow patterns. However, in my understanding you’ll have to implement scheduling yourself since TaskFlow does not support it now, at least I didn’t find anything like that (Joshua can provide more details on that). Thanks. Renat Akhmerov @Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
I guess this isn't a new discussion. I did some more digging and apparently this is what came out of the last discussion: https://wiki.openstack.org/wiki/EventScheduler That definitely seems like it would be something simple we could use, since it only provides scheduling and that's all we need, but it doesn't appear that it's had any traction since that time. I guess that got absorbed along with Convection into Mistral (is that right?). I tend to agree with the sentiment that the scheduling component should live outside the workflow service, especially for use cases like this where we just need scheduling and not the workflow portions as we're just scheduling things that are already achievable via single API calls (to my knowledge). It seems like we basically have 4 options at this point: 1. Wait for/help finish the scheduling component of mistral and use that 2. Build Qonos workers to do the trove needful and integrate with that 3. Build this proposed EventScheduler thing and have trove use that 4. Build something simple internal to trove for now and revisit when things have matured more Despite my initial enthusiasm about Qonos after it was mentioned earlier today, the more I look into it, the more it seems like the wrong fit. It could definitely do it, but the way it's structured, it appears that we'd have to add code to qonos/worker for each action we wanted to schedule, which just seems like a pain for what amounts to make this API call to trove. My gut says working on EventScheduler is probably the best/most ideal option, but time constraints and what-not make build it into trove the most likely course of action for now. Greg On Dec 31, 2013, at 10:06 AM, Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote: Agreed taskflow doesn't currently provide scheduling as it was thought that reliable execution that can be restarted and resumed is the foundation that someone using taskflow can easily provide scheduling ontop of... Better IMHO to have a project doing this foundation well (since openstack would benefit from this) than try to pack so many features in that it does none of them well (this kind of kitchen sink approach seems to happen more often than not, sadly). But in reality it's all about compromises and finding the solution that makes sense and works, and happy new year!! :P Sent from my really tiny device... On Dec 30, 2013, at 9:03 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: Greg, Georgy is right. We’re now actively working on PoC and we’ve already implemented the functionality we initially planned, including cron-based scheduling. You can take a look at our repo and evaluate what we’ve done, we’d be very glad to hear some feedback from anyone potentially interested in Mistral. We were supposed to deliver PoC in the end of December, however, we decided not to rush and include several really cool things that we came up with while working on PoC, they should demonstrate the whole idea of Mistral much better and expose functionality for more potential use cases. A couple of days ago I sent out the information about additional changes in DSL that we want to implement (etherpad: https://etherpad.openstack.org/p/mistral-poc), so if you’d like please join the discussion and let us know how we can evolve the project to better fit your needs. In fact, even though we call it PoC it’s already in a good shape and pretty soon (~1.5 month) is going to be mature enough to use it as a dependency for other projects. As far as security, we thought about this and and we have a vision of how it could be implemented. Generally, later on we’re planning to implement sort of Role Based Access Control (RBAC) to, first of all, isolate user workbooks (definition of tasks, actions, events) from each other and deal with access patterns to OpenStack services. We would encourage you to file a BP with a description of what would be needed by Trove in that regard. I looked at https://wiki.openstack.org/wiki/Trove/scheduled-tasks and at the first glance Mistral looks a good fit here, especially if you’re interested in a standalone REST service with its capabilities like execution monitoring, history, language independence and HA (i.e. you schedule backups via Mistral and Trove itself shouldn’t care about availability of any functionality related to scheduling). TaskFlow may also be helpful in case your scheduled jobs are representable as flows using one of TaskFlow patterns. However, in my understanding you’ll have to implement scheduling yourself since TaskFlow does not support it now, at least I didn’t find anything like that (Joshua can provide more details on that). Thanks. Renat Akhmerov @Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [trove][mistral] scheduled tasks
Any reason to not use taskflow (https://wiki.openstack.org/wiki/TaskFlow) to help u here?? I think it could be easily adapted to do what u want, and would save u from having to recreate the same task execution logic that everyone seems to rebuild… -Josh From: Greg Hill greg.h...@rackspace.commailto:greg.h...@rackspace.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, December 30, 2013 at 9:59 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [trove][mistral] scheduled tasks I've begun working on the scheduled tasks feature that will allow automated backups (and other things) in trove. Here's the blueprint: https://wiki.openstack.org/wiki/Trove/scheduled-tasks I've heard some mention that mistral might be an option rather than building something into trove. I did some research and it seems like it *might* be a good fit, but it also seems like a bit of overkill for something that could be built in to trove itself pretty easily. There's also the security concern of having to give mistral access to the trove management API in order to allow it to fire off backups and other tasks on behalf of users, but maybe that's just my personal paranoia and it's really not much of a concern. My current plan is to not use mistral, at least for the original implementation, because it's not yet ready and we have a fairly urgent need for the functionality. We could make it an optional feature later for people who are running mistral and want to use it for this purpose. I'd appreciate any and all feedback before I get too far along. Greg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
I accidentally sent this reply to Josh directly. Greg On Dec 30, 2013, at 12:17 PM, Greg Hill greg.h...@rackspace.commailto:greg.h...@rackspace.com wrote: Taskflow seems like it would be a good fit for implementation or re-implementation of some of the tasks we hope to automate, but the first set of desired tasks are already built. We're simply building the scheduling logic to automate them, so I don't see what taskflow would buy us there. Greg On Dec 30, 2013, at 12:14 PM, Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote: Any reason to not use taskflow (https://wiki.openstack.org/wiki/TaskFlow) to help u here?? I think it could be easily adapted to do what u want, and would save u from having to recreate the same task execution logic that everyone seems to rebuild… -Josh From: Greg Hill greg.h...@rackspace.commailto:greg.h...@rackspace.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, December 30, 2013 at 9:59 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [trove][mistral] scheduled tasks I've begun working on the scheduled tasks feature that will allow automated backups (and other things) in trove. Here's the blueprint: https://wiki.openstack.org/wiki/Trove/scheduled-tasks I've heard some mention that mistral might be an option rather than building something into trove. I did some research and it seems like it *might* be a good fit, but it also seems like a bit of overkill for something that could be built in to trove itself pretty easily. There's also the security concern of having to give mistral access to the trove management API in order to allow it to fire off backups and other tasks on behalf of users, but maybe that's just my personal paranoia and it's really not much of a concern. My current plan is to not use mistral, at least for the original implementation, because it's not yet ready and we have a fairly urgent need for the functionality. We could make it an optional feature later for people who are running mistral and want to use it for this purpose. I'd appreciate any and all feedback before I get too far along. Greg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
Greg, This might be useful: https://wiki.openstack.org/wiki/Qonos-scheduling-service There's a link to the code repository at the bottom of the document. It might be kind of overkill for what you want, though. cheers, brian From: Greg Hill [greg.h...@rackspace.com] Sent: Monday, December 30, 2013 12:59 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [trove][mistral] scheduled tasks I've begun working on the scheduled tasks feature that will allow automated backups (and other things) in trove. Here's the blueprint: https://wiki.openstack.org/wiki/Trove/scheduled-tasks I've heard some mention that mistral might be an option rather than building something into trove. I did some research and it seems like it *might* be a good fit, but it also seems like a bit of overkill for something that could be built in to trove itself pretty easily. There's also the security concern of having to give mistral access to the trove management API in order to allow it to fire off backups and other tasks on behalf of users, but maybe that's just my personal paranoia and it's really not much of a concern. My current plan is to not use mistral, at least for the original implementation, because it's not yet ready and we have a fairly urgent need for the functionality. We could make it an optional feature later for people who are running mistral and want to use it for this purpose. I'd appreciate any and all feedback before I get too far along. Greg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
Hi, I would suggest to talk with Mistral team about scheduling. As I know right now thay are writing a PoC code which will have scheduling functionality. Mistral in contrary to Qonos does not require worker process. In Mistral you can create a task with scheduler and API call back so that Mistral can call some API endpoint with some specific parameters in predefined time. I suspect that it covers your use case. Security concerns can be addressed and it will be great if you submit specific BP to Mistral with all security requirements and scheduling features required by Trove. Thanks Georgy On Mon, Dec 30, 2013 at 12:26 PM, Brian Rosmaita brian.rosma...@rackspace.com wrote: Greg, This might be useful: https://wiki.openstack.org/wiki/Qonos-scheduling-service There's a link to the code repository at the bottom of the document. It might be kind of overkill for what you want, though. cheers, brian -- *From:* Greg Hill [greg.h...@rackspace.com] *Sent:* Monday, December 30, 2013 12:59 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* [openstack-dev] [trove][mistral] scheduled tasks I've begun working on the scheduled tasks feature that will allow automated backups (and other things) in trove. Here's the blueprint: https://wiki.openstack.org/wiki/Trove/scheduled-tasks I've heard some mention that mistral might be an option rather than building something into trove. I did some research and it seems like it *might* be a good fit, but it also seems like a bit of overkill for something that could be built in to trove itself pretty easily. There's also the security concern of having to give mistral access to the trove management API in order to allow it to fire off backups and other tasks on behalf of users, but maybe that's just my personal paranoia and it's really not much of a concern. My current plan is to not use mistral, at least for the original implementation, because it's not yet ready and we have a fairly urgent need for the functionality. We could make it an optional feature later for people who are running mistral and want to use it for this purpose. I'd appreciate any and all feedback before I get too far along. Greg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
Sorry for being a little bit verbose :) … Renat ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
