Re: [openstack-dev] [mistral] Best practices for the DB maintanence in production
On 03 Jul 2015, at 12:14, Lingxian Kong anlin.k...@gmail.com wrote: What do you think if we add a script(may be under tools or cmd package) to help doing this? What script do you mean? To launch a separate clean-up component. I see that, first of all, it’s a subsystem of Mistral consisting at least of things like various eviction policies (people may want to cleanup differently) and an active manager that implements the logic, maybe there should be stat collector if we want to evaluate the work of this cleanuper. So, IMO, it should be a package which most likely contain several modules and classes and then we may want to have a script to launch it separately if needed. Or it can start automatically say within an engine instance. Anyway, let’s not run ahead of train and gather what we need to do first. Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] Best practices for the DB maintanence in production
What do you think if we add a script(may be under tools or cmd package) to help doing this? On Thu, Jul 2, 2015 at 7:59 PM, ELISHA, Moshe (Moshe) moshe.eli...@alcatel-lucent.com wrote: Thanks, Renat. I also believe the right place to do it is in Mistral. I will create a blueprint and we will discuss the details in the spec. Thanks. *From:* Renat Akhmerov [mailto:rakhme...@mirantis.com] *Sent:* יום ה 02 יולי 2015 12:34 *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [mistral] Best practices for the DB maintanence in production Hi Elisha, Currently Mistral doesn’t support any expiration policies for workflow/task/action runtime objects. It keeps them forever until someone deletes them manually. I see the following ways of addressing your need: - Implement some cleanup component within Mistral (how to call it?) using its Scheduler component to periodically query and delete objects based on a criteria provided in a config. - Just implement something on top of Mistral API to do the same. The cons of this approach is that Mistral now doesn’t provide any flexible mechanism to do criteria-based search of its objects. There’s an adjacent BP for that [1]. Generally, there’s a number of things in Mistral API we are not satisfied with and we’ve been planning to design and suggest API v3 for Mistral to support those features (don’t confuse with DSL v3, there’s no plan for now to implement a new backwards incompatible DSL). So this option may not be effective from performance perspective. I think it deserves its own blueprint so that we can discuss nuances. [1] https://blueprints.launchpad.net/mistral/+spec/mistral-items-filtering Renat Akhmerov @ Mirantis Inc. On 02 Jul 2015, at 13:37, ELISHA, Moshe (Moshe) moshe.eli...@alcatel-lucent.com wrote: Hey, We are planning to use Mistral in production in the next few months. We noticed that having even a simple workflow with a cron-trigger (For example monitor and heal workflow) can create large amounts of data in the DB (MariaDB). Does Mistral have a mechanism / configuration of automatic deletion of old executions? What is the best practice to handle this type of challenge? Thanks. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- *Regards!* *---* *Lingxian Kong* __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] Best practices for the DB maintanence in production
OK, Renat, I see, thanks for clarification. Let's discuss it together after the blueprint is proposed. On Fri, Jul 3, 2015 at 4:41 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 03 Jul 2015, at 12:14, Lingxian Kong anlin.k...@gmail.com wrote: What do you think if we add a script(may be under tools or cmd package) to help doing this? What script do you mean? To launch a separate clean-up component. I see that, first of all, it’s a subsystem of Mistral consisting at least of things like various eviction policies (people may want to cleanup differently) and an active manager that implements the logic, maybe there should be stat collector if we want to evaluate the work of this cleanuper. So, IMO, it should be a package which most likely contain several modules and classes and then we may want to have a script to launch it separately if needed. Or it can start automatically say within an engine instance. Anyway, let’s not run ahead of train and gather what we need to do first. Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- *Regards!* *---* *Lingxian Kong* __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Best practices for the DB maintanence in production
Hey, We are planning to use Mistral in production in the next few months. We noticed that having even a simple workflow with a cron-trigger (For example monitor and heal workflow) can create large amounts of data in the DB (MariaDB). Does Mistral have a mechanism / configuration of automatic deletion of old executions? What is the best practice to handle this type of challenge? Thanks. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] Best practices for the DB maintanence in production
Hi Elisha, Currently Mistral doesn’t support any expiration policies for workflow/task/action runtime objects. It keeps them forever until someone deletes them manually. I see the following ways of addressing your need: Implement some cleanup component within Mistral (how to call it?) using its Scheduler component to periodically query and delete objects based on a criteria provided in a config. Just implement something on top of Mistral API to do the same. The cons of this approach is that Mistral now doesn’t provide any flexible mechanism to do criteria-based search of its objects. There’s an adjacent BP for that [1]. Generally, there’s a number of things in Mistral API we are not satisfied with and we’ve been planning to design and suggest API v3 for Mistral to support those features (don’t confuse with DSL v3, there’s no plan for now to implement a new backwards incompatible DSL). So this option may not be effective from performance perspective. I think it deserves its own blueprint so that we can discuss nuances. [1] https://blueprints.launchpad.net/mistral/+spec/mistral-items-filtering https://blueprints.launchpad.net/mistral/+spec/mistral-items-filtering Renat Akhmerov @ Mirantis Inc. On 02 Jul 2015, at 13:37, ELISHA, Moshe (Moshe) moshe.eli...@alcatel-lucent.com wrote: Hey, We are planning to use Mistral in production in the next few months. We noticed that having even a simple workflow with a cron-trigger (For example monitor and heal workflow) can create large amounts of data in the DB (MariaDB). Does Mistral have a mechanism / configuration of automatic deletion of old executions? What is the best practice to handle this type of challenge? Thanks. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] Best practices for the DB maintanence in production
Thanks, Renat. I also believe the right place to do it is in Mistral. I will create a blueprint and we will discuss the details in the spec. Thanks. From: Renat Akhmerov [mailto:rakhme...@mirantis.com] Sent: יום ה 02 יולי 2015 12:34 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [mistral] Best practices for the DB maintanence in production Hi Elisha, Currently Mistral doesn’t support any expiration policies for workflow/task/action runtime objects. It keeps them forever until someone deletes them manually. I see the following ways of addressing your need: * Implement some cleanup component within Mistral (how to call it?) using its Scheduler component to periodically query and delete objects based on a criteria provided in a config. * Just implement something on top of Mistral API to do the same. The cons of this approach is that Mistral now doesn’t provide any flexible mechanism to do criteria-based search of its objects. There’s an adjacent BP for that [1]. Generally, there’s a number of things in Mistral API we are not satisfied with and we’ve been planning to design and suggest API v3 for Mistral to support those features (don’t confuse with DSL v3, there’s no plan for now to implement a new backwards incompatible DSL). So this option may not be effective from performance perspective. I think it deserves its own blueprint so that we can discuss nuances. [1] https://blueprints.launchpad.net/mistral/+spec/mistral-items-filtering Renat Akhmerov @ Mirantis Inc. On 02 Jul 2015, at 13:37, ELISHA, Moshe (Moshe) moshe.eli...@alcatel-lucent.commailto:moshe.eli...@alcatel-lucent.com wrote: Hey, We are planning to use Mistral in production in the next few months. We noticed that having even a simple workflow with a cron-trigger (For example monitor and heal workflow) can create large amounts of data in the DB (MariaDB). Does Mistral have a mechanism / configuration of automatic deletion of old executions? What is the best practice to handle this type of challenge? Thanks. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
