Re: [openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Roman Sokolkov]

Dmitry,

Q1. Yes.

> where do you plan to actually perform settings manipulation?

It was one of the critical blockers. Most of the settings are baked inside
fuel-library. Your feature [1] partially fixes this BTW. Which is good.
Partially, because only limited number of tasks has defined overrides.

> scheduled basis run nailgun-cm-agent

Currently i see better way. nailgun-cm-agent (or whatever) should just check
system status (i.e. puppet apply --noop) and report back. User will decide
apply changes or not.

Q2.
Yes, i did. One more use case covered. Please see first table in [2].

Q4. Agree. Here is the bug [3]

Q3,Q5, Q6
Good.

[1] https://blueprints.launchpad.net/fuel/+spec/openstack-config-change
[2]
https://docs.google.com/document/d/1bVVZJR73pBWB_WbfOzC-fh84pVsi20v4n3rvn38F4OI/edit
(limited access)
[3] https://bugs.launchpad.net/fuel/+bug/1525872



On Mon, Dec 14, 2015 at 4:39 AM, Dmitriy Novakovskiy <
dnovakovs...@mirantis.com> wrote:

> Roman,
>
> Thanks a lot for the feedback. We'll be planning improvements for [1] in
> upcoming 9.0 cycle, so your input and this discussion are very helpful and
> much appreciated.
>
> In overall, the concept for nailgun-cm-agent looks interesting, but I
> think you'll face some problems with it:
> - idempotency of puppet modules
> - lack of exposed parameters (fuel-lib hacking)
> - speed of re-runs of configuration mgmt (that we're already working on in
> 8.0)
>
> Now, my comments and questions.
>
> *1) nailgun-cm-agent concept*
> Q1. Do I understand correctly that the planned UX is:
> - Allow user to change configuration as dictated by Fuel (btw, where do
> you plan to actually perform settings manipulation? Directly in Puppet
> modules/manifests?)
> - On scheduled basis run nailgun-cm-agent and let it bring overall system
> state to be consistent with latest changes
> ?
>
> *2) "Advanced settings" [1] feature feedback*
> Q2. Please share the details about 13 real world tasks that you used for
> testing. Have you had a chance to test this same list against [1], as you
> did with fuel-cm-agent approach? I need to know what from real world is
> doable and what not with current state of [1]
> Q3. "It allows just apply, not track changes" - that's true, 8.0 has
> first MVP of this feature in place, and we don't yet have much tracking
> capability (other than looking at logs in DB, when what config change yaml
> was uploaded). We will be improving it in 9.0 cycle
> Q4. "Moreover works weird, if multiple changes uploaded, applying not the
> latest, but initial config change." - can you please share the detailed
> example? I'm not sure I understood it, but so far sounds like a bug that
> needs to be fixed.
> Q5. "Just limited number[1] of resources/tasks has support." - this is
> the limitation of what configs are shipped out of the box. When 8.0 is
> released, we'll have a documented way to add support for any OpenStack
> config file that Fuel tasks can reach
> Q6. "Can we  start moving all (non orchestrating) data into CMDB? yaml
> under git or any existing solution." We're now discussing major
> refactoring effort to be done in Fuel to integrate with Solar and solve
> some of the long standing
>
> [1] https://blueprints.launchpad.net/fuel/+spec/openstack-config-change
>
> On Fri, Dec 11, 2015 at 6:21 PM, Roman Sokolkov 
> wrote:
>
>> Oleg,
>>
>> thanks. I've tried it [1], looks like it works.
>>
>> - GOOD. "override_resource" resource. Like "back door" into puppet
>> modules.
>> - BAD. It allows just apply, not track changes. Moreover works weird,
>> if multiple changes uploaded, applying not the latest, but initial
>> config change.
>> - BAD. Just limited number[1] of resources/tasks has support.
>>
>> BTW, my feeling that we should NOT develop this approach in the same way.
>>
>> I'm not an expert, but as long-term
>> - Can we  start moving all (non orchestrating) data into CMDB? yaml under
>> git
>> or any existing solution.
>> - Can we track nodes state? For example, start by cron all puppet tasks
>> with --noop option
>> and check puppet state. Then if "out of sync" node start blinking YELLOW
>> and user
>> can push button, if needed.
>>
>> Thanks
>>
>> [1] https://blueprints.launchpad.net/fuel/+spec/openstack-config-change
>> [2] http://paste.openstack.org/show/481677/
>>
>> On Fri, Dec 11, 2015 at 4:34 PM, Oleg Gelbukh 
>> wrote:
>>
>>> Roman,
>>>
>>> Changing arbitrary parameters supported by respective Puppet manifests
>>> for OpenStack services is implemented in this blueprint [1]. It is being
>>> landed in release 8.0.
>>>
>>> [1] https://blueprints.launchpad.net/fuel/+spec/openstack-config-change
>>>
>>> --
>>> Best regards,
>>> Oleg Gelbukh
>>>
>>> On Thu, Dec 3, 2015 at 5:28 PM, Roman Sokolkov 
>>> wrote:
>>>
 Folks,

 little bit more research done in regards #2 usability.

 I've selected 13 real-world tasks from customer (i.e. update flag X in
 

Re: [openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Oleg Gelbukh]

Roman,

Changing arbitrary parameters supported by respective Puppet manifests for
OpenStack services is implemented in this blueprint [1]. It is being landed
in release 8.0.

[1] https://blueprints.launchpad.net/fuel/+spec/openstack-config-change

--
Best regards,
Oleg Gelbukh

On Thu, Dec 3, 2015 at 5:28 PM, Roman Sokolkov 
wrote:

> Folks,
>
> little bit more research done in regards #2 usability.
>
> I've selected 13 real-world tasks from customer (i.e. update flag X in
> nova.conf):
> - 6/13 require fuel-library patching (or is #2 unusable)
> - 3/13 are OK and can be done with #2
> - 4/13 can be done with some limitations.
>
> If needed i'll provide details.
>
> Rough statistics is that *only ~20-25% of use cases can be done with #2*.
>
> Let me give a very popular use case that will fail with #2. Assume we'r
> executing whole task graph every two hours.
> We want to change nova.conf "DEFAULT/amqp_durable_queues" from False to
> True.
>
> There is no parameter in hiera for "amqp_durable_queues". We have two
> solutions here (both are bad):
> 1) Redefine "DEFAULT/amqp_durable_queues" = True in plugin task. What will
> happen on the node. amqp_durable_queues will continue changing value
> between True and False on every execution. We shouldn't do it this way.
> 2) Patch fuel-library. Value for amqp_durable_queues should be taken from
> hiera. This is also one way ticket.
>
> Thanks
>
>
>
>
>
> On Thu, Dec 3, 2015 at 11:28 AM, Oleg Gelbukh 
> wrote:
>
>> Roman,
>>
>> Thank you. This is great research.
>>
>> Could we have a conversation to discuss this? I'm especially interested
>> in idempotency problems of the fuel-library modules and the common way to
>> provide serialised data to the deployment.
>>
>> --
>> Best regards,
>> Oleg Gelbukh
>> Mirantis Inc
>>
>>
>> On Tue, Dec 1, 2015 at 6:38 PM, Roman Sokolkov 
>> wrote:
>>
>>> Hello, folks.
>>>
>>> We need any kind of CM for Fuel 7.0. Otherwise new project with 800+
>>> nodes
>>> will be near impossible to support. Customer always wants to change
>>> something.
>>>
>>> In our opinion, there are two major approaches for CM:
>>>
>>> #1 Independent CM (Puppet master, Chef, Ansible, whatever)
>>> #2 Fuel-based CM
>>>
>>> Solution for #2
>>> --
>>>
>>> Fuel has all info about configuration. So we've tried to
>>> unlock "Settings" [0] and push "deploy" button.
>>>
>>> Major findings:
>>>
>>> * Task idem-potency. Looks like most of the tasks are idempotent.
>>> We've skipped 3 tasks on controller and were able to get NO downtime
>>> for Horizon and "nova list". BTW deeper QA required.
>>>
>>> * Standard changes. Operator can change parameters via WebUI, CLI or API.
>>> For example, i was able to deploy Sahara. Unfortunately there is not
>>> foolproof.
>>> I mean some changes can lead to broken cloud...
>>>
>>> * Non-standard changes. Any other changes can be done with plugins.
>>> We can modify plugin tasks and scripts (all except UI flags). And then
>>> just
>>> do "--update" + "--sync". BTW, we can change UI for particular env via
>>> API
>>> by modifying "clusters/X/attributes".
>>>
>>> Conclusion
>>> --
>>>
>>> - This works (We have service under cron that runs tasks) [1]
>>> - NOT ready for production (in current state)
>>> - This requires much deeper testing
>>>
>>>
>>> I want to hear thoughts about approach above?
>>> What is the current status/plans for CM? I saw this discussion [2]
>>>
>>> References
>>> --
>>>
>>> [0]
>>> https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d
>>> [1]
>>> https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p
>>> [2] https://etherpad.openstack.org/p/lcm-use-cases
>>>
>>> --
>>> Roman Sokolkov,
>>> Deployment Engineer,
>>> Mirantis, Inc.
>>> Skype rsokolkov,
>>> rsokol...@mirantis.com
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Roman Sokolkov,
> Deployment Engineer,
> Mirantis, Inc.
> Skype rsokolkov,
> rsokol...@mirantis.com
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>

Re: [openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Roman Sokolkov]

Oleg,

thanks. I've tried it [1], looks like it works.

- GOOD. "override_resource" resource. Like "back door" into puppet modules.
- BAD. It allows just apply, not track changes. Moreover works weird,
if multiple changes uploaded, applying not the latest, but initial
config change.
- BAD. Just limited number[1] of resources/tasks has support.

BTW, my feeling that we should NOT develop this approach in the same way.

I'm not an expert, but as long-term
- Can we  start moving all (non orchestrating) data into CMDB? yaml under
git
or any existing solution.
- Can we track nodes state? For example, start by cron all puppet tasks
with --noop option
and check puppet state. Then if "out of sync" node start blinking YELLOW
and user
can push button, if needed.

Thanks

[1] https://blueprints.launchpad.net/fuel/+spec/openstack-config-change
[2] http://paste.openstack.org/show/481677/

On Fri, Dec 11, 2015 at 4:34 PM, Oleg Gelbukh  wrote:

> Roman,
>
> Changing arbitrary parameters supported by respective Puppet manifests for
> OpenStack services is implemented in this blueprint [1]. It is being landed
> in release 8.0.
>
> [1] https://blueprints.launchpad.net/fuel/+spec/openstack-config-change
>
> --
> Best regards,
> Oleg Gelbukh
>
> On Thu, Dec 3, 2015 at 5:28 PM, Roman Sokolkov 
> wrote:
>
>> Folks,
>>
>> little bit more research done in regards #2 usability.
>>
>> I've selected 13 real-world tasks from customer (i.e. update flag X in
>> nova.conf):
>> - 6/13 require fuel-library patching (or is #2 unusable)
>> - 3/13 are OK and can be done with #2
>> - 4/13 can be done with some limitations.
>>
>> If needed i'll provide details.
>>
>> Rough statistics is that *only ~20-25% of use cases can be done with #2*.
>>
>> Let me give a very popular use case that will fail with #2. Assume we'r
>> executing whole task graph every two hours.
>> We want to change nova.conf "DEFAULT/amqp_durable_queues" from False to
>> True.
>>
>> There is no parameter in hiera for "amqp_durable_queues". We have two
>> solutions here (both are bad):
>> 1) Redefine "DEFAULT/amqp_durable_queues" = True in plugin task. What
>> will happen on the node. amqp_durable_queues will continue changing value
>> between True and False on every execution. We shouldn't do it this way.
>> 2) Patch fuel-library. Value for amqp_durable_queues should be taken from
>> hiera. This is also one way ticket.
>>
>> Thanks
>>
>>
>>
>>
>>
>> On Thu, Dec 3, 2015 at 11:28 AM, Oleg Gelbukh 
>> wrote:
>>
>>> Roman,
>>>
>>> Thank you. This is great research.
>>>
>>> Could we have a conversation to discuss this? I'm especially interested
>>> in idempotency problems of the fuel-library modules and the common way to
>>> provide serialised data to the deployment.
>>>
>>> --
>>> Best regards,
>>> Oleg Gelbukh
>>> Mirantis Inc
>>>
>>>
>>> On Tue, Dec 1, 2015 at 6:38 PM, Roman Sokolkov 
>>> wrote:
>>>
 Hello, folks.

 We need any kind of CM for Fuel 7.0. Otherwise new project with 800+
 nodes
 will be near impossible to support. Customer always wants to change
 something.

 In our opinion, there are two major approaches for CM:

 #1 Independent CM (Puppet master, Chef, Ansible, whatever)
 #2 Fuel-based CM

 Solution for #2
 --

 Fuel has all info about configuration. So we've tried to
 unlock "Settings" [0] and push "deploy" button.

 Major findings:

 * Task idem-potency. Looks like most of the tasks are idempotent.
 We've skipped 3 tasks on controller and were able to get NO downtime
 for Horizon and "nova list". BTW deeper QA required.

 * Standard changes. Operator can change parameters via WebUI, CLI or
 API.
 For example, i was able to deploy Sahara. Unfortunately there is not
 foolproof.
 I mean some changes can lead to broken cloud...

 * Non-standard changes. Any other changes can be done with plugins.
 We can modify plugin tasks and scripts (all except UI flags). And then
 just
 do "--update" + "--sync". BTW, we can change UI for particular env via
 API
 by modifying "clusters/X/attributes".

 Conclusion
 --

 - This works (We have service under cron that runs tasks) [1]
 - NOT ready for production (in current state)
 - This requires much deeper testing


 I want to hear thoughts about approach above?
 What is the current status/plans for CM? I saw this discussion [2]

 References
 --

 [0]
 https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d
 [1]
 https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p
 [2] https://etherpad.openstack.org/p/lcm-use-cases

 --
 Roman Sokolkov,
 Deployment Engineer,
 Mirantis, Inc.
 Skype rsokolkov,

Re: [openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Roman Sokolkov]

Hi there,

one small step into this direction. I've checked how idempotent
controller's tasks. As result bugs below were reported:

https://bugs.launchpad.net/fuel/+bug/1524759 NEW
https://bugs.launchpad.net/fuel/+bug/1524747 NEW
https://bugs.launchpad.net/fuel/+bug/1524731 NEW
https://bugs.launchpad.net/fuel/+bug/1524727 NEW
https://bugs.launchpad.net/fuel/+bug/1524724 NEW
https://bugs.launchpad.net/fuel/+bug/1524719 NEW
https://bugs.launchpad.net/fuel/+bug/1524713 NEW
https://bugs.launchpad.net/fuel/+bug/1524687 NEW
https://bugs.launchpad.net/fuel/+bug/1524630 NEW
https://bugs.launchpad.net/fuel/+bug/1524327 CONFIRMED
https://bugs.launchpad.net/fuel/+bug/1522857 IN PROGRESS

If it's interesting i can go thru other roles and tasks? Please let me know.

Thanks



On Thu, Dec 3, 2015 at 10:33 PM, Yuriy Taraday  wrote:

> Hi, Roman.
>
> On Thu, Dec 3, 2015 at 5:36 PM Roman Sokolkov 
> wrote:
>
>> I've selected 13 real-world tasks from customer (i.e. update flag X in
>> nova.conf):
>> - 6/13 require fuel-library patching (or is #2 unusable)
>> - 3/13 are OK and can be done with #2
>> - 4/13 can be done with some limitations.
>>
>> If needed i'll provide details.
>>
>> Rough statistics is that *only ~20-25% of use cases can be done with #2*.
>>
>> Let me give a very popular use case that will fail with #2. Assume we'r
>> executing whole task graph every two hours.
>> We want to change nova.conf "DEFAULT/amqp_durable_queues" from False to
>> True.
>>
>> There is no parameter in hiera for "amqp_durable_queues". We have two
>> solutions here (both are bad):
>> 1) Redefine "DEFAULT/amqp_durable_queues" = True in plugin task. What
>> will happen on the node. amqp_durable_queues will continue changing value
>> between True and False on every execution. We shouldn't do it this way.
>> 2) Patch fuel-library. Value for amqp_durable_queues should be taken from
>> hiera. This is also one way ticket.
>>
>
> You are describing one of use cases we want to cover in future with Config
> Service. If we store all configuration variables consumed by all deployment
> tasks in the service, one will be able to change (override) the value in
> the same service and let deployment tasks apply config changes on nodes.
>
> This would require support from the deployment side (source of all config
> values will become a service, not static file) and from Nailgun (all data
> should be stored in the service). In the future this approach will allow us
> to clarify which value goes where and to define new values and override old
> ones in a clearly manageable fashion.
>
> Config Service would also allow us to use data defined outside of Nailgun
> to feed values into deployment tasks, such as external CM services (e.g.
> Puppet Master).
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Roman Sokolkov,
Deployment Engineer,
Mirantis, Inc.
Skype rsokolkov,
rsokol...@mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Oleg Gelbukh]

Roman,

Thank you. This is great research.

Could we have a conversation to discuss this? I'm especially interested in
idempotency problems of the fuel-library modules and the common way to
provide serialised data to the deployment.

--
Best regards,
Oleg Gelbukh
Mirantis Inc


On Tue, Dec 1, 2015 at 6:38 PM, Roman Sokolkov 
wrote:

> Hello, folks.
>
> We need any kind of CM for Fuel 7.0. Otherwise new project with 800+ nodes
> will be near impossible to support. Customer always wants to change
> something.
>
> In our opinion, there are two major approaches for CM:
>
> #1 Independent CM (Puppet master, Chef, Ansible, whatever)
> #2 Fuel-based CM
>
> Solution for #2
> --
>
> Fuel has all info about configuration. So we've tried to
> unlock "Settings" [0] and push "deploy" button.
>
> Major findings:
>
> * Task idem-potency. Looks like most of the tasks are idempotent.
> We've skipped 3 tasks on controller and were able to get NO downtime
> for Horizon and "nova list". BTW deeper QA required.
>
> * Standard changes. Operator can change parameters via WebUI, CLI or API.
> For example, i was able to deploy Sahara. Unfortunately there is not
> foolproof.
> I mean some changes can lead to broken cloud...
>
> * Non-standard changes. Any other changes can be done with plugins.
> We can modify plugin tasks and scripts (all except UI flags). And then just
> do "--update" + "--sync". BTW, we can change UI for particular env via API
> by modifying "clusters/X/attributes".
>
> Conclusion
> --
>
> - This works (We have service under cron that runs tasks) [1]
> - NOT ready for production (in current state)
> - This requires much deeper testing
>
>
> I want to hear thoughts about approach above?
> What is the current status/plans for CM? I saw this discussion [2]
>
> References
> --
>
> [0]
> https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d
> [1]
> https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p
> [2] https://etherpad.openstack.org/p/lcm-use-cases
>
> --
> Roman Sokolkov,
> Deployment Engineer,
> Mirantis, Inc.
> Skype rsokolkov,
> rsokol...@mirantis.com
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Roman Sokolkov]

Folks,

little bit more research done in regards #2 usability.

I've selected 13 real-world tasks from customer (i.e. update flag X in
nova.conf):
- 6/13 require fuel-library patching (or is #2 unusable)
- 3/13 are OK and can be done with #2
- 4/13 can be done with some limitations.

If needed i'll provide details.

Rough statistics is that *only ~20-25% of use cases can be done with #2*.

Let me give a very popular use case that will fail with #2. Assume we'r
executing whole task graph every two hours.
We want to change nova.conf "DEFAULT/amqp_durable_queues" from False to
True.

There is no parameter in hiera for "amqp_durable_queues". We have two
solutions here (both are bad):
1) Redefine "DEFAULT/amqp_durable_queues" = True in plugin task. What will
happen on the node. amqp_durable_queues will continue changing value
between True and False on every execution. We shouldn't do it this way.
2) Patch fuel-library. Value for amqp_durable_queues should be taken from
hiera. This is also one way ticket.

Thanks





On Thu, Dec 3, 2015 at 11:28 AM, Oleg Gelbukh  wrote:

> Roman,
>
> Thank you. This is great research.
>
> Could we have a conversation to discuss this? I'm especially interested in
> idempotency problems of the fuel-library modules and the common way to
> provide serialised data to the deployment.
>
> --
> Best regards,
> Oleg Gelbukh
> Mirantis Inc
>
>
> On Tue, Dec 1, 2015 at 6:38 PM, Roman Sokolkov 
> wrote:
>
>> Hello, folks.
>>
>> We need any kind of CM for Fuel 7.0. Otherwise new project with 800+ nodes
>> will be near impossible to support. Customer always wants to change
>> something.
>>
>> In our opinion, there are two major approaches for CM:
>>
>> #1 Independent CM (Puppet master, Chef, Ansible, whatever)
>> #2 Fuel-based CM
>>
>> Solution for #2
>> --
>>
>> Fuel has all info about configuration. So we've tried to
>> unlock "Settings" [0] and push "deploy" button.
>>
>> Major findings:
>>
>> * Task idem-potency. Looks like most of the tasks are idempotent.
>> We've skipped 3 tasks on controller and were able to get NO downtime
>> for Horizon and "nova list". BTW deeper QA required.
>>
>> * Standard changes. Operator can change parameters via WebUI, CLI or API.
>> For example, i was able to deploy Sahara. Unfortunately there is not
>> foolproof.
>> I mean some changes can lead to broken cloud...
>>
>> * Non-standard changes. Any other changes can be done with plugins.
>> We can modify plugin tasks and scripts (all except UI flags). And then
>> just
>> do "--update" + "--sync". BTW, we can change UI for particular env via API
>> by modifying "clusters/X/attributes".
>>
>> Conclusion
>> --
>>
>> - This works (We have service under cron that runs tasks) [1]
>> - NOT ready for production (in current state)
>> - This requires much deeper testing
>>
>>
>> I want to hear thoughts about approach above?
>> What is the current status/plans for CM? I saw this discussion [2]
>>
>> References
>> --
>>
>> [0]
>> https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d
>> [1]
>> https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p
>> [2] https://etherpad.openstack.org/p/lcm-use-cases
>>
>> --
>> Roman Sokolkov,
>> Deployment Engineer,
>> Mirantis, Inc.
>> Skype rsokolkov,
>> rsokol...@mirantis.com
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Roman Sokolkov,
Deployment Engineer,
Mirantis, Inc.
Skype rsokolkov,
rsokol...@mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Yuriy Taraday]

Hi, Roman.

On Thu, Dec 3, 2015 at 5:36 PM Roman Sokolkov 
wrote:

> I've selected 13 real-world tasks from customer (i.e. update flag X in
> nova.conf):
> - 6/13 require fuel-library patching (or is #2 unusable)
> - 3/13 are OK and can be done with #2
> - 4/13 can be done with some limitations.
>
> If needed i'll provide details.
>
> Rough statistics is that *only ~20-25% of use cases can be done with #2*.
>
> Let me give a very popular use case that will fail with #2. Assume we'r
> executing whole task graph every two hours.
> We want to change nova.conf "DEFAULT/amqp_durable_queues" from False to
> True.
>
> There is no parameter in hiera for "amqp_durable_queues". We have two
> solutions here (both are bad):
> 1) Redefine "DEFAULT/amqp_durable_queues" = True in plugin task. What will
> happen on the node. amqp_durable_queues will continue changing value
> between True and False on every execution. We shouldn't do it this way.
> 2) Patch fuel-library. Value for amqp_durable_queues should be taken from
> hiera. This is also one way ticket.
>

You are describing one of use cases we want to cover in future with Config
Service. If we store all configuration variables consumed by all deployment
tasks in the service, one will be able to change (override) the value in
the same service and let deployment tasks apply config changes on nodes.

This would require support from the deployment side (source of all config
values will become a service, not static file) and from Nailgun (all data
should be stored in the service). In the future this approach will allow us
to clarify which value goes where and to define new values and override old
ones in a clearly manageable fashion.

Config Service would also allow us to use data defined outside of Nailgun
to feed values into deployment tasks, such as external CM services (e.g.
Puppet Master).
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Vladimir Kuklin]

Hi, Roman

So here is how we think we are going to tackle this. The whole idea is to
make Fuel architecture much more flexible. We are actively working on this
and are going to present draft of initial documents pretty shortly before
the end of the year.


* *Short (very-very short) summary of our plan*

*** Nailgun will be split into pieces*

*** *Managers split-off*
Business logic things such as network allocation, volume allocation and so
on are going to become services with their own APIs

 Serializers externalization and pluggability*

Current serializers are going to become separate entities that will be
pluggable and easily extendable. These serializers will be able to consume
data from any service you register in Nailgun as a data source. These data
sources could be:

1) Nailgun network/volume/openstack_cluster managers
2) any 3rd party service with an API:
  a) LDAP
  b) Inventory
  c) external puppet master
  d) etc.

*** There is going to be a configuration storage*

All the stuff created by serializers is going to be stored in a database in
versioned and hierarchical format.
This storage will be a source for deployment tasks execution.

*** Difference in configuration is going to be handled by new components*

Right now we have some PoC code to make it happen and once it reaches
working stage, we will upload it to upstream and continue working on it in
open format. It is going to consume the difference between cluster
configuration and orchestrate execution of particular deployment tasks from
Fuel Library or any other 'runnable' resource wrapped into its format, e.g.
container, ansible playbook, whatever.

** How it is all related to you problem*

In this case you should be able to proof-test this approach - you could
take any simple DB and make current serializers store data into this DB.
This will allow you to fetch additional data from other sources and write
also to this config DB. E.g. if you want to fetch some data from puppet
master - do the following:
1) send a request to puppet master to compile the catalogue for a
particular node
2) parse this catalogue and extract required things into config DB
3) run deployment with new values from config db

** Existing feature tackling this issue*

BTW, there is already a feature which partially addresses your demands:
https://blueprints.launchpad.net/fuel/+spec/openstack-config-change

** Idempotency fixes*

Well, if you find any problems - please file bugs. There are some execs in
puppet code which make tasks not 100% idempotent, but these execs are
actually harmless. If you find any other issues with idempotency - please,
file bugs for them.

** Issues like '**amqp_durable_queues**' redefinition*

These issues are going to be tackled by strict responsibilities
distribution between 'serializers' part and deployment tasks part.

E.g. there should be no more than 1 task doing one thing. 'amqp_durable_queues'
should be configured exactly with one task, not with 2 tasks.

To achieve this the plugin you mentioned should only change the value of '
amqp_durable_queues' in the config DB, while the regular task that will be
called will be the one you have in the core of Fuel Library.

** Issues with deployment time*

We are working hard on introducing improved orchestration in experimental
mode in 8.0 - please check out
https://blueprints.launchpad.net/fuel/+spec/task-based-deployment-astute

Hope, my answer was not too long. Feel free to ask questions.

On Thu, Dec 3, 2015 at 5:28 PM, Roman Sokolkov 
wrote:

> Folks,
>
> little bit more research done in regards #2 usability.
>
> I've selected 13 real-world tasks from customer (i.e. update flag X in
> nova.conf):
> - 6/13 require fuel-library patching (or is #2 unusable)
> - 3/13 are OK and can be done with #2
> - 4/13 can be done with some limitations.
>
> If needed i'll provide details.
>
> Rough statistics is that *only ~20-25% of use cases can be done with #2*.
>
> Let me give a very popular use case that will fail with #2. Assume we'r
> executing whole task graph every two hours.
> We want to change nova.conf "DEFAULT/amqp_durable_queues" from False to
> True.
>
> There is no parameter in hiera for "amqp_durable_queues". We have two
> solutions here (both are bad):
> 1) Redefine "DEFAULT/amqp_durable_queues" = True in plugin task. What will
> happen on the node. amqp_durable_queues will continue changing value
> between True and False on every execution. We shouldn't do it this way.
> 2) Patch fuel-library. Value for amqp_durable_queues should be taken from
> hiera. This is also one way ticket.
>
> Thanks
>
>
>
>
>
> On Thu, Dec 3, 2015 at 11:28 AM, Oleg Gelbukh 
> wrote:
>
>> Roman,
>>
>> Thank you. This is great research.
>>
>> Could we have a conversation to discuss this? I'm especially interested
>> in idempotency problems of the fuel-library modules and the common way to
>> provide serialised data to the deployment.
>>
>> --
>> Best regards,

[openstack-dev] [Fuel] Configuration management for Fuel 7.0

[Roman Sokolkov]

Hello, folks.

We need any kind of CM for Fuel 7.0. Otherwise new project with 800+ nodes
will be near impossible to support. Customer always wants to change
something.

In our opinion, there are two major approaches for CM:

#1 Independent CM (Puppet master, Chef, Ansible, whatever)
#2 Fuel-based CM

Solution for #2
--

Fuel has all info about configuration. So we've tried to
unlock "Settings" [0] and push "deploy" button.

Major findings:

* Task idem-potency. Looks like most of the tasks are idempotent.
We've skipped 3 tasks on controller and were able to get NO downtime
for Horizon and "nova list". BTW deeper QA required.

* Standard changes. Operator can change parameters via WebUI, CLI or API.
For example, i was able to deploy Sahara. Unfortunately there is not
foolproof.
I mean some changes can lead to broken cloud...

* Non-standard changes. Any other changes can be done with plugins.
We can modify plugin tasks and scripts (all except UI flags). And then just
do "--update" + "--sync". BTW, we can change UI for particular env via API
by modifying "clusters/X/attributes".

Conclusion
--

- This works (We have service under cron that runs tasks) [1]
- NOT ready for production (in current state)
- This requires much deeper testing


I want to hear thoughts about approach above?
What is the current status/plans for CM? I saw this discussion [2]

References
--

[0]
https://github.com/rsokolkov/fuel-web/commit/366daaa2eb874c8e54c2d39be475223937cd317d
[1]
https://docs.google.com/presentation/d/12kkh1hu4ZrY9S6XXsY_HWaesFwESfxbl5czUwde8isM/edit#slide=id.p
[2] https://etherpad.openstack.org/p/lcm-use-cases

-- 
Roman Sokolkov,
Deployment Engineer,
Mirantis, Inc.
Skype rsokolkov,
rsokol...@mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev