Re: [openstack-dev] [fuel-library] Nominate Stanislaw Bogatkin to fuel-library core

[Matthew Mosesohn]

+1

On Sun, Sep 11, 2016 at 10:20 PM, Alexey Shtokolov
 wrote:
> +1
>
> Stanislaw made a great contribution!
> I would like to mention SSL-support, YAQL expressions for data-driven
> orchestration
> and his awesome live YAQL evaluator for Fuel Master node [0]
>
> [0] https://github.com/sorrowless/fuyaql
>
> ---
> WBR, Alexey Shtokolov
> Fuel Development Lead
>
> On Thu, Sep 8, 2016 at 1:17 PM, Denis Egorenko 
> wrote:
>>
>> +1
>>
>> 2016-09-08 13:06 GMT+03:00 Aleksandr Didenko :
>>>
>>> +1
>>>
>>> On Thu, Sep 8, 2016 at 11:06 AM, Sergey Vasilenko
>>>  wrote:

 +1




 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

>>>
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Egorenko Denis,
>> Senior Deployment Engineer
>> Mirantis
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel] Propose Denis Egorenko for fuel-library core

[Matthew Mosesohn]

+1 Denis is excellent at reviews and spotting CI failure root causes.
I definitely support him.

On Wed, Aug 24, 2016 at 11:36 PM, Alex Schultz  wrote:
> Ahoy Fuel Cores,
>
> I would like to propose Denis Egorenko for fuel-library core.  Denis is
> always providing great reviews[0] and continues to help keep Fuel moving
> forward.  He's the #3 reviewer and committer of the last 90 days[1] and 180
> days[2].
>
> Please vote with a +1/-1. Voting will close on August 31st.
>
> Thanks,
> -Alex
>
> [0] http://stackalytics.com/?user_id=degorenko
> [1] http://stackalytics.com/report/contribution/fuel-library/90
> [2] http://stackalytics.com/report/contribution/fuel-library/180
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] - Nominate Maksim Malchuk to Fuel Library Core

[Matthew Mosesohn]

+1. Maksim is an excellent reviewer.

On Mon, Jun 27, 2016 at 6:06 PM, Alex Schultz  wrote:
> +1
>
> On Mon, Jun 27, 2016 at 9:04 AM, Bogdan Dobrelya 
> wrote:
>>
>> On 06/27/2016 04:54 PM, Sergii Golovatiuk wrote:
>> > I am very sorry for sending without subject. I am adding subject to
>> > voting and my +1
>>
>> +1 from my side!
>>
>> >
>> > --
>> > Best regards,
>> > Sergii Golovatiuk,
>> > Skype #golserge
>> > IRC #holser
>> >
>> > On Mon, Jun 27, 2016 at 4:42 PM, Sergii Golovatiuk
>> > > wrote:
>> >
>> > Hi,
>> >
>> > I would like to nominate Maksim Malchuk to Fuel-Library Core team.
>> > He’s been doing a great job so far [0]. He’s #2 reviewer and #2
>> > contributor with 28 commits for last 90 days [1][2].
>> >
>> > Fuelers, please vote with +1/-1 for approval/objection. Voting will
>> > be open until July of 4th. This will go forward after voting is
>> > closed if there are no objections.
>> >
>> > Overall contribution:
>> > [0] http://stackalytics.com/?user_id=mmalchuk
>> > Fuel library contribution for last 90 days:
>> > [1]
>> > http://stackalytics.com/report/contribution/fuel-library/90
>> > http://stackalytics.com/report/users/mmalchuk
>> > List of reviews:
>> > [2]
>> > https://review.openstack.org/#/q/reviewer:%22Maksim+Malchuk%22+status:merged,n,z
>> > --
>> > Best regards,
>> > Sergii Golovatiuk,
>> > Skype #golserge
>> > IRC #holser
>> >
>> >
>> >
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>>
>> --
>> Best regards,
>> Bogdan Dobrelya,
>> Irc #bogdando
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Plugins] Tasks ordering between plugins

[Matthew Mosesohn]

Hi Simon,

For 8.0 and earlier, I would deploy ElasticSearch before deploy_end
and LMA collector after post_deploy_start

For Mitaka and Newton releases, the task graph now skips dependencies
that are not found for the role being processed. Now this "requires"
dependency will work that previously errored.

Best Regards,
Matthew Mosesohn

On Tue, May 17, 2016 at 6:27 PM, Simon Pasquier <spasqu...@mirantis.com> wrote:
> I'm resurrecting this thread because I didn't manage to find a satisfying
> solution to deal with this issue.
>
> First let me provide more context on the use case. The Elasticsearch/Kibana
> and LMA collector plugins need to synchronize their deployment. Without too
> many details, here is the workflow when both plugins are deployed:
> 1. [Deployment] Install the Elasticsearch/Kibana primary node.
> 2. [Deployment] Install the other Elasticsearch/Kibana nodes.
> 3. [Post-Deployment] Configure the Elasticsearch cluster.
> 4. [Post-Deployment] Install and configure the LMA collector.
>
> Task #4 should happen after #3 so we've specified the dependency in
> deployement_tasks.yaml [0] but when the Elasticsearch/Kibana plugin isn't
> deployed in the same environment (which is a valid case), it fails [1] with:
>
> Tasks 'elasticsearch-kibana-configuration, influxdb-configuration' can't be
> in requires|required_for|groups|tasks for [lma-backends] because they don't
> exist in the graph
>
> To workaround this restriction, we're using 'upload_nodes_info' as an anchor
> task [2][3] since it is always present in the graph but this isn't really
> elegant. Any suggestion to improve this?
>
> BR,
> Simon
>
> [0]
> https://github.com/openstack/fuel-plugin-lma-collector/blob/fd9337b43b6bdae6012f421e22847a1b0307ead0/deployment_tasks.yaml#L123-L139
> [1] https://bugs.launchpad.net/lma-toolchain/+bug/1573087
> [2]
> https://github.com/openstack/fuel-plugin-lma-collector/blob/56ef5c42f4cd719958c4c2ac3fded1b08fe2b90f/deployment_tasks.yaml#L25-L37
> [3]
> https://github.com/openstack/fuel-plugin-elasticsearch-kibana/blob/4c5736dadf457b693c30e20d1a2679165ae1155a/deployment_tasks.yaml#L156-L173
>
> On Fri, Jan 29, 2016 at 4:27 PM, Igor Kalnitsky <ikalnit...@mirantis.com>
> wrote:
>>
>> Hey folks,
>>
>> Simon P. wrote:
>> > 1. Run task X for plugin A (if installed).
>> > 2. Run task Y for plugin B (if installed).
>> > 3. Run task Z for plugin A (if installed).
>>
>> Simon, could you please explain do you need this at the first place? I
>> can imagine this case only if your two plugins are kinda dependent on
>> each other. In this case, it's better to do what was said by Andrew W.
>> - set 'Task Y' to require 'Task X' and that requirement will be
>> satisfied anyway (even if Task X doesn't exist in the graph).
>>
>>
>> Alex S. wrote:
>> > Before we get rid of tasks.yaml can we provide a mechanism for plugin
>> > devs could leverage to have tasks executes at specific points in the
>> > deploy process.
>>
>> Yeah, I think that may be useful sometime. However, I'd prefer to
>> avoid anchor usage as much as possible. There's no guarantees that
>> other plugin didn't make any destructive actions early, that breaks
>> you later. Anchors is good way to resolve possible conflicts, but they
>> aren't bulletproof.
>>
>> - igor
>>
>> On Thu, Jan 28, 2016 at 1:31 PM, Bogdan Dobrelya <bdobre...@mirantis.com>
>> wrote:
>> > On 27.01.2016 14:44, Simon Pasquier wrote:
>> >> Hi,
>> >>
>> >> I see that tasks.yaml is going to be deprecated in the future MOS
>> >> versions [1]. I've got one question regarding the ordering of tasks
>> >> between different plugins.
>> >> With tasks.yaml, it was possible to coordinate the execution of tasks
>> >> between plugins without prior knowledge of which plugins were installed
>> >> [2].
>> >> For example, lets say we have 2 plugins: A and B. The plugins may or
>> >> may
>> >> not be installed in the same environment and the tasks execution should
>> >> be:
>> >> 1. Run task X for plugin A (if installed).
>> >> 2. Run task Y for plugin B (if installed).
>> >> 3. Run task Z for plugin A (if installed).
>> >>
>> >> Right now, we can set task priorities like:
>> >>
>> >> # tasks.yaml for plugin A
>> >> - role: ['*']
>> >>   stage: post_deployment/1000
>> >>   type: puppet
>> >>   parameters:
>> >> puppet_manifest: puppet/manifests/task_X.pp
>> >> puppet_modul

Re: [openstack-dev] [fuel] switch to upstream haproxy module

[Matthew Mosesohn]

Hi Alex,

Collapsing our haproxy tasks makes it a bit trickier for plugin
developers. We would still be able to control it via hiera, but it
means more effort for a plugin developer to run haproxy for a given
set of services, but explicitly exclude all those it doesn't intend to
run on a custom role. Maybe you can think of some intermediate step
that wouldn't add a burden to a plugin developer that would want to
just proxy keystone and mysql, but not nova/neutron/glance/cinder?

On Thu, May 12, 2016 at 5:34 PM, Alex Schultz  wrote:
> Hey Fuelers,
>
> We have been using our own fork of the haproxy module within fuel-library
> for some time. This also includes relying on a MOS specific version of
> haproxy that carries the conf.d hack.  Unfortunately this has meant that
> we've needed to leverage the MOS version of this package when deploying with
> UCA.  As far as I can tell, there is no actual need to continue to do this
> anymore. I have been working on switching to the upstream haproxy module[0]
> so we can drop this custom haproxy package and leverage the upstream haproxy
> module.
>
> In order to properly switch to the upstream haproxy module, we need to
> collapse the haproxy tasks into a single task. With the migration to
> leveraging classes for task functionality, this is pretty straight forward.
> In my review I have left the old tasks still in place to make sure to not
> break any previous dependencies but they old tasks no longer do anything.
> The next step after this initial merge would be to cleanup the haproxy code
> and extract it from the old openstack module.
>
> Please be aware that if you were relying on the conf.d method of injecting
> configurations for haproxy, this will break you. Please speak up now so we
> can figure out an alternative solution.
>
> Thanks,
> -Alex
>
>
> [0] https://review.openstack.org/#/c/307538/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] [Ci] Re-trigger by keyword in comment

[Matthew Mosesohn]

Aleksey, actually I want to extend the test group we run there. Many
changes coming out of nailgun are actually creating BVT failures that
can only be prevented by such tests. One such extension would be
adding a plugin to the deployment to ensure that basic plugins are
still deployable.

I'm ok with tweaking recheck flags, but we should not try to avoid
using the CI that saves us from regressions.

On Fri, Apr 22, 2016 at 3:43 PM, Aleksey Kasatkin
 wrote:
> Hi Dmitry,
>
> Thank you for update.
> Is it intended that master.fuel-web.pkgs.ubuntu.review_fuel_web_deploy job
> for code requests to fuel-web runs at every recheck now?
> Before the change it was executed for new patch/rebase only.
> Its run takes about 1.5 hour and there is little sense to run it more than
> once for the same patch.
>
> Thanks,
>
>
>
> Aleksey Kasatkin
>
>
> On Fri, Apr 22, 2016 at 10:59 AM, Dmitry Kaiharodsev
>  wrote:
>>
>> Hi to all,
>>
>> please be informed that recently we've merged a patch[0]
>> that allow to re-trigger fuel-ci[1] tests by commenting review with
>> keywords "fuel: recheck"[2]
>>
>> For now actual list of Jenkins jobs with retrigger by "fuel: recheck"[2]
>> keyword looks like:
>>
>> 7.0.verify-python-fuelclient
>> 8.0.fuel-library.pkgs.ubuntu.neutron_vlan_ha
>> 8.0.fuel-library.pkgs.ubuntu.smoke_neutron
>> 8.0.verify-docker-fuel-web-ui
>> 8.0.verify-fuel-web
>> 8.0.verify-fuel-web-ui
>> fuellib_noop_tests
>> master.fuel-agent.pkgs.ubuntu.review_fuel_agent_one_node_provision
>> master.fuel-astute.pkgs.ubuntu.review_astute_patched
>> master.fuel-library.pkgs.ubuntu.neutron_vlan_ha
>> master.fuel-library.pkgs.ubuntu.smoke_neutron
>> master.fuel-ostf.pkgs.ubuntu.gate_ostf_update
>> master.fuel-web.pkgs.ubuntu.review_fuel_web_deploy
>> master.python-fuelclient.pkgs.ubuntu.review_fuel_client
>> mitaka.fuel-agent.pkgs.ubuntu.review_fuel_agent_one_node_provision
>> mitaka.fuel-astute.pkgs.ubuntu.review_astute_patched
>> mitaka.fuel-library.pkgs.ubuntu.neutron_vlan_ha
>> mitaka.fuel-library.pkgs.ubuntu.smoke_neutron
>> mitaka.fuel-ostf.pkgs.ubuntu.gate_ostf_update
>> mitaka.fuel-web.pkgs.ubuntu.review_fuel_web_deploy
>> mitaka.python-fuelclient.pkgs.ubuntu.review_fuel_client
>> old.verify-nailgun_performance_tests
>> verify-fuel-astute
>> verify-fuel-devops
>> verify-fuel-docs
>> verify-fuel-library-bats-tests
>> verify-fuel-library-puppetfile
>> verify-fuel-library-python
>> verify-fuel-library-tasks
>> verify-fuel-nailgun-agent
>> verify-fuel-plugins
>> verify-fuel-qa-docs
>> verify-fuel-stats
>> verify-fuel-ui-on-fuel-web
>> verify-fuel-web-docs
>> verify-fuel-web-on-fuel-ui
>> verify-nailgun_performance_tests
>> verify-puppet-modules.lint
>> verify-puppet-modules.syntax
>> verify-puppet-modules.unit
>> verify-python-fuelclient
>> verify-python-fuelclient-on-fuel-web
>> verify-sandbox
>>
>>
>> [0] https://review.fuel-infra.org/#/c/17916/
>> [1] https://ci.fuel-infra.org/
>> [2] without quotes
>> --
>> Kind Regards,
>> Dmitry Kaigarodtsev
>> Mirantis, Inc.
>>
>> +38 (093) 522-09-79 (mobile)
>> +38 (057) 728-4214 (office)
>> Skype: d1mas85
>>
>> 38, Lenin avenue
>> Kharkov, Ukraine
>> www.mirantis.com
>> www.mirantis.ru
>> dkaiharod...@mirantis.com
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel] Unrelated changes in patches

[Matthew Mosesohn]

Hi Dmitry,

I've seen several cases where core reviewers bully contributors into
refactoring a particular piece of logic because it contains common
lines relating to some non-ideal code, even if the change doesn't
relate to this logic.
In general, I'm ok with formatting issues, but changing how a piece of
existing code works is over the line. It should be handled as a
separate bug.

But yes, in general, if someone complains about something unrelated to
your patch, he or she should just file a bug with what is required.

-Matthew


On Mon, Apr 4, 2016 at 3:46 PM, Dmitry Guryanov  wrote:
> Hello, colleagues!
>
> It's often not so easy to decide, if you should include some unrelated
> changes to your patch, like fixing spaces, renaming variables or something
> else, which don't change logic. On the one hand you see something's wrong
> with the code and you'd like to fix it, on the other hand reviewers can vote
> of -1 and you'll have to fix you patch and upload it again and this is very
> annoying. You can also create separate review for such changes, but it will
> require additional effort from you and reviewers.
>
> If you are a reviewer, and you've noted unrelated changes you may hesitate,
> if you should ask an author to remove them and upload new version of the
> patch or not. Also such extra changes may confuse you sometimes.
>
> So I suggest creating separate patches for unrelated changes if they add new
> chucks to patch. And I'd like to ask authors to clearly state in the subject
> of a commit message, that this patch just fixes formatting. And reviewers
> shouldn't check such patches too severely, so that they'll get into repo as
> soon as possible.
>
> What do you think?
>
>
> --
> Dmitry Guryanov
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel] FFE for fuel-openstack-tasks and fuel-remove-conflict-openstack

[Matthew Mosesohn]

Andrew,

The stubs + deprecation warning is exactly the approach I believewe should
take for renaming/moving tasks.

If it was possible for a plugin to override a task, but preserve the fields
from the original task, we could avoid such scenarios. What I mean is that
if the following task:

- id: workloads_collector_add
  type: puppet
  version: 2.0.0
  groups: [primary-controller]
  required_for: [deploy_end]
  requires: [keystone, primary-keystone]
  parameters:
puppet_manifest:
/etc/puppet/modules/osnailyfacter/modular/keystone/workloads_collector_add.pp
puppet_modules: /etc/puppet/modules
timeout: 1800

If we could override the groups field only, a plugin developer would not
need to copy and paste the dependencies and other parameters. But until
that works, we should effectively deprecate top level tasks whenever
possible.
On Mar 22, 2016 2:52 AM, "Andrew Woodward"  wrote:

> I've mocked up the change to implementation using the already landed
> changes to ceph as an example
>
> https://review.openstack.org/295571
>
> On Mon, Mar 21, 2016 at 3:44 PM Andrew Woodward  wrote:
>
>> We had originally planned for the FFEs for both fuel-openstack-tasks[1]
>> and fuel-remove-conflict-openstack to [2] to close on 3/20, This would have
>> placed them before changes that conflict with
>> fuel-refactor-osnailyfacter-for-puppet-master-compatibility [3].
>>
>> [1]
>> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088297.html
>> [2]
>> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088298.html
>> [3]
>> http://lists.openstack.org/pipermail/openstack-dev/2016-March/089028.html
>>
>> However we found this morning that the changes from [2], and more of
>> issue [1] will result in further issues such as [4], where as the task
>> files move, any task that explicitly relied on it, now no longer is in the
>> same path.
>>
>> [4] https://review.openstack.org/#/c/295170/
>>
>> Due to this newly identified issue with backwards comparability. It
>> appears that [4] shows that we have plugins using interfaces that we don't
>> have formal coverage for so If we introduce this set of changes, we will
>> cause breakage for plugins that use fuel's current tasks.
>>
>> From a deprecation standpoint we don't have a way to deal with this,
>> unless  fuel-openstack-tasks [1] lands after
>> fuel-refactor-osnailyfacter-for-puppet-master-compatibility [3]. In this
>> case we can take advantage of the class include stubs, leaving a copy in
>> the old location (osnailyfacter/modular/roles/compute.pp) pointing to the
>> new include location (include openstack_tasks::roles::compute) and adding a
>> warning for deprecation. The tasks includes in the new location
>> openstack_tasks/examples/roles/compute.pp would simply include the updated
>> class location w/o the warning.
>>
>> This would take care of [1] and it's review [5]
>>
>> [5] https://review.openstack.org/283332
>>
>> This still leaves [2] un-addressed, we still have 3 open CR for it:
>>
>> [6] Compute https://review.openstack.org/285567
>> [7] Cinder https://review.openstack.org/294736
>> [8] Swift https://review.openstack.org/294979
>>
>> Compute [6] is in good shape, while Cinder [7] and Swift [8] are not. For
>> these do we want to continue to land them, if so what do we want to do
>> about the now deprecated openstack:: tasks? We could leave them in place
>> with a warning since we would not be using them
>>
>> --
>>
>> --
>>
>> Andrew Woodward
>>
>> Mirantis
>>
>> Fuel Community Ambassador
>>
>> Ceph Community
>>
> --
>
> --
>
> Andrew Woodward
>
> Mirantis
>
> Fuel Community Ambassador
>
> Ceph Community
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [fuel] UCA deployment landed and working

[Matthew Mosesohn]

Hi all,

Thanks to Alex Schultz, we landed the Ubuntu Cloud Archive as a
possible deployment source last week just in time for the feature
freeze exception. No sooner than landing it, we have a new task to
switch it to a separate Fuel release[1]. Patches are on review to fix
this now[2]. But now it seems that deployment should pass using Ubuntu
Cloud Archive with Fuel.

There were some packaging issues blocking deployment last week, but
now with openstackclient 2.2.0 in trusty-proposed/mitaka, it should
all be working. I want to thank Corey Bryant and James Page from the
UCA team for rebuilding client libraries so quickly.

I'm still running through testing cycles, and I'll report when we have
a passed BVT using UCA.

[1] https://bugs.launchpad.net/fuel/+bug/1556011
[2] https://review.openstack.org/293044 https://review.openstack.org/#/c/292340/

Best Regards,
MAtthew Mosesohn

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel][plugins] Should we maintain example plugins?

[Matthew Mosesohn]

Mike,

#1 - If you truly agree with that, you should weigh in here:
https://review.openstack.org/#/c/287286/
#2 - Requires a blueprint and new docs, but okay.
#3 - Yes! We have very poor CI for fuel plugin builder. All it does is
ensure it makes a plugin, not that it can be installed and deployed.

On Thu, Mar 10, 2016 at 10:44 AM, Mike Scherbakov
<mscherba...@mirantis.com> wrote:
> Folks,
> here is what I think:
> 1) I suggest to fix what is broken now. So that people who come across
> examples would not have to deal with issues. We may debate for other few
> days (hopefully not more), and all plugin devs will be suffering during this
> time. Also, according to Matt,
>
>> I should add that this is the only automated daily test that will verify
> that our plugin framework actually works.
> simply means that we must fix it in order to catch any possible regressions
> we may introduce later (if not already).
>
> 2) I like idea Ilya K. has proposed. To rephrase it (to put extra accent
> into it and get feedback), we may not need to have example plugins. However,
> we can have fpb generating template plugin, with commented code in there. If
> you uncomment, you a get a comprehensive example of a working plugin.
> To ensure that uncommented code would actually work, we must have a test for
> it (which would uncomment - run - ensure everything deploys).
>
> 3) Ideally, we need to catch issues at pre-commit stage. So I'd suggest to
> think if there is a way to have tests, which would run such examples against
> pluggable framework for every change to the framework, so that we can have
> -1 right away if something goes wrong.
>
> I've started separate thread on general thoughts about backward
> compatibility and multiple releases support, which actually affects
> examples: [1]
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088921.html
>
> Thanks,
>
> On Wed, Mar 9, 2016 at 7:59 AM Evgeniy L <e...@mirantis.com> wrote:
>>
>> Because it doesn't make much sense for a plugin developer to have scripts
>> to build packages for installation on slave nodes. For default template it's
>> better to have something minimalistic and the rest of the tasks commented,
>> otherwise it may create a lot of confusion.
>>
>> On Wed, Mar 9, 2016 at 6:37 PM, Ilya Kutukov <ikutu...@mirantis.com>
>> wrote:
>>>
>>> Talking about templates I mean plugins generated by FBP from the
>>> `templates` folder using command you mentioned above.
>>>
>>> Why not to extend (not replace) template-generated plugins with
>>> additional data to provide right coverage?
>>>
>>> On Wed, Mar 9, 2016 at 6:23 PM, Evgeniy L <e...@mirantis.com> wrote:
>>>>
>>>> Ilya,
>>>>
>>>> What do you mean by "templates" the plugin which is create by just "fpb
>>>> --create plugin-name"?
>>>> It doesn't cover enough, package installation and all range of tasks
>>>> executions.
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, Mar 9, 2016 at 6:16 PM, Ilya Kutukov <ikutu...@mirantis.com>
>>>> wrote:
>>>>>
>>>>> Igor, i completely agree, actually plugin examples is almost a
>>>>> copy-paste.
>>>>>
>>>>> The only thing i see useful in the built plugins example is the ability
>>>>> to point some code lines, discussing plugin design and writing
>>>>> documentation. Why not to generate this examples automatically from
>>>>> templates?
>>>>>
>>>>> Also, as developer and administrator i love to use examples/templates
>>>>> where all essential settings and options are persist but commented (e.g.
>>>>> ProFTPD or Apache) and i could uncomment and copy-paste them not being
>>>>> afraid of typos. Also it allows to keep documentation actual and head
>>>>> documentation small. Duplication of inline documentation between examples
>>>>> and templates making things more weird and overshadows idea of inline
>>>>> documentation itself.
>>>>>
>>>>> Eugeny, why not to run integration tests against plugins generated from
>>>>> templates? It's will be even better integration tests.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 7, 2016 at 4:49 PM, Igor Kalnitsky
>>>>> <ikalnit...@mirantis.com> wrote:
>>>>>>
>>>>>> > and really lowering barriers for people who just begin create
>>>

Re: [openstack-dev] [Fuel] Compatibility of fuel plugins and fuel versions

[Matthew Mosesohn]

Hi Mike,

Normally I would support your idea, but the reality is any plugin from
7.0 that defines a plugin role isn't going to work in 8.0 or 9.0. We
changed too many task names and you just can't make a plugin support
both versions not without something incredibly clever that I
haven't thought of already. If I'm a plugin developer, I'm not going
to advertise support for versions that won't work or expect it to
work when I haven't stated it explicitly.

The example plugins are quite simple and have no real tasks, so
enabling this for plugins is ok.  For real plugins that do more then
install 1 package and enable 1 service, version limiting is the only
thing to keep your users from losing their hair trying to figure out
why it doesn't work.

Best Regards,
Matthew Mosesohn

On Thu, Mar 10, 2016 at 10:30 AM, Mike Scherbakov
<mscherba...@mirantis.com> wrote:
> Hi folks,
> in order to make a decision whether we need to support example plugins, and
> if actually need them [1], I'd suggest to discuss more common things about
> plugins.
>
> My thoughts:
> 1) This is not good, that our plugins created for Fuel 8 won't even install
> on Fuel 9. By default, we should assume that plugin will work at newer
> version of Fuel. However, for proper user experience, I suggest to create
> meta-field "validated_against", where plugin dev would provide versions of
> Fuel this plugin has been tested with. Let's say, it was tested against 7.0,
> 8.0. If user installs plugin in Fuel 9, I'd suggest to show a warning saying
> about risks and the fact that the plugin has not been tested against 9. We
> should not restrict intsallation against 9, though.
>
> 2) We need to keep backward compatibility of pluggable interface for a few
> releases. So that plugin developer can use pluggable interface of version x,
> which was supported in Fuel 6.1. If we still support it, it would mean (see
> next point) compatibility of this plugin with 6.1, 7.0, 8.0, 9.0. If we want
> to deprecate pluggable interface version, we should announce it, and
> basically follow standard process of deprecation.
>
> 3) Plugin's ability to work against multiple releases of Fuel (multi-release
> support). If if..else clauses to support multiple releases are fairly
> minimal, let's say take less that 10% of LOC, I'd suggest to have this
> supported. Just because it will be easier for plugin devs to support their
> plugin code (no code duplication, single repo for multiple releases).
>
> Thoughts?
>
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088211.html
> --
> Mike Scherbakov
> #mihgen
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] proposal to create puppet-neutron-core and add Sergey Kolekonov

[Matthew Mosesohn]

I'm not core, but I would like to say his contributions for Mitaka
were invaluable and I've greatly benefited from his efforts :)

On Fri, Mar 4, 2016 at 6:49 PM, Cody Herriges  wrote:
> Emilien Macchi wrote:
>> Hi,
>>
>> To scale-up our review process, we created pupept-keystone-core and it
>> worked pretty well until now.
>>
>> I propose that we continue this model and create puppet-neutron-core.
>>
>> I also propose to add Sergey Kolekonov in this group.
>> He's done a great job helping us to bring puppet-neutron rock-solid for
>> deploying OpenStack networking.
>>
>> http://stackalytics.com/?module=puppet-neutron=marks
>> http://stackalytics.com/?module=puppet-neutron=commits
>> 14 commits and 47 reviews, present on IRC during meetings & bug triage,
>> he's always helpful. He has a very good understanding of Neutron &
>> Puppet so I'm quite sure he would be a great addition.
>>
>> As usual, please vote!
>
> +1 from me.  Excited to continue seeing neutron get better.
>
> --
> Cody
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel][plugins] Should we maintain example plugins?

[Matthew Mosesohn]

Igor,

It seems you are proposing an IKEA approach to plugins. Take Fuel's
example plugin, add in the current Fuel release, and then build it. We
maintained these plugins in the past, but now it should a manual step
to test it out on the current release.

What would be a more ideal situation that meets the needs of users and
QA? Right now we have failed tests until we can decide on a solution
that works for everybody.

On Fri, Mar 4, 2016 at 1:26 PM, Igor Kalnitsky <ikalnit...@mirantis.com> wrote:
> No, this is a wrong road to go.
>
> What if in Fuel 10 we drop v1 plugins support? What should we do?
> Remove v1 example from source tree? That doesn't seem good to me.
>
> Example plugins are only examples. The list of supported releases must
> be maintained on system test side, and system tests must inject that
> information into plugin's metadata.yaml and test it.
>
> Again, I don't say we shouldn't test plugins. I say, tests should be
> responsible for preparing plugins. I can say even more: tests should
> not rely on what is produced by plugins, since it's something that
> could be changed and tests start failing.
>
> On Thu, Mar 3, 2016 at 7:54 PM, Swann Croiset <scroi...@mirantis.com> wrote:
>> IMHO it is important to keep plugin examples and keep testing them, very
>> valuable for plugin developers.
>>
>> For example, I've encountered [0] the case where "plugin as role" feature
>> wasn't easily testable with fuel-qa because not compliant with the last
>> plugin data structure,
>> and more recently we've spotted a regression [1] with "vip-reservation"
>> feature introduced by a change in nailgun.
>> These kind of issues are time consuming for plugin developers and can/must
>> be avoided by testing them.
>>
>> I don't even understand why the question is raised while fuel plugins are
>> supposed to be supported and more and more used [3], even by murano [4] ...
>>
>> [0] https://bugs.launchpad.net/fuel/+bug/1543962
>> [1] https://bugs.launchpad.net/fuel/+bug/1551320
>> [3]
>> http://lists.openstack.org/pipermail/openstack-dev/2016-February/085636.html
>> [4] https://review.openstack.org/#/c/286310/
>>
>> On Thu, Mar 3, 2016 at 3:19 PM, Matthew Mosesohn <mmoses...@mirantis.com>
>> wrote:
>>>
>>> Hi Fuelers,
>>>
>>> I would like to bring your attention a dilemma we have here. It seems
>>> that there is a dispute as to whether we should maintain the releases
>>> list for example plugins[0]. In this case, this is for adding version
>>> 9.0 to the list.
>>>
>>> Right now, we run a swarm test that tries to install the example
>>> plugin and do a deployment, but it's failing only for this reason. I
>>> should add that this is the only automated daily test that will verify
>>> that our plugin framework actually works. During the Mitaka
>>> development  cycle, we already had an extended period where plugins
>>> were broken[1]. Removing this test (or leaving it permanently red,
>>> which is effectively the same), would raise the risk to any member of
>>> the Fuel community who depends on plugins actually working.
>>>
>>> The other impact of abandoning maintenance of example plugins is that
>>> it means that a given interested Fuel Plugin developer would not be
>>> able to easily get started with plugin development. It might not be
>>> inherently obvious to add the current Fuel release to the
>>> metadata.yaml file and it would likely discourage such a user. In this
>>> case, I would propose that we remove example plugins from fuel-plugins
>>> GIT repo if they are not maintained. Non-functioning code is worse
>>> than deleted code in my opinion.
>>>
>>> Please share your opinions and let's decide which way to go with this
>>> bug[2]
>>>
>>> [0] https://github.com/openstack/fuel-plugins/tree/master/examples
>>> [1] https://bugs.launchpad.net/fuel/+bug/1544505
>>> [2] https://launchpad.net/bugs/1548340
>>>
>>> Best Regards,
>>> Matthew Mosesohn
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: o

[openstack-dev] [fuel][plugins] Should we maintain example plugins?

[Matthew Mosesohn]

Hi Fuelers,

I would like to bring your attention a dilemma we have here. It seems
that there is a dispute as to whether we should maintain the releases
list for example plugins[0]. In this case, this is for adding version
9.0 to the list.

Right now, we run a swarm test that tries to install the example
plugin and do a deployment, but it's failing only for this reason. I
should add that this is the only automated daily test that will verify
that our plugin framework actually works. During the Mitaka
development  cycle, we already had an extended period where plugins
were broken[1]. Removing this test (or leaving it permanently red,
which is effectively the same), would raise the risk to any member of
the Fuel community who depends on plugins actually working.

The other impact of abandoning maintenance of example plugins is that
it means that a given interested Fuel Plugin developer would not be
able to easily get started with plugin development. It might not be
inherently obvious to add the current Fuel release to the
metadata.yaml file and it would likely discourage such a user. In this
case, I would propose that we remove example plugins from fuel-plugins
GIT repo if they are not maintained. Non-functioning code is worse
than deleted code in my opinion.

Please share your opinions and let's decide which way to go with this bug[2]

[0] https://github.com/openstack/fuel-plugins/tree/master/examples
[1] https://bugs.launchpad.net/fuel/+bug/1544505
[2] https://launchpad.net/bugs/1548340

Best Regards,
Matthew Mosesohn

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel][FFE] Enable UCA repositories for deployment

[Matthew Mosesohn]

Hi all,

I would like to request a feature freeze exception for "Deploy with
UCA packages" feature.

I anticipate 2 more days to get tests green and add some depth to the
existing test.

https://blueprints.launchpad.net/fuel/+spec/deploy-with-uca-packages

The impact to BVT stability is quite small because it only touches 1
task in OpenStack deployment, and by default it is not enabled.

Open reviews:
https://review.openstack.org/#/c/281762/
https://review.openstack.org/#/c/279556/
https://review.openstack.org/#/c/279542/
https://review.openstack.org/#/c/284584/

Best Regards,
Matthew Mosesohn

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Fuel-Library] Nominating Matthew Mosesohn for Fuel Library Core

[Matthew Mosesohn]

Hi all,

Thank you for the nominations and +1s. I would like to propose Michael
Polenchuk to add as a maintainer to fuel-library to take my spot when
I leave the maintainers list.

Best Regards,
Matthew Mosesohn

On Fri, Feb 26, 2016 at 3:54 PM, Kyrylo Galanov <kgala...@mirantis.com> wrote:
> Finally! +2 !
>
> On Fri, Feb 26, 2016 at 9:08 PM, Roman Vyalov <rvya...@mirantis.com> wrote:
>>
>> +1
>>
>> On Fri, Feb 26, 2016 at 12:31 PM, Aleksey Kasatkin
>> <akasat...@mirantis.com> wrote:
>>>
>>> +1
>>>
>>>
>>> Aleksey Kasatkin
>>>
>>>
>>> On Thu, Feb 25, 2016 at 11:59 PM, Sergey Vasilenko
>>> <svasile...@mirantis.com> wrote:
>>>>
>>>> +1
>>>>
>>>>
>>>> /sv
>>>>
>>>>
>>>>
>>>> __
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] is puppet-keystone using v3 credentials correctly ?

[Matthew Mosesohn]

Hi Michal,

Just add --os-identity-api-version=3 to your command it will work. The
provider uses v3 openstackclient via env var
OS_IDENTITY_API_VERSION=3. The default is still 2.

Best Regards,
Matthew Mosesohn

On Fri, Feb 19, 2016 at 5:25 PM, Matt Fischer <m...@mattfischer.com> wrote:
> What version of openstack client do you have? What version of the module are
> you using?
>
> On Feb 19, 2016 7:20 AM, "Ptacek, MichalX" <michalx.pta...@intel.com> wrote:
>>
>> Hi all,
>>
>>
>>
>> I was playing some time with puppet-keystone deployments,
>>
>> and also reported one issue related to this:
>>
>> https://bugs.launchpad.net/puppet-keystone/+bug/1547394
>>
>> but in general my observations are that keystone_service is using v3
>> credentials with openstack cli commands that are not compatible
>>
>>
>>
>> e.g.
>>
>> Error: Failed to apply catalog: Execution of '/bin/openstack service list
>> --quiet --format csv --long' returned 2: usage: openstack service list [-h]
>> [-f {csv,table}] [-c COLUMN]
>>   [--max-width ]
>>   [--quote {all,minimal,none,nonnumeric}]
>> openstack service list: error: unrecognized arguments: --long
>>
>>
>>
>>
>>
>> It can’t be bug, because whole module will not work due to this J
>>
>> I think I miss something important somewhere …
>>
>>
>>
>> My latest manifest file is :
>>
>>
>>
>> Exec { logoutput => 'on_failure' }
>>
>> package { 'curl': ensure => present }
>>
>>
>>
>> node keystone {
>>
>>
>>
>>   class { '::mysql::server': }
>>
>>   class { '::keystone::db::mysql':
>>
>> password => 'keystone',
>>
>>   }
>>
>>
>>
>>   class { '::keystone':
>>
>> verbose => true,
>>
>> debug   => true,
>>
>> database_connection => 'mysql://keystone:keystone@127.0.0.1/keystone',
>>
>> catalog_type=> 'sql',
>>
>> admin_token => 'admin_token',
>>
>>   }
>>
>>
>>
>>   class { '::keystone::roles::admin':
>>
>> email=> 'exam...@abc.com',
>>
>> password => 'ChangeMe',
>>
>>   }
>>
>>
>>
>>   class { '::keystone::endpoint':
>>
>> public_url => "http://${::fqdn}:5000/v2.0;,
>>
>> admin_url  => "http://${::fqdn}:35357/v2.0;,
>>
>>   }
>>
>> }
>>
>>
>>
>> Env variables looks as follows(before service list is called with --long)
>>
>> {"OS_IDENTITY_API_VERSION"=>"3", "OS_TOKEN"=>"admin_token",
>> "OS_URL"=>"http://127.0.0.1:35357/v3"}
>>
>> Debug: Executing: '/bin/openstack service list --quiet --format csv
>> --long'
>>
>>
>>
>> Thanks for any hint,
>>
>> Michal
>>
>> --
>> Intel Research and Development Ireland Limited
>> Registered in Ireland
>> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
>> Registered Number: 308263
>>
>> This e-mail and any attachments may contain confidential material for the
>> sole use of the intended recipient(s). Any review or distribution by others
>> is strictly prohibited. If you are not the intended recipient, please
>> contact the sender and delete all copies.
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Fuel 9.0 (Mitaka) deployment with Ubuntu UCA packages

[Matthew Mosesohn]

Hi Fuelers and Stackers,

I am pleased to announce the first possibility to deploy Mitaka using
Fuel as a deployment tool. I am taking advantage of Alex Schultz's
plugin, fuel-plugin-upstream[0], along with a series of patches
currently on review[1]. I have not had a chance to do destructive
tests yet, but OSTF health checks are (nearly all) passing. Tests show
that we can complete deployment with either ceph or swift
successfully.

The positive side of all of this experience shows that we can deploy
both Liberty and Mitaka (b1) on the same manifests.

One item to note is that the Nova EC2 API has gone away completely in
Mitaka, and haproxy configuration is updated to compensate for this.

Finally, I should add that our current custom automated BVT script
can't install a plugin, so I've written 2 patches[2] to hack in
fuel-plugin-upstream's tasks.

The only failure during deployment is OSTF reported that nova metadata
and nova osapi_compute services are down. Other tests pass just fine.
For those interested, I've attached a link to the logs from the
deployment[3].


This achievement moves us closer to the goal of enabling Fuel to
deploy OpenStack using non-Mirantis packages.


[0] https://github.com/mwhahaha/fuel-plugin-upstream
[1] Mitaka support patches (in order):
https://review.openstack.org/#/c/267697/11
https://review.openstack.org/#/c/268147/13
https://review.openstack.org/#/c/269564/5
https://review.openstack.org/#/c/268214/11
https://review.openstack.org/#/c/268082/9
https://review.openstack.org/#/c/267448/7
https://review.openstack.org/#/c/272557/2

[2] https://review.openstack.org/#/c/269752/ https://review.openstack.org/269749

[3] 
https://drive.google.com/file/d/0B0UMyn5tu8EUdkUxMUs3Z0FxbGs/view?usp=sharing
(.tar.xz format)

Best Regards,
Matthew Moseoshn

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [oslo] Proposal of adding puppet-oslo to OpenStack

[Matthew Mosesohn]

I would personally like to see Keystone get transitioned first, but it
really doesn't matter where we start if we reach the right goal in the end.
Since Emelien's work on refactoring all the providers for puppet-keystone,
it has become a test bed for project-wide features. I'm really excited to
see consistency in oslo config across services, so keep up the good work!

On Sun, Jan 24, 2016 at 7:05 AM, Xingchao Yu  wrote:

> Hi, all:
>
> I spend some times to collect oslo.* versions of openstack
> projects(which has related puppet module), please check it in following
> table:
>
> https://github.com/openstack/puppet-oslo#module-description
>
> From the table, we can find most of oslo.* libraries are the same
> among the openstack projects(except aodh, gnocchi).
>
> So from the table, we could use puppet-oslo to replace configuration
> of oslo.* in related modules gradually.
>
> Thanks & Regards.
>
>
> 2016-01-21 23:58 GMT+08:00 Emilien Macchi :
>
>>
>>
>> On 01/21/2016 08:15 AM, Doug Hellmann wrote:
>> > Excerpts from Cody Herriges's message of 2016-01-19 15:50:05 -0800:
>> >> Colleen Murphy wrote:
>> >>> On Tue, Jan 19, 2016 at 9:57 AM, Xingchao Yu > >>> > wrote:
>> >>>
>> >>> Hi, Emilien:
>> >>>
>> >>>  Thanks for your efforts on this topic, I didn't attend V
>> >>> release summit and missed related discussion about puppet-oslo.
>> >>>
>> >>>  As the reason for not using a unified way to manage oslo_*
>> >>> parameters is there maybe exist different oslo_* version between
>> >>> openstack projects.
>> >>>
>> >>>  I have an idea to solve this potential problem,we can
>> maintain
>> >>> several versions of puppet-oslo, each module can map to different
>> >>> version of puppet-oslo.
>> >>>
>> >>> It would be something like follows: (the map info is not true,
>> >>> just for example)
>> >>>
>> >>> In Mitaka release
>> >>> puppet-nova maps to puppet-oslo with 8.0.0
>> >>> puppet-designate maps to puppet-oslo with 7.0.0
>> >>> puppet-murano maps to puppet-oslo with 6.0.0
>> >>>
>> >>> In Newton release
>> >>> puppet-nova maps to puppet-oslo with 9.0.0
>> >>> puppet-designate maps to puppet-oslo with 9.0.0
>> >>> puppet-murano maps to puppet-oslo with 7.0.0
>> >>>
>> >>> For the simplest case of puppet infrastructure configuration, which
>> is a
>> >>> single puppetmaster with one environment, you cannot have multiple
>> >>> versions of a single puppet module installed. This means you
>> absolutely
>> >>> cannot have an openstack infrastructure depend on having different
>> >>> versions of a single module installed. In your example, a user would
>> not
>> >>>  be able to use both puppet-nova and puppet-designate since they are
>> >>> using different versions of the puppet-oslo module.
>> >>>
>> >>> When we put out puppet modules, we guarantee that version X.x.x of a
>> >>> given module works with the same version of every other module, and
>> this
>> >>> proposal would totally break that guarantee.
>> >>>
>> >>
>> >> How does OpenStack solve this issue?
>> >>
>> >> * Do they literally install several different versions of the same
>> >> python library?
>> >> * Does every project vendor oslo?
>> >> * Is the oslo library its self API compatible with older versions?
>> >
>> > Each Oslo library has its own version. Only one version of each
>> > library is installed at a time. We use the global requirements list
>> > to sync compatible requirements specifications across all OpenStack
>> > projects to make them co-installable. And we try hard to maintain
>> > API compatibility, using SemVer versioning to indicate when that
>> > was not possible.
>> >
>> > If you want to have a single puppet module install all of the Oslo
>> > libraries, you could pull the right versions from the
>> upper-constraints.txt
>> > file in the openstack/requirements repository. That file lists the
>> > versions that were actually tested in the gate.
>>
>> Thanks for this feedback Doug!
>> So I propose we create the module in openstack namespace, please vote for:
>> https://review.openstack.org/#/c/270872/
>>
>> I talked with xingchao on IRC #puppet-openstack and he's doing
>> project-config patch today.
>> Maybe could we start with Nova, Neutron, Cinder, Glance, Keystone, see
>> how it works and iterate later with other modules.
>>
>> Thoughts are welcome,
>> --
>> Emilien Macchi
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Xingchao Yu
>
>
>
> __
> OpenStack Development Mailing List 

Re: [openstack-dev] [Fuel] Relieving CI/gate jenkins bottleneck

[Matthew Mosesohn]

+1 for defaulting to upstream for CI. If we have a strong case where we
need to make a patch in order to make unit tests pass, we could switch a
module back to review.fuel-infra.org/puppet-modules/ with a FIXME and a
LP bug ID so we can switch it back quickly once the upstream issue is
resolved.  As far as I know, we don't have to worry about that scenario.

-Matthew

On Fri, Jan 22, 2016 at 12:51 PM, Bogdan Dobrelya <bdobre...@mirantis.com>
wrote:

> Another point is to use upstream links for modules w/o downstream patches.
> I noticed we *always* put it to the deployment/Puppetfile [0] as
>  "https://review.fuel-infra.org/puppet-modules/...;. Why should we?
> Let's just do the best to reuse upstream modules as is, eventually.
>
> [0]
> https://github.com/openstack/fuel-library/blob/master/deployment/Puppetfile
>
> Regards,
> Bogdan Dobrelya.
> Irc #bogdando
>
> 2016-01-21 11:09 GMT+01:00 Bartlomiej Piotrowski <bpiotrow...@mirantis.com
> >:
>
>> Let's drop 3.3 as well. 3.4 is oldschool enough for vintage lovers.
>>
>> BP
>>
>> On Thu, Jan 21, 2016 at 11:03 AM, Aleksandr Didenko <
>> adide...@mirantis.com> wrote:
>>
>>> Hi,
>>>
>>> > I also think 3.3 is the version that ships with 14.04.
>>>
>>> 3.4.3 is shipped with Ubuntu-14.04. I think 3.4, 3.8 and 4 should be
>>> enough.
>>>
>>> Regards,
>>> Alex
>>>
>>> On Wed, Jan 20, 2016 at 6:38 PM, Sergii Golovatiuk <
>>> sgolovat...@mirantis.com> wrote:
>>>
>>>> +1 for 3.3, 3.4, 3.8 and 4
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Sergii Golovatiuk,
>>>> Skype #golserge
>>>> IRC #holser
>>>>
>>>> On Wed, Jan 20, 2016 at 6:12 PM, Alex Schultz <aschu...@mirantis.com>
>>>> wrote:
>>>>
>>>>> On Wed, Jan 20, 2016 at 9:02 AM, Matthew Mosesohn
>>>>> <mmoses...@mirantis.com> wrote:
>>>>> > Hi all,
>>>>> >
>>>>> > Unit tests on CI and gate bottleneck are really slowing down commit
>>>>> > progress. We recently had a meeting to discuss possible ways to
>>>>> improve
>>>>> > this, including symlinks, caching git repositories, etc, but one
>>>>> thing we
>>>>> > can do much faster is to simply disable 3.3-3.7 puppet jobs. We
>>>>> don't deploy
>>>>> > Fuel 9.0 (or 8.0) on earlier Puppet versions, so what value is there
>>>>> to the
>>>>> > checks? I propose we remove these tests, and hopefully we will see
>>>>> some
>>>>> > immediate relief.
>>>>> >
>>>>>
>>>>> How about we reduce to 3.3, 3.4, 3.8 and 4?  We would remove  3.6 and
>>>>> 3.7 which would reduce the number of jobs by a third  The goal of
>>>>> keeping the others was to ensure that if/when we are able to install
>>>>> fuel-library without our version of puppet that a user could use
>>>>> whatever version their environment has. There were some changes
>>>>> between 3.3 and 3.4 (if I remember correctly) so we should keep
>>>>> checking that as it's also the oldest version supported by the
>>>>> upstream puppet openstack modules.  I also think 3.3 is the version
>>>>> that ships with 14.04.  Additionally we used 3.4 in fuel 7 and below
>>>>> so we should keep those around.
>>>>>
>>>>> -Alex
>>>>>
>>>>> > Best Regards,
>>>>> > Matthew Mosesohn
>>>>> >
>>>>> >
>>>>> __
>>>>> > OpenStack Development Mailing List (not for usage questions)
>>>>> > Unsubscribe:
>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> >
>>>>>
>>>>>
>>>>> __
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>
>>>

[openstack-dev] [Fuel] Relieving CI/gate jenkins bottleneck

[Matthew Mosesohn]

Hi all,

Unit tests on CI and gate bottleneck are really slowing down commit
progress. We recently had a meeting to discuss possible ways to improve
this, including symlinks, caching git repositories, etc, but one thing we
can do much faster is to simply disable 3.3-3.7 puppet jobs. We don't
deploy Fuel 9.0 (or 8.0) on earlier Puppet versions, so what value is there
to the checks? I propose we remove these tests, and hopefully we will see
some immediate relief.

Best Regards,
Matthew Mosesohn
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel][plugin] node_role only need when attribute false - where is the fuel plugin parser code?

[Matthew Mosesohn]

Hi Nikolas,

I'm not exactly sure about your case, but you should try something like
this:
https://github.com/openstack/fuel-plugin-detach-keystone/blob/master/node_roles.yaml#L14-L15
restrictions:
- condition: "settings:opendaylight_plugin:use_external_odl == false"
- message: "OpenDaylight role can only be used without external ODL"

On Wed, Jan 20, 2016 at 9:41 PM, Nikolas Hermanns <
nikolas.herma...@ericsson.com> wrote:

> Hey,
>
> I am developing on a fuel plugin at the moment. (fuel-plugin-opendaylight)
> In node_roles.yaml I would like to define something similar to:
> opendaylight:
>   limits:
> max: 1
> min: if "attributes:use_external_odl == true" then 0 else 1
>
> attributes:use_external_odl comes from the environment_config. Is such
> thing possible? And where is the code where that logic is actually build
> up. Which repo is having that?
>
> BR Nikolas
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel] RabbitMQ in dedicated network

[Matthew Mosesohn]

I agree. As far as I remember, rabbit needs fqdns to work and map
correctly. I think it means we should disable the ability to move the
internal messaging network role in order to fix this bug until we can add
extra dns entries per network role (or at least addr)
On Dec 23, 2015 8:42 PM, "Andrew Maksimov"  wrote:

> Hi Kirill,
>
> I don't think we can give up on using fqdn node names for RabbitMQ because
> we need to support TLS in the future.
>
> Thanks,
> Andrey Maximov
> Fuel Project Manager
>
> On Wed, Dec 23, 2015 at 8:24 PM, Kyrylo Galanov 
> wrote:
>
>> Hello,
>>
>> I would like to start discussion regarding the issue we have discovered
>> recently [1].
>>
>> In a nutshell, if RabbitMQ is configured to run in separate
>> mgmt/messaging network it fails on building cluster.
>> While RabbitMQ is managed by Pacemaker and OCF script, the cluster is
>> built using FQDN. Apparently, FQDN resolves to admin network which is
>> different in this particular case.
>> As a result, RabbitMQ on secondary controller node fails to join to
>> primary controller node.
>>
>> I can suggest two ways to tackle the issue: one is pretty simple, while
>> other is not.
>>
>> The first way is to accept by design using admin network for RabbitMQ
>> internal communication between controller nodes.
>>
>> The second way is to dig into pacemaker and RabbitMQ reconfiguration.
>> Since it requires to refuse from using common fqdn/node names, this
>> approach can be argued.
>>
>>
>> --
>> [1] https://bugs.launchpad.net/fuel/+bug/1528707
>>
>> Best regards,
>> Kyrylo
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Proposal to Delay Docker Removal From Fuel Master Node

[Matthew Mosesohn]

Hi all,

I have one concern related to 8.0 bugfixing until GA. If we disable docker
deployment in master, we will need to target docker-specific patches for
8.0 first to master and then backport to stable/8.0, which is still in line
with our standard bugfixing strategy. It will mean we should leave docker
bits in master for now, but mark the deployment mode to "non-docker" as the
default. I doubt that raises concerns, but it will affect CI.

In the short term, we should consider adding a CI job that includes Fuel
Master deployment *with docker enabled* to support speedy bugfix patches.

On Fri, Dec 18, 2015 at 3:07 PM, Aleksandr Didenko 
wrote:

> Hi,
>
> having or not having docker on Fuel master affects mostly Fuel node
> deployment related scripts/manifests. If we need to fix a bug in Nailgun or
> OSTF - we can fix it in their codebase and it does not really matter
> whether we run Nailgin or OSTF inside docker or not. The same goes for all
> the OpenStack deployment related manifests - fix will be the same in 9.0
> without docker and in 8.0 with docker.
> Removing docker may introduce a bunch of new bugs, but they will be only
> 9.0 specific and won't affect 8.0. So I don't think that removing docker in
> master branch will affect bugfixing velocity for stable branch a lot.
>
> Regards,
> Alex
>
> On Fri, Dec 18, 2015 at 1:45 AM, Sergii Golovatiuk <
> sgolovat...@mirantis.com> wrote:
>
>> +1
>>
>> --
>> Best regards,
>> Sergii Golovatiuk,
>> Skype #golserge
>> IRC #holser
>>
>> On Thu, Dec 17, 2015 at 3:43 PM, Mike Scherbakov <
>> mscherba...@mirantis.com> wrote:
>>
>>> If there are no concrete points why we should wait, I'm +1 to go ahead
>>> with merges.
>>>
>>> On Thu, Dec 17, 2015 at 1:32 AM Oleg Gelbukh 
>>> wrote:
>>>
 Evgeniy,

 True, and I fully support merging this particular change as soon as
 possible, i.e. the moment the 'master' is open for 9.0 development.

 -Oleg

 On Thu, Dec 17, 2015 at 12:28 PM, Evgeniy L  wrote:

> Hi Oleg,
>
> With the same degree of confidence we can say that anything we have in
> the beginning of
> the release cycle is not urgent enough. We pushed early branching
> specifically for
> such big changes as Docker removal/Changing repos structures and
> merging invasive patches
> for new release features.
>
> Vladimir Kuklin,
>
> I'm not sure what do you mean by "fixing 2 different environments"?
> With environment without
> containers it will simplify debugging process.
>
> Thanks,
>
> On Wed, Dec 16, 2015 at 10:12 PM, Oleg Gelbukh 
> wrote:
>
>> Hi
>>
>> Although I agree that it should be done, the removal of Docker
>> doesn't seem an urgent feature to me. It is not blocking anything besides
>> moving to full package-based deployment of Fuel, as far as I understand. 
>> So
>> it could be easily delayed for one milestone, especially if it is already
>> almost done and submitted for review, so it could be merged fast before 
>> any
>> other significant changes land in 'master' after it is open.
>>
>> --
>> Best regards,
>> Oleg Gelbukh
>>
>> On Wed, Dec 16, 2015 at 8:56 PM, Vladimir Kozhukalov <
>> vkozhuka...@mirantis.com> wrote:
>>
>>> Vladimir,
>>>
>>> I have other activities planned for the time immediately after SCF
>>> (separating UI from fuel-web, maybe it is even more invasive :-)) and 
>>> it is
>>> not a big deal to postpone this feature or another. I am against the
>>> approach itself of postponing something because it is too invasive. If 
>>> we
>>> create stable branch master becomes open. That was our primary 
>>> intention to
>>> open master earlier than later when we decided to move stable branch
>>> creation.
>>>
>>>
>>>
>>>
>>> Vladimir Kozhukalov
>>>
>>> On Wed, Dec 16, 2015 at 8:28 PM, Vladimir Kuklin <
>>> vkuk...@mirantis.com> wrote:
>>>
 Vladimir

 I am pretty much for removing docker, but I do not think that we
 should startle our developers/QA folks with additional efforts on 
 fixing 2
 different environments. Let's just think from the point of development
 velocity here and at delay such changes for at least after NY. Because 
 if
 we do it immediately after SCF there will be a whole bunch of holidays 
 and
 Russian holidays are Jan 1st-10th and you (who is the SME for docker
 removal) will be offline. Do you really want to fix things instead of
 enjoying holidays?

 On Wed, Dec 16, 2015 at 4:09 PM, Evgeniy L 
 wrote:

> +1 to Vladimir Kozhukalov,
>
> Entire point of moving branches creation to SCF was to 

Re: [openstack-dev] [Fuel] Getting rid of Docker containers on the Fuel master node

[Matthew Mosesohn]

Bogdan brings up a good point. fuel-createmirror in its current state pulls
down an Ubuntu container and uses apt utilities inside. I know it's being
replaced, but it's one instance of an item that might be overlooked by this
process.

On Mon, Nov 23, 2015 at 3:27 PM, Bogdan Dobrelya 
wrote:

> On 23.11.2015 12:47, Aleksandr Maretskiy wrote:
> > Hi all,
> >
> > as you know, Rally runs inside docker on Fuel master node, so docker
> > removal (good improvement) is a problem for Rally users.
> >
> > To solve this, I'm planning to make native Rally installation on Fuel
> > master node that is running on CentOS 7,
> > and then make a step-by-step instruction how to make this installation.
> >
> > So I hope docker removal will not make issues for Rally users.
>
> I believe the most backwards compatible scenario is to keep the docker
> installed while removing the fuel-* docker things back to the host OS.
> So nothing would prevent user from pulling and running whichever docker
> containers he wants to put on the Fuel master node. Makes sense?
>
> >
> > On Mon, Nov 23, 2015 at 12:28 PM, Vladimir Kozhukalov
> > > wrote:
> >
> > ETA:
> >
> > experimental ISO w/o docker: tonight
> > spec: tomorrow night
> >
> >
> >
> > Vladimir Kozhukalov
> >
> > On Mon, Nov 23, 2015 at 9:25 AM, Anastasia Urlapova
> > > wrote:
> >
> > @Vova,
> > thanks for bringing this up.
> > The new approach without docker containers on master node really
> > has many advantages.
> >
> > @Igor,
> > regarding your question, I would say that we have many
> > dependencies from containers in CI/CD systems and test's
> > codebase. The test alignment to the new implementation won't be
> > easy. In such things we should move forward step by step.
> > As you know the first step is a spec file, can you give us a
> > link to it?
> >
> >
> > Nastya.
> >
> > On Sat, Nov 21, 2015 at 6:10 AM, Oleg Gelbukh
> > > wrote:
> >
> > With CentOS7 we will have python2.7 at Fuel Admin node as a
> > default version, I believe.
> >
> > --
> > Best regards,
> > Oleg Gelbukh,
> > Principal Engineer
> > Mirantis
> >
> > On Fri, Nov 20, 2015 at 6:27 AM, Timur Nurlygayanov
> >  > > wrote:
> >
> > Hi Andrey,
> >
> > As far as I remember from the last usage of fuel
> > master node, there was Centos + py26 installation.
> > Python 2.6 is old enough and sometimes it is hard to
> > launch some application on fuel node without docker
> > (image with py27/py3). Are you planning to provide
> > py27 at least or my note is outdated and I can
> > already use py27 from the box?
> >
> > We can install docker on master node anyway to run Rally
> > / Tempest or other test suites and scripts from master
> > node with Python 2.7 or something also.
> >
> > On Fri, Nov 20, 2015 at 5:20 PM, Andrey Kurilin
> > >
> > wrote:
> >
> > Hi!
> > I'm not fuel developer, so opinion below is based on
> > user-view.
> > As far as I remember from the last usage of fuel
> > master node, there was Centos + py26 installation.
> > Python 2.6 is old enough and sometimes it is hard to
> > launch some application on fuel node without docker
> > (image with py27/py3). Are you planning to provide
> > py27 at least or my note is outdated and I can
> > already use py27 from the box?
> >
> > On Thu, Nov 19, 2015 at 4:59 PM, Vladimir Kozhukalov
> >  > > wrote:
> >
> > Dear colleagues,
> >
> > As might remember, we introduced Docker
> > containers on the master node a while ago when
> > we implemented first version of Fuel upgrade
> > feature. The motivation behind was to make it
> > possible to rollback upgrade process if
> > something goes wrong.
> >
> > Now we are at the point where we can not use our
> > 

Re: [openstack-dev] [Fuel] Getting rid of Docker containers on the Fuel master node

[Matthew Mosesohn]

Vladimir,

The old site.pp is long out of date and should just be recreated from the
content of all the other $service-only.pp files.

My main question is how do we propose to do a rollback from an update (in
theory, from 8.0 to 9.0, then back to 8.0)? Should we hardcode persistent
data directories (or symlink them?) to
/var/lib/fuel/$fuel_version/$service_name, as we are doing behind the
scenes currently with Docker? If we keep that mechanism in place, all the
existing puppet modules can be used without any modifications. On the same
note, upgrade/rollback is the same as backup and restore, that means our
restore should follow a similar approach.
-Matthew

On Thu, Nov 19, 2015 at 6:36 PM, Bogdan Dobrelya 
wrote:

> On 19.11.2015 15:59, Vladimir Kozhukalov wrote:
> > Dear colleagues,
> >
> > As might remember, we introduced Docker containers on the master node a
> > while ago when we implemented first version of Fuel upgrade feature. The
> > motivation behind was to make it possible to rollback upgrade process if
> > something goes wrong.
> >
> > Now we are at the point where we can not use our tarball based upgrade
> > approach any more and those patches that deprecate upgrade tarball has
> > been already merged. Although it is a matter of a separate discussion,
> > it seems that upgrade process rather should be based on kind of backup
> > and restore procedure. We can backup Fuel data on an external media,
> > then we can install new version of Fuel from scratch and then it is
> > assumed backed up Fuel data can be applied over this new Fuel instance.
>
> A side-by-side upgrade, correct? That should work as well.
>
> > The procedure itself is under active development, but it is clear that
> > rollback in this case would be nothing more than just restoring from the
> > previously backed up data.
> >
> > As for Docker containers, still there are potential advantages of using
> > them on the Fuel master node, but our current implementation of the
> > feature seems not mature enough to make us benefit from the
> > containerization.
> >
> > At the same time there are some disadvantages like
> >
> >   * it is tricky to get logs and other information (for example, rpm
> > -qa) for a service like shotgun which is run inside one of
> containers.
> >   * it is specific UX when you first need to run dockerctl shell
> > {container_name} and then you are able to debug something.
> >   * when building IBP image we mount directory from the host file system
> > into mcollective container to make image build faster.
> >   * there are config files and some other files which should be shared
> > among containers which introduces unnecessary complexity to the
> > whole system.
> >   * our current delivery approach assumes we wrap into rpm/deb packages
> > every single piece of the Fuel system. Docker images are not an
> > exception. And as far as they depend on other rpm packages we forced
> > to build docker-images rpm package using kind of specific build
> > flow. Besides this package is quite big (300M).
> >   * I'd like it to be possible to install Fuel not from ISO but from RPM
> > repo on any rpm based distribution. But it is double work to support
> > both Docker based and package based approach.
>
> There is another point, the containers long build time when installing
> the master node.
>
> >
> > Probably some of you can give other examples. Anyway, the idea is to get
> > rid of Docker containers on the master node and switch to plane package
> > based approach that we used before.
> >
> > As far as there is nothing new here, we just need to use our old site.pp
> > (with minimal modifications), it looks like it is possible to implement
> > this during 8.0 release cycle. If there are no principal objections,
> > please give me a chance to do this ASAP (during 8.0), I know it is a
> > huge risk for the release, but still I think I can do this.
> >
> >
> >
> >
> > Vladimir Kozhukalov
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> Best regards,
> Bogdan Dobrelya,
> Irc #bogdando
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] API services available on public VIP

[Matthew Mosesohn]

I haven't seen any more discussion on this topic. It looks like since we
default to enabling SSL/TLS on deployments, there's no reason to block
access to public API endpoints.

On Fri, Nov 13, 2015 at 5:15 PM, Vladimir Kuklin 
wrote:

> Adam
>
> I think, the answer is realtively simple - if user does not want to expose
> those APIs, he can easily configure his infra to filter this traffic. We
> just need to mention this in Ops Guide.
>
> On Fri, Nov 13, 2015 at 4:02 PM, Adam Heczko  wrote:
>
>> Hello fuelers,
>>
>> today I'd like to raise a questions about Fuel deployment practice
>> related to Public (external) network.
>> Current approach is to expose by default over public IP openstack API
>> endpoints like nova, cinder, glance, neutron etc. These API services are
>> exposed through HAProxy with TLS support, so this approach seems to be
>> relatively secure.
>> OTOH industry practice is to don't expose over public IPs too much and
>> rather rely on user action / decision to expose API access to the public.
>> I'd like to ask for your opinions regarding this topic and approach taken
>> by Fuel.
>>
>> Thank you,
>>
>> --
>> Adam Heczko
>> Security Engineer @ Mirantis Inc.
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com 
> www.mirantis.ru
> vkuk...@mirantis.com
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Running Fuel node as non-superuser

[Matthew Mosesohn]

Dmitry,

We really shouldn't put "user" creation into a single package and then
depend on it for daemons. If we want nailgun service to run as nailgun
user, it should be created in the fuel-nailgun package.
I think it makes the most sense to create multiple users, one for each
service.

Lastly, it makes a lot of sense to tie a "fuel" CLI user to
python-fuelclient package.

On Thu, Nov 12, 2015 at 6:42 PM, Dmitry Nikishov 
wrote:

> Stanislaw,
>
> I agree that this approch would work well. However, does Puppet allow
> managing capabilities and/or file ACLs? Or can they be easily set up when
> installing RPM package? (is there a way to specify capabilities/ACLs in the
> RPM spec file?) This doesn't seem to be supported out of the box.
>
> I'm going to research if it is possible to manage capabilities and  ACLs
> with what we have out of the box (RPM, Puppet).
>
> On Wed, Nov 11, 2015 at 4:29 AM, Stanislaw Bogatkin <
> sbogat...@mirantis.com> wrote:
>
>> Dmitry, I propose to give needed linux capabilities
>> (like CAP_NET_BIND_SERVICE) to processes (services) which needs them and
>> then start these processes from non-privileged user. It will give you
>> ability to run each process without 'sudo' at all with well fine-grained
>> permissions.
>>
>> On Tue, Nov 10, 2015 at 11:06 PM, Dmitry Nikishov > > wrote:
>>
>>> Stanislaw,
>>>
>>> I've been experimenting with 'capsh' on the 6.1 master node and it
>>> doesn't seem to preserve any capabilities when setting SECURE_NOROOT bit,
>>> even if explicitely told to do so (via either --keep=1 or
>>> "SECURE_KEEP_CAPS" bit).
>>>
>>> On Tue, Nov 10, 2015 at 11:20 AM, Dmitry Nikishov <
>>> dnikis...@mirantis.com> wrote:
>>>
 Bartolomiej, Adam,
 Stanislaw is correct. And this is going to be ported to master. The
 goal currently is to reach an agreement on the implementation so that
 there's going to be a some kinf of compatibility during upgrades.

 Stanislaw,
 Do I understand correctly that you propose using something like sucap
 to launch from root, switch to a different user and then drop capabilities
 which are not required?

 On Tue, Nov 10, 2015 at 3:11 AM, Stanislaw Bogatkin <
 sbogat...@mirantis.com> wrote:

> Bartolomiej, it's customer-related patches, they, I think, have to be
> done for 6.1 prior to 8+ release.
>
> Dmitry, it's nice to hear about it. Did you consider to use linux
> capabilities on fuel-related processes instead of just using non-extended
> POSIX privileged/non-privileged permission checks?
>
> On Tue, Nov 10, 2015 at 10:11 AM, Bartlomiej Piotrowski <
> bpiotrow...@mirantis.com> wrote:
>
>> We don't develop features for already released versions… It should be
>> done for master instead.
>>
>> BP
>>
>> On Tue, Nov 10, 2015 at 7:02 AM, Adam Heczko 
>> wrote:
>>
>>> Dmitry,
>>> +1
>>>
>>> Do you plan to port your patchset to future Fuel releases?
>>>
>>> A.
>>>
>>> On Tue, Nov 10, 2015 at 12:14 AM, Dmitry Nikishov <
>>> dnikis...@mirantis.com> wrote:
>>>
 Hey guys.

 I've been working on making Fuel not to rely on superuser privileges
 at least for day-to-day operations. These include:
 a) running Fuel services (nailgun, astute etc)
 b) user operations (create env, deploy, update, log in)

 The reason for this is that many security policies simply do not
 allow root access (especially remote) to servers/environments.

 This feature/enhancement means that anything that currently is being
 run under root, will be evaluated and, if possible, put under a
 non-privileged
 user. This also means that remote root access will be disabled.
 Instead, users will have to log in with "fueladmin" user.

 Together with Omar  we've put together a blueprint[0]
 and a
 spec[1] for this feature. I've been developing this for Fuel 6.1,
 so there
 are two patches into fuel-main[2] and fuel-library[3] that can give
 you an
 impression of current approach.

 These patches do following:
 - Add fuel-admin-user package, which creates 'fueladmin'
 - Make all other fuel-* packages depend on fuel-admin-user
 - Put supervisord under 'fueladmin' user.

 Please review the spec/patches and let's have a discussion on the
 approach to
 this feature.

 Thank you.

 [0] https://blueprints.launchpad.net/fuel/+spec/fuel-nonsuperuser
 [1] https://review.openstack.org/243340
 [2] https://review.openstack.org/243337
 [3] https://review.openstack.org/243313

 --
 Dmitry Nikishov,
 Deployment Engineer,
 Mirantis, 

Re: [openstack-dev] [Fuel][Fuel-QA][Fuel-TechDebt] Code Quality: Do Not Hardcode - Fix Things Instead

[Matthew Mosesohn]

Vladimir,

Bugfixes and minor refactoring often belong in separate commits. Combining
"extending foo to enable bar in XYZ" with "ensuring logs from service abc
are sent via syslog" often makes little sense to code reviewers. In this
case it is a feature enhancement + a bugfix.

Looking at it from one perspective, if the bugfix is made poorly without a
feature commit, then it looks like the scenario you described. However, it
has the benefit that it can be cleanly backported. If we simply reverse the
order of the commits (untangling the workaround), we get the same result,
but get flamed.

Sometimes both approaches are necessary. I agree that not growing tech debt
is important, but perceptions really depend on trends over 3+ weeks. It's
possible that such tech debt bugs are created and solved within 2-3 days of
the workaround. I know that's the exception, but I think we should be most
concerned about what happens when we carry tech debt across entire Fuel
releases.
On Nov 11, 2015 10:28 AM, "Aleksandr Didenko"  wrote:

> +1 from me
>
> On Tue, Nov 10, 2015 at 6:38 PM, Stanislaw Bogatkin <
> sbogat...@mirantis.com> wrote:
>
>> I think that it is excellent thought.
>> +1
>>
>> On Tue, Nov 10, 2015 at 6:52 PM, Vladimir Kuklin 
>> wrote:
>>
>>> Folks
>>>
>>> I wanted to raise awareness about one of the things I captured while
>>> doing reviews recently - we are sacrificing quality to bugfixing and
>>> feature development velocity, essentially moving from one heap to another -
>>> from bugs/features to 'tech-debt' bugs.
>>>
>>> I understand that we all have deadlines and need to meet them. But,
>>> folks, let's create the following policy:
>>>
>>> 1) do not introduce hacks/workarounds/kludges if it is possible.
>>> 2) while fixing things if you have a hack/workaround/kludge that you
>>> need to work with - think of removing it instead of enhancing and extending
>>> it. If it is possible - fix it. Do not let our technical debt grow.
>>> 3) if there is no way to avoid kludge addition/enhancing, if there is no
>>> way to remove it - please, add a 'TODO/FIXME' line above it, so that we can
>>> collect them in the future and fix them gradually.
>>>
>>> I suggest to add this requirement into code-review policy.
>>>
>>> What do you think about this?
>>>
>>> --
>>> Yours Faithfully,
>>> Vladimir Kuklin,
>>> Fuel Library Tech Lead,
>>> Mirantis, Inc.
>>> +7 (495) 640-49-04
>>> +7 (926) 702-39-68
>>> Skype kuklinvv
>>> 35bk3, Vorontsovskaya Str.
>>> Moscow, Russia,
>>> www.mirantis.com 
>>> www.mirantis.ru
>>> vkuk...@mirantis.com
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Master node upgrade

[Matthew Mosesohn]

Oleg,

All the volatile information, including a DB dump, are contained in the
small Fuel Master backup. There should be no information lost unless there
was manual customization done inside the containers (such as puppet
manifest changes). There shouldn't be a need to back up the entire
containers.

The information we would lose would include the IP configuration interfaces
besides the one used for the Fuel PXE network and any custom configuration
done on the Fuel Master.

I want #1 to work smoothly, but #2 should also be a safe route.

On Fri, Nov 6, 2015 at 5:39 PM, Oleg Gelbukh  wrote:

> Evgeniy,
>
> On Fri, Nov 6, 2015 at 3:27 PM, Evgeniy L  wrote:
>
>> Also we should decide when to run containers
>> upgrade + host upgrade? Before or after new CentOS is installed? Probably
>> it should be done before we run backup, in order to get the latest
>> scripts for
>> backup/restore actions.
>>
>
> We're working to determine if we need to backup/upgrade containers at all.
> My expectation is that we should be OK with just backup of DB, IP addresses
> settings from astute.yaml for the master node, and credentials from
> configuration files for the services.
>
> --
> Best regards,
> Oleg Gelbukh
>
>
>>
>> Thanks,
>>
>> On Fri, Nov 6, 2015 at 1:29 PM, Vladimir Kozhukalov <
>> vkozhuka...@mirantis.com> wrote:
>>
>>> Dear colleagues,
>>>
>>> At the moment I'm working on deprecating Fuel upgrade tarball.
>>> Currently, it includes the following:
>>>
>>> * RPM repository (upstream + mos)
>>> * DEB repository (mos)
>>> * openstack.yaml
>>> * version.yaml
>>> * upgrade script itself (+ virtualenv)
>>>
>>> Apart from upgrading docker containers this upgrade script makes copies
>>> of the RPM/DEB repositories and puts them on the master node naming these
>>> repository directories depending on what is written in openstack.yaml and
>>> version.yaml. My plan was something like:
>>>
>>> 1) deprecate version.yaml (move all fields from there to various places)
>>> 2) deliver openstack.yaml with fuel-openstack-metadata package
>>> 3) do not put new repos on the master node (instead we should use online
>>> repos or use fuel-createmirror to make local mirrors)
>>> 4) deliver fuel-upgrade package (throw away upgrade virtualenv)
>>>
>>> Then UX was supposed to be roughly like:
>>>
>>> 1) configure /etc/yum.repos.d/nailgun.repo (add new RPM MOS repo)
>>> 2) yum install fuel-upgrade
>>> 3) /usr/bin/fuel-upgrade (script was going to become lighter, because
>>> there should have not be parts coping RPM/DEB repos)
>>>
>>> However, it turned out that Fuel 8.0 is going to be run on Centos 7 and
>>> it is not enough to just do things which we usually did during upgrades.
>>> Now there are two ways to upgrade:
>>> 1) to use the official Centos upgrade script for upgrading from 6 to 7
>>> 2) to backup the master node, then reinstall it from scratch and then
>>> apply backup
>>>
>>> Upgrade team is trying to understand which way is more appropriate.
>>> Regarding to my tarball related activities, I'd say that this package based
>>> upgrade approach can be aligned with (1) (fuel-upgrade would use official
>>> Centos upgrade script as a first step for upgrade), but it definitely can
>>> not be aligned with (2), because it assumes reinstalling the master node
>>> from scratch.
>>>
>>> Right now, I'm finishing the work around deprecating version.yaml and my
>>> further steps would be to modify fuel-upgrade script so it does not copy
>>> RPM/DEB repos, but those steps make little sense taking into account Centos
>>> 7 feature.
>>>
>>> Colleagues, let's make a decision about how we are going to upgrade the
>>> master node ASAP. Probably my tarball related work should be reduced to
>>> just throwing tarball away.
>>>
>>>
>>> Vladimir Kozhukalov
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Creating puppet-keystone-core and proposing Richard Megginson core-reviewer

[Matthew Mosesohn]

While I am not core, I would like to say that Rich's leadership in
improving our Keystone Puppet module has been immensely valuable. +1 from
me.

-Matthew
On Oct 31, 2015 5:58 PM, "Emilien Macchi"  wrote:

> At the Summit we discussed about scaling-up our team.
> We decided to investigate the creation of sub-groups specific to our
> modules that would have +2 power.
>
> I would like to start with puppet-keystone:
> https://review.openstack.org/240666
>
> And propose Richard Megginson part of this group.
>
> Rich is leading puppet-keystone work since our Juno cycle. Without his
> leadership and skills, I'm not sure we would have Keystone v3 support in
> our modules.
> He's a good Puppet reviewer and takes care of backward compatibility. He
> also has strong knowledge at how Keystone works. He's always willing to
> lead our roadmap regarding identity deployment in OpenStack.
>
> Having him on-board is for us an awesome opportunity to be ahead of other
> deployments tools and supports many features in Keystone that real
> deployments actually need.
>
> I would like to propose him part of the new puppet-keystone-core group.
>
> Thank you Rich for your work, which is very appreciated.
> --
> Emilien Macchi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][puppet] CI gate for regressions detection in deployment data

[Matthew Mosesohn]

Bogdan,

I don't want to maintain catalog resources 10 (or 20) times. It's an
incredible amount of work to upkeep. There has to be a better way to ensure
we don't lose some things. The approach I had in mind would have covered
these cases:
1 - track all hiera lookups and make sure we catch new/lost lookups
2 - ensure all classes called from top level tasks are passed these values
from hiera

We lost in this case a hiera lookup. Such a test would cover this, rather
than comparing resulting puppet resources.

On Thu, Oct 29, 2015 at 5:39 PM, Bogdan Dobrelya 
wrote:

> On 29.10.2015 15:24, Bogdan Dobrelya wrote:
> > Hello.
> > There are few types of a deployment regressions possible. When changing
> > a module version to be used from upstream (or internal module repo), for
> > example from Liberty to Mitaka. Or when changing the composition layer
> > (modular tasks in Fuel). Specifically, adding/removing/changing classes
> > and a class parameters.
> >
> > An example regression for swift deployment data [0]. Something was
> > changed unnoticed by existing noop tests and as a result
> > the swift data became being stored in root partition.
> >
> > Suggested per-commit based regressions detection [1] for deployment data
> > assumes to automatically detect if a class in a noop catalog run has
> > gained or lost a parameter or if it has been updated to another value by
> > a patch under test. Later, this check could even replace existing noop
> > tests, everything will be checked automatically, unless every deployment
> > scenario is covered by a corresponding template, which are represented
> > as YAML files [2] in Fuel.
> > Note: The tool [3] can help to get all deployment cases (-Y) and all
> > deployment tasks (-S) as well.
> >
> > I propose to review the patch [1], understand how it works (see tl;dr
> > section below) and to start using it ASAP. The earlier we commit the
> > "initial" data layer state, less regressions would pop up.
> >
> > (tl;dr)
> > The check should be done for every modular component (aka deployment
> > task). Data generated in the noop catalog run for all classes and
> > defines of a given deployment task should be verified against its
> > "acknowledged" (committed) state.
> > And fail the test gate, if changes has been found, like new parameter
> > with a defined value, removed a parameter, changed a parameter's value.
> >
> > In order to remove a regression, a patch author will have to add (and
> > reviewers should acknowledge) detected changes in the committed state of
> > the deployment data. This may be done manually, with a tool like [3] or
> > by a pre-commit hook, or even at the CI side!
> > The regression check should show the diff between committed state and a
> > new state proposed in a patch. Changed state should be *reviewed* and
> > accepted with a patch, to became a committed one. So the deployment data
> > will evolve with *only* approved changes. And those changes would be
> > very easy to be discovered for each patch under review process!
> > No more regressions, everyone happy.
> >
> > Examples:
> >
> > - A. A patch author removed the mpm_module parameter from the
> > composition layer (apache modular task). The test should fail with a
> >
> > Diff:
> >   @@ -90,7 +90,7 @@
> >  manage_user=> 'true',
> >  max_keepalive_requests => '100',
> >  mod_dir=> '/etc/httpd/conf.d',
> >   -  mpm_module => 'false',
> >   +  mpm_module => 'prefork',
> >  name   => 'Apache',
> >  package_ensure => 'installed',
> >  ports_file => '/etc/httpd/conf/ports.conf',
> >
> > It illustrates that the mpm_module's committed value was "false".
> > But the new one came as the 'prefork', likely from the apache class
> > defaults.
> > The solution:
> > Follow the failed build link and see for detected changes (a diff).
> > Acknowledge the changes and include rebuilt templates in the patch as a
> > new revision. The tool [3] (use -h for help) example command:
> > ./utils/jenkins/fuel_noop_tests.rb -q -b -s api-proxy/api-proxy_spec.rb
> >
> > Or edit the committed templates manually and include data changes in the
> > patch as well.
> >
> > -B. An upstream module author added the new parameter mpm_mode with a
> > default '123'. The test should fail with a
> >
> > Diff:
> >@@ -90,6 +90,7 @@
> >   manage_user=> 'true',
> >   max_keepalive_requests => '100',
> >   mod_dir=> '/etc/httpd/conf.d',
> >+  mpm_mode   => '123',
> >   mpm_module => 'false',
> >   name   => 'Apache',
> >   package_ensure => 'installed',
> >
> > It illustrates that the composition layer is not consistent with the
> > upstream module data schema, and that could be a potential regression in
> > deployment (new parameter 

Re: [openstack-dev] [Fuel][Plugins] Plugin deployment questions

[Matthew Mosesohn]

Hi Patrick,

During the 7.0 development cycle we made a lot of enhancements to what
environment characteristics can be modified through a plugin. One item that
plugins cannot directly modify is the default Fuel roles and their
metadata. That having been said, there is an open-ended post_install.sh
script you can use for your plugin to "hack" this value. I know of one
project that currently disables the requirement for controller role in a
deployment. This may be helpful in testing a given standalone role that
doesn't depend on a controller.

Here's a link to the script: http://paste.openstack.org/show/476821/
Note that this doesn't reflect "enabled" status of a plugin. It will make
controller min count 0 for all environments. That won't break them, but it
just removes the restriction.

Best Regards,
Matthew Mosesohn

On Mon, Oct 19, 2015 at 3:29 PM, Dmitry Mescheryakov <
dmescherya...@mirantis.com> wrote:

> Hello folks,
>
> I second Patrick's idea. In our case we would like to install standalone
> RabbitMQ cluster with Fuel reference architecture to perform destructive
> tests on it. Requirement to install controller is an excessive burden in
> that case.
>
> Thanks,
>
> Dmitry
>
> 2015-10-19 13:44 GMT+03:00 Patrick Petit <ppe...@mirantis.com>:
>
>> Hi There,
>>
>> There are situations where we’d like to deploy only Fuel plugins in an
>> environment.
>> That’s typically the case with Elasticsearch and InfluxDB plugins of LMA
>> tools.
>> Currently it’s not possible because you need to at least have one
>> controller.
>> What exactly is making that limitation? How hard would it be to have it
>> removed?
>>
>> Thanks
>> Patrick
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Testing before switching to upstream librarian

[Matthew Mosesohn]

Hi Fuelers,

It seems we have a regression on two critical bugs because of switching
Fuel to puppet-openstacklib:
https://bugs.launchpad.net/fuel/+bug/1507685

This regressed to patches that were in Fuel Library that addressed two bugs
marked as Critical.

We should improve the acceptance criteria for moving to upstream modules to
ensure no bugs are regressed that relate to the particular Puppet module
being migrated.

Secondly, what should our policy be? Revert the switch to upstream module?
Or just work on cherry-picking the appropriate fixes?

Best Regards,
Matthew Mosesohn
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] backporting before merging to master

[Matthew Mosesohn]

Hi Fuelers,

I would like to address a concern I have with backporting policy. I'm sure
all of you know that we should always land patches to master before it
reaches stable/X.X branch. What you are not aware of probably is that many
people are making cherry picks well in advance of gathering reviews and
getting the patch landed in master. Some argue that it "saves time waiting
on CI", but in reality it's quite the opposite. Adding a cherry pick before
merging master causes the following workflow to take place:
1 - Propose to master and to stable/7.0
2 - CI runs on 2 patches
3 - Reviewer A comments on master patch
4 - owner adjusts both patches and runs CI
5 - Reviewer B comments on stable patch
6 - owner adjusts both patches and runs CI
(repeat 3-6 in varying degrees until enough patches are gathered)
7 - rebase stable/7.0 patch again... wait for CI again

This doubles the burden on CI and complicates the overall review process
where we are accepting feedback for the initial solution on two (nearly)
identical patches. What's worse is it's possible that the two solutions
merged won't be identical and introduce potential regressions.

I propose we avoid raising any stable/X.X patches before a patch is
_merged_ into master to avoid this scenario. Additionally, if a core sees
that this is happening, he or she should mark it -2 and discourage
submission of new patchsets.

I welcome your thoughts and feedback.

Best Regards,
Matthew Mosesohn
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-ansible] To NTP, or not to NTP, that is the question

[Matthew Mosesohn]

Major,

in Fuel, we've dealt with this problem for a long time in its varying
degrees of unpleasantness. Some virtualization platforms, such as
VirtualBox, are very prone to time drift. Hardware nodes, thankfully, don't
suffer so badly.

Time sync is very important for RabbitMQ, Corosync, and Ceph, in addition
to those items you mentioned above. I haven't seen swift itself break due
to time issues, but you may be right.

The most ideal situation is to point all hosts to public NTP pool servers.
Barring that, elect 1 host to base its time by its hardware clock, and then
direct all other hosts to sync time against that one host. This has major
issues when you're doing virtual deployments with snapshot/revert and
experiencing major time skew, so you may need extra VM management scripts
to manually sync time again after revert.


Best Regards,
Matthew Mosesohn

On Fri, Sep 18, 2015 at 4:03 PM, Major Hayden <ma...@mhtx.net> wrote:

> Hey there,
>
> I start working on a bug[1] last night about adding a managed NTP
> configuration to openstack-ansible hosts.  My patch[2] gets chrony up and
> running with configurable NTP servers, but I'm still struggling to meet the
> "Proposal" section of the bug where the author has asked for non-infra
> physical nodes to get their time from the infra nodes.  I can't figure out
> how to make it work for AIO builds when one physical host is part of all of
> the groups. ;)
>
> I'd argue that time synchronization is critical for a few areas:
>
>   1) Security/auditing when comparing logs
>   2) Troubleshooting when comparing logs
>   3) I've been told swift is time-sensitive
>   4) MySQL/Galera don't like time drift
>
> However, there's a strong argument that this should be done by deployers,
> and not via openstack-ansible.  I'm still *very* new to the project and I'd
> like to hear some feedback from other folks.
>
> [1] https://bugs.launchpad.net/openstack-ansible/+bug/1413018
> [2] https://review.openstack.org/#/c/225006/
>
> --
> Major Hayden
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Puppet] Keystone V2/V3 service endpoints

[Matthew Mosesohn]

Gilles,

I already considered this when looking at another openstackclient issue.
Version 1.0.4 has almost no changes from 1.0.3, which is the official
release for Kilo. Maybe we can get this keystone URL handling fix
backported to the 1.0.X branch of openstackclient?

-Matthew

On Fri, Aug 14, 2015 at 3:47 PM, Gilles Dubreuil gil...@redhat.com wrote:



 On 14/08/15 20:45, Gilles Dubreuil wrote:
 
 
  On 13/08/15 23:29, Rich Megginson wrote:
  On 08/13/2015 12:41 AM, Gilles Dubreuil wrote:
  Hi Matthew,
 
  On 11/08/15 01:14, Rich Megginson wrote:
  On 08/10/2015 07:46 AM, Matthew Mosesohn wrote:
  Sorry to everyone for bringing up this old thread, but it seems we
 may
  need more openstackclient/keystone experts to settle this.
 
  I'm referring to the comments in
  https://review.openstack.org/#/c/207873/
  Specifically comments from Richard Megginson and Gilles Dubreuil
  indicating openstackclient behavior for v3 keystone API.
 
 
  A few items seem to be under dispute:
  1 - Keystone should be able to accept v3 requests at
  http://keystone-server:5000/http://keystone-server:5000/
  I don't think so.  Keystone requires the version suffix /v2.0 or
  /v3.
 
  Yes, if the public endpoint is set without a version then the service
  can deal with either version.
 
  http://paste.openstack.org/raw/412819/
 
  That is not true for the admin endpoint (authentication is already
 done,
  the admin services deals only with tokens), at least for now, Keystone
  devs are working on it.
 
  I thought it worked like this - the openstack cli will infer from the
  arguments if it should do v2 or v3 auth.  In the above case for v3,
  since you specify the username/password, osc knows it has to use
  password auth (as opposed to token auth), and since all of the required
  v3 arguments are provided (v3 api version, domains for user/project), it
  can use v3 password auth.  It knows it has to use the /v3/auth/token
  path to get a token.
 
  Similarly for v2, since it only has username/password, no v3 api or v3
  domain arguments, it knows it has to use v2 password auth.  It knows it
  has to use /v2.0/token to get a token.
 
  With the puppet-keystone code, since it uses TOKEN/URL, osc cannot infer
  if it can use v2 or v3, so it requires the /v2.0 or /v3 suffix, and
  the api version.
 
 
  First of my apologies because I confused admin enpdoint with the admin
  service (or whatever that's dubbed).
 
  As of Keystone v3 (not the API, the latest version of Keystone which
  supports both API v2.0 and V3), the OS_AUTH_URL doesn't need the
  version. That can be effectively any of the endpoints, either the main
  (or public) by default on port 5000 or the admin by default on port
  35357, the third one internal pointing to either of the first two ones.
 
  This is a behavior at Keystone level not at the OSC. I mean OSC will
  just reflect the http-api behavior.
 
  Now the admin service, the one needed for the OS-TOKEN (used for
  bootstrapping) needs a URL (OS_URL) with a version to work.
 
  ATM, the issue with puppet keystone is that endpoints, OS_URL and
  OS_AUTH_URL are walking on each others.
 
 

 My latest update on this one, is that I found out while testing beaker
 which is using OSC 1.0.3 is not handling OS_AUTH_URL properly.

 I had been testing with OSC 1.5.1 and now latest 1.6.1 from Dolerean
 repo, where the version-less URLs are working, but not with OSC 1.0.1:

 --

 # cat openrc
 export OS_AUTH_URL=http://127.0.0.1:5000
 export OS_USERNAME=adminv3
 export OS_PASSWORD=testing
 export OS_PROJECT_NAME=openstackv3
 export OS_USER_DOMAIN_NAME=admin_domain
 export OS_PROJECT_DOMAIN_NAME=admin_domain
 export OS_IDENTITY_API_VERSION=3

 # openstack endpoint list -f csv
 ID,Region,Service Name,Service Type,Enabled,Interface,URL

 87b7db1b23df487bb4ec96de5aa3c271,RegionOne,keystone,identity,True,internal,
 http://127.0.0.1:5000;

 d9b345109d8a4320ac0dd832d2532cce,RegionOne,keystone,identity,True,admin,
 http://127.0.0.1:35357;

 f3a579a64f0241ef9aef3dc983e0fd4a,RegionOne,keystone,identity,True,public,
 http://127.0.0.1:5000;

 --

 # cat openrc_v2
 export OS_AUTH_URL=http://[::1]:5000
 export OS_USERNAME=admin
 export OS_PASSWORD=a_big_secret
 export OS_TENANT_NAME=openstack

 # openstack endpoint list -f csv --long
 ID,Region,Service Name,Service
 Type,PublicURL,AdminURL,InternalURL
 cf8825c44bd041109d5c7438ba9db8f3,RegionOne,keystone,identity,
 http://127.0.0.1:5000,http://127.0.0.1:35357,http://127.0.0.1:5000;




 
  2 - openstackclient should be able to interpret v3 requests and
 append
  v3/ to OS_AUTH_URL=http://keystone-server.5000/ or rewrite the URL
  if it is set as
  OS_AUTH_URL=http://keystone-server.5000/
 http://keystone-server.5000/
  It does, if it can determine from the given authentication arguments
 if
  it can do v3 or v2.0.
 
  It effectively does, again, assuming the path doesn't contain a version
  number (x.x.x.x:5000)
 
  3 - All deployments

Re: [openstack-dev] [Fuel][Puppet] Keystone V2/V3 service endpoints

[Matthew Mosesohn]

Sorry to everyone for bringing up this old thread, but it seems we may need
more openstackclient/keystone experts to settle this.

I'm referring to the comments in https://review.openstack.org/#/c/207873/
Specifically comments from Richard Megginson and Gilles Dubreuil indicating
openstackclient behavior for v3 keystone API.


A few items seem to be under dispute:
1 - Keystone should be able to accept v3 requests at
http://keystone-server:5000/
2 - openstackclient should be able to interpret v3 requests and append
v3/ to OS_AUTH_URL=http://keystone-server.5000/ or rewrite the URL if it
is set as OS_AUTH_URL=http://keystone-server.5000/
3 - All deployments require /etc/keystone/keystone.conf with a token (and
not simply use openrc for creating additional endpoints, users, etc beyond
keystone itself and an admin user)

I believe it should be possible to set v2.0 keystone OS_AUTH_URL in openrc
and puppet-keystone + puppet-openstacklib should be able to make v3
requests sensibly by manipulating the URL.
Additionally, creating endpoints/users/roles shouldbe possible via openrc
alone. It's not possible to write single node tests that can demonstrate
this functionality, which is why it probably went undetected for so long.

If anyone can speak up on these items, it could help influence the outcome
of this patch.

Thank you for your time.

Best Regards,
Matthew Mosesohn

On Fri, Jul 31, 2015 at 6:32 PM, Rich Megginson rmegg...@redhat.com wrote:

 On 07/31/2015 07:18 AM, Matthew Mosesohn wrote:

 Jesse, thanks for raising this. Like you, I should just track upstream
 and wait for full V3 support.

 I've taken the quickest approach and written fixes to
 puppet-openstacklib and puppet-keystone:
 https://review.openstack.org/#/c/207873/
 https://review.openstack.org/#/c/207890/

 and again to Fuel-Library:
 https://review.openstack.org/#/c/207548/1

 I greatly appreciate the quick support from the community to find an
 appropriate solution. Looks like I'm just using a weird edge case
 where we're creating users on a separate node from where keystone is
 installed and it never got thoroughly tested, but I'm happy to fix
 bugs where I can.


 Most puppet deployments either realize all keystone resources on the
 keystone node, or drop an /etc/keystone/keystone.conf with admin token onto
 non-keystone nodes where additional keystone resources need to be realized.



 -Matthew

 On Fri, Jul 31, 2015 at 3:54 PM, Jesse Pretorius
 jesse.pretor...@gmail.com wrote:

 With regards to converting all services to use Keystone v3 endpoints,
 note
 the following:

 1) swift-dispersion currently does not support consuming Keystone v3
 endpoints [1]. There is a patch merged to master [2] to fix that, but a
 backport to kilo is yet to be done.
 2) Each type (internal, admin, public) of endpoint created with the
 Keystone
 v3 API has its own unique id, unlike with the v2 API where they're all
 created with a single ID. This results in the keystone client being
 unable
 to read the catalog created via the v3 API when querying via the v2 API.
 The
 solution is to use the openstack client and to use the v3 API but this
 obviously needs to be noted for upgrade impact and operators.
 3) When glance is setup to use swift as a back-end, glance_store is
 unable
 to authenticate to swift when the endpoint it uses is a v3 endpoint.
 There
 is a review to master in progress [3] to fix this which is unlikely to
 make
 it into kilo.

 We (the openstack-ansible/os-ansible-deployment project) are tracking
 these
 issues and doing tests to figure out all the bits. These are the bugs
 we've
 hit so far. Also note that there is a WIP patch to gate purely on
 Keystone
 v3 API's which is planned to become voting (hopefully) by the end of this
 cycle.

 [1] https://bugs.launchpad.net/swift/+bug/1468374
 [2] https://review.openstack.org/195131
 [3] https://review.openstack.org/193422


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi

Re: [openstack-dev] [Fuel][Puppet] Keystone V2/V3 service endpoints

[Matthew Mosesohn]

Jesse, thanks for raising this. Like you, I should just track upstream
and wait for full V3 support.

I've taken the quickest approach and written fixes to
puppet-openstacklib and puppet-keystone:
https://review.openstack.org/#/c/207873/
https://review.openstack.org/#/c/207890/

and again to Fuel-Library:
https://review.openstack.org/#/c/207548/1

I greatly appreciate the quick support from the community to find an
appropriate solution. Looks like I'm just using a weird edge case
where we're creating users on a separate node from where keystone is
installed and it never got thoroughly tested, but I'm happy to fix
bugs where I can.

-Matthew

On Fri, Jul 31, 2015 at 3:54 PM, Jesse Pretorius
jesse.pretor...@gmail.com wrote:
 With regards to converting all services to use Keystone v3 endpoints, note
 the following:

 1) swift-dispersion currently does not support consuming Keystone v3
 endpoints [1]. There is a patch merged to master [2] to fix that, but a
 backport to kilo is yet to be done.
 2) Each type (internal, admin, public) of endpoint created with the Keystone
 v3 API has its own unique id, unlike with the v2 API where they're all
 created with a single ID. This results in the keystone client being unable
 to read the catalog created via the v3 API when querying via the v2 API. The
 solution is to use the openstack client and to use the v3 API but this
 obviously needs to be noted for upgrade impact and operators.
 3) When glance is setup to use swift as a back-end, glance_store is unable
 to authenticate to swift when the endpoint it uses is a v3 endpoint. There
 is a review to master in progress [3] to fix this which is unlikely to make
 it into kilo.

 We (the openstack-ansible/os-ansible-deployment project) are tracking these
 issues and doing tests to figure out all the bits. These are the bugs we've
 hit so far. Also note that there is a WIP patch to gate purely on Keystone
 v3 API's which is planned to become voting (hopefully) by the end of this
 cycle.

 [1] https://bugs.launchpad.net/swift/+bug/1468374
 [2] https://review.openstack.org/195131
 [3] https://review.openstack.org/193422

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel][Puppet] Keystone V2/V3 service endpoints

[Matthew Mosesohn]

Hi all,

It seems that I've reached an impasse with
https://review.openstack.org/#/q/topic:bp/detach-components-from-controllers,n,z
in Keystone with regards to Kilo puppet manifests. One of the
objectives is the ability to deploy Keystone on a separate node from
the controllers.

Here is what we know:

We are using Keystone v2.0 API everywhere currently.

Most Keystone providers for users, services, etc, use V3 API through
openstackclient

Keystone provider for service endpoints is still on V2. This is
because v2.0 clients can't see v3 endpoints. It's by design to not
be forward compatible[0].

There is a patch upstream[1] that enables V3 service endpoint
creation, but v2.0 users/clients will not see these endpoints.

Identity v2 and v3 behavior of openstackclient are vastly different.
There is nothing backward/forward compatible among the two, so it's a
hassle to deal with them in parallel.

openstackclient fails on v3 parameters unless version 3 is explicitly enabled.

What we can do to go forward?

Option 1: Keep v2.0 API data in openrc and hack v3 keystone providers,
updating ENV with 2 vars:
OS_IDENTITY_API_VERSION=3
OS_AUTH_URL=$(echo $OLD_OS_AUTH_URL | sed -e s'/v2.0/v3/')
or their equivalent in command line parameters

Option 2: Update to v3 Identity API for all services and accept the
unmerged patch[1]. This route requires the most disruptive changes
because of [0] and I would like to avoid this.

Option 3: Revert puppet-keystone to version 5.1.0 which is before v3
became mandatory.


I'd like to see what is possible with Option 1 because it should be
possible to use the existing providers in puppet-keystone master
without too many hoops to make them all work cleanly. I'd really
prefer being able to provide all these parameters to the keystone
provider, rather than relying on the /root/openrc file or exporting
shell vars, but getting this issue unstuck is really the most
important.


[0] https://review.openstack.org/#/c/196943/
[1] https://review.openstack.org/#/c/178456/24

Best Regards,
Matthew Mosesohn

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Puppet] Keystone V2/V3 service endpoints

[Matthew Mosesohn]

Hi Rich,

Sorry, I meant to link [0] to https://bugs.launchpad.net/keystone/+bug/1470635
More responses inline.


On Thu, Jul 30, 2015 at 5:38 PM, Rich Megginson rmegg...@redhat.com wrote:
 There is a patch upstream[1] that enables V3 service endpoint
 creation, but v2.0 users/clients will not see these endpoints.


 Right.  I'm not sure what the problem is - v3 clients can see the endpoints
 created with v2.

But not vice versa.

 But you said We are using Keystone v2.0 API everywhere currently. - Are
 you trying to move to use v3?

Not yet.

 I'm still not sure what the problem is.  Are you trying to move to use v3
 for auth, and use v3 resources like domains?

No. Avoiding that is better for now.


 Option 1: Keep v2.0 API data in openrc and hack v3 keystone providers,
 updating ENV with 2 vars:
 OS_IDENTITY_API_VERSION=3
 OS_AUTH_URL=$(echo $OLD_OS_AUTH_URL | sed -e s'/v2.0/v3/')
 or their equivalent in command line parameters


 I don't understand.  When you say v3 keystone providers are you talking
 about the puppet-keystone openstack.rb providers?  If so, they already do
 something similar to the above.
Yes, the puppet-keystone openstack.rb providers. Almost, except they don't
update the identity_api_version. It just passes the version from ENV
or $HOME/openrc

https://github.com/openstack/puppet-openstacklib/blob/master/lib/puppet/provider/openstack/credentials.rb



 Option 2: Update to v3 Identity API for all services and accept the
 unmerged patch[1]. This route requires the most disruptive changes
 because of [0] and I would like to avoid this.


 I don't understand why you need [1] which makes keystone_endpoint use the v3
 api.  v3 clients can see endpoints created with the v2 api.

Updating all clients to v3 is more effort at this point and v3
keystone is not targeted for Fuel 7.0.


 Option 3: Revert puppet-keystone to version 5.1.0 which is before v3
 became mandatory.


 I'd like to see what is possible with Option 1 because it should be
 possible to use the existing providers in puppet-keystone master
 without too many hoops to make them all work cleanly. I'd really
 prefer being able to provide all these parameters to the keystone
 provider, rather than relying on the /root/openrc file or exporting
 shell vars, but getting this issue unstuck is really the most
 important.


 I'm still not sure what the issue is, what you are prevented from doing.

The issue, concisely, is creating service_endpoints with v2.0 API and
other keystone resources with v3 API using one /root/openrc file.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Puppet] Keystone V2/V3 service endpoints

[Matthew Mosesohn]

The problem appears to be much more concise. The provider sets
identity_api_version okay, but doesn't alter OS_AUTH_URL api version.
That's the only reason why this is breaking.

It is broken in 2 places: in openstacklib's credentials class and in
keystone base provider. The keystone auth_endpoint logic seems to just
duplicate that of openstacklib's credentials class, so I think it
would make sense to consolidate that. I will prepare a patch to
puppet-keystone and puppet-openstacklib to address this.

On Thu, Jul 30, 2015 at 6:14 PM, Rich Megginson rmegg...@redhat.com wrote:
 On 07/30/2015 08:53 AM, Matthew Mosesohn wrote:

 Hi Rich,

 Sorry, I meant to link [0] to
 https://bugs.launchpad.net/keystone/+bug/1470635
 More responses inline.


 On Thu, Jul 30, 2015 at 5:38 PM, Rich Megginson rmegg...@redhat.com
 wrote:

 There is a patch upstream[1] that enables V3 service endpoint
 creation, but v2.0 users/clients will not see these endpoints.


 Right.  I'm not sure what the problem is - v3 clients can see the
 endpoints
 created with v2.

 But not vice versa.

 But you said We are using Keystone v2.0 API everywhere currently. - Are
 you trying to move to use v3?

 Not yet.

 I'm still not sure what the problem is.  Are you trying to move to use v3
 for auth, and use v3 resources like domains?

 No. Avoiding that is better for now.

 Option 1: Keep v2.0 API data in openrc and hack v3 keystone providers,
 updating ENV with 2 vars:
 OS_IDENTITY_API_VERSION=3
 OS_AUTH_URL=$(echo $OLD_OS_AUTH_URL | sed -e s'/v2.0/v3/')
 or their equivalent in command line parameters


 I don't understand.  When you say v3 keystone providers are you talking
 about the puppet-keystone openstack.rb providers?  If so, they already do
 something similar to the above.

 Yes, the puppet-keystone openstack.rb providers. Almost, except they don't
 update the identity_api_version. It just passes the version from ENV
 or $HOME/openrc


 https://github.com/openstack/puppet-openstacklib/blob/master/lib/puppet/provider/openstack/credentials.rb


 Ok, I see.  The intention of that code is that, if you specify something in
 ENV/openrc, it will override the default settings in the provider.

 What if the puppet-keystone openstack providers did not allow you to
 override the api version/url version with ENV/openrc?  Would that solve the
 problem?


 Option 2: Update to v3 Identity API for all services and accept the
 unmerged patch[1]. This route requires the most disruptive changes
 because of [0] and I would like to avoid this.


 I don't understand why you need [1] which makes keystone_endpoint use the
 v3
 api.  v3 clients can see endpoints created with the v2 api.

 Updating all clients to v3 is more effort at this point and v3
 keystone is not targeted for Fuel 7.0.

 Option 3: Revert puppet-keystone to version 5.1.0 which is before v3
 became mandatory.


 I'd like to see what is possible with Option 1 because it should be
 possible to use the existing providers in puppet-keystone master
 without too many hoops to make them all work cleanly. I'd really
 prefer being able to provide all these parameters to the keystone
 provider, rather than relying on the /root/openrc file or exporting
 shell vars, but getting this issue unstuck is really the most
 important.


 I'm still not sure what the issue is, what you are prevented from doing.

 The issue, concisely, is creating service_endpoints with v2.0 API and
 other keystone resources with v3 API using one /root/openrc file.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] version.yaml in the context of packages

[Matthew Mosesohn]

 2) production - It is always equal to docker which means we deploy docker 
 containers on the master node. Formally it comes from one of fuel-main 
 variables, which is set into docker by default, but not a single job in CI 
 customizes this variable. Looks like it makes no sense to have this.
This gets set to docker-build during fuel ISO creation because several
tasks cannot be done in the containers during docker build phase. We
can replace this by moving it to astute.yaml easily enough.
 4) openstack_version - It is just an extraction from openstack.yaml [2].
Without installing nailgun, it's impossible to know what the repo
directories should be. Abstracting it buried in some other package
makes puppet tasks laborious. Keeping it in a YAML allows it to be
accessible.

The rest won't impact Fuel Master deployment significantly.

On Fri, Jul 24, 2015 at 8:21 PM, Vladimir Kozhukalov
vkozhuka...@mirantis.com wrote:
 Dear colleagues,

 Although we are focused on fixing bugs during next few weeks I still have to
 ask everyone's opinion about /etc/fuel/version.yaml. We introduced this file
 when all-inclusive ISO image was the only way of delivering Fuel. We had to
 have somewhere the information about SHA commits for all Fuel related git
 repos. But everything is changing and we are close to flexible package based
 delivery approach. And this file is becoming kinda fifth wheel.

 Here is how version.yaml looks like

 VERSION:
   feature_groups:
 - mirantis
   production: docker
   release: 7.0
   openstack_version: 2015.1.0-7.0
   api: 1.0
   build_number: 82
   build_id: 2015-07-23_10-59-34
   nailgun_sha: d1087923e45b0e6d946ce48cb05a71733e1ac113
   python-fuelclient_sha: 471948c26a8c45c091c5593e54e6727405136eca
   fuel-agent_sha: bc25d3b728e823e6154bac0442f6b88747ac48e1
   astute_sha: b1f37a988e097175cbbd14338286017b46b584c3
   fuel-library_sha: 58d94955479aee4b09c2b658d90f57083e668ce4
   fuel-ostf_sha: 94a483c8aba639be3b96616c1396ef290dcc00cd
   fuelmain_sha: 68871248453b432ecca0cca5a43ef0aad6079c39


 Let's go through this file.

 1) feature_groups - This is, in fact, runtime parameter rather then build
 one, so we'd better store it in astute.yaml or other runtime config file.
 2) production - It is always equal to docker which means we deploy docker
 containers on the master node. Formally it comes from one of fuel-main
 variables, which is set into docker by default, but not a single job in CI
 customizes this variable. Looks like it makes no sense to have this.
 3) release - It is the number of Fuel release. Frankly, don't need this
 because it is nothing more than the version of fuel meta package [1].
 4) openstack_version - It is just an extraction from openstack.yaml [2].
 5) api - It is 1.0 currently. And we still don't have other versions of API.
 Frankly, it contradicts to the common practice to make several different
 versions available at the same time. And a user should be able to ask API
 which versions are currently available.
 6) build_number and build_id - These are the only parameters that relate to
 the build process. But let's think if we actually need these parameters if
 we switch to package based approach. RPM/DEB repositories are going to
 become the main way of delivering Fuel, not ISO. So, it also makes little
 sense to put store them, especially if we upgrade some of the packages.
 7) X_sha - This does not even require any explanation. It should be rpm -qa
 instead.

 I am raising this topic, because it is kind of blocker for switching to
 package based upgrades. Our current upgrade script assumes we have this file
 version.yaml in the tarball and we put this new file instead of old one
 during upgrade. But this file could not be packaged into rpm because it can
 only be built together with ISO.


 [1] https://github.com/stackforge/fuel-main/blob/master/specs/fuel-main.spec
 [2]
 https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/fixtures/openstack.yaml

 Vladimir Kozhukalov

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Get rid of fuelmenu

[Matthew Mosesohn]

We had that before and had very poor validation. Removing fuelmenu
would make the experience quite manual and prone to errors.

This topic comes up once a year only from Fuel Python developers
because they rarely use it and no dev cycles have been invested in
improving it.

The actual Fuel deployers use it and appreciate its validation and
wish to extend it.

I'd like to hear more feedback.

On Thu, Jul 23, 2015 at 2:23 PM, Vladimir Kozhukalov
vkozhuka...@mirantis.com wrote:
 Dear colleagues,

 What do you think of getting rid of fuelmenu and substituting it with
 thoroughly commented text file + some validation + vim? The major pro of
 this is that text file is easier to extend and edit. Many people prefer vim
 UX instead of wandering through the semi-graphical interface. And it is not
 so hard to implement syntax and logic checking for the text file.

 Please give your opinions about this.

 Vladimir Kozhukalov

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Get rid of fuelmenu

[Matthew Mosesohn]

How much effort are we spending? I'm not so sure it's a major development drain.

Since Fuel 6.0 dev cycle (Sept 2014) until now there have been 34
commits into Fuelmenu:
* New features/functionality: 12
* Bugfix: 15
* Other: 7 (version bumps, and commits without bug ID)

Across 3 releases, that's only ~11 commits per release. We've added
features like generating random passwords for services, warnings about
setting credentials apart from the default, adding a hook for CI for
testing custom manifests on Fuel Master, and duplicate IP address
checks.

These improved user experience. If you take it away and replace it
with a config file with basic validation, we will see users fail to
deploy due to things that Fuelmenu already checks easily. Imagine
you're an existing user of Fuel and suddenly you install the newest
version of Fuel and see a large configuration file which you have to
set by hand. Here's a relic of what users used to have to configure by
hand:
https://github.com/stackforge/fuel-library/blob/b015ed975b58dddff3b8da0ce34d9a638c22d032/deployment/puppet/openstack/examples/site_simple.pp

Am I alone in thinking it's not the best use of our development
resources to throw it away and replace it with a text file that is
edited by hand?

On Thu, Jul 23, 2015 at 3:33 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote:
 Hello,

 Here's my 2 cents on it.

 I think the effort we put to support fuelmenu doesn't worth it. I used
 to deploy fuel too often in previous release, and I never used
 features of fuelmenu? Why? Because I prefer to apply changes on
 already deployed node. Moreover, I don't like that users are prompted
 with fuelmenu by default. I want to deploy fuel automatically, without
 any manual actions (though it's another topic).

 I'm agree with Vladimir, vim + config files are enough, since Fuel is
 not a product for housewives. It's a product for those who do not
 hesitate to use Vim for soft configuration.

 Thanks,
 Igor



 On Thu, Jul 23, 2015 at 2:27 PM, Matthew Mosesohn
 mmoses...@mirantis.com wrote:
 We had that before and had very poor validation. Removing fuelmenu
 would make the experience quite manual and prone to errors.

 This topic comes up once a year only from Fuel Python developers
 because they rarely use it and no dev cycles have been invested in
 improving it.

 The actual Fuel deployers use it and appreciate its validation and
 wish to extend it.

 I'd like to hear more feedback.

 On Thu, Jul 23, 2015 at 2:23 PM, Vladimir Kozhukalov
 vkozhuka...@mirantis.com wrote:
 Dear colleagues,

 What do you think of getting rid of fuelmenu and substituting it with
 thoroughly commented text file + some validation + vim? The major pro of
 this is that text file is easier to extend and edit. Many people prefer vim
 UX instead of wandering through the semi-graphical interface. And it is not
 so hard to implement syntax and logic checking for the text file.

 Please give your opinions about this.

 Vladimir Kozhukalov

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Get rid of fuelmenu

[Matthew Mosesohn]

Oleg, those parameters are no longer documented (because there was no
validation) and are absent from the Fuel User Guide. They are,
however, still used by fuel-qa scripts.

On Thu, Jul 23, 2015 at 5:26 PM, Oleg Gelbukh ogelb...@mirantis.com wrote:
 Unless I am mistaken, it is possible to set most of the parameters supported
 by Fuel menu as kernel boot parameters. Isn't it sufficient replacement for
 fuelmenu for dev's purposes?

 -Oleg

 On Thu, Jul 23, 2015 at 4:05 PM, Matthew Mosesohn mmoses...@mirantis.com
 wrote:

 How much effort are we spending? I'm not so sure it's a major development
 drain.

 Since Fuel 6.0 dev cycle (Sept 2014) until now there have been 34
 commits into Fuelmenu:
 * New features/functionality: 12
 * Bugfix: 15
 * Other: 7 (version bumps, and commits without bug ID)

 Across 3 releases, that's only ~11 commits per release. We've added
 features like generating random passwords for services, warnings about
 setting credentials apart from the default, adding a hook for CI for
 testing custom manifests on Fuel Master, and duplicate IP address
 checks.

 These improved user experience. If you take it away and replace it
 with a config file with basic validation, we will see users fail to
 deploy due to things that Fuelmenu already checks easily. Imagine
 you're an existing user of Fuel and suddenly you install the newest
 version of Fuel and see a large configuration file which you have to
 set by hand. Here's a relic of what users used to have to configure by
 hand:

 https://github.com/stackforge/fuel-library/blob/b015ed975b58dddff3b8da0ce34d9a638c22d032/deployment/puppet/openstack/examples/site_simple.pp

 Am I alone in thinking it's not the best use of our development
 resources to throw it away and replace it with a text file that is
 edited by hand?

 On Thu, Jul 23, 2015 at 3:33 PM, Igor Kalnitsky ikalnit...@mirantis.com
 wrote:
  Hello,
 
  Here's my 2 cents on it.
 
  I think the effort we put to support fuelmenu doesn't worth it. I used
  to deploy fuel too often in previous release, and I never used
  features of fuelmenu? Why? Because I prefer to apply changes on
  already deployed node. Moreover, I don't like that users are prompted
  with fuelmenu by default. I want to deploy fuel automatically, without
  any manual actions (though it's another topic).
 
  I'm agree with Vladimir, vim + config files are enough, since Fuel is
  not a product for housewives. It's a product for those who do not
  hesitate to use Vim for soft configuration.
 
  Thanks,
  Igor
 
 
 
  On Thu, Jul 23, 2015 at 2:27 PM, Matthew Mosesohn
  mmoses...@mirantis.com wrote:
  We had that before and had very poor validation. Removing fuelmenu
  would make the experience quite manual and prone to errors.
 
  This topic comes up once a year only from Fuel Python developers
  because they rarely use it and no dev cycles have been invested in
  improving it.
 
  The actual Fuel deployers use it and appreciate its validation and
  wish to extend it.
 
  I'd like to hear more feedback.
 
  On Thu, Jul 23, 2015 at 2:23 PM, Vladimir Kozhukalov
  vkozhuka...@mirantis.com wrote:
  Dear colleagues,
 
  What do you think of getting rid of fuelmenu and substituting it with
  thoroughly commented text file + some validation + vim? The major pro
  of
  this is that text file is easier to extend and edit. Many people
  prefer vim
  UX instead of wandering through the semi-graphical interface. And it
  is not
  so hard to implement syntax and logic checking for the text file.
 
  Please give your opinions about this.
 
  Vladimir Kozhukalov
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org

Re: [openstack-dev] [Fuel] Getting rid of upgrade tarball

[Matthew Mosesohn]

One item that will impact this separation is that fuel_upgrade
implicitly depends on the openstack.yaml release file from
fuel-nailgun. Without it, the upgrade process won't work. We should
refactor fuel-nailgun to implement this functionality on its own, but
then have fuel_upgrade call this piece. Right now, we're copying the
openstack.yaml for the target version of Fuel and embedding it in the
tarball[1].
Instead, the version should be taken from the new version of
fuel-nailgun that is installed inside the nailgun container.

The other file which gets embedded in the upgrade tarball is the
version.yaml file, but I think that's okay to embed during RPM build.

[1]https://github.com/stackforge/fuel-web/blob/master/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/engines/openstack.py#L211-L213

On Thu, Jul 16, 2015 at 3:55 PM, Oleg Gelbukh ogelb...@mirantis.com wrote:
 Vladimir,

 Good, thank you for extended answer.

 --
 Best regards,
 Oleg Gelbukh

 On Thu, Jul 16, 2015 at 3:30 PM, Vladimir Kozhukalov
 vkozhuka...@mirantis.com wrote:

 Oleg,

 Yes, you are right. At the moment all docker containers are packaged into
 a single rpm package. Yes, it would be great to split them into several
 one-by-one rpms, but it is not my current priority. I'll definitely think of
 this when I'll be moving so called late packages (which depend on other
 packages) into perestroika. Yet another thing is that eventually all those
 packages and containers will be artifacts and will be treated differently
 according to their nature. That will be the time when we'll be thinking of a
 docker registry and other stuff like this.






 Vladimir Kozhukalov

 On Thu, Jul 16, 2015 at 2:58 PM, Oleg Gelbukh ogelb...@mirantis.com
 wrote:

 Vladimir,

 Thank you, now it sounds concieving.

 My understanding that at the moment all Docker images used by Fuel are
 packaged in single RPM? Do you plan to split individual images into separate
 RPMs?

 Did you think about publishing those images to Dockerhub?

 --
 Best regards,
 Oleg Gelbukh

 On Thu, Jul 16, 2015 at 1:50 PM, Vladimir Kozhukalov
 vkozhuka...@mirantis.com wrote:

 Oleg,

 All docker containers currently are distributed as rpm packages. A
 little bit surprising, isn't it? But it works and we can easily deliver
 updates using this old plain rpm based mechanism. The package in 6.1GA is
 called fuel-docker-images-6.1.0-1.x86_64.rpm So, upgrade flow would be like
 this
 0) add new (say 7.0) repository into /etc/yum.repos.d/some.repo
 1) install fuel-upgrade package (yum install fuel-upgrade-7.0)
 2) fuel-upgrade package has all other packages (docker, bootstrap image,
 target images, puppet modules) as its dependencies
 3) run fuel-upgrade script (say /usr/bin/fuel-upgrade) and it performs
 all necessary actions like moving files, run new containers, upload 
 fixtures
 into nailgun via REST API.

 It is necessary to note that we are talking here about Fuel master node
 upgrades, not about Openstack cluster upgrades (which is the feature you 
 are
 working on).

 Vladimir Kozhukalov

 On Thu, Jul 16, 2015 at 1:22 PM, Oleg Gelbukh ogelb...@mirantis.com
 wrote:

 Vladimir,

 I am fully support moving fuel-upgrade-system into repository of its
 own. However, I'm not 100% sure how docker containers are going to appear 
 on
 the upgraded master node. Do we have public repository of Docker images
 already? Or we are going to build them from scratch during the upgrade?

 --
 Best regards,
 Oleg Gelbukh

 On Thu, Jul 16, 2015 at 11:46 AM, Vladimir Kozhukalov
 vkozhuka...@mirantis.com wrote:

 By the way, first step for this to happen is to move
 stackforge/fuel-web/fuel_upgrade_system into a separate repository.
 Fortunately, this directory is not the place where the code is 
 continuously
 changing (changes are rather seldom) and moving this project is going to
 barely affect the whole development flow. So, action flow is as follows

 0) patch to openstack-infra for creating new repository (workflow -1)
 1) patch to Fuel CI to create verify jobs
 2) freeze stackforge/fuel-web/fuel_upgrade_system directory
 3) create upstream repository which is to be sucked in by openstack
 infra
 4) patch to openstack-infra for creating new repository (workflow +1)
 5) patch with rpm spec for fuel-upgrade package and other
 infrastructure files like run_tests.sh
 6) patch to perestroika to build fuel-upgrade package from new repo
 7) patch to fuel-main to remove upgrade tarball
 8) patch to Fuel CI to remove upgrade tarball
 9) patch to fuel-web to remove fuel_upgrade_system directory



 Vladimir Kozhukalov

 On Thu, Jul 16, 2015 at 11:13 AM, Vladimir Kozhukalov
 vkozhuka...@mirantis.com wrote:

 Dear colleagues,

 I'd like to suggest to get rid of Fuel upgrade tarball and convert
 this thing into fuel-upgrade rpm package. Since we've switched to online
 rpm/deb based upgrades, it seems we can stop packaging rpm/deb 
 repositories
 and docker containers into tarball and instead package upgrade python 
 

[openstack-dev] [Fuel][Plugins] Separating services from controller role update

[Matthew Mosesohn]

Hi fellow Fuelers and Stackers,

I'm writing to you with an update on the project I am working on
regarding separating one or more services from a Fuel controller node.
The scope of this project is limited at this point to separation of
database, keystone, RabbitMQ, and horizon from the default controller.

Spec: 
https://github.com/stackforge/fuel-specs/blob/master/specs/7.0/separate-services.rst

The current implementation requires creation of two new roles:
controller_minus_service and service
While we are waiting for Role as a Plugin[0] feature to become ready,
we have a workaround to create roles via plugin postinstall[1]

The main objective of implementing this function is to enable a
deployer to move services off the controller. With respect to Fuel
Library, this means changing two things: tasks and hiera.

Skipping tasks is currently accomplished by defining new roles. There
is a high chance that this will be covered in Role as a Plugin feature
and we can retain the original controller role.

Hiera is used to override default parameters. For example, if I want
an external DB, I need to replace the metadata Fuel generates for nova
DB with a specified management_vip, db_name, db_user, db_password in
hiera for the nova hiera hash. That can't be accomplished right now
via an API call because plugins have no deployment hooks and can only
write in astute.yaml inside its own plugin hash. Instead, a Puppet
task should be run to take the plugin's metadata and create a new YAML
file in Hiera. I understand that it provides a huge learning curve for
a plugin developer to write a task and puppet file to enable
replacement of default features, but it's the only capability
available right now.

Once the proper values have been set in Hiera, the tasks will be more
flexible for selecting the appropriate endpoint for services. In this
example[2], if you set 'service_endpoint' in hiera, it gets used for
Keystone. If not, management_vip gets used.

The ultimate goal here is to allow any specific service to be deployed
on its own role or even externally from a single Fuel environment. It
could be used to enable multi-region deployments for distributed
scale.



I should mention we have a few setbacks that impact user experience on
Fuel Python side for this feature.

The first is OSTF would need to be trained to identify the new
location of these endpoints. Since hiera becomes the source of truth,
it should be queried for the proper endpoints and credentials. Without
OSTF support for this feature, a deployer or plugin developer could
not easily verify that the deployment was successful.

The second is the complexity of the task graph to create new roles. A
plugin developer would need to copy the task graph and manually create
a new role, group, and list of tasks for the customized roles. This is
a little unintuitive. It would be helpful if we could have the ability
in a plugin to skip a task to enable this feature. As an
alternative, a way to generate a task graph (in order to generate a
role), skipping $STEPS, would serve developers as well.

I already mentioned the API call from a plugin step as something that
isn't supported yet, so I won't repeat that item.

Thanks for reading my update on this project. I look forward to your
feedback and comments.

Best Regards,
Matthew Mosesohn

[0] https://blueprints.launchpad.net/fuel/+spec/role-as-a-plugin
[1] https://review.openstack.org/#/c/189262/
[2] 
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/openstack-cinder/openstack-cinder.pp#L14

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [fuel] more collaboration request

[Matthew Mosesohn]

Hi Emilien,

I can see why you might be unhappy with Fuel's actions with regards to
the OpenStack Puppet modules. You could make this argument about many
components in Fuel. The heart of the matter is that we bundle the
upstream OpenStack Puppet modules with all the other modules,
developed both upstream and by Fuel's developers in one single git
repository. This decision, along with all the other decisions to put
Fuel's components under its own repositories, was intended to add
stability and granular control to the product. I'm not saying it's the
most community-oriented approach, but Fuel would have never evolved
and matured without it. The attribution in commits is lost because our
directory namespace differs such that it can't just be merged cleanly.
Revisiting submodules is an option, but it led to maintenance problems
in the past.

Secondly, I'd like to point out that Fuel is not so different from
what other teams are doing. At the Summit, I heard from others who all
maintain internal Gerrits and internal forks of the modules. The
difference is that Fuel is being worked on in the open in StackForge.
Anyone is free to contribute to Fuel as he or she wishes, take our
patches, or review changesets.

Starting in October 2014, the Fuel team has adopted a policy that we
cannot merge any patches into the core Puppet OpenStack modules of
Fuel without submitting a patch or at least a bug upstream. Our
reviewers block patches consistently. The truth is that the upstream
modules are quite excellent and we don't need to make changes so
often. Our goal is to work with upstream modules or in the issue where
upstream integration is impossible, we place that config in our own
separate modules.

The point you raised about fixing bugs in Fuel and not in Puppet
OpenStack is definitely valid and something we need to collaborate on.
The first and easiest option when a bug is applicable to both, we
could use Launchpad to assign bugs to both Fuel project and
puppet-$project so it gains visibility. If a bug is discovered in
Puppet OpenStack after it's been reported and/or fixed in Fuel, then
it would be best to ping someone in #fuel-dev on IRC and we can try to
figure out how to get this applied upstream correctly. Our best
results come from fixing upstream and I want to make sure that is
clear.

If you have specific bugs or commits you'd like to discuss, let's work
them out. I believe I can get the Fuel Library team to agree to do a
walk through all the open bugs in Puppet OpenStack and see if we have
any related fixes or bug reports.

Best Regards,
Matthew Mosesohn

On Thu, Jun 11, 2015 at 2:34 PM, Sanjay Upadhyay san...@gmail.com wrote:
 +1 for the thread, I would also like to hear from Mirantis on this.

 The Fork on fuel/puppet has been actively seen patching and consolidation.It
 seems like parallel effort why not merge it.

 regards
 /sanjay

 On Thu, Jun 11, 2015 at 9:12 AM, Emilien Macchi emil...@redhat.com wrote:

 Hi,

 Before reading this e-mail, please keep in mind:

 * I have a lot of admiration for Fuel and since I'm working on OpenStack
 Installers (at eNovance and now Red Hat), Fuel is something I always
 consider a good product.
 * This e-mail is about Fuel and Puppet, nothing about Mirantis.
 * I'm writing on behalf of my thoughts, and not on our group.
 * I'm using open mailing-list for open discussion. There is not bad
 spirit in this e-mail and I want to have a productive thread.

 I have some concerns I would like to share with you and hopefully find
 some solutions together.

 Since I've been working on Puppet OpenStack (2 years now), I see some
 situations that happen - according to me - too often:

 * A bug is reported in both Fuel Library and the Puppet module having
 trouble. A patch is provided in Fuel Library (your fork of Puppet
 OpenStack modules) but not in Puppet upstream module. That means you fix
 the bug for Fuel, and not for Puppet OpenStack community. It does not
 happen all the time but quite often.

 * A patch is submitted in a Puppet module and quite often does not land
 because there is no activity, no tests or is abandonned later because
 fixed in Fuel Library. I've noticed the patch is fixed in Fuel Library
 though.

 * RAW copy/paste between upstream modules code and your forks. In term
 of Licensing, I'm even not sure you have the right to do that (I'm not a
 CLA expert though) but well... in term of authorship and statistics on
 code, I'm not sure it's fair. Using submodules with custom patches would
 have been great to respect the authors who created the original code and
 you could have personalize the manifests.

 Note: you can see that I don't give any example because I'm not here to
 blame people or judge anyone.

 So the goal of my e-mail is to open the discussion and have a *real*
 collaboration between Fuel team which seems to have a lot of good Puppet
 engineers and Puppet OpenStack team.

 We had this kind of discussion at the Summit (in Vancouver and also
 Paris

Re: [openstack-dev] [Fuel][docker] A FFE request for docker improvements for the 6.0.1 release

[Matthew Mosesohn]

This should be for 6.1, not 6.0.1.
On Mar 13, 2015 2:10 PM, Bogdan Dobrelya bdobre...@mirantis.com wrote:

 Hello.

 I'd like to request a FFE for Docker host networking improvements [0]
 for the Fuel 6.0.1 feature freeze.

 This is really important to have it implemented for the 6.0.1 release as
 it would increase the overall user experience for deploying and scaling
 Openstack environments and would drastically improve Docker
 implementation, Fuel Master node upgrades story, and its overall
 performance experience.

 In order to make it happen, we should finish the merging of two
 remaining patches [1], [2] for master branch and backport them for the
 6.0.1.

 The merging is currently blocked by CI issue [3] and we're expecting it
 to be fixed after the devops version on CI nodes updated and the fix for
 [3] applied. That should not take longer than a two work days.

 [0]

 https://blueprints.launchpad.net/fuel/+spec/fuel-master-docker-host-networking
 [1] https://review.openstack.org/#/c/121559/
 [2] https://review.openstack.org/#/c/146885/
 [3] https://bugs.launchpad.net/fuel/+bug/1431477

 --
 Best regards,
 Bogdan Dobrelya,
 Skype #bogdando_at_yahoo.com
 Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Change diagnostic snapshot compression algoritm

[Matthew Mosesohn]

I did this exercise over many iterations during Docker container
packing and found that as long as the data is under 1gb, it's going to
compress really well with xz. Over 1gb and lrzip looks more attractive
(but only on high memory systems). In reality, we're looking at log
footprints from OpenStack environments on the order of 500mb to 2gb.

xz is very slow on single-core systems with 1.5gb of memory, but it's
quite a bit faster if you run it on a more powerful system. I've found
level 4 compression to be the best compromise that works well enough
that it's still far better than gzip. If increasing compression time
by 3-5x is too much for you guys, why not just go to bzip? You'll
still improve compression but be able to cut back on time.

Best Regards,
Matthew Mosesohn

On Mon, Nov 24, 2014 at 3:14 PM, Vladimir Kuklin vkuk...@mirantis.com wrote:
 IMO, we should get a bunch of snapshots and decide which compression to use
 according to the results of an experiment. XZ compression takes much longer,
 so you will need to use parallel xz compression utility.

 On Fri, Nov 21, 2014 at 9:09 PM, Tomasz Napierala tnapier...@mirantis.com
 wrote:


  On 21 Nov 2014, at 16:55, Dmitry Pyzhov dpyz...@mirantis.com wrote:
 
  We have a request for change compression from gz to xz. This simple
  change halfs our snapshots. Does anyone has any objections? Otherwise we
  will include this change in 6.0 release.

 I agree with the change, but it shouldn’t be high

 Regards,
 --
 Tomasz 'Zen' Napierala
 Sr. OpenStack Engineer
 tnapier...@mirantis.com







 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Yours Faithfully,
 Vladimir Kuklin,
 Fuel Library Tech Lead,
 Mirantis, Inc.
 +7 (495) 640-49-04
 +7 (926) 702-39-68
 Skype kuklinvv
 45bk3, Vorontsovskaya Str.
 Moscow, Russia,
 www.mirantis.com
 www.mirantis.ru
 vkuk...@mirantis.com

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] fuel master monitoring

[Matthew Mosesohn]

I'm okay with Sensu or Monit, just as long as the results of
monitoring can be represented in a web UI and has a configurable
option for email alerting. Tight integration with Fuel Web is a
nice-to-have (via AMQP perhaps), but anything that can solve our
out-of-disk scenario is ideal. I did my best to tune our logging and
logs rotation, but monitoring is the most sensible approach.

-Matthew

On Fri, Nov 21, 2014 at 12:21 PM, Przemyslaw Kaminski
pkamin...@mirantis.com wrote:
 BTW, there's also Monit

 http://mmonit.com/monit/

 (though it's in C) that looks quite nice. Some config examples:

 http://omgitsmgp.com/2013/09/07/a-monit-primer/

 P.

 On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote:

 Guys, maybe we can use existing software, for example Sensu [1]?
 Maybe i am wrong, but i dont like the idea to start writing our small
 monitoring applications..
 Also something well designed and extendable can be reused for statistic
 collector


 1. https://github.com/sensu

 On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com
 wrote:


 On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com
 wrote:

  I didn't mean a robust monitoring system, just something simpler.
  Notifications is a good idea for FuelWeb.

 I’m all for that, but if we add it, we need to document ways to clean up
 space.
 We could also add some kind of simple job to remove rotated logs, obsolete
 spanshots, etc., but this is out of scope for 6.0 I guess.

 Regards,
 --
 Tomasz 'Zen' Napierala
 Sr. OpenStack Engineer
 tnapier...@mirantis.com







 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Power management in Cobbler

[Matthew Mosesohn]

Tomasz, Vladimir, others,

The way I see it is we need a way to discover the corresponding IPMI
address for a given node for out-of-band power management. The
ultimate ipmitool command is going to be exactly the same whether it
comes from Cobbler or Ironic, and all we need to do is feed
information to the appropriate utility when it comes to power
management. If it's the same command, it doesn't matter who does it.
Ironic of course is a better option, but I'm not sure where we are
with discovering ipmi IP addresses or prompting admins to enter this
data for every node. Without this step, neither Cobbler nor Ironic is
capable of handling this task.

Best Regards,
Matthew Mosesohn

On Wed, Nov 19, 2014 at 7:38 PM, Tomasz Napierala
tnapier...@mirantis.com wrote:

 On 19 Nov 2014, at 16:10, Vladimir Kozhukalov vkozhuka...@mirantis.com 
 wrote:

 I am absolutely -1 for using Cobbler for that. Lastly, Ironic guys became 
 much more open for adopting new features (at least if they are implemented 
 in terms of Ironic drivers). Currently, it looks like we are  probably able 
 to deliver zero step Fuel Ironic driver by 6.1. Ironic already has working 
 IPMI stuff and they don't oppose ssh based power management any more. 
 Personally, I'd prefer to focus our efforts towards  Ironic stuff and 
 keeping in mind that Cobbler will be removed in the nearest future.

 I know that due to time constraints we would be better to go with Cobbler, 
 but I also think we should be closer to the community and switch to Ironic as 
 soon as possible.

 Regards,
 --
 Tomasz 'Zen' Napierala
 Sr. OpenStack Engineer
 tnapier...@mirantis.com







 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] CentOS falls into interactive mode: Unsupported hardware

[Matthew Mosesohn]

Hi Mike,

I actually reported this to CentOS back in May:
https://bugs.centos.org/view.php?id=7136
It's a bug/feature in Anaconda. It can be worked around quite easily
by adding unsupported_hardware to kernel params or to the kickstart
file.

I reported the bug because there's no support for CentOS (except from
the community), so this error message has no true value in a
non-commercial OS.

Best Regards,
Matthew Mosesohn

On Mon, Nov 17, 2014 at 4:30 PM, Mike Scherbakov
mscherba...@mirantis.com wrote:
 Hi all,
 I was skimming through a nicely written blogpost about Fuel experience [1],
 and noticed This hardware ... not supported by CentOS [2] on one of the
 screenshots. Looks like CentOS goes into interactive mode and complains
 about unsupported hardware.
 Can we do anything with this? I can hardly imagine clicking Ok for 100 nodes
 deployment...

 It will be fixed by image based provisioning of course, but the question is
 can we do anything to fix it in the current release?

 [1]
 http://ehaselwanter.com/en/blog/2014/10/15/deploying-openstack-with-mirantis-fuel-5-1/
 [2] http://ehaselwanter.com/images/article-images/mirantis-35-blog-780x.png
 --
 Mike Scherbakov
 #mihgen


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Failed upgrade chain - 5.1 - 5.1.1 - 6.0

[Matthew Mosesohn]

Igor,

I'm trying to figure out where the failure is. The 5.1.1 and 6.0
prebuilt containers contain the correct version of nailgun RPM
pre-installed. It shouldn't matter if the versions are identical
(which is a problem for sure).

Can you open a bug and post upgrade logs? I'm interested to see which
pieces cause the failure and this is some area I didn't plan for in
container upgrades.

Best Regards,
Matthew Mosesohn

On Fri, Nov 14, 2014 at 3:43 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote:
 Hi folks,

 Yesterday I performed the following upgrade chain:

 5.1 - 5.1.1 - 6.0

 and upgrading to 6.0 was failed. I did some research and figure out
 that Nailgun's container doesn't have Nailgun at all. So here's what
 happened.

 1. When we're upgrading to 6.0 we already have three repos in the
 system: 5.1, 5.1.1 and 6.0.

 2. When we start a new Nailgun container, it runs start.sh which runs
 puppet [1].

 3. And puppet fails to install a Nailgun package. Why it's happened?
 We have the same Nailgun version in both 5.1.1 and 6.0 repos, but
 those packages have different dependencies. From yum's point of view,
 the packages are same (since they have the same version), so yum tries
 to install the Nailgun from 5.1.1 repo, but the container contains
 already installed dependencies for the Nailgun from 6.0 (they were
 installed during building container), and obviously it fails with
 failed dependencies errors (we have higher versions in the system).

 Well, the issue blocks complex upgrade chains and definitely we have
 to do something ASAP.

 In order to protect us from similar issues in future I propose next changes:

 1. Increase versions of all Fuel packages at least once per release.

 2. Remove a package installation code from manifests - the container
 should not install packages, they should be installed during container
 building process.

 What do you think, folks?


 [1]: 
 https://github.com/stackforge/fuel-main/blob/master/docker/nailgun/start.sh#L11

 Thanks,
 Igor

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] NTP settings.

[Matthew Mosesohn]

Andrew,
That just shifts the error into Nailgun which is forced to examine NTP
settings of the host. With docker containerization, it makes detecting
this much more difficult. Erroring during fuelmenu is the only chance
where a user can effectively set the NTP parameters correctly. We
could write a ton of infrastructure around this capability, but all it
wins us is a more beautiful error that blocks a user from proceeding.

On Wed, Nov 12, 2014 at 7:21 PM, Andrew Woodward xar...@gmail.com wrote:
 Should we just block the deployment until the NTP is fixed so they
 know they need to fix it?

 On Wed, Nov 12, 2014 at 6:06 PM, Stanislaw Bogatkin
 sbogat...@mirantis.com wrote:
 Hi all,
 Today we have a next workflow about NTP in Fuel:

 1. When someone deploy a master node, we need to somehow operate with NTP
 and set upstream NTP servers to Fuel NTP daemon on master node.
 2. NTP will enabled by default and set to default upstream values (from
 ntp.org pool).
 3. If user will enter to fuelmenu then then he can change that values and it
 will stored as new ones.

 That can lead to some errors at deploy, some as:
 1. User set upstream NTP (or use defaults).
 2. By some reasons that NTP servers get unreacheable (i.e. network down).
 3. User creates environment and push 'deploy' button.
 4. In the middle of deploy process upstream NTP become reacheable and master
 node sync with them.
 If time on master node before that sync will have big shift regarding to
 real time, then we will have 2 problems:
 1. Deploy can fail, cause slave nodes will sync with master one time only -
 right after provisioning and if master node resync with upstream in the
 middle of deploy - some services can stop works (e.g. mcollective).
 3. It will be more complicate to understand logs, cause it will shifted too.

 As I see, we have two ways to solve this:
 1. Check upstream NTP reachability and disable upstream servers if they
 unreachable at fuelmenu step. It will cost us some shift with real time in
 the end.

  or

 2. Disable NTP upstream servers if they unreachable right after user press
 'deploy' button and enable them after deploy is over.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Andrew
 Mirantis
 Ceph community

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Propose adding Igor K. to core reviewers for fuel-web projects

[Matthew Mosesohn]

+1. I'm not core, but he has done the most thorough reviews lately and
shows great initiative in maintaining quality in Fuel.

On Mon, Oct 13, 2014 at 5:53 PM, Evgeniy L e...@mirantis.com wrote:
 Hi everyone!

 I would like to propose Igor Kalnitsky as a core reviewer on the
 Fuel-web team. Igor has been working on openstack patching,
 nailgun, fuel upgrade and provided a lot of good reviews [1]. In
 addition he's also very active in IRC and mailing list.

 Can the other core team members please reply with your votes
 if you agree or disagree.

 Thanks!

 [1]
 http://stackalytics.com/?project_type=stackforgerelease=junomodule=fuel-web

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Let's remove fuelweb from repo paths

[Matthew Mosesohn]

+1. It was a legacy item and it should go away. I'll review the patches.

On Wed, Oct 8, 2014 at 9:08 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote:
 Hi fuelers,

 I'm going to propose you remove fuelweb word from repos' paths. What
 am I talking about? Let me show you.

 Currently we have the following paths to repos:

 /var/www/nailgun/2014.2-6.0/centos/fuelweb/x86_64/
 /var/www/nailgun/2014.2-6.0/ubuntu/fuelweb/x86_64/

 Obviously, the word fuelweb is redundant here and doesn't reflect
 reality, because our repos contain not only fuel packages, but
 openstack.

 Moreover, fuel-upgrade script installs repos without that word
 (fuelweb, I mean) so we have inconsistent file structure for repos,
 which may lead to problems in future.

 So I propose to do it now, while we can do it without risks and
 safety. I prepared a set of patches

 https://review.openstack.org/#/c/126885/
 https://review.openstack.org/#/c/126886/
 https://review.openstack.org/#/c/126887/

 and built an ISO #508 [1] - both master node and centos cluster was
 deployed successfully.

 Folks, please, take a look over patches above and let's merge it.

 Thanks,
 Igor


 [1]: 
 http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_master_iso/508/

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?

[Matthew Mosesohn]

Dmitry, we already use lrzuntar in deploying Docker containers. Use a lower
compression ratio and it will decompress faster on virtual envs and it
takes under two mins on my virtual env.

Compress:
https://github.com/stackforge/fuel-main/blob/master/docker/module.mk#L27

Decompress:
https://github.com/stackforge/fuel-main/blob/master/iso/bootstrap_admin_node.docker.sh#L63
On Aug 21, 2014 5:54 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote:

 Fuelers,

 Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by
 2Gb with lrzip tool (ticket https://bugs.launchpad.net/fuel/+bug/1356813,
 change in build system https://review.openstack.org/#/c/114201/, change
 in docs https://review.openstack.org/#/c/115331/), but it will
 dramatically increase unpacking time. I've run unpack on my virtualbox
 environment and got this result:
 [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz
 Decompressing...
 100%7637.48 /   7637.48 MB
 Average DeCompression Speed:  8.014MB/s
 [OK] - 8008478720 bytes
 Total time: 00:15:52.93

 My suggestion is to reject this change, release 5.1 with big tarball and
 find another solution in next release. Any objections?

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?

[Matthew Mosesohn]

Mike, others,

I believe Jesse was proposing an upgrade that downloads all the files
separately on the Fuel Master itself. This is a move that we've gone
away from since Fuel 2.0 because of intermittent issues with 3rd party
mirrors. It's often better to consume 1 large file that has everything
and can be verified than to try to pull hundreds of separate bits that
can't be verified, plus trying to track down errors when something
small doesn't work. Locking down to a single install base really
contributed to Fuel's success early on, so I don't think moving back
to separate file downloads is a good idea.

However, clearly we should do our best to compress the upgrade package
file as best as possible so it is less expensive to transfer and also
consume.

On Fri, Aug 22, 2014 at 1:43 PM, Mike Scherbakov
mscherba...@mirantis.com wrote:
 I think 15 minutes is not too bad. Additionally, it will reduce download
 time and price for bandwidth.
 +1

 But let's see if we can find some other solution still in 5.1 (hardlinks,
 whatever else), and we obviously need to seriously address it in next
 release.

 Perhaps for 6 an option can be made to allow the Fuel master to use
 package repositories instead of an upgrade file - and the option can be used
 for development and production?
 Jessee, could please clarify this? Do you mean to use remote repository with
 packages, instead of tarballing everything into single bundle?



 On Fri, Aug 22, 2014 at 12:39 PM, Matthew Mosesohn mmoses...@mirantis.com
 wrote:

 Dmitry, we already use lrzuntar in deploying Docker containers. Use a
 lower compression ratio and it will decompress faster on virtual envs and it
 takes under two mins on my virtual env.

 Compress:
 https://github.com/stackforge/fuel-main/blob/master/docker/module.mk#L27

 Decompress:
 https://github.com/stackforge/fuel-main/blob/master/iso/bootstrap_admin_node.docker.sh#L63

 On Aug 21, 2014 5:54 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote:

 Fuelers,

 Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by
 2Gb with lrzip tool (ticket, change in build system, change in docs), but it
 will dramatically increase unpacking time. I've run unpack on my virtualbox
 environment and got this result:
 [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz
 Decompressing...
 100%7637.48 /   7637.48 MB
 Average DeCompression Speed:  8.014MB/s
 [OK] - 8008478720 bytes
 Total time: 00:15:52.93

 My suggestion is to reject this change, release 5.1 with big tarball and
 find another solution in next release. Any objections?

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Mike Scherbakov
 #mihgen


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Using host networking for docker containers

[Matthew Mosesohn]

Moving to host networking would reduce our ability to do zero downtime
upgrades in the future. It means you must kill the old container in
order to start the new one, rather than allowing for the possibility
to remap the network configuration in iptables. It's something we
don't have now, but we may be able to do in the future.

With regards to security issues, we have some more restrictive
firewall rules in place now. I don't think this is a major issue.

I don't think it makes a huge difference in performance to switch to
host networking, but it's worth testing.

On Mon, Aug 11, 2014 at 1:16 PM, Aleksandr Didenko
adide...@mirantis.com wrote:
 Hi,

 we're running only 3 containers in privileged mode: cobbler, rsyslog and
 mcollective. Running all the containers in privileged mode is not a good
 idea for security reasons. Docker manages DNAT forwarding itself, so it does
 not create any overhead for us.


 Is there any real benefits of using separate namespaces in security terms?

 Of course, for example only ports specified in EXPOSE line in Dockerfile are
 exposed to the host network. So if you start any additional tcp/udp
 listeners inside the containers, their ports won't be accessible from the
 host network.



 On Sat, Aug 9, 2014 at 10:39 AM, Dmitriy Shulyak dshul...@mirantis.com
 wrote:

 Hi team,

 I want to discuss benefits of using host networking [1] for docker
 containers, on master node.

 This feature was added in docker 0.11 and basicly means - reuse host
 networking stack, without
 creating separate namespace for each container.

 In my opinion it will result in much more stable install/upgrade of master
 node.

 1. There will be no need for dhcrelay/dhcrelay_monitor on host
 2. No dnat port forwarding
 3. Performance improvement for pxe boot ???

 Is there any real benefits of using separate namespaces in security terms?

 To implement this we will need:

 1. Update docker to recent version 0.12/1.x, we will do it anyway, yes?
 2. Run docker containers with --net=host

 Ofcourse it will require running containers in privileged mode, but afaik
 we are already doing this for other reasons.

 So, what do you think?

 [1] https://github.com/docker/docker/issues/2012
 [2] https://docs.docker.com/articles/networking/

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Blueprints process

[Matthew Mosesohn]

+1

Keeping features separate as blueprints (even tiny ones with no spec)
really will let us focus on the volume of real bugs.

On Tue, Jun 24, 2014 at 5:14 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote:
 Guys,

 We have a beautiful contribution guide:
 https://wiki.openstack.org/wiki/Fuel/How_to_contribute

 However, I would like to address several issues in our blueprints/bugs
 processes. Let's discuss and vote on my proposals.

 1) First of all, the bug counter is an excellent metric for quality. So
 let's use it only for bugs and track all feature requirement as blueprints.
 Here is what it means:

 1a) If a bug report does not describe a user’s pain, a blueprint should be
 created and bug should be closed as invalid
 1b) If a bug report does relate to a user’s pain, a blueprint should be
 created and linked to the bug
 1c) We have an excellent reporting tool, but it needs more metrics: count of
 critical/high bugs, count of bugs assigned to each team. It will require
 support of team members lists, but it seems that we really need it.


 2) We have a huge amount of blueprints and it is hard to work with this
 list. A good blueprint needs a fixed scope, spec review and acceptance
 criteria. It is obvious for me that we can not work on blueprints that do
 not meet these requirements. Therefore:

 2a) Let's copy the nova future series and create a fake milestone 'next' as
 nova does. All unclear blueprints should be moved there. We will pick
 blueprints from there, add spec and other info and target them to a
 milestone when we are really ready to work on a particular blueprint. Our
 release page will look much more close to reality and much more readable in
 this case.
 2b) Each blueprint in a milestone should contain information about feature
 lead, design reviewers, developers, qa, acceptance criteria. Spec is
 optional for trivial blueprints. If a spec is created, the designated
 reviewer(s) should put (+1) right into the blueprint description.
 2c) Every blueprint spec should be updated before feature freeze with the
 latest actual information. Actually, I'm not sure if we care about spec
 after feature development, but it seems to be logical to have correct
 information in specs.
 2d) We should avoid creating interconnected blueprints wherever possible. Of
 course we can have several blueprints for one big feature if it can be split
 into several shippable blocks for several releases or for several teams. In
 most cases, small parts should be tracked as work items of a single
 blueprint.


 3) Every review request without a bug or blueprint link should be checked
 carefully.

 3a) It should contain a complete description of what is being done and why
 3b) It should not require backports to stable branches (backports are
 bugfixes only)
 3c) It should not require changes to documentation or be mentioned in
 release notes



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Symlinks to new stuff for OpenStack Patching

[Matthew Mosesohn]

Hi Igor,

The repo directory its is too large to fit in a docker container and
work reliably. It is just a symlink inside the repo storage container
from host:/var/www/nailgun to repo-container:/repo. This /repo folder
is shared out to containers, such as nginx, and then symlinks are
created for each subdir in /var/www/nailgun. If you need more links
without rebuilding your environment, you would need to symbolically
link your new repository from /var/www/nailgun/newlink to
/var/lib/docker/devicemapper/mnt/$(docker nspect -f='{{.ID}}'
__containername__)/rootfs/repo. (replace __containername__ with the
container you're trying to work on).

I hope this helps.

-Matthew

On Sun, Jun 15, 2014 at 2:32 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote:
 Hello fuelers,

 I'm working on openstack patching for 5.1 and I've met some problems.
 The problems I've met are in repos/puppets installing process.

 The problems are almost same, so I describe it on repos example.

 The repos data are located in /var/www/nailgun. This folder is mounted
 as /repo into Nginx container. Nginx container has own /var/www/nailgun
 with various symlinks to /repo's content.

 So the problem is that we need to add symlinks to newest repos in Nginx
 container. How this problem should be solved? Should our fuel-upgrade
 script add these symlinks or we'll ship new docker containers which
 already contain these symlinks?


 Thanks,
 Igor

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Problem with kombu version.

[Matthew Mosesohn]

Dmitry, I don't think you should drop kombu.five so soon.
We haven't heard directly from Fuel python team, such as Dmitry
Pyzhov, what reason we have to lock kombu at version 2.5.14.
I wrote to him earlier today out of band, so hopefully he will get
back to this message soon.

On Wed, Apr 9, 2014 at 3:27 PM, Dmitry Teselkin dtesel...@mirantis.com wrote:
 Hi again,

 So there is a reply from the Dmitry Burmistrov which for some reason was
 missed in this thread:
 Nailgun requires exact version of kombu ( == 2.5.14 ).
 This is the only reason why we can't update it.
 I think you should talk to Dmitry P. about this version conflict.
 I want to take this opportunity to remind everyone that we should
 adhere to the global-requirements.txt in order to avoid such
 conflicts.

 Hopefully our developers decided to get rid of kombu.five usage what looks
 an easy task.

 Thanks, everyone.



 On Mon, Apr 7, 2014 at 8:33 PM, Dmitry Teselkin dtesel...@mirantis.com
 wrote:

 Hello,

 I'm working on Murano integration into FUEL-5.0, and have faced the
 following problem: our current implementation depends on 'kombu.five'
 module, but this module (actually a single file) is available only starting
 at kombu 3.0. So this means that murano-api component depends on kombu
 =3.0. This meets the OpenStack global requirements list, where kombu
 =2.4.8 is declared. Unfortunately, this also means that system-wide
 version upgrade is required.

 So the question is - what is the right way to solve the promblem? I see
 the following options:
 1. change kombu version requirement to =3.0 for entire FUEL installation
 - it doesn't break global requirements constraint but some other FUEL
 components could be affected.
 2. replace calls to functions from 'python.kombu' and use existing version
 - I'm not sure if it's possible, I'm awaiting answer from our developers.

 Which is the most suitable variant, or are there any other solutions for
 the problem?


 --
 Thanks,
 Dmitry Teselkin
 Deployment Engineer
 Mirantis
 http://www.mirantis.com




 --
 Thanks,
 Dmitry Teselkin
 Deployment Engineer
 Mirantis
 http://www.mirantis.com

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] dhcp-all-interfaces changes reverted

[Matthew Mosesohn]

Robert,

I have noticed that trying to DHCP on all interfaces at once in Ubuntu
12.04 results in wrong interfaces getting particular reservations. It is
better to do one at a time (with all interfaces down first) with a pause in
between.
On Feb 14, 2014 6:03 AM, Robert Collins robe...@robertcollins.net wrote:

 Dan - your DHCP-all-interfaces changes broke on Ubuntu 'testenv'
 environments - we've backed them out to give them time to be fixed
 without it being a firedrill.

 -Rob

 --
 Robert Collins rbtcoll...@hp.com
 Distinguished Technologist
 HP Converged Cloud

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev